US diplomats query the accuracy of opinion polls

By MURITHI MUTIGA mmutiga@ke.nationmedia.com
Posted Saturday, March 19 2011 at 22:00

In Summary

• Embassy’s own analysis in 2007 election showed Kibaki and Raila in a neck-and-neck race

American diplomats doubted the accuracy of opinion poll projections in 2007 giving ODM leader Raila Odinga a wide lead in the presidential race and feared the polls could fuel tension in the country.

A cable sent to Washington in mid November 2007 said the poll findings diverged from the embassy’s own analysis which indicated a close race between Mr Odinga and President Kibaki.

The cable, one of thousands of documents released by anti-secrecy website WikiLeaks, may raise new questions about the methodology employed by opinion pollsters in the country.

The view of the US diplomats was that the sampling formula the opinion poll firms were using was faulty.

“Our election analysis indicates a close race for the presidency. This analysis is based on voter registration data and traditional turnout rates as well as polling on presidential candidate preferences,” the cable says.

“We were concerned that widely published public opinion polls, which showed ODM’s Raila Odinga well ahead of President Kibaki, did not accurately reflect the true status of the contest. Given the rising political temperature, partially due to the use of blatant ethnic appeals by both sides, we were concerned about the reaction of ODM supporters should their candidate lose in a close outcome when they were led by public opinion polls to expect a landslide victory.”

The diplomats argued that the pollsters were wrong to base their sample of interviewees on population alone.

“The Mission quietly reached out to polling firms and their clients to suggest that poll sampling distribution should be based on the regional distribution of registered voters, not on raw population (there are wide variances in voter registration rates around the country, with the high population density areas, such as Kibaki’s native Central Province, enjoying higher rates). Polling on this basis would be a more accurate predictor of the outcome.

“When one polling firm (Consumer Insight) then started limiting its respondents on political preference polls to those who could produce an actual voter identification card, the change was dramatic: Odinga: 41 per cent, Kibaki: 40.6 per cent. Once the Steadman Group re-distributed their sampling according to the August voter registration figures, the Kibaki/Odinga gap shrank from 11 points to 4 points.”

Dr Tom Wolf, a political scientist who consults for opinion pollsters Synovate (formerly Steadman) says the analysis by the US officials was broadly accurate.

Dr Wolf says Steadman used raw population data while the registration of voters was going on but switched to a sampling formula that takes into account the number of registered voters after the roll was ready.

“It is a matter of choice and debate whether pollsters should rely on the number of registered voters,” he says. “Either raw population data or total number of registered voters is acceptable, but once the voters roll is ready no pollster can ignore the number of registered voters.”

Dr Wolf says the last elections were “a tragically lost opportunity” for pollsters because the confusion over the actual outcome of the vote makes it impossible to tell how accurate the polls were.

“What is important is for the pollster to state the method they used to compile their data,” he says.

Voter surveys

The Nation commissioned several opinion poll firms to conduct voter surveys before the last election. The final polls published before the elections in December 2007 were wildly divergent.

Infotrak and Strategic PR gave Mr Odinga the widest lead, predicting that he would defeat President Kibaki by about 10 percentage points.

Steadman and Consumer Insight projected that Mr Odinga was two percentage points ahead of President Kibaki.

American firm Gallup was alone in predicting a Kibaki victory, putting the incumbent one percentage point ahead of his main challenger.

Those differences raise questions about the methods used by Kenyan opinion pollsters. While the opinion polling industry in the country is one of the most vibrant in sub-Saharan Africa, with Kenyan firms active in several countries in the region, critics say the polls are not always reliable.

This was the view taken by US diplomats, although, unlike politicians unhappy with their popularity rating, the officials pinpointed specific issues with the methodology used by pollsters.

In a separate cable sent in the immediate aftermath of the disputed presidential election, US embassy officials told Washington it was impossible to tell who won, a view that was reinforced by a probe led by South African judge Johann Kriegler.

The embassy officials argued there were major irregularities in the vote tallying in PNU and ODM strongholds, noting that both Mr Kibaki and Mr Odinga received about half a million votes from voters who did not cast a ballot for a parliamentary candidate. This was seen as an indication of possible vote padding in Nyanza and Central provinces.

US ambassador Michael Ranneberger used that assessment to urge Washington not to support a demand for a re-tallying of the vote or a repeat of the election.

Dr Wolf says while many may take different opinions on the contents of the cables, the ones released on Kenya paint a broadly accurate and fair reflection of events.

“The guys at the US embassy did their homework. I have not seen anything that is totally out of reality, and, in fact, you can see that some of the analysis is quite measured despite the fact the authors seem to have had a lot of information. It would be good for opinion poll firms to assess the analysis and discuss