I write to you further to the open letter sent by DBOL to all Calderdale Councillors the contents of which appear also on your website under the headline “Doubt cast over Halifax library survey” (7 November 2012). There are a number of points that were made in the letter that require clarification and correction.
As you are aware, Ipsos MORI is a specialist, independent research agency with extensive experience of carrying out market, social and opinion research including research and other work in support of consultations such as the one commissioned by Calderdale Metropolitan Borough Council. Ipsos MORI provide such services to a wide variety of clients. We are not the Council. Our role in this consultation exercise is completely impartial.
DBOL make a number of specific points within their open letter which we have dealt with in turn below:
1. “Bear in mind also that Ipsos MORI quote a 95% confidence in these figures to an accuracy of around +/-3.5%. In other words, the returns indicate that there is a 95% probability that the number in support of Option A is between 35.5% and 42.5%.”
DBOL is entirely correct to state that there is always a margin of error when talking about any sample survey results – this allows for the fact that we have only spoken to a sample and not the whole population of Calderdale.
It is standard practice in market, social and opinion research to talk to a random sample of residents/stakeholders, since talking to everybody in a specific area would be prohibitively expensive. Further there is a large body of scientific, academic and statistical evidence that has shown that speaking to a large enough sample provides statistically robust results which can be extrapolated, within margins of error, to the total population in question – in this case the population of Calderdale.
Statistical theory states that with a sample of 1,000 the margin of error (at 95% confidence) is +/- 3.1%, that is if 50% select a particular option the true figure 19 out of 20 times will be between 46.9% and 53.1%. For a sample of 1,500 the margin of error would be +/- 2.5% with a pure random sample (ie between 47.5% and 52.5%). However, the sample for the representative sample survey in Calderdale has been weighted to ensure it matches the actual population profile of Calderdale; that is to correct for fewer younger people and more older people responding. This represents standard practice for a survey of this nature.
Weighting the data (see below) also has an impact on its statistical reliability. Calculating an effective base size accounts for this impact by reducing the “effective” sample size and hence widening the margins of error. For this representative sample survey the effective base size is 730, with a corresponding margin of error of +/- 3.5% for a figure of 39% supporting Option A.
2. “The first thing to note is the small print that explains that the numbers have been weighted to the profile of adults resident across Calderdale. It is not explained how this is done …”
Yes, the results from the representative survey have been weighted and the process used is explained on page 34 of the combined report. The weighting factors applied are: age, gender, ethnicity, disability and ward. The recently released data from Census 2011 were used to ensure the results are representative of the Calderdale population. It is normal practice to use these factors in making the results representative – and is used to correct for the fact that, for example, and as is usual in postal self-completion surveys, fewer younger people and more older people take part in a research exercise such as this.
We would re-iterate that only these standard factors were included in the weighting. Weighting is not an attempt to manipulate the results but to ensure they are robust and representative of the population in question. This is standard practice in a self-completion survey of this nature. For more information see the explanation on http://www.esds.ac.uk/government/docs/weighting.pdf
3. “Where is the justification for assuming that the 150 Under 25s who didn’t reply would have voted the same way as the 30 who did? … This is what gives statistics and statisticians a bad name. Manipulating the figures to give the answer they want – or their customer wants.”
We have made absolutely no assumptions about how those who did not respond would have responded. We would strongly refute the second comment that we would in any way invent or “manipulate the figures” to meet the needs of our client. We are an independent research agency whose work is closely regulated by the Market Research Society (MRS). We abide fully with the MRS Code of Conduct (http://www.mrs.org.uk/standards/code_of_conduct/) when undertaking our work.
There has been extensive investigation and publication of thoughts and opinions about non-response bias (whether those who do respond share similar views to those who do not respond). There is not definitive answer to this, but the weight of expert opinion is that the views are broadly in line and that given a sufficiently large sample size (1,000 or more responses from a random sample) the margins of error applied to the results will allow for any discrepancies.
4.“The picture is further muddied, however, if we delve into the full, 141 page report issued by Ipsos MORI, which contains interesting detail, conveniently omitted from the Executive Summary … These figures seem to contradict the views expressed about Options A and B and, indeed, contradict each other”
It is not unusual to find figures within a survey which contradict each other, nor to find statements where equal proportions agree and disagree. People do hold often seemingly contradictory opinions (termed “cognitive polyphasia”): for example we consistently find that people both want to pay lower taxes and also have good levels of local services or that they want local services to reflect local needs but also do not want a “postcode lottery” when it comes to local service delivery.
All results from the representative sample survey and the open consultation were fully analysed in the combined report. By definition, not all results find their way into a summary – rather a summary serves to highlight the key results and to explain the basis of any conclusions or recommendations drawn from the wider analysis.
5. “The Ipsos MORI report says “While this different in the profile of the responses does not mean the responses to the open consultation are less important …” We challenge Ipsos MORI or the Cabinet to table any evidence that any importance, let alone equal importance, has been attached to the open consultation responses.
Ipsos MORI also say “to reconcile the two opposing results from the representative sample survey and the open consultation, the profile of those who responded to each strand needs to be taken into account.” They offer no suggestion however, of how this should be done.”
Ipsos MORI were appointed by Calderdale Council to conduct and manage the project. Our role was to ensure that the representative sample survey and the open consultation were robust, impartial, balanced and fair and that the analysis of responses to both strands was also robust, impartial and fair. It is not our role to tell the council how to reconcile the opposing results from the two strands nor how much importance to place on the different strands within the project, this would be a political decision. Our remit was to report what Calderdale residents told us. It is not to decide on what the outcome should be.
Finally, DBOL start their open letter to the Courier with the statement “If ever anybody wondered what prompted Mark Twain to coin the phrase Lies, Damned Lies and Statistics, look no further than the results just published of the independent consultation carried out by Ipsos MORI into the future of Halifax Town Centre”
We would refute this statement completely. Ipsos MORI’s reputation as a provider of robust, representative and reliable research and consultation services to our clients is based on our knowledge and application of the theories and practices associated with conducting research and consultations. Ipsos MORI prides itself on the impartial, independent expertise of its research staff and takes great care to assure research is carried out in compliance with all relevant legal, regulatory and industry codes of practice requirements.
While we are active in developing and refining research techniques as appropriate (and publishing the results of these innovations), we work within the existing, tried and tested approaches which have themselves been extensively tested by researchers, statisticians and academics alike to establish their qualifications as robust tools for gathering opinions about issues and proposals no matter how controversial.
We would answer DBOL’s question by simply stating that the representative sample survey and the open consultation have been carried out and the results of each analysed in an open and transparent way. We have used established and rigorous approaches to ensure that the results are robust, allowing Calderdale Council to make decisions based on reliable and solid evidence.
Ipsos MORI Social Research Institute
 Updated 2010 mid-year estimates based on Census 2001 findings were used for the ward populations and ethnicity as this is the most up-to-date available information. Information about disability comes from Census 2001 data, again the most up-to-date source.