Online Data Collection: Are We There Yet?

Anne Crassweller, President, NADbank, Toronto
Presented to the CARF Seminar - Market Research Surveys: Online Versus Traditional – Jan. 18, 2008
(First presented at the Worldwide Readership Research Symposia, Vienna. October 21-24, 2007)

For many years, NADbank has been using RDD telephone-based methodology to collect data on newspaper readership and consumer behaviour of adults 1*+ in urban markets across Canada. However, the decline in response rates (45% in 2006) and the increase in cell phone only households (5% in 2006) have motivated NADbank to explore alternatives to its traditional methodology.

Parallel with its 2006 fieldwork, NADbank conducted an online survey using the same questionnaire as their telephone research. The purpose of the test was to obtain top-line estimates of similarities and differences between online and traditional telephone survey methodology. The test compared the results using the TNS/Canadian Facts online panel (Toronto only members) with the NADbank telephone research in the Toronto market. Observations from the test are reported below.

Differences in sample composition – Generally speaking, the online panel was composed of more male respondents, fewer 18- to 24-year-old respondents, and more 35-to 64-year-olds than the sample collected via telephone survey. Compared to the respondents in the telephone survey, the respondents in the online test tended to have a higher level of education, particularly at the “some post secondary” level with fewer university graduates, but lower average household incomes. They were also less likely to own their own home and were more likely to speak English. The online panel had fewer respondents classified as blue-collar workers compared to the telephone sample.

Differences in media exposure – Generally speaking, online respondents said they spent about 50% more time (hours per week) with individual media than those in the telephone sample and also more time online than the “wired” respondents in the telephone sample (accessed the internet in the past 3 months). Compared to the full telephone sample and the “wired” respondents in the telephone sample, the online respondents typically read more weekday issues of a newspaper (4.7 issues per week versus 4.0 for the “wired” group) and more weekend issues (4.5 issues per week versus 3.6 for the “wired” group).

Readership of “any” and individual daily newspaper titles was higher for the online panellists than for those in the telephone sample. The differences were not consistent across titles or metrics. Even when the demographic profile of the online panel was adjusted to match the telephone-based sample using rim weighting, the differences in readership persisted.

Test conclusions – NADbank concluded that there were behavioural differences between those participating in the online panel and those reached via RDD telephone sampling. Further, it was concluded that such differences couldn’t be overcome by simply adjusting or weighting the panel to match known demographics. Differences in results gleaned from traditional versus online panels are likely due, in part, to the fundamental differences in the research platforms.

Next steps – NADbank will continue its research into issues relating to web-based panels. A second test with four panels in three markets was conducted last fall in order to look at consistency of survey to survey results, ability to conduct research in markets outside of Toronto, and comparability of different panels. NADbank will continue to conduct its study using the traditional telephone-based protocol.