Learn how the MR industry can re-build respondent relationships and fight disengagement in a time of fierce online competition. Read Rules of Engagement by Kees de Jong, SSI CEO, in Research World.
Read Simon Chadwick's interview on the "Future of Insight." Then log in and join the discussion!
GfK NOP Embraces Research Voice "The most important challenge...is how to meet people in a progressive research atmosphere..." Mike Cooke, Global Director: Online Development. Read more.
Gloria Mellinger
Survey Sampling International
Member
October 7, 2009 / Read/Post Comments
Research participants are six times as likely to give poor quality responses on a 30 minute study compared to a 15 minute study, according to The ARF's Foundations of Quality Study. These results confirm findings of past research. They are in line with what people in the industry have been saying for years: we should avoid fielding surveys longer than 15 minutes. What's the barrier to taking action on this? We could, for example, split surveys longer than 15 minutes into two parts -- either recontacting the same people again, or using two matched samples? What are other thoughts on limiting survey length and restoring participation?
Jul 14, 2010
Rules of Engagement
May 7, 2010
Slider Scales in Online Surveys
Mar 30, 2010
Questionnaire Length, Fatigue Effects and Response Quality Revisited
© Copyright 2010 Survey Sampling International. All Rights Reserved
Comments
I see several obstacles, starting with the main one: a lack of economic incentive. Today, for most vendors, it would cost almost twice more to field two surveys of 15 minutes than one of 30 minutes. Most price grids, starting with sample costs, are discontinuous functions of the survey length. Your study will not be more or less expensive if you add or remove a few more questions. Do you know the difference in cost between a 12-minute and a 14-minute survey? There isn’t. So why would there be a difference between 14 and 16? You get the picture. So, clients add more questions, because they can, and because there is no benefit for them to be shorter. Sure, if the survey is much longer, some vendors will charge more. But most of the time, nothing will change. Which leads to my second point.
The second obstacle is a lack of good measurement of survey length. This post and the industry in general measure survey length in minutes, which has the convenience of being easy to understand but unfortunately lacks a clear absolute and unambiguous definition. In reality, a survey does not have a length, it has as many lengths as there are respondents, it has an average length, a median length, etc. Which one to take? A look at the number of questions would be more objective, but again there is a difference between a single-punch and a long grid. One could imagine a system where word count, number of questions, number of answers, media to watch, etc, could be computed into an objective function. This sounds difficult, but is not impossible if there is a will. Maybe the members of this forum could agree on an objective definition of survey duration based on a number of questions and answers.
Once respondents get more incentive for each question they answer, once sample providers charge more for every additional question asked, then optimization will begin. Clients will remove these few questions that are not so important after all. Data collection vendors will try not to ask questions from respondents when the sample provider already knows the answer.
If survey length can get an objective definition, and if data collection costs become a continuous function of survey length, then there will be an incentive to ask less questions from respondents. Is that possible? Maybe. Is that enough? Probably not. Does anyone feel like giving it a try?
Olivier, your points are well made and you are right to point to pricing practices as part of the problem.
The issue with pricing currently for sample suppliers is that it is not a "cost+" pricing model. Sure there are some fixed costs associated with a project - and these are often itemised - but what is the cost to each project in terms of respondent acquisition? The costs of acquiring the respondent occured in the past and will be recouped by a stream of future revenue. The individual project just makes some contribution to covering the costs. Therefore it is always possible to reduce costs per interview to the level where fixed costs are covered. This is what competition forces us to do. If we knew how the project would impact panellist's lifetime and engagement levels perhaps we could price against that - a sort of "the polluter pays" principle. But anyone going it alone on this basis would soon find themselves short of projects as their prices would almost certainly be the highest in the market. There are only two ways that I can think of that *real* pricing can be introduced and sustained. One is that the panel companies set up a cartel (which of course is illegal) the second is that Government intervenes to set prices (which is certainly not going to happen any time soon).
Anyone think of any other way?
Until the respondents fight back, it ain't going to change any time soon. Unless someone comes up with 'Internet research methodology 2.0'. Back to the lab, then...
The classic response I hear is, "The client made me do it." Maybe some of that goes back to a point Simon Chadwick is making on another board about training on the client side but I also suspect that researchers on the supplier side are at least equally to blame. How many of us can make cogent, fact-based arguments to clients about the impact of survey length on data quality? That data are out there but mostly what one hears people say doesn't go much beyond, "Long surveys are bad." And when we lose the argument and field that 40 minute survey how many of us close the loop by gently pointing out to that client the problems in the data we've collected and the risk to the business decision to be made?
A lot of the problem is fundamental to the way we do quant research. We tend to ask ask our questions in a single survey with a respondent, so we have to ask everything we might need to know. We are not sure what we will need to know, is the brand cheap or cheerful, or rich or feminine, and does feminine matter as much as cheerful?
The two options are
a) conduct the research in waves, asking what is needed at each stage (this might be via communities, or via databases where we build our picture of each person over time).
b) have a conversation with the respondent. If I talk to you I can find out everything I want to know about your holiday in 5-10 minutes. If I write a quant survey I will hardly scratch the surface in 20 minutes.
Either way we need to move away from models based on assumptions of respodents answering grids, with no missing data, using meaningless 5 and 7 point scales.
There's another issue here, building on Ray's comment. That is that we are constrained to ask participants again and again information that they have given us a zillion times before. The day when we can append rich information to the participant's profile that would obviate the need for these questions will be the day that we can shorten the questionnaire. Until then, clients will continue to insist we ask the same old stuff time and again.
I agree with Simon that appending respondent profile data is a proven way to save time in the survey process. We and others have been doing some appending for some time, but issues have arisen with the lack of standards between how the profile is built by the sample provider and how the agency or a particular client wants the question asked, resulting in funky data being collected. If major advertisers, agencies and sample providers could agree on standards for the top 10-20 demographics in top 5 markets, it would open the door to more integration. I think that's a collaboration project which could achieve a measurable impact in a short time frame. If anyone else in this forum is interested to work with me on this standardization project, let me know.
Isn't it weird how a client will not accept essentially the same question worded slightly differently but they will accept "funky" data from the last 10 minutes of a 25 minute survey? To Reg's point I would also ask: How many researchers working today would know how to analyse out let alone present the problem data? If we don't know this stuff then it's just our "feeling" against that of the client, and since they have the big cheque book....
I agree with what you are saying, but I also think that the we also need to push online surveys to be more engaging.
I agree with some of the comments above. The cost argument could be valid but I've seen many cases when clients will be willing to pay "anything" as long as we provide the completes for a 45 or 50 minute survey (no matter how unfeasible we tell them this is) so cost doesn't seem to be the main driver.
As a panel company we rarely deal with the end clients so sometimes it's really difficult to understand what is behind these decisions. A lot of the times we get the feeling that "the client" is a robot (picture a 70s robot) that cannot reason or understand or even take any suggestions regarding questionnaires, because of course we are not researchers and we don't know anything about research.
In my opinion, the real reason behind long questionnaires has to do with inertia and the history of research. Until online research came along long surveys were not really a problem. People on the phone are very unlikely to just hang up on you and people on the street are very unlikely to start running when after 25 minutes you are still asking. And now it seems that some of these surveys cannot be changed because then data wouldn't be comparable (no matter how crappy the data is).
I cannot see this changing in the short term unless an organisation like ESOMAR or CASRO "outlaw" long online surveys. If there is clear evidence that they produce bad data then why do we stick to them? It's funny how sample buyers are so "strict" about the ESOMAR 26 but then "kill" a survey with a 50 minute questionnaire :-)
Can anyone think of any other solutions?