Learn how the MR industry can re-build respondent relationships and fight disengagement in a time of fierce online competition. Read Rules of Engagement by Kees de Jong, SSI CEO, in Research World.
Read Simon Chadwick's interview on the "Future of Insight." Then log in and join the discussion!
GfK NOP Embraces Research Voice "The most important challenge...is how to meet people in a progressive research atmosphere..." Mike Cooke, Global Director: Online Development. Read more.
Ian Kiernan
MRops, Inc.
Member
November 4, 2009 / Read/Post Comments
They are in most surveys....lengthy grid questions. It is not uncommon for us to ask respondents to rate 15 - 20 different attributes on a single screen. And, sometimes we have multiple grids within a survey.
We all agree that these grids are tedious and boring to complete. So we add survey traps and timers to catch the respondents that may not be paying attention or maybe straightlining through the survey. It doesn't make a lot of sense to me that we knowingly design these boring grids and then we go to great lengths to catch the people that were not paying attention.
There have been a number of companies (including MRops) that have leveraged technology (sliders, drag & drop, etc.) to make some of these grids a little more visually appealing and engaging to complete. But, this is just a band-aid and in some cases can add length to the survey. The real solution is designing questions that are easier to answer.
Sounds to me that this might be a place where we could work together to do some research on research to develop best practices. It would be great if we had some data to help clients understand what the ideal number of attributes on a single screen, the dangers of too many grids in a survey, the impact on data quality, etc.
Has anyone done any work in this area already? If so, can you share your findings?
Jul 14, 2010
Rules of Engagement
May 7, 2010
Slider Scales in Online Surveys
Mar 30, 2010
Questionnaire Length, Fatigue Effects and Response Quality Revisited
© Copyright 2010 Survey Sampling International. All Rights Reserved
Comments
Absolutely right, Ian! Researchers at SSI agree that the cumbersome grid in survey research questionnaires isn't serving its intended purpose. SSI designed and conducted research on the grid design as part of Research Voice and the Respondent Preservation Initiative. Download the Grid Test Summary in Recent Documents.
Are grids the problem or are we just trying to collect more information than is practical in a self-administered mode? A well-trained interviewer can keep a respondent on the line and focused. There simply may not be the magic bullet to do the same in a Web survey.
I think Reg has a great point here and I also believe that this problem was exacerbated by the fact that many clients added questions, attributes and brands to surveys as they moved from telephone to online, realizing that they could get much more data, quicker and at lower costs. I think that it is one of those issues, where we as researchers need to be a bit tougher on our clients. Which brands on the list are absolutely critical to benchmark against – and which are just “nice to have” included? Obviously it would be good if we could back up our approach with substantial research on research, although I think everyone would agree that experience tells us that large grids equals drop-out and that drop-out equals potential bias. If we can convince (and document to) the client that while their suggested approach will certainly generate more data, the results will nonetheless be less reliable and may suffer participation bias, I don’t believe that many clients will insist on fielding as is (I'll admit to being a bit naïve on this).
The referenced paper from SSI suggests that there are some fairly good and relatively straight forward alternatives (be ‘moderate’ or ‘ideal’) - and on the technology side we’ve seen the development of rather smooth “grid replacement tools” (e.g. in flash).
In my opinion, the way forward would be a combination of the above. Indeed, I would say that this is not really a choice? If we don’t change our approach in this respect, the “ever-decreasing-response-rates-challenge”, will only accelerate and we’ll end up with no data at all... save it for what we’re able to collect from the few eager, forgiving and persistent survey takers - who hardly reflect the general population.
Reg & Dan --- you have both made excellent points. Thank you for your input.
I believe that the majority of our industry will agree that these lengthy grids create many problems (lower response and drop off rates, data quality issues, etc.). And, I 100% agree that a key part of the solution is for researchers to be tougher on clients.
But, I think we can do a better job of helping the researchers push back and more importantly, educate our clients. We need to arm them with data to support our point of view and simple alternatives / best practices that they can follow (real live examples, screen shots of what to do, what not to do, etc.).
The work that SSI has done is a good start. And, I am hoping that other firms may have conducted research on research and developed documents that could be shared to help with this education effort.
Does anyone have a real attitude battery that has real predictive power? This as opposed to just a bunch of beliefs made up by the client or the researcher? If we can get our hands on some such we could test the various ways this could be administered (in a grid versus one by one) and see which one produced the best quality data. I'm sure we would also all enjoy making up extraneous stuff to add into this grid to make it ever longer. We then have an experimental design that has length of battery (number of statements) and adminstration method varying while the answer scale remains unchanged. The weakness of Philip (Garland)'s approach (IMHO) was that it mixed question types into administration methods. I'm sure the "best" data will come from the shortest battety adminstered as single questions.We might also be able to compare results to other modes of interview. Anyone got anything like this in their toolbox?