Learn how the MR industry can re-build respondent relationships and fight disengagement in a time of fierce online competition. Read Rules of Engagement by Kees de Jong, SSI CEO, in Research World.
Read Simon Chadwick's interview on the "Future of Insight." Then log in and join the discussion!
GfK NOP Embraces Research Voice "The most important challenge...is how to meet people in a progressive research atmosphere..." Mike Cooke, Global Director: Online Development. Read more.
Ray Poynter
The Future Place
Member
November 1, 2009 / Read/Post Comments
Reg Baker has uploaded a really interesting set of slides from a study looking at consumers' reactions to MaxDiff, compared with other alternatives [see it here http://www.research-voice.com/sites/all/files/documents/Tradeoffs%20with%20MaxDif.pdf]. What makes it interesting is that the study finds that respondents do not like the technique - which makes it a very interesting set of results.
I think we need to look carefully at the study, ideally replicating it in other countries (as the authors suggest).
I tend to use MaxDiff quite a lot for three reasons:
1) It tends to work well across cultures, it is not prone to centre tick people versus extreme tick people
2) It produces scores that are less flat than other techniques
3) Respondents seem to like it.
I would love to discuss the study further with the authors as I have a couple of queries, in particular
a) the study appeared to show each 20 screens for the pairs section on the Problems with US Healthcare section, so that should have meant showing 10 screens for the MaxDiff section. With only 50% as many screens it is surprising that the MaxDiff took nearly twice as long to answer.
b) The layout of the MaxDiff seemed counter-intuitive, I would normally put the four options across the screen, making the layout more attractive, rather than one above the other (most of the pairs options were laid out attractively).
c) In the mean importance scores for Exercise 3 the range of scores for MaxDiff is much higher than for the other five techniques. Did it score much worse in the other two exercises?
I will contact Reg to pursure these questions, but the team are to be congratulated on looking at these techniques from the most important angle, the respondents'.
Jul 14, 2010
Rules of Engagement
May 7, 2010
Slider Scales in Online Surveys
Mar 30, 2010
Questionnaire Length, Fatigue Effects and Response Quality Revisited
© Copyright 2010 Survey Sampling International. All Rights Reserved
Comments
Good questions, Ray.
First off, I agree with your first two reasons for using MaxDif. I think that's why it's been widely adopted. As you note, this research seems to take issue with respondent enjoyment.
I'll need to go back to my co-authors to respond to your points a) and b) but in the meantime would be interested in your preferred presentation (point c)). What you seem to describe is a vertically oriented grid rather than the standard horizontally oriented grid. If that's the case, I would argue that's counterintuitive for respondents who see horizontally oriented grids in the great majority of surveys they do.
Hi Reg
In the slide set the Max diff is shown as taking 4 rows, with two radio buttons stretched out to the right. This means the respondent has to read down the four statements, then go back up and track across.
This grid layout works fine where each statement is being rated on the grid, but here the task you are asking them to do is vertical, not horizontal.
Remeber, the first thing the respondent does is to read the four statements. Once they have picked their best and worst they then use the scales.
My most preferred option is to show 4 boxes side by side, with the descriptions in them. The respondent is asked first to click on the best or most important box (which then changes colour), they are then asked to click on the worst or least important.
When that degree of scipting is not posisble, I show them in four boxes and put the radio buttons below the boxes.
Can I also be so impertinant to point out that (from English English eyes at least) the question on your MaxDiff questionnaire is a little odd. It asks for the "reforms that are more attractive" - which implies a multicode - and then only gives a single possible coded response. It also doesn't make it clear that these are 4 of many possible reforms. Perhaps the respondent got confused or hacked off, accounting for (some of) the high drop out? A pedant such as myself would say that once I have told you the least attractive you know by definition which ones are more attractive (all the rest).
Ray's point first. Answers to his queries a) and c) below:
a) Respondents saw 20 screens for the MaxDiff exercise, exactly as many as they saw for the pair exercises.
c) MaxDiff has the largest range in Exercise 2 (19.4), but is not much higher than Q-Sort (18.1), and Two-by-four is not far behind (14.5). In Exercise 1, Q-Sort has the largest range (31.7), with MaxDiff comfortably in second (20.0).
As for Ray's alternate display for MaxDif, maybe we need an experiment that tests the usability of different presentation formats since it has become so popular in multicultural research.
Pete -- to which treatment are you referring in your query? I'm having trouble responding.
Back to Ray, my marketing science colleagues tell me that their goal with MaxDif is to have sufficient exercises so that every R gets to see an option 3-5 times, this based on a recommendation from Sawtooth. I might argue that is still another example of the statisticians winning out over respondents or, at best, shows the difficulty of trading off statistical precision and respondent burden.
one page 12, method 4. Question states:
"which reforms are more important and which are least important to improving the American Healthcare system?"
I think it should be written:
"Out of the four possible reforms listed below which one is most important and whioch one is least important in order to reform the American Healthcare system?"
don't think it will account for all the drop out but it is is muddying factor (IMHO)
Hi Reg, if the methodologists wanted to ensure that every respondent saw every option 3-5 times in the MaxDiff, then the equivalent treatment for the pairs is for each respondent to see each item 4 times. In the study the MaxDiff shows each item 4 times and in the pairs is shows them twice, which IMHO is an issue with the comparison.
Maybe, maybe not. I might argue that one of the advantages of paired comparisons is that it is definitive: R likes a more than b and by this much. with MaxDif R may like a more than b, c, and d and like b less than c and d but we don't k now by how much nor do we know how she feels about d vs. c. Granted, we are more efficient by putting four options on the screen rather than two but are we getting as much information in return?
Pete has a good catch on the question wording. I also expect he's write that some Rs might have been put off but I doubt enough to impact the results.
Reg, a very good initiative this study. It also makes me happy to see that Q-sort methodology is included in this, since it is usually left out in most methodological comparisons. Ons thing that worries me though is the form in which Q-sort is presented to respondents. The radio buttons shown on the slides where quite different from what a Q-sort looks like in my experience.
Can I suggest to have a look at the following (fictive) example we used at a presentation at Rotterdam Erasmus University < www.q-research.nl/sample_eur >
best regards
Florent