Don’t End with Open-Ended Questions!

By: Michael Carli

{Thinking to Self}:

“Ok, just about done with this 100 question survey!

Phew… just a couple more……. WHAT!?

NOW, you want me to write a few sentences about my thoughts!?”

Isn’t that always the case? You’re nearly at the end of the survey that you didn’t want to take in the first place. You’re ready to pass it in and you’re mentally exhausted from the relentless barrage of questions related to your age, marital status, income, feelings, and thoughts. It’s just then on the last page that you realize the most difficult question of all is lurking: the open-ended question.

After taking Professor Smith’s mid-semester, feedback survey, I became interested in discovering if open-ended questions positioned at the end of the survey were more likely to be skipped by respondents. I decided to do some online searching to see if any research had been conducted on this matter. Interestingly to me, I found an academic study posted on the website, Survey Practice (www.surveypractice.org)).

Indiana University Ph.D. students, Angie Miller and Amber Lambert, used data from a Strategic National Arts Alumni Project (SNAAP) electronic survey to analyze over 27,000 submittals of an 82 question survey. The survey asked alumni to reflect on their educational institutions from which they’ve graduated. Miller and Lambert targeted 3 open-ended questions to conclude whether the placement of these types of questions received higher response rates depending on where they were chronologically situated within the survey.  They chose a question from the beginning, middle, and end to measure the rates of responses. It’s important to note that the electronic survey didn’t force any questions to be answered before the respondent could move on. Also, Miller and Lambert used the respondents’ demographics to determine if certain types of individuals were more or less likely to respond to open-ended questions.

As a whole, the study found that the respondents were more likely to answer open-ended questions when the questions were situated in the beginning or in the middle of the survey. The results from the study clearly show a drop-off in respondents’ willingness to answer open-ended questions at the end of the survey. Statistically, the open-ended responses were 68% from the beginning, 79% from the middle, and 24% from the end. Clearly, the findings show that the respondents grew tired of answering survey questions and opted to skip the open-ended questions when they were nearing completion of the survey.

What’s also interesting is that Miller and Lambert were able to use the demographic information of each respondent and identify which groups of people are more or less likely to respond to an open-ended question all together. The study shows that females were more likely than males to respond to open-ended questions in the beginning or the middle of the survey. However, there was little difference in gender when comparing response rates of open-ended questions towards the end of the survey. Other findings demonstrate that people over the age of 50 were significantly more likely to answer open-ended questions as compared to their younger counterparts. One could assume that open-ended responses require more time and effort. Thus, those people who may be retired or have less dependents to care for and/or may have more time to allocate to the survey. Similarly, unemployed individuals were more likely to respond to open-ended questions. Perhaps, it’s because they have more free time spend on the survey.

Another finding showed that those with negative opinions were shown to have a higher response rate than those with neutral or positive opinions. Most interesting to me, the respondents with lower income were more likely to answer all three of the open-ended questions. Psychologically, this is very interesting. Is it because people with higher incomes feel more important and prefer to spend less of their time answering survey questions? Lastly, another finding was that married people were more likely to respond compared to single people. Perhaps, single people are too busy looking for the loves of their lives?

From the research results, it’s very convincing that open-ended questions should be placed at an earlier stage to generate the most responses. Since many good ideas and concepts can be derived from qualitative, open-ended questions, I’m perplexed as to why so many surveyors put these important questions only at the end of their surveys. In the business world of marketing, marketers should be aware of the psychological behaviors pertaining to respondents and surveyors should be more informed of how to appropriately design a survey.   Certainly, it’s vital to collect qualitative data and from the aforementioned research it’s evident that surveyors are missing out on valuable information.

On Thursday, March 26th, I plan to survey my classmates with a similar, but shorter survey compared to Miller and Lambert’s survey.   I plan to spread out three open-ended questions in the survey and determine if my shorter survey can corroborate the findings of the academic article. I plan to post these findings in the comments section of this post. Please check back shortly to put an end to the case of misplaced open-ended questions.

Advertisements

5 thoughts on “Don’t End with Open-Ended Questions!

  1. Linsey Walker March 30, 2015 / 4:02 pm

    Hi Mike,

    I really enjoyed your presentation. I think this is valuable information not only for marketers, but for anyone conducting a survey containing open-ended questions for any reason. The issue of getting responses to open-ended questions is not one I have put much thought into, but anecdotally I could have guessed from experience that open-ended questions at the end of surveys are likely to go unanswered. It’s valuable to have researchers actually examine this question to see what factors affect the response rate to open-ended questions, so that survey administrators can both ensure an acceptable response rate and also take into consideration what characteristics those respondents likely share.

    Like

  2. McGee Hines April 25, 2015 / 8:11 pm

    I could not agree more with your thoughts Michael! Now thinking of this theory, it almost seems silly that a survey or questionnaire would ever be laid out in such a manner. Personally, I find that multiple choice, or rating-based questions are a pain to fill out, but can be very helpful to one’s research. However, when going about the survey if one is not totally interested or committed to getting a helpful, solid outcome by filling out the questionnaire, they are also less likely to want to dedicate their time to come up with thought-out answers to open-ended questions in addition. I, too feel as thought persons conducting research should construct a better way for people to elaborate on their responses when filling out surveys, that were obviously given to them for a reason- to get their individual, varying opinion!

    Like

  3. Charlie O'Connor May 7, 2015 / 4:13 am

    I’m not surprised that the majority of open ended questions at the end of a survey go unanswered. I’m certainly guilty Of skipping the open ended portions, aren’t they opional? Marketers and survey administrators should definitely be aware of the psychological behaviors of respondents when faced with an open ender. Great write up Michael, where’s the follow up comment?

    Like

  4. Nick Pappalardo May 12, 2015 / 7:08 pm

    I feel like this is very accurate. I know when I’m given a survey to fill out I’m much less likely to answer the open ended questions at the end. By the time I get to the end I don’t want to be writing a small paragraph to answer the questions. We actually talked about this idea in one of my undergraduate course, Marketing Research, where we made questionnaires of our own. We were told the exact same thing, That if we were going to ask open ended questions, we should do it at the beginning rather than the end.

    Like

  5. Megan Lac May 6, 2016 / 12:26 am

    As a psychology major who took psychology inquiry and methodology, a class based on surveying and research we learned that the order of questions does matter. The placement of certain questions affects the answer of questions that comes after it. That is why in a well written survey questions may be asked in a few different ways throughout so the answer is more likely to be accurate. Which is also why the placement of an open ended question matters as well. People’s attention span and willingness to help fades after so long, therefore putting the questions at the end is not beneficial. Thats why its important for marketers to take psychology into account.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s