Don’t End with Open-Ended Questions!

By: Michael Carli

{Thinking to Self}:

“Ok, just about done with this 100 question survey!

Phew… just a couple more……. WHAT!?

NOW, you want me to write a few sentences about my thoughts!?”

Isn’t that always the case? You’re nearly at the end of the survey that you didn’t want to take in the first place. You’re ready to pass it in and you’re mentally exhausted from the relentless barrage of questions related to your age, marital status, income, feelings, and thoughts. It’s just then on the last page that you realize the most difficult question of all is lurking: the open-ended question.

After taking Professor Smith’s mid-semester, feedback survey, I became interested in discovering if open-ended questions positioned at the end of the survey were more likely to be skipped by respondents. I decided to do some online searching to see if any research had been conducted on this matter. Interestingly to me, I found an academic study posted on the website, Survey Practice (www.surveypractice.org)).

Indiana University Ph.D. students, Angie Miller and Amber Lambert, used data from a Strategic National Arts Alumni Project (SNAAP) electronic survey to analyze over 27,000 submittals of an 82 question survey. The survey asked alumni to reflect on their educational institutions from which they’ve graduated. Miller and Lambert targeted 3 open-ended questions to conclude whether the placement of these types of questions received higher response rates depending on where they were chronologically situated within the survey.  They chose a question from the beginning, middle, and end to measure the rates of responses. It’s important to note that the electronic survey didn’t force any questions to be answered before the respondent could move on. Also, Miller and Lambert used the respondents’ demographics to determine if certain types of individuals were more or less likely to respond to open-ended questions.

As a whole, the study found that the respondents were more likely to answer open-ended questions when the questions were situated in the beginning or in the middle of the survey. The results from the study clearly show a drop-off in respondents’ willingness to answer open-ended questions at the end of the survey. Statistically, the open-ended responses were 68% from the beginning, 79% from the middle, and 24% from the end. Clearly, the findings show that the respondents grew tired of answering survey questions and opted to skip the open-ended questions when they were nearing completion of the survey.

What’s also interesting is that Miller and Lambert were able to use the demographic information of each respondent and identify which groups of people are more or less likely to respond to an open-ended question all together. The study shows that females were more likely than males to respond to open-ended questions in the beginning or the middle of the survey. However, there was little difference in gender when comparing response rates of open-ended questions towards the end of the survey. Other findings demonstrate that people over the age of 50 were significantly more likely to answer open-ended questions as compared to their younger counterparts. One could assume that open-ended responses require more time and effort. Thus, those people who may be retired or have less dependents to care for and/or may have more time to allocate to the survey. Similarly, unemployed individuals were more likely to respond to open-ended questions. Perhaps, it’s because they have more free time spend on the survey.

Another finding showed that those with negative opinions were shown to have a higher response rate than those with neutral or positive opinions. Most interesting to me, the respondents with lower income were more likely to answer all three of the open-ended questions. Psychologically, this is very interesting. Is it because people with higher incomes feel more important and prefer to spend less of their time answering survey questions? Lastly, another finding was that married people were more likely to respond compared to single people. Perhaps, single people are too busy looking for the loves of their lives?

From the research results, it’s very convincing that open-ended questions should be placed at an earlier stage to generate the most responses. Since many good ideas and concepts can be derived from qualitative, open-ended questions, I’m perplexed as to why so many surveyors put these important questions only at the end of their surveys. In the business world of marketing, marketers should be aware of the psychological behaviors pertaining to respondents and surveyors should be more informed of how to appropriately design a survey.   Certainly, it’s vital to collect qualitative data and from the aforementioned research it’s evident that surveyors are missing out on valuable information.

On Thursday, March 26^th, I plan to survey my classmates with a similar, but shorter survey compared to Miller and Lambert’s survey.   I plan to spread out three open-ended questions in the survey and determine if my shorter survey can corroborate the findings of the academic article. I plan to post these findings in the comments section of this post. Please check back shortly to put an end to the case of misplaced open-ended questions.