Stay true to your Form

In my most recent post I showed what happened when you make a change in one of the answer options in a choice question in Forms. That case was based on a real-life scenario.

You will know by now that these things trigger my curiosity, so I decided to find out what happens to other question types when you change them halfway?

I thought about question types that can be changed significantly and came up with Text, Rating, Ranking and Likert.
So I started out with this questionnaire:

The survey start – 4 changeable questions

And after entering 10 responses randomly, this was the first result:

Results 1

With a score of 4.20 this must be a good blog post! πŸ™‚

Then I made the first set of changes:

  • In the Text question, I changed from short text to long answer
  • In the Rating question, I changed the 5 stars to 5 numbers
  • In the Ranking question, I replaced “Darts” with “Figure-skating”
  • In the Likert question, I changed “Forms” to “Visio”

Again, I added 5 random responses, and ended up with these results:

Results 2

Results after the first change

You only see the last 3 responses for the Text question, the answers appear a bit more wordy. The word cloud is a bit fuller, but that’s it.

Exchanging stars in the Rating question does not make much difference, you get a number rather than stars, but that is not significantly different.

For the Ranking, the change is rather more alarming as the results for Darts are replaced by those for Figure skating, and the name Darts completely disappears. I had expected that the results would be separated, and that Darts would be called “Other”, like with the choice options, but apparently not.

For the Likert question, again all votes for Forms are now turned into votes for Visio, and the word Visio has disappeared, there’s no “Other” like in the choice question.

So, it was time to make some more changes:

  • In the Text question, I changed to a Number
  • In the Rating question, I changed the 5 numbers to 10 numbers
  • In the Ranking question, I added “Cycling” as an option
  • In the Likert question, I added an option “N.A. or don’t know”

And when I had added 5 responses, the results looked like this:

Results 3

Results after the second change

The Text question now shows the last 3 numbers PLUS the old word cloud. Most confusing!

For the Rating question, the average was 4.9. I had made several entries in the 6-10 region, but as there were at least15 entries with a lower number, the average had not increased that much.
But…when you have a 5-point scale and score a 4.2 (as in the second result) you are doing very well. If you have a 10-point scale, 4.9 is pretty low.

For the Ranking question, the new sports gets added to the results. There’s only 5 responses for this option, which skews the results. 20% firsts for this sport is not as many as 20% for the sports that have been there for the entire ride. πŸ™‚

For the Likert question, a change of colour may confuse you. The “Yes” option was blue, but has turned to grey. This makes perfect sense from a visual design perspective, but is a bit confusing otherwise. Again, as with the Ranking, there’s only max. 5 responses of “N.A. or don’t know” possible, compared to max. 20 for the other options, so this will skew results.

I also looked at the Excel.

The detailed results in Excel

The Text results are a mess. Yes, you could score all answers as a number, but then you might have as well sent an email or interviewed everyone.

You see that the first 15 Rating scale answers are not multiplied by 2 after moving from a 5-point scale to a 10-point scale. This will result in a much lower score than you would have had if you had started out with a 10-point scale straight away. This will lead to the wrong conclusion: that people do not like this blog topic, while in fact they do!

For the Ranking, you see that the word “Darts” no longer appears in the responses, it gets overwritten after the change. So that darts fan is now suddenly a lover of figure skating. πŸ€¦β€β™‚οΈ It also means that cycling fans will hardly ever be in the majority as that option was added quite late in the process.

For the Likert question, the results between the options are not comparable. Everyone who thought they were answering a question about Forms, have now answered the question for Visio. And “N.A. or Don’t know” may have scored differently when the option had been there throughout the survey.

I could have made more changes, but I trust you will get the gist.

Conclusions

  1. Mid-survey changes are technically very easy to do.
  2. Mid-survey changes are not visible as they are not logged (not even in the Excel file). This will make it very hard to interpret the results. It will even be difficult for the owner, as they do not know at exactly which moment between entries the change has been made (except when adding an extra option).
  3. The good news is, that for Rating scale and Likert questions you can easily correct a typo. In the Choice question types even that creates a new option, as we have seen before.
  4. Mid-survey changes can create havoc in the results. In fact, you are changing the rules in the middle of the game. Results become incomparable or and conclusions may be completely incorrect (such as changing Rating from a 5- to a 10-point scale).
  5. You may want to check the Excel for the exact results, but that does not help when drawing conclusions. Are you sure that someone who likes to watch Formula 1, would have the same preference when (s)he would have had the option to select Cycling?

This will not come as a surprise, of course. I only hope I have made things more clear by having shown what happens exactly when you make a change. It is tempting as it may look like a small and easy change, but it can have unpleasant consequences.

As mentioned in my earlier post, I would rather suggest that before launching your survey, you

  • discuss the setup of your survey with your colleagues – e.g. which sports do they want included in the answer options?
  • ask one or more colleagues to complete the survey to see if they understand your questions and the answer options are logical, complete and spelled correctly
  • remove all test entries or copy your survey before it goes live
  • resist any push to make changes when the survey is running, even if it comes from higher management. Show them this blog when they insist. πŸ™‚

Do you have any experiences with making mid-survey changes?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s