Intro to Survey Design

Chapter 3

Intro to Survey Design

By Ginette Law

Most people think conducting a survey is as simple as writing a bunch of questions and asking people to answer them. Easy, right? Well, sort of: it’s a little more complicated than that if you want to collect the best data possible. Don’t worry, though: we’re about to go over some key elements of survey design, starting with the purpose of your survey.

Purpose of a Survey

The first step in designing a good survey is to identify its purpose before you create it. A good survey collects accurate and verifiable data that allow you to make concrete claims. This is easier to do if you have a clear purpose to guide you when deciding what information you want to collect your respondents.

A survey generally helps you do one or more of the following:

• describe something;

• describe how things are related;

• explain a relationship; or

• influence something

So how do you come up with a nice, well-defined purpose for your survey? Try asking yourself the following questions:

• What are you trying to achieve with your survey?

• What, precisely, do you want to know?

• Why is this important to know?

• Is there other information that could be useful? Why?

• Is conducting a survey the right method for the type of data you’re collecting?

To see this process in action, let’s say you’re a media tycoon trying to increase your profits from television subscribers, so you’re interested in the potential returns from expanding to an internet market. You decide the main question and purpose of your survey are:

Research Question	Purpose
What percentage of television viewers watch their favorite television shows online?	Describe a variable The percentage of people who watch television programs online.

After thinking about it, you realize that it would be helpful to get more details, so you ask yourself some of those questions we just suggested:

Question	Answer
What are you trying to achieve with your survey?	To evaluate the profit potential of the internet market.
What do you want to know precisely?	How many people currently or would like to be able to stream shows online.
Why is this important to know?	It will help determine where we can improve our online service and whether it’s worth the investment.
Is there other information that could be useful? Why?	Which age group is more likely to watch TV online.
Is conducting a survey the right method for the type of data you’re collecting?	Yes.

Based on your answers to these questions, you expand the scope of your survey slightly:

Research Question	Purpose
How can we maximize profits from online viewership?	Influence something: Find out what needs to be done to improve our online service.
Do younger television users tend to stream more television programs online than older viewers?	Describe how things are related: Describe how a viewer’s age is related to how much TV he or she watches online.
If one group watches more programs online, why?	Explain a relationship: Explain why one age group prefers watching TV online more than another age group does.

Now you have a nice set of clearly-defined questions that your survey should address. The next step is to choose the best type of survey to help you expand your business.

Data and numbers can be very powerful, but sometimes people conduct surveys for the wrong reasons. You will miss the opportunity to gain meaningful insight with your survey if:

• You don’t really care about the results; you just want to show people you’ve got numbers.

• You misuse the data collected.

• You are more concerned by how people will receive your findings than you are with having reliable results.

• You have already determined what the results “should” be.

Remember, we do research to gain insight and to test hypotheses. That means it’s important to try and collect the most accurate and representative data possible and not just the data that support your own preconceived ideas and biases.

Types of Surveys

Now that you’ve identified the purpose of your survey, you can use this to help you choose a type of survey. When it comes to choosing a survey type, the first big decision you’ll have to make is how you want to distribute your survey to respondents. Surveys can be self-administered or administered.

Self-Administered and Administered Surveys

Self-administered simply means that respondents fill out a questionnaire by themselves, whereas administered means that an interviewer asks the questions. Each of these methods has advantages and disadvantages, summarized below.

Self-Administered Surveys

Pros	Cons
Good for limited budgets Large geographic reach Respondents can answer when convenient and at their own pace	More difficult to distribute to populations with limited literacy Risk of respondents misinterpreting questions Risk of lower completion rate Requires extra effort to verify that the respondent was actually the person to whom the survey was addressed

Administered Surveys

Pros	Cons
Tighter quality control Interviewers can clarify questions for respondents Can collect richer and more in-depth information Easier to reach specific or marginalized populations (e.g., the elderly or the homeless)	Can be expensive Can be time consuming Risk of interviewer effect (e.g., an interviewer’s interpretation of a question could bias some of the results)

The table above mentions the interviewer effect. This is a phenomenon whereby the interviewers themselves influence the responses. This could happen because of the way they ask questions or phrase explanations, or because of another social factor altogether. For example, maybe the interviewer is incredibly good-looking and the respondent unintentionally gives answers that are biased towards impressing the interviewer!

Self-administered and administered surveys are each associated with certain types of surveys, as shown below:

Types of Self-Administered Surveys

Web and Email Surveys

One of the easiest and quickest ways you can conduct a survey is online. You can either have people fill out questionnaires directly on a website or send them a questionnaire via e-mail. There are even survey sites that let you design questionnaire forms, collect data, and analyze responses in real-time at an affordable rate. You’ll find links to a few of these sites in the resources section.

Don’t forget to read the Privacy Policies and Terms of Use before signing up though—especially if you’re dealing with sensitive data—because these services are often subject to the laws of the countries in which they’re based.

The downside to all of this convenience is that only regular internet or computer users are likely to fill out your survey.

Web and email

Pros	Cons
Fairly fast results Integrated support tools for images, sounds, video Relatively cheap	Limited to people who have internet access Respondents can be a self-selected group of regular computer-users

Web

Pros	Cons
Can invite participants through social media Can use survey sites	Survey sites are subject to the laws of the countries in which they’re based

Email

Pros	Cons
Easier to survey a specific group	Need email addresses in order to distribute survey

Mail Surveys

Mail surveys are much like email surveys except that you’re sending a paper version of the questionnaire by mail. They aren’t limited to the population of internet users, but they can be more expensive because you have to pay for printing, paper, envelopes, and postage. You may also end up with a lower response rate because some people think it’s a pain to mail anything, not just your survey. Including a pre-addressed, stamped envelope with the survey will often improve your overall response rate, but will, again, increase the overall cost to you.

Mail

Pros	Cons
Open to a wider population	More expensive Lower response rate

Types of Administered Surveys

Phone Surveys

Compared to online surveys, phone surveys may take less time for you to collect all the data you need because respondents answer questions immediately. However, if you need to complete a certain number of surveys and most people hang up on you, it might take longer to collect data by phone!

It helps if questions are short and clear so that respondents can easily understand them over the phone. Try also to not take more than 15 to 20 minutes of your respondents’ time (both to be respectful of their time and to avoid losing their attention).

Another thing to keep in mind is that the interviewer and the respondent can’t see each other. This can be both good and bad: good because it lessens the interviewer effect but bad because both parties may miss non-verbal cues.

Pros	Cons
Relatively fast results Higher quality control because trained interviewers administer the survey	Can be more expensive Can’t use images or video when asking questions Need a list of reliable phone numbers

Face-to-Face Surveys

Unlike phone surveys, face-to-face surveys allow the interviewer and the respondent to see each other’s facial expressions and body language. This can be helpful because the additional physical cues can help the interviewer and the respondent understand better understand each other; however, it can also lead to the respondent being further influenced by the interviewer’s behavior and appearance.

Face-to-face surveys are limited geographically if interviewers and respondents have to meet in person. If they don’t have to meet in person, the surveys can be conducted online using video conferencing software such as Skype or Google Hangouts.

Pros	Cons
Can collect more in-depth information Good for specific or hard-to-reach populations Better for long surveys	Can be more expensive May have a stronger interviewer effect Can take longer to train interviewers and to complete enough surveys

Handouts

So far, we’ve talked about self-administered and administered surveys, but one of the most frequently-encountered type of survey is actually a combination of these. Say you want to hand out paper surveys and have people complete and return them immediately. The survey itself is self-administered, but since you have a trained person there who is available to answer questions, it also has features of an administered survey.

For this reason, handouts can be a good option if you have a limited budget but want to make sure you have someone available when the survey is completed to clarify any questions your respondents might have.

One of the disadvantages of handouts is that people may be rushed to complete the survey if you are catching them in passing, which can affect the quality of your data. You will also be limited to the population that is physically present in the location where you are giving the survey. This may not be an issue if you are targeting a specific group, such as college students, shoppers at a particular store, or residents of a certain region. If you’re looking for a more general audience, however, you may consider handing the survey out in several different locations to reach a more diverse audience.

Choosing a Survey Type

Whew! That is a lot of information so far. As you can see, there are many types of surveys, each with its own pros and cons. With so many factors to consider, how do you decide which one to choose? Let’s walk through a few scenarios of choosing a survey type. Remember, you’re a media tycoon trying to expand your media empire.

Let’s say you want to evaluate the potential of the internet market. If you’re only interested in the online viewing habits of internet users, then a web or email survey cheaply and conveniently targets the group you’re interested in. On the other hand, if you want to know how much of the general television viewing audience watches shows online, but you don’t want to spend the time or the money to train interviewers, then a mail survey is a better option.

But wait a minute: you’re a media tycoon! You have lots of money and you’re a total data geek. You want to collect more in-depth information like:

• How do families watch television programs in their homes?

• Which family members tend to watch TV online?

• What type of television program does each family member like to watch?

• Do they watch programs together?

• If so, what types of programs do they watch together? Do they ever fight about what to watch?

In this case, a face-to-face survey is a great option because you can interview all the members of a household at the same time and collect higher quality data about the dynamics of television viewing in families.

Now let’s say you’ve analyzed the data and the numbers look promising, so you decide to create a new online viewing experience for your users. You want to emphasize how great and cutting-edge your new service is by creating a catchy name and logo for it. If you want to see what people think about different logos, then phone surveys won’t be helpful. On the other hand, if you want to see what people think about different names and slogans, then phone surveys are fine. If your new service is specifically targeting a younger audience of avid internet users, then you can avoid the extra cost of phone surveys and use an online survey instead.

If you have the time and resources to do so, you can also consider using more than one type of survey. For example, you may want to do an online and phone version of the same survey to increase response rates and improve your sample in both size and diversity. Or you may start by doing an online survey and then conduct a face-to-face survey afterwards to gain more insight from your initial results.

We hope these scenarios show you that choosing a survey type isn’t a one-size-fits-all process. There isn’t a nice and tidy formula that you can use to pick the perfect survey. Instead, it’s up to your good judgment to balance the resources you have with the goals you’re trying to achieve. </div>