Support

1. Try to keep it as short as possible. The longer it is, the more your respondents will get bored and the less accurate your answers are likely to be. Therefore, try to minimise the questions that would be "nice to know" and concentrate on the really important stuff wherever possible;
2. Only ask questions that will give you information you can use - again to keep the questionnaire as short as possible;
3. Always tell people how long the survey will take them before they start - this will minimise the drop out rate and maximise the response rate. Besides, it's considered best practice and it's polite;
4. Be careful how you structure your questions. It's all too easy to lead the respondent by asking questions in the wrong way. For example, don't ask "When would you buy this?" or "how much would you pay for this?" unless they have already said that they would buy it. Instead, ask "Would you buy this?" first. Similarly, don't ask "How much do you agree with the following statements?" - there is evidence that you will get more accurate results if instead you ask "How much do you agree or disagree with the following statements?";
5. Don't use jargon - always use straightforward language - particularly in consumer surveys. You may be spending all your time thinking about your product or service, but you should use only language that someone who is not involved in it will understand. Often surveys, even from large companies who really ought to know better, include questions that are completely incomprehensible. Don't use jargon or buzz words unnecessarily.;
6. Give people the right options. You should not insist that people complete each question without some thought to the options you are offering. Too often, even market research professionals working for some very large companies who ought to know better are guilty of insisting that the respondent should click on an answer before they can move on through the survey. If you haven't given the right options, the only way the respondent can move on is to click on the wrong answer. Effectively you're making the respondent lie in order to complete the survey - which they may do in an effort to be helpful, or to earn any incentive you're offering. Unfortunately you will never know if this is the case, so give careful consideration to including an "other" option on each question. It's easy to do this, and to ask what the "other" is if that option is clicked;
7. Try to keep personal questions to a minimum - only ask those that you really need. Also, you may get a better response if, for example, you allow people to place themselves into brackets or ranges for questions such as "How much do you earn?" or "How old are you?", rather than asking them to be specific.
8. Ask personal questions at the end of your survey if possible. People are more likely to give you this information once they've spent some time on the questionnaire and many will find it intrusive when it's asked up front, so unless you need to filter out your respondents, our advice would be to keep questions about age, location, income and so on to the end.
9. Bear in mind that a lot of respondents will probably take part in your survey on a mobile device. Design your questionnaire accordingly – for example, avoid "grid" or "matrix" questions, as these are difficult to complete on a phone. It's best to use simple click on the button questions wherever possible, and also avoid free text responses.

Sample size

How many do I need?

A small sample is better than nothing, but of course, the bigger the better.

Sampling theory is based on probability analysis, and it is possible to calculate how big a sample you need for a given level of reliability. Of course, the only way to be completely sure of your results is to interview everyone in your target audience, but this is rarely achievable. That's why we use samples and it's possible to generate sample that are very reliable, even though they are only a fraction of the total population.

For example, you might want to generate a sample that will give you a high level of confidence in the results and usually market researchers work to 95% or 99% confidence levels. However, there is still a margin of error, even with large samples, and it's important to know what this is.

Let's say you want to sample a population so that you have a 95% confidence level and a margin of error of 5%. This would mean that you can be 95% sure that your sample's responses will be within 5% of any other sample drawn from the same population. This also means that any differences in responses which are less than 5% are unreliable.

You should also bear in mind that any sub samples within your overall sample will be less reliable, and ideally you'd want to sample around 400 of every group within your sample (so you might want to ensure you have 400 women and 400 men in your sample, so the results are reliable for each sub group).

Here are some guidelines on sample sizes and their relative levels of reliability (based on large populations):

In gathering your sample, it's important to ensure that the people you're sampling are representative of your target group. It's very easy to introduce a bias or skew the results without realising it. A common example of this is when businesses offer one of their own products as an incentive for taking part in a customer satisfaction survey. If a customer is not satisfied (and one would obviously want to know this) they are unlikely to want another product from the business with whom they're not satisfied and so will be unlikely to take part in the survey. Too often, businesses think their customers are all very happy, when it's only the happy ones who take part in the survey.

Another mistake might be trying to research people who are buyers of a competitor's product. There may be a whole untapped market of people who would buy your product but who would not buy your competitor's, perhaps because it's too expensive, badly designed, too difficult to use, or whatever.

People now have more ways than ever to access the web. When you're designing your questionnaire, bear in mind that your respondents may be accessing it on a smart phone or tablet.

Therefore, you need to consider that some types of questions are easier to read and answer than others. For example, a matrix question with several columns may not fit on the screen unless it's so reduced in size that it becomes unreadable.

So, if you think some or all of your respondents may be accessing your questionnaire via a mobile device, you need to make it as easy as possible, so try to stick to questions in list or drop down format rather than matrices.

In any case - it's always sensible to preview your survey on a screen and also on a mobile device.

There are a number of things you should consider when trying to maximize your response rates:

1. Keep your questionnaire as short as possible;
2. Keep it simple and easy to complete;
3. Only ask things that are relevant;
4. Let people know how long it's going to take (and be realistic), so you don't lose respondents half way through;
5. Consider offering an incentive - you're asking people to give you some of their time, so it's only reasonable to offer something in return (see "incentives" below);
6. Avoid repetition - few things make a survey more boring than asking the same thing (or appearing to) again and again;
7. Let people know why you're asking them. What's it for? What will depend on the outcome?
8. If it's for genuine market research, don't use your questionnaire to sell - and tell people you won't be using it to sell to them;
9. Promise confidentiality (and mean it!). Unless you particularly need to know who said what, then keeping the responses confidential will not affect the usefulness of the survey;
10. Don't forget to send a reminder! Sending a carefully worded reminder to those who have not yet responded can have a very dramatic effect on your response rate - as much as doubling it. But we'd advise not bothering people more than two or three times...

Depending on what kind of survey you're running, you should consider offer your respondents something in return for the time they spend helping you. This will improve your response rates and, after all, it's only polite to say "thank you"! Incentives are not necessary for employee surveys, and if you're a charity, it may not be appropriate to offer anything as an incentive, unless you can persuade someone to sponsor it.

The incentive could be a prize in a draw (i.e. one person gets a big prize) or vouchers or coupons (where everyone gets something). The latter option is potentially more expensive as you'll need to pay everyone who takes part, which could get expensive if your response rate is high, so don't forget to set your quota limit (in your survey settings).

Be careful of incentives in surveys aimed at people at work, as many organizations have policies which forbid employees accepting gifts. In this case, you could offer an incentive that would be useful for the organization, or a copy of the report from your survey. You could also offer a donation to a specified charity for each completed questionnaire, though there is research that suggests this is not highly effective.

Be careful that your chosen incentive doesn't skew your results. For example, offering tickets to certain venues or events, or particular kinds of gifts, could let you down because they only appeal to certain segments of the population, so choose with care.

Do not offer your own goods or services, or those of your client, as an incentive. It's bad practice, will get you unreliable results and could be interpreted as direct marketing.

Given that you're dealing with people who are online, then it's probably safe to offer vouchers for an online store that offers a wide range of products, so that there will be something to appeal to everyone - even business respondents.

So - you have your survey complete, with plenty of results - now what?

The simplest thing is to let our software do the work for you. Once you have your survey complete, click on the "Results" page and look at your results question by question, changing the charts as you wish.

You can filter any question by any other question, so if you've asked about height and nationality, you can choose to look at the results for only one country's respondents. Alternatively, you can compare by sub-sets - for example, comparing height for two specific countries.

Or - you could use the auto analysis feature, which will find statistically significant results for you. It will automatically spot differences in sub sets that are large enough to be above the margin of error for that particular sub-sample, at a level of confidence of 95%. It will also find correlations between different groups:

• Look at the "Significant differences" option to find, for each question, any statistically significant differences among sub-samples who have answered other questions differently;
• Look at the "Median deviation analysis" to find sub-samples who answered a question in a particular way which differ by a significant degree from the median of the sample;
• Look at "Correlative analysis" to see associations between sub-samples who answer other questions to a higher level than the norm.

When conducting your analysis (or letting us do it for you), concentrate on the results that are going to be really useful. It won't be much help knowing that people of a certain height may prefer your product if you have no way of targeting them, for example!

Many people who design online surveys make it a requirement for the respondent to answer each question - usually just because they can. Often this is unnecessary and at worst it can annoy the respondent and lead to misleading information. If you don't offer the right options for the respondent to choose from, and then make it a requirement to answer the question, the respondent will get stuck at a point in the survey with the only choice being to discontinue the survey or give wrong information to continue - neither of which you want. For that reason, we tend to discourage required answers for anything other than filter questions, and if you do need to make it a requirement, offer an "other" or "don't know" option.

You will get a much better response to your surveys, especially on personal subjects, if you assure the respondent of anonymity. Apart from that, it is a requirement of the code of conduct of the major market research associations and societies to protect the identity of respondents. Specifically, you must ensure that if you're conducting a survey for a client or other third party, that responses are only available anonymously or as part of an aggregated analysis.

You should only reveal respondents' identities when they have given specific permission for you to do so.

You can use social media for your surveys and these can be a good source of respondents - often at little or no cost. You simply need to paste your survey's web address and wait for people to come by your survey. However, it's important to think about whether the people who might take part are representative of your target group. Also, if the people taking part are in your own circle, are they truly representative? Are the media you're using representative? For example, if you're selling to all consumers, can you be sure that the sample you get from social media is not skewed in any way?

Even with social media, it can be difficult to get a large sample. If you have a database you can email them via the Invitation Manager, which will send out your email invitations, personally addressed, automatically.

You can also paste the link to your survey on websites, or publish it in newsletters, etc. Wherever possible, we recommend using electronic means to publicize your survey, as this means that people can click on a link and be involved instantly - at very little cost.

Alternatively, you might consider buying a database of email addresses of people to whom you can send your survey invitation, but make sure it's a reputable supplier (both from a legal and results point of view). There are also companies that run "panels" of people who are willing to participate in surveys and who are paid for doing so.

When developing your questionnaire, keep your language simple and easily understood.

Avoid using jargon and industry-specific buzz words and phrases, along with business clichés such "going forward". You may well be asking respondents questions about a product or service you work with all day, every day, but it may not be something your respondent thinks about as often, so you need to make sure you’re not confusing them with terminology they won’t understand, or asking questions that will be meaningless to them.

Keep your questions and sentences short and simple, that way the respondent can easily understand what you’re asking about.

You can invite people to take part in your survey by email (our Invitation Manager makes this very quick and simple).

The way your email is written can make a big difference in the numbers of people who decide to take your survey. Make your subject line interesting, without looking like spam - it’s probably best to avoid the use of the word "survey" in the subject line, along with words like "win" and $ or £ signs.

Keep your email short and to the point – resist the temptation to write a long and wordy explanation. If you’re running a genuine survey you should assure the recipient that you are only contacting them for that purpose and that you will not be attempting to sell them anything.

Tell the recipient roughly how long the survey will take (be honest) and tell them what’s in it for them – this may be an incentive, a copy of your results, or just the satisfaction of helping you. Try to do this near the start of the email.

Did you know that when you create a survey with Survey Mechanics you can make live edits – even after testing it and delivering it to your respondents? This is invaluable if you’ve launched your survey and then find you’ve forgotten an option in a question, or early results are showing that you’ve missed something.

If you haven’t collected any responses yet, you can obviously edit to the n^th degree – within reason, of course, but if you have there are a few things that you should consider:

• You can easily add an answer as well as edit the text of your answer options, but bear in mind that your survey should still be straight-forward and simple, you don’t want to add too many answer options and confuse your respondents.
• You can also add pages to your survey, but, just as when you add questions, you need to review it before sending it out – the order of your survey should still flow and make logical sense.
• Any changes and edits that you make will take immediate effect, and the survey link does not change so you don’t need to send respondents a new link.
• If you are making changes to surveys that have already been sent out, any respondents taking part while you are live editing may experience issues, so you may want to close the survey temporarily.
• Always be mindful of any responses you have already collected – when you delete a question, you also delete the answers attached to it. To avoid this, you can simply add a new page to substitute for the page you’re changing without the question you’re deleting, and then alter your survey’s routing so it skips the old page and includes the new one in the right place. This way you’ll keep your data.
• If you have only a few responses, you could always copy your survey and make the edits on that new version, and then re-send the new and improved copy.
• Whatever edits you make, we strongly recommend making a note of the date and time you made them, so you can tell if any blank answers are before your edit, or simply people ignoring your question!
• And last, but certainly not least, if your survey has only recently been sent out, chances are your respondents won’t notice any changes, but if that’s not the case it could have a big impact in the data you’ve collected so far and drastically affect your survey results.