From Creation to Launch

Overview

This walkthrough offers guidance on best practice for creating and launching an experiment in Gorilla. Check out the topics in the menu for advice on each step.

Ethics/IRB Approval

You may need ethical approval from your Internal Review Board or Ethics Committee to run your experiment. If so, you will find a useful template on our Ethics Applications page.

Create the questionnaires/tasks

The first thing to do is to create the different parts of your study: these are your individual Questionnaires and Tasks (where you can record response times and accuracy etc).

Even a simple experiment is likely to have three parts:

1. A consent form
2. A demographics questionnaire
3. A task.

As you create these parts, use the preview and play functionalities a lot! These will help you get everything working exactly as you want. Particularly, make sure you've tested your experiment thoroughly across all the browsers and devices that you intend to support. We have created a helpful tutorial to preview Tasks and Questionnaires on different devices (iPad, iPhone etc).

If you have different media files throughout your experiment, check out our Technical Checklist to ensure your audio / video / image files will be displayed correctly.

Once the individual questionnaires and tasks of your experiment are working, make sure you preview each of them and look at the data. Scrutinising the data will help you be confident that Gorilla is collecting the metrics that you need. For example, check that you can identify each condition and each dependent variable in your task.

When you commit (save) versions in Gorilla, you can write a commit message. Use this to write a note to your future self about what is working and what is left to be done. This helps when you return to building the task or questionnaire after a break. Your future self will thank you!

Learn about Questionnaire Builder 2

Learn about Task Builder 2

Learn about the Experiment Tree

Build your Experiment

Once you have created the individual tasks and surveys, put them together in the experiment tree! Simply add new nodes to your tree and link them together to map out the journey that your participant takes through your experiment.

See our Tree Nodes Guide for details on all the ways you can control how participants enter, progress through, and exit your experiment.

Previewing

Once you have created your experiment, you can experience it almost as a participant would by previewing. You can then download the data for each questionnaire/task individually at the end. We recommend you do this at least once before piloting your study.

Take your time looking over your data output, making sure you understand the data columns, and looking at our troubleshooting guide if your data looks wrong.

• If you have Branch Nodes, test the experiment for each possible response, to ensure your branching is working correctly. We have a guide which explains common Branch Node mistakes and how to fix them.
• If you are using Randomiser Nodes or Counterbalance Nodes, the preview will assign you to a condition at random, without taking previous previews into account.
• If you are using Quota Nodes, the preview will not send you along the reject path, as previewing does not consume tokens, and so will not fill your Quotas.
• If you are using an Order Node, the preview will only give you one of the possible orders.
• If you are using a Counterbalance Node, ensure you have checked your spreadsheets or manipulations are configured correctly.
• If you have a Delay Node, set the delay to 1 minute for the purposes of previewing, then change it to the final duration before you commit your experiment.

Committing

After previewing your experiment, you may find you need to make changes to one of the Questionnaires or Tasks. Once you have made these changes you’ll need to commit them, update your nodes in the experiment tree, and then commit the experiment.

Auto-commit and update nodes on experiment commit

In New Experiment Builder, when you commit the experiment, Gorilla will alert you to any nodes that you are currently editing:

Clicking 'Commit all changes and update all nodes' commits all open changes to all activities in your experiment, and updates all nodes to use the newly committed versions.

If you do not click this button before committing your experiment, the experiment uses the latest committed versions before you started making your changes.

Gorilla will also alert you to any nodes that are committed, but where the experiment tree is not using the latest version:

Clicking 'Update all nodes to latest versions' updates all nodes in your experiment tree to use the latest versions.

If you do not click this button before committing your experiment, the experiment uses the version currently selected in the experiment tree.

Checkpoint Nodes

Checkpoint Nodes are a great way to track participants' progress through your experiment and simplify the process of deciding whether to reject them or include them in your downloadable data.

Place Checkpoint Nodes after consent and demographics questionnaires and at the beginning of different paths through your experiment from Branch or Randomiser Nodes. Checkpoints can help you identify any potential problems in your experimental design. They are also a great help when it comes to analysing your data.

This video shows you how to use Checkpoint Nodes to monitor participant progress and determine whether to reject or include participants.

Length (mins): 2:37

Pilot Your Experiment

Before you launch your experiment you should collect some pilot data, such as from friends or colleagues, so that you can get some feedback. We suggest using the Pilot Recruitment Policy. Because you are now running an experiment and collecting real data, this process will consume tokens.

If you want to test your experiment live without collecting data, you can use a Reject node to avoid consuming tokens. However, we would recommend running at least a small pilot (see below) where you do collect data so that you can see what it looks like and how you might analyse it.

The Pilot Recruitment policy requires participants to type in some text as an ID. You can create IDs that help you remember what functionality you are testing. For example, Test_1 or Testing_Branch_1.

The pilot ID can also be useful when you want feedback from your whole lab. You can send out the link and each person can use their name. We use this recruitment policy internally at Gorilla to test experiments, too!

Remember to set a Recruitment Target.

Learn more about the different Recruitment Policies.

You should also complete a small pilot with some real participants. Let’s imagine you’re using Facebook to recruit participants. We’d recommend initially launching just a few participants (5 to 10) to allows participants to raise any questions. Don't forget to change the Recruitment Policy!

You may want to include an additional feedback questionnaire at the end of your experiment. Check out our sample: Generic Pilot Feedback Questionnaire

Once you've finished piloting and taken the feedback on board you can then remove this questionnaire from your Experiment Tree.

Learn more about your Data and how to understand your data and find out what each column means.

Check Your Data

Once your colleagues or friends have tested your experiment, download the data files from the Data Tab and check that you have everything you need to run your analysis. We have guides on how to understand your data, or what to do if your data doesn't look as you expect.

Always check:

• Scoring is working correctly. Learn more about scoring in Task Builder 2 and scoring in Questionnaire Builder 2 in the relevant How-To guides.
• Data is saving to the Store correctly. You can find some more information about the Store in Task Builder 2 and the Store in Questionnaire Builder 2 to check your configuration.
• You can easily identify each independent and dependent variable for your analysis.
• Checkpoint Nodes are all in the right places.

Check your data workflow

Now that you've got pilot data, it’s time to run through how you are going to do your data pre-processing and data analysis. This can also be a good time to define exclusion criteria so that you end up with high quality data.

Running through your data analysis workflow gives you an opportunity to add metadata, checkpoints, and make corrections - without using up a lot of tokens and losing potential participants (and their data!).

Data pre-processing:

Data pre-processing is the process of taking data from Gorilla and manipulating it into the right format so that it can be analysed in your software of choice (such as SPSS, RStudio, or JASP). This usually involves combining data files and calculating summary data for each participant.

Gorilla can provide questionnaire data in short form or long form. Task data is always provided in long form as only you know how you want to process your data and what summary statistics you want to calculate, such as which measure of central tendency you need (e.g. the mean or median), how different conditions are defined in your study, and which information you want to aggregate (e.g. accuracy, response times, or slider ratings to name a few).

As of April 2025, you can now automatically combine data across multiple tasks or across multiple questionnaires when downloading your data from the Data Tab. We cannot aggregate data for you across tasks and questionnaires, as National Institute for Health and Care Research (NIHR) and British Psychological Society (BPS) guidelines are to keep demographic and performance data separate.

Check our our useful tutorials about using Excel and RStudio, including a video guide to combining task and questionnaire data in Excel and an R script that transforms long-form data to short-form.

Launch Your Experiment

If you're happy that the following two points are true, then you're ready to launch your experiment for real:

• Your experiment is working smoothly for real participants.
• You are collecting all the data you need for analysis from real participants.

The next steps are:

1. Set any requirements for accessing your experiment. For example, you may want to limit access to participants using computers, or from a particular location.
2. Set a time limit for your study. This is optional, but helps manage participant attrition.
3. Set a recruitment target for your experiment. This assigns the right number of tokens for the number of participants you want to recruit.
4. Select a recruitment policy that meets your study's requirements.
   • Crowdsource your participants by posting a Simple Link on Facebook or other social media channels.
   • If you already have a list of participants, then Email ID or Email Shot may be the right recruitment policies for you.
   • Many researchers use Third Party Recruitment Services. Check out our guidance on using recruitment services.

You can browse our full list of available Recruitment Policies.

If participants are having trouble accessing your experiment, check out our Troubleshooting Guide.

We have also created a short video that walks you through the process of launching your experiment:

You could consider adding specific Requirements to your experiment. This could be to limit the devices, browsers, or connection speeds of participants you recruit.

Attrition

Participant attrition - the rate of participant dropout - is a factor in any experiment, whether in the lab or online.

A major benefit of conducting research online is the increased scale and reach. The result: participant sample sizes in the 1000s rather than 10s or 100s is now an achievable possibility!

However, the upshot of it being easier to join an experiment is that it is also easier to leave one. You should expect the attrition rate for online experiments to be higher than for the same experiment conducted in a lab. There is no way of knowing why they have stopped, and for ethical reasons, participants have the right to withdraw at any time.

When you set a recruitment target, participants who are currently doing your experiment ('Live' participants) contribute towards your target.

Example: Your recruitment target is 20. 8 participants have completed your experiment. 12 participants are Live. In this situation, Gorilla will mark your experiment as FULL and prevent further participants from joining your study. This is because all 12 Live participants might still complete the experiment.

However, if any of these participants leaves your study without completing it, they will still appear to be Live. While they are Live, they reserve a token that you can't yet use on another participant.

This means it is important to reject participants who have dropped out. You can do this:

• Automatically by setting a Time Limit
  • The Time Limit feature is not recommended for longitudinal studies.
  • If you're recruiting via Prolific, make sure that your Gorilla experiment Time Limit matches the time limit set in Prolific.
• Manually on the Participants tab
  • For experiments that are completed in one sitting, a good rule of thumb is to reject participants that started over 24 hours ago.
  • Check a participant's progress by clicking 'View Progress' to assess if they have completed enough of the experiment for you to use their data.

You can find more information about participant status in our Participants guide.

Adding a Checkpoint Node at the point in the experiment where you would be happy to still include a participant's data is a helpful way to speed up rejection and inclusion of multiple participants. The video below gives an overview of how to use Checkpoint Nodes to easily identify the participants you want to include or reject.

Technical Attrition

The Attrition section of this guide covers any instance where a participant drops out of a study for any reason. This section focuses specifically on technical attrition - when participants are unable to complete an experiment due to technical issues with their device, browser, or internet connection.

While Gorilla is designed to function correctly and reliably on modern browsers, there are some factors we cannot account for. Some participants may encounter difficulties that prevent them from successfully completing a study that are beyond our control. These issues can arise for various reasons, including (but not limited to):

• Outdated or corrupt browser installations
• Incompatible browser extensions or add-ons
• Unstable or intermittent internet connections
• Underlying browser bugs affecting specific functionality

Online experiments require a higher level of precision and connectivity than standard web pages. They require 1) precise control of stimuli loading and timing, 2) careful control of content positioning and 3) constant, uninterrupted communication with the Gorilla server. As such, online research has more in common with online gaming that typical web pages, making it more vulnerable to technical disruption.

Although technical attrition cannot be eliminated entirely, planning for a percentage of participant dropouts (such as over-recruiting participants) can help to mitigate its impact. Just as some participants may give low-effort/unusable responses, some may experience technical problems that prevent them from completing the experiment, or require their end results to be discarded.

From our experience in supporting researchers with their experiments, we consider a technical attrition rate of around 10% as 'normal' for online research, though this may vary between studies. To minimise technical attrition as much as possible, we recommend considering the following before recruiting participants:

• Thoroughly preview your experiment to ensure everything is functioning as expected across different devices and browsers.
• Compress uploaded files as much as possible to prevent excessive loading delays. You can find recommended file types and sizes in our Technical Checklist as well as some guidance on how to compress files.
• Ask participants to avoid completing the experiment in private or incognito browsers, and to disable browser add-ons such as VPNs and AdBlockers which might interfere with the experiment.

• Ask participants to stay on the Gorilla tab for the duration of the experiment, and avoid switching to other tabs or applications.
  • If you are on a Standard or Department subscription, you can track if participants switched tabs during the experiment using the TAB_HIDDEN and TAB_VISIBLE metrics provided in the data file as part of our bot detection features.

Using Recruitment Services

Often the fastest way of getting participants for your study is to work with a Recruitment Service who provides them.

Recruitment Services will take care of both finding the participant and also paying them. For this they will take a commission for the work that they have done.

We highly recommend Prolific; they are specialists for behavioural scientists. We also have a full list of integrated recruitment services, but you can also integrate with other 3rd party recruitment services yourself such as market research agencies.

Market Research agencies exist all over the world and in nearly every jurisdiction. They are more expensive than recruitment services like mTurk and Prolific, and their participants are used to questionnaires about products, but this can be a good option if you need participants fast.

There are a number of challenges to manage when using a recruitment service:

1. Fully informing your participants how to interact with Gorilla and complete your study.
2. Setting participant recruitment numbers in both Gorilla and the recruitment service.
3. Keeping on top of participant attrition.
4. Taking account of server downtime - limiting live participants.

Informing your participants about Gorilla

Participating in a Gorilla study via a recruitment service should be a seamless experience for the participant. Gorilla has been designed to reduce participant 'barriers to entry' and protect participant anonymity; Gorilla does NOT require participants to sign-up for a Gorilla account. Nor are participants required to download anything to their computer in order to run your study.

While a participant may be familiar with taking studies through a particular recruitment service, they may not have taken part in a Gorilla study before. Mentioning that they don't need to sign up or download anything can improve uptake of your study!

Setting Participant recruitment numbers

When using a Recruitment Service, you are using two paid-for services that must both protect you from overspending: Gorilla, and the recruitment service. Consequently, you’ll need to set the number of participants you want to recruit in both services. In Gorilla, you do this by setting the Recruitment Target to be the total number of participants that you want to recruit in your study. You can find out more about this from our Pricing FAQ page.

Problems can occur when the two different systems get ‘out of balance’. For example, Gorilla may think you have finished participant recruitment while the recruitment service may still send participants to Gorilla. The result is a frustrating experience for a participant as they will encounter a message saying 'this experiment is not currently available'.

How can this happen? A recruitment service may have a way of keeping track of participant attrition that we don’t have in Gorilla (and vice-versa):

Scenario 1:

1. A participant clicks on a link to your study in a recruitment service. The recruitment service logs them as an active participant.
2. They click the Gorilla Start button. At this point, in Gorilla, the participant reserves a participant token.
3. The participant consents and come to a screening questionnaire which they fail to pass.
4. They are sent to a Gorilla Reject Node. At this point in Gorilla, the participant token is returned to the pool, and the participant is not counted towards the Recruitment Target.
5. Depending upon how you have set up your Reject Node the participant may or may not be redirected back to the recruitment service. Therefore, it's possible that the recruitment service still believes they are an active participant.

Scenario 2:

1. A participant clicks on a link to your study in a recruitment service. The recruitment service logs them as an active participant.
2. They click the Gorilla Start button. At this point, in Gorilla, the participant reserves a token.
3. The participant consents and they start the task but, for whatever reason, they decide to drop out. The participant goes back to the recruitment service to tell them they've dropped out of your experiment.
4. At this point Gorilla may not be told by the recruitment service that the participant has withdrawn. As such the participant's status will remain 'Live, their token is still reserved, and the researcher must manually reject this participant.

Continue reading below to learn how to avoid this and stay on top of participant attrition!

Keeping on top of participant attrition

When you set a specific Recruitment Target in Gorilla (in addition to the recruitment target set in the recruitment service itself) you will need to make sure you keep the number of recruited participants on both websites in check:

1. Be sure to read our Participant Tokens Guide so you fully understand when and where tokens will be reserved, spent, and returned in Gorilla.
2. Be aware of how participants are considered 'recruited' within your chosen recruitment service. For example: commonly the recruitment service requires that they need to be returned to their site and/or submit a code to be considered finished.
3. Keep a close eye on participants who have started your study and reserved a token in Gorilla, but who have since dropped out (their status remains 'Live').
4. You may need to manually reject participants who look to have dropped out in order to return their 'reserved' token to the pool.

Another way you can alleviate this issue if you don't wish to manually monitor your participants is to use over-allocate participant tokens to your study e.g. if you want 200 complete participants and you are expecting ~30% attrition then assign 300 tokens to your experiment.

You can also set a maximum time limit for a participant to complete a study. If they have not completed within this time, they are automatically rejected. If your study takes 20 minutes on average you could set this to 30 minutes (and risk rejecting slower participants) or 2 or 24 hours if you wanted to be more generous.

Recruit in batches

Microsoft Azure guarantees that our servers will be working 99.95% of the time, but they can still go down. See our Server Downtime page for more details.

If you are paying a lot for your recruited participants or recruiting a hard to reach demographic, then low participant attrition may be a crucial factor in both your experimental design and recruitment phase.

In these cases, we highly recommend launching experiments in small enough batches that you can afford to lose every participant that is currently active.

To do this, set the Recruitment Target to 20 (for example) and once these are all complete, update the Recruitment Target to 40. Continue adding batches of 20 until you reach your total.

In parallel, do the same with your recruitment service. Start with a smaller batch of 20 participants, and once that data is in, release a further 20. This way you can protect yourself from the cost of participant attrition.

Some participant recruitment services give you the option of limiting how many participants can take part simultaneously.