So you're ready to launch your experiment on Gorilla. You've bought tokens or requested them from your subscription, you've assigned tokens to your experiment by setting a Recruitment Target, and you've chosen your preferred Recruitment Policy. What happens to your tokens as participants move through your experiment?
This guide will lay out how participants progress through your experiment and when their tokens are consumed. It will also explain ways you can manually change participant status and the effect this has on your tokens.
If you prefer to watch a video, the following guide covers the contents of this page and shows a live example of how to Reject and Include Participants in your Experiment.
Length (mins): 13:22
In the Participants tab of the Experiment Builder the Status column refers to how far your participant has moved within the experiment.
Live - the participant is still taking part, or has dropped out (i.e. did not reach the 'Finish' node).
Complete - the participant has completed the experiment (i.e. reached the 'Finish' node); their data is stored and ready to download.
Rejected - the participant has reached the Reject Node.
RejectedManual - the participant was manually rejected, or the participant returned the study or timed out on Prolific.
RejectedTime Limit - the participant has been automatically rejected because they exceeded the Time Limit set for the experiment.
Pending - the participant was added using specific recruitment polices (Email ID, Email Shot, Supervised), but has not yet been activated.
Activated - the participant was activated using specific recruitment polices (Email ID, Email Shot, Supervised), but has not yet started the experiment.
You can download all the information on the participant tab as a spreadsheet by clicking on "Global Actions" and "Download All".
You can further monitor how far the participant has got through your experiment by clicking ‘View Progress’ under the Progress column.
Checkpoint Nodes are a useful way to track how far through your experiment a participant has progressed. Based on this information, you can then decide whether to manually Include or Reject participants who are still Live. However, first it's important to understand how these actions affect token consumption.
Gorilla charges one participant token per participant per experiment. The token represents the data that can be downloaded.
When a participant starts your experiment, the Start Node reserves a token for them. When that same participant enters the Finish Node, that token is then consumed. The token this participant consumed is then permanently removed from your account's token total.
If the participant does not enter the Finish Node, and does not pass through a Reject Node, the participant's token will remain reserved until they are Included or Rejected (either manually or via a timelimit).
The total number of participant tokens that can be consumed during an experiment is set using the Recruitment Target. The experiment will continue to accept participants until the combined number of consumed and reserved tokens reaches your recruitment target - the Recruitment Status is then 'Full'. If any currently live participant is Rejected, their reserved token will be returned, and the experiment will accept another participant until the Recruitment Target is reached. When the number of consumed tokens matches the Recruitment Target, the Recruitment Status will then be 'Complete'.
Including a participant permanently consumes a token. Once a token is consumed, it cannot be reused or reclaimed.
Rejecting a participant returns their reserved token (unless you have previously included them). Rejected participants can later be Included, as long as you have sufficient remaining tokens in your recruitment target. However, Included participants cannot be subsequently Rejected.
Tokens are consumed when:
If a participant consumed a token, you can download their data for analysis.
Tokens are not consumed when:
If a participant is Rejected, you cannot download their data for analysis unless you manually Include them, which will consume a token - see above.
Read on to find out more about how to Reject, Include, or Delete participants.
The Recruitment Progress table is available in the Recruitment tab and can be used to track token usage. Essentially, this table provides a record of where tokens have been allocated.
| Row Name | Description |
|---|---|
| Target | The number of tokens you have assigned to the study: Target = Participants + Still to recruit |
| Participants | The total number of tokens consumed in your study: Participants = Complete + Live + Rejected + Deleted |
| Complete | The number of tokens consumed by participants who fully completed your study. |
| Live | The number of tokens reserved for participants currently completing your study. |
| Live (included) | The number of tokens consumed by participants who are Live in the study and have been manually included. |
| Rejected | The number of tokens consumed by participants who have been manually included after originally being rejected. |
| Deleted | The number of tokens consumed by participants that fully completed your study but were then deleted. |
| Still to recruit | The number of tokens that are still available for the study. |
Recruiting participants using Prolific? Participants who return the study on Prolific, or time out on the Prolific end, will be automatically rejected in Gorilla. Their status in your Gorilla Participants tab will be RejectedManual.
Sometimes participants will not meet your requirements, or will drop out before completing your study. Gorilla allows you to 'reject' these participants, meaning they will not consume a token.
Rejected participants are:
Rejected participants can still be manually included in your downloadable data if you change your mind.
Please note that consumed tokens cannot be reclaimed, i.e. if your participant completed the experiment or you manually Included them in your dataset, you cannot subsequently reclaim their token by rejecting them. Find out more in the When are tokens consumed? section above.
To make an educated decision about whether or not to reject a 'Live' participant, we strongly recommend you make use of Checkpoint Nodes in order to know how far through your experiment your participant progressed.
If after looking at the Participant Progress you have decided that a Live participant does not fulfil your study requirements, has dropped out, or is taking longer to complete your experiment than is reasonable, you can reject them. After a participant is rejected, the token they were occupying is released and you can recruit another participant in their place.
Live participants can be rejected (and therefore stopped from consuming a token) in the following ways:
See more information about each of the rejection methods below.
You may wish to screen certain participants out early on in your experiment (for example, based on demographic questions or perfomance in a task) and not include them in your downloadable data. This can be achieved by directing these participants to a Reject Node instead of a Finish Node in your Experiment Tree design. Participants who reach a Reject Node will be automatically rejected and will be given a status of Rejected on your Participants tab.
Learn how to set up a Reject Node in your experiment.
The classic use-case for Time Limit Rejection is to reduce the number of participants with 'Live' Status who temporarily reserve your tokens. To do that, you may wish to set a Time Limit on your Experiment. This means that participants who take more than the allowed time will be automatically rejected. This will prevent you having to manually reject participants who are still marked as 'Live' because they left the experiment part-way through.
Participants who reach the Time Limit will be automatically rejected and will be given a status of RejectedTime Limit on your Participants tab. However, they will be allowed to finish their current task before being redirected to the Finish Node.
Read more about setting a Time Limit for your experiment, including important factors to consider when deciding how much time to allow your participants.
While a participant has 'Live' status, you can choose to manually reject the participant from your experiment. Participants can be rejected manually by selecting 'Reject' from the Action column in the Participants Tab of your Experiment, as shown in the image below. Rejected participants will be given a status of RejectedManual on your Participants tab.
After the participant has been rejected by one of the ways described above, you can still later include them and use their data in your analysis. Including participants will consume a token, i.e. you will pay for them. Note that including a participant is irreversible. Find out more about Including Participants in the next section.
There are two situations in which you can manually Include a participant:
Choosing to manually Include a Participant will:
Including a participant is a permanent action; once a participant has been included in your downloadable data, and a token consumed, the decision cannot be reversed. Be sure to only include participants whose data you wish to download and use.
To make educated decisions about whether or not to manually include a participant, we strongly recommend you make use of Checkpoint Nodes in order to know how far through your experiment your participant progressed.
Note: If your recruitment progress is at 100% (Complete), you must first increase your Recruitment Target before you can continue to manually Include participants. For Pay-per-Participant accounts this allows you to keep tight control of how many tokens you spend on an experiment.
You may choose to include participants who have 'Live' status at any time. This will allow you to download their data and view it while they are still progressing through your experiment. This can be useful in teaching situations or supervised trials using the 'Supervised' Recruitment Policy.
Another situation where you may want to include live participants is during a longitudinal study. Including a participant means you can access the data they have already generated, and also access their future data without having to consume another token. Simply regenerate your data once the participant has completed the next stage of your study, and you will be able to download all the data they have produced so far at no extra cost.
You may also choose to manually Include participants who completed most of your experiment but did not reach a Finish Node.
You can include Live participants by selecting 'Include' from the Action column as shown in the image below. This will immediately consume one token per participant from your experiment's allocation.
Sometimes you may wish to include rejected participants in your data even though they took longer than the Time Limit to complete your experiment, or ended your experiment at a Reject Node. Even though you have marked these participants as 'rejected', you always retain the option to manually Include these participants in your downloadable data.
To include participants that you have previously Rejected, select 'Include' from the Action column as shown on the image below. This will immediately consume one token per participant from your experiment's allocation.
After you've included the Live/Rejected participant, you will be able to see a green tick in the 'Included' column, as shown below. You will then be able to download data from these participants for your analysis. Included participants permanently consume your tokens.
Participants with any status can be Deleted, as shown in the image below:
However, we strongly advise you not to do this, as you will permanently lose your data. The Delete option should only be used to comply with your ethics, i.e., to withdraw data from participants who no longer wish to be part of your experiment, or to delete participants' data after the time you specified in your ethics application/IRB. The Delete option could also be used for deleting pilot data that you used for testing your experiment, but please use it carefully so as not to lose your valuable data. Deleting participants does not reclaim Tokens as these have already been consumed.
If you Delete a participant who has already been included in your data, the participant will still count towards your Recruitment Progress being Complete. You will need to increase your Recruitment Target if you want to recruit another participant in their place.
You can also delete all participant data in bulk using the 'Delete All' option in the Global Actions menu.
The 'Delete All' option should be exercised with great care to avoid unintentional data loss!
To delete a subset of your participants, first use the search filters on the Participants tab to show only the participants you want to delete.
Once you click 'Delete All', the status of all participants selected by your current filters will change to Deletion Scheduled. You then have a 24-hour grace period within which to change your mind. At any point during this 24-hour period, you can cancel the deletion by using the 'Cancel Deletion' option in the Global Actions menu. Note that the 24 hours starts immediately you confirm the Delete All action, and does not discriminate between working days and weekends. Once the 24-hour period has elapsed, all your participant data will be permanently deleted, and there will be no way to retrieve it.
We have made the option of bulk participant data deletion available because it is an important requirement of many university ethics/data protection regulations to delete data after a certain time period. However, please take care using this option and ensure you only do so if you actually want your data to be irrecoverably deleted.
