Amazon Mechanical Turk (often abbreviated to AMT or MTurk) allows individuals around the world to take part in online tasks in return for payment. The basic way it works is:
You can sign up for an MTurk requester account.
It's worth reading Gleibs (2017) to help you work with Turkers effectively.
This line is particularly important: 'Rejections leave workers with a mark counting against them on their “permanent record” at MTurk, which may take them below the 95 % threshold.'
Wherever possible, have workers return HITs themselves rather than rejecting them via Gorilla.
The simple way of listing your experiment on AMT is:
While this approach is suitable for a small pilot studies, there are two drawbacks:
In order to mitigate these two problems we need to capture the AMT worker ID, so that Gorilla metrics line up with AMT reports.
AMT gives you full access to the HTML on the HIT listing page, and as a result, doesn't offer a way to automatically append the worker ID to the URL. Therefore, we need to do this ourselves, with a bit of JavaScript.
In Gorilla:
In AMT:
On the Design Layout page, click Source to see the HTML source for the page.
Find the line containing the URL for your survey - by default it is:
<a class="dont-break-out" href="http://[example.com/survey345.html]" target="_blank">http://example.com/survey345.html</a>Replace it with:
<a id="gorilla-link" target="_blank" href="https://research.sc/participant/login/dynamic/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX">Survey Link</a>Next, scroll down to the bottom where you should find some JavaScript:
$(document).ready(function() {
// Instructions expand/collapse
var content = $('#instructionBody');
var trigger = $('#collapseTrigger');
content.hide();
$('.collapse-text').text('(Click to expand)');
trigger.click(function(){
content.toggle();
var isVisible = content.is(':visible');
if(isVisible){
$('.collapse-text').text('(Click to collapse)');
} else {
$('.collapse-text').text('(Click to expand)');
}
});
// end expand/collapse
});Change it to:
function appendWorkerIDToURL(linkElem, workerIDName) {
var url = window.location.href;
var name = 'workerId';
var regex = new RegExp("[?&]" + name + "(=([^&#]*)|&|#|$)"),
results = regex.exec(url);
if(results && results[2]) {
var workerID = decodeURIComponent(results[2].replace(/\+/g, " "));
var linkURL = $(linkElem).attr('href');
var appendChar = linkURL.indexOf('?') >= 0 ? '&' : '?';
$(linkElem).attr('href', linkURL + appendChar + workerIDName + '=' + workerID);
}
}
$(document).ready(function() {
// Instructions expand/collapse
var content = $('#instructionBody');
var trigger = $('#collapseTrigger');
content.hide();
$('.collapse-text').text('(Click to expand)');
trigger.click(function() {
content.toggle();
var isVisible = content.is(':visible');
if(isVisible) {
$('.collapse-text').text('(Click to collapse)');
} else {
$('.collapse-text').text('(Click to expand)');
}
});
// end expand/collapse
appendWorkerIDToURL('#gorilla-link', 'external_id');
});This creates a function which looks at the current URL, retrieves the workerId and appends it to our survey link. We then call that function from within $(document).ready(), passing in our survey link that we created earlier.
If your experimental design contains conditions where you wish to reject or screen out a participant, direct them to a Reject Node.
This can also show them a completion code.
You may also want to include a simple questionnaire before the secondary reject node which tells the participant to return to AMT and return the HIT.
By default, AMT will automatically prevent the same worker doing the same HIT multiple times.
However, some researchers deploy their experiments in batches of 9 HITs to avoid paying AMT a higher rate of commission. When researchers do this, AMT doesn't prevent the same worker doing multiple HITs because AMT has no way of knowing that the batches are related.
Nevertheless, on the Gorilla side, it is just one experiment. If you have followed the Advanced Set Up instruction above, Gorilla will detect that the same worker is accessing the experiment again and resumes them where they left off. If they have reached the finish node, with the completion code, it will resume there.
From the participant's perspective, they would click on the HIT, press start and immediately be given a completion code.
If this happens, you want these participants to return the HIT, so that you don't have to reject it. To encourage this, make it clear in the task listing on AMT that if they get the experience above, they should return the HIT. It would be ideal to make this section of the instructions salient to the worker.
BEWARE: If you use ALL CAPS or RED workers may get put off your task completely!
An alternative approach, which is more reliable, is to issue a Qualification to each participant that completes one of your batches. You can then set this qualification as an exclusion criterion. This ensures that subjects who have already participated would not see the HITs again.
Novice Turkers, without a reputation, may not have access to tasks that require a good reputation.
Novice Turkers are particularly valuable to researchers. To attract them, put out a HIT with a 0 cent payment and an appropriate bonus.
It is possible to run longitudinal studies in AMT.
If you want more details, don't hesitate to get in touch.