In online research, the main sources of inaccuracy in regards to timing come from peripheral devices, which affect both native applications (i.e. installed on a computer in the lab) and browser-based ones alike. Woods et al. (2015) give a good overview of the issues involved in online research, and contains a detailed section on timing.
In Gorilla, we use high resolution timers for accurate reaction times and frame counting for accurate stimuli presentation times - techniques that have only recently become available in all major browsers.
Click through sections on the menu to the left to learn all about timing in Gorilla and make sure to check out further resources for more information on timing in online research!
When conducting research online, rather than in the lab, it's important to understand how timing accuracy changes.
The schematic below shows the main differences:
Modern techniques refers to the Performance.now() function common in all major browsers which provides microsecond timing (read more on Mozilla's Performance.now() page). We do not have control
of the underlying operating system, so we cannot make the same OS-level timing calls that native software
can. While clearly there will always be some experiments that require extremely high fidelity timing, the
precision offered by Gorilla is appropriate for a wide range of research.
The latency between the cloud storage and local computer is irrelevant because Gorilla downloads all task information (i.e. stimuli) to the local machine in advance and collects local timestamps.
To talk through each element in the picture above:
See the further resources section for more information on timing in online research.
Much of the criticism of browser-based timings stem from the inaccuracy of JavaScript's setTimeout(), setInterval() and Date.now() functions.
Within the last few years, more accurate techniques have become available:
performance.now() offers microsecond-precision timestamps. This was first introduced in Chrome in 2012, and iOS was the last to adopt it in 2015.requestAnimationFrame() allows us to run logic every time the screen is about to be refreshed, and should be synchronised to the refresh rate. This was first introduced in Firefox in 2012, and Android was the last to adopt it in 2013.Both features are now standard in all major browsers, but you can see a detailed breakdown of performance.now()'s usability here and of requestAnimationFrame()'s usability here.
The key issue with displaying stimuli is that we are fundamentally limited to the refresh rate of the display. Most modern displays refresh at 60Hz, which is every 16.67 milliseconds. This means that stimuli can only ever be displayed for multiples of 16.67 milliseconds - without specialist equipment, you cannot show a stimuli image for 40ms; you can either have 33.3ms (two frames) or 50ms (three frames).
This issue affects native programs and browser-based ones equally, as they are both ending up on the same display.
Gorilla uses requestAnimationFrame() to count frames and ensure that a stimuli is shown
for as accurate a time as possible, by evaluating for each frame whether it would be more accurate to
display for another frame (because there is more than 8ms left to display) or to stop on this frame (because there is less than 8ms left to show).
Whenever a new screen is displayed in Gorilla, a timestamp is recorded using performance.now().
When a response is recorded (e.g. a keypress or a mouse click), a second timestamp is taken, and the reaction time
is then the difference between the two.
Exactly how accurate the response is depends chiefly on the latency of the input device. The default USB polling rate in Windows is 120Hz, or every 8.3ms. Therefore, there could be up to an 8.3ms error on keyboard presses or mouse clicks for peripherals connected via USB.
Touchscreens have higher latency still. The fastest devices are now at around 20ms, and getting faster all the time.
We published our own paper on timing accuracy in Gorilla (Anwyl-Irvine et al., 2019) where we used a simplified flanker task taken from the attentional network task (Rueda, Posner, & Rothbart, 2004). We replicated the “conflict network” effect in all these populations, demonstrating the platform’s capability to run reaction-time-sensitive experiments - the effect we replicated was under 100ms.
We've also published a paper that focuses on timing accuracies on various online platforms, browsers, and devices (Anwyl-Irvine et al., 2021).
UCL are using Gorilla for teaching first year undergraduate students. One of the first research methods lectures was on the IAT. 108 participants took part in an IAT that is included in the Starter Project: Examples Project folder. This IAT investigates whether participants have an implicit association between young people and good words, and old people and bad words.
The results of this test replicated previous findings that the participants do have such an implicit association.
Below are some highly recommended resources to help you learn more about timing in online research:
Reimers & Steward (2014) examined response timing accuracy in Adobe Flash and HTML5/JavaScript.
Hilbig (2015) presents the effect of lab- versus web-based research on reaction times.
Woods et al. (2015) wrote an excellent summary of the strength and weakness of online research, and the paper contains a detailed section on timing.
Our own paper, Anwyl-Irvine et al. (2019), discusses timing accuracy in Gorilla.
Our recent paper, Anwyl-Irvine et al. (2021), compares timing accuracies on various online platforms, browsers, and devices. This BeOnline talk "All about timing accuracy" presents the results from the pre-print and discusses the accuracy and consistency of visual presentation and recording of keyboard responses online. We also have a blog post summarising our timing performance article.