Skip to main content

Conducting Web-Based Experiments for Numerical Cognition Research Cover

Conducting Web-Based Experiments for Numerical Cognition Research

Journal of Cognition

Volume 2 (2019): Issue 1

By: Arnold R. Kochari

Open Access

|Sep 2019

Abstract

It is becoming increasingly popular and straightforward to collect data in cognitive psychology through web-based studies. In this paper, I review issues around web-based data collection for the purpose of numerical cognition research. Provided that the desired type of data can be collected through a web-browser, such online studies offer numerous advantages over traditional forms of physical lab-based data collection, such as gathering data from larger sample sizes in shorter time-windows and easier access to non-local populations. I then present results of two replication studies that employ classical paradigms in numerical cognition research: the number-size congruity paradigm and comparison to a given standard, which also included a masked priming manipulation. In both replications, reaction times and error rates were comparable to original, physical lab-based studies. Consistent with the results of original studies, a distance effect, a congruity effect, and a priming effect were observed. Data collected online thus offers a level of reliability comparable to data collected in a physical lab when it comes to questions in numerical cognition.

References

Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2019). Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods. DOI: 10.3758/s13428-019-01237-x
Open DOI Search in Google Scholar Back to article
Baker, D. H., Vilidaite, G., Lygo, F. A., Smith, A. K., Flack, T. R., Gouws, A. D., & Andrews, T. J. (2019). Power contours: Optimising sample size and precision in experimental psychology and human neuroscience. arXiv:1902.06122 [q-bio, stat].
Search in Google Scholar Back to article
Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015). QRTEngine: An easy solution for running online reaction time experiments using Qualtrics. Behavior Research Methods, 47(4), 918–929. DOI: 10.3758/s13428-014-0530-7
Open DOI Search in Google Scholar Back to article
Besner, D., & Coltheart, M. (1979). Ideographic and alphabetic processing in skilled reading of English. Neuropsychologia, 17(5), 467–472. DOI: 10.1016/0028-3932(79)90053-8
Open DOI Search in Google Scholar Back to article
Birnbaum, M. H. (2000). Introduction to psychological experiments on the internet. In: M. H. Birnbaum (Ed.), Psychological Experiments on the Internet (pp. xv–xx). San Diego: Academic Press. DOI: 10.1016/B978-012099980-4/50001-0
Open DOI Search in Google Scholar Back to article
Brysbaert, M. (2019). Power analysis and effect size in mixed effects models: A tutorial. DOI: 10.31234/osf.io/fahxc
Open DOI Search in Google Scholar Back to article
Brysbaert, M., & Stevens, M. (2018). Power Analysis and Effect Size in Mixed Effects Models: A Tutorial. Journal of Cognition, 1(1), 9. DOI: 10.5334/joc.10
Open DOI Search in Google Scholar Back to article
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3–5. DOI: 10.1177/1745691610393980
Open DOI Search in Google Scholar Back to article
Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use. Perspectives on Psychological Science, 13(2), 149–154. DOI: 10.1177/1745691617706516
Open DOI Search in Google Scholar Back to article
Callaway, E. (2013). Dyscalculia: Number games. Nature News, 493(7431), 150. DOI: 10.1038/493150a
Open DOI Search in Google Scholar Back to article
Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130. DOI: 10.3758/s13428-013-0365-7
Open DOI Search in Google Scholar Back to article
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using Nonnaive Participants Can Reduce Effect Sizes. Psychological Science, 26(7), 1131–1139. DOI: 10.1177/0956797615585115
Open DOI Search in Google Scholar Back to article
Cipora, K., Soltanlou, M., Reips, U.-D., & Nuerk, H.-C. (2019). The SNARC and MARC effects measured online: Large-scale assessment methods in flexible cognitive effects. Behavior Research Methods. DOI: 10.3758/s13428-019-01213-5
Open DOI Search in Google Scholar Back to article
Cipora, K., Willmes, K., Szwarc, A., & Nuerk, H.-C. (2018). Norms and Validation of the Online and Paper-and-Pencil Versions of the Abbreviated Math Anxiety Scale (AMAS) For Polish Adolescents and Adults. Journal of Numerical Cognition, 3(3), 667–693. DOI: 10.5964/jnc.v3i3.121
Open DOI Search in Google Scholar Back to article
Cohen Kadosh, R., & Henik, A. (2006). A Common Representation for Semantic and Physical Properties. Experimental Psychology, 53(2), 87–94. DOI: 10.1027/1618-3169.53.2.87
Open DOI Search in Google Scholar Back to article
Cohen Kadosh, R., Henik, A., & Rubinsten, O. (2008). Are Arabic and verbal numbers processed in different ways? Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(6), 1377–1391. DOI: 10.1037/a0013413
Open DOI Search in Google Scholar Back to article
Cohen Kadosh, R., Lammertyn, J., & Izard, V. (2008). Are numbers special? An overview of chronometric, neuroimaging, developmental and comparative studies of magnitude representation. Progress in Neurobiology, 84(2), 132–147. DOI: 10.1016/j.pneurobio.2007.11.001
Open DOI Search in Google Scholar Back to article
Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. DOI: 10.1126/science.aac4716
Open DOI Search in Google Scholar Back to article
Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLOS ONE, 8(3), e57410. DOI: 10.1371/journal.pone.0057410
Open DOI Search in Google Scholar Back to article
Dehaene, S., Dupoux, E., & Mehler, J. (1990). Is numerical comparison digital? Analogical and symbolic effects in two-digit number comparison. Journal of Experimental Psychology: Human Perception and Performance, 16(3), 626–641. DOI: 10.1037/0096-1523.16.3.626
Open DOI Search in Google Scholar Back to article
Dehaene, S., Naccache, L., Le Clec’H, G., Koechlin, E., Mueller, M., Dehaene-Lambertz, G., Le Bihan, D., et al. (1998). Imaging unconscious semantic priming. Nature, 395(6702), 597–600. DOI: 10.1038/26967
Open DOI Search in Google Scholar Back to article
de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47, 1–12. DOI: 10.3758/s13428-014-0458-y
Open DOI Search in Google Scholar Back to article
de Leeuw, J. R., & Motz, B. A. (2016). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods, 48(1), 1–12. DOI: 10.3758/s13428-015-0567-2
Open DOI Search in Google Scholar Back to article
Elze, T., & Tanner, T. G. (2012). Temporal Properties of Liquid Crystal Displays: Implications for Vision Science Experiments. PLOS ONE, 7(9), e44048. DOI: 10.1371/journal.pone.0044048
Open DOI Search in Google Scholar Back to article
Eriksson, K., & Lindskog, M. (2017). Encoding of numerical information in memory: Magnitude or nominal? DOI: 10.5964/jnc.v3i1.68
Open DOI Search in Google Scholar Back to article
Ferguson, A. M., Maloney, E. A., Fugelsang, J., & Risko, E. F. (2015). On the relation between math and spatial ability: The case of math anxiety. Learning and Individual Differences, 39, 1–12. DOI: 10.1016/j.lindif.2015.02.007
Open DOI Search in Google Scholar Back to article
Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon mechanical turk: Gold mine or coal mine? Computational Linguistics, 37(2), 413–420. DOI: 10.1162/COLI_a_00057
Open DOI Search in Google Scholar Back to article
Gleibs, I. H. (2017). Are all “research fields” equal? Rethinking practice for the use of data from crowdsourcing market places. Behavior Research Methods, 49(4), 1333–1342. DOI: 10.3758/s13428-016-0789-y
Open DOI Search in Google Scholar Back to article
Gökaydin, D., Brugger, P., & Loetscher, T. (2018). Sequential Effects in SNARC. Scientific Reports, 8(1), 10996. DOI: 10.1038/s41598-018-29337-2
Open DOI Search in Google Scholar Back to article
Gosling, S. D., & Mason, W. (2015). Internet Research in Psychology. Annual Review of Psychology, 66(1), 877–902. DOI: 10.1146/annurev-psych-010814-015321
Open DOI Search in Google Scholar Back to article
Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should We Trust Web-Based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires. American Psychologist, 59(2), 93–104. DOI: 10.1037/0003-066X.59.2.93
Open DOI Search in Google Scholar Back to article
Halberda, J., Ly, R., Wilmer, J. B., Naiman, D. Q., & Germine, L. (2012). Number sense across the lifespan as revealed by a massive Internet-based sample. Proceedings of the National Academy of Sciences, 109(28), 11116–11120. DOI: 10.1073/pnas.1200196109
Open DOI Search in Google Scholar Back to article
Henik, A., & Tzelgov, J. (1982). Is three greater than five: The relation between physical and semantic size in comparison tasks. Memory & Cognition, 10(4), 389–395. DOI: 10.3758/BF03202431
Open DOI Search in Google Scholar Back to article
Henninger, F., Shevchenko, Y., Mertens, U. K., Kieslich, P. J., & Hilbig, B. E. (2019). Lab.js: A free, open, online study builder (Preprint). PsyArXiv. DOI: 10.31234/osf.io/fqr49
Open DOI Search in Google Scholar Back to article
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. DOI: 10.1017/S0140525X0999152X
Open DOI Search in Google Scholar Back to article
Hilbig, B. E. (2016). Reaction time effects in lab- versus Web-based research: Experimental evidence. Behavior Research Methods, 48(4), 1718–1724. DOI: 10.3758/s13428-015-0678-9
Open DOI Search in Google Scholar Back to article
Hinrichs, J. V., Yurko, D. S., & Hu, J.-M. (1981). Two-digit number comparison: Use of place information. Journal of Experimental Psychology: Human Perception and Performance, 7(4), 890–901. DOI: 10.1037/0096-1523.7.4.890
Open DOI Search in Google Scholar Back to article
Huber, S., Nuerk, H.-C., Reips, U.-D., & Soltanlou, M. (2017). Individual differences influence two-digit number processing, but not their analog magnitude processing: A large-scale online study. Psychological Research. DOI: 10.1007/s00426-017-0964-5
Open DOI Search in Google Scholar Back to article
Irvine, A. (2018). The Unity game engine for a large-cohort, developmental study. Behavioral Science Online conference, London.
Search in Google Scholar Back to article
Kaufmann, L., Koppelstaetter, F., Delazer, M., Siedentopf, C., Rhomberg, P., Golaszewski, S., Ischebeck, A., et al. (2005). Neural correlates of distance and congruity effects in a numerical Stroop task: An event-related fMRI study. NeuroImage, 25(3), 888–898. DOI: 10.1016/j.neuroimage.2004.12.041
Open DOI Search in Google Scholar Back to article
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Nosek, B. A., et al. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. DOI: 10.1177/2515245918810225
Open DOI Search in Google Scholar Back to article
Kochari, A., & Schriefers, H. (in preparation). Processing symbolic magnitude information conveyed by number words and scalar adjectives: Parallel size-congruity effects.
Search in Google Scholar Back to article
Landy, D., Silbert, N., & Goldin, A. (2013). Estimating Large Numbers. Cognitive Science, 37(5), 775–799. DOI: 10.1111/cogs.12028
Open DOI Search in Google Scholar Back to article
Lin, C.-Y., & Göbel, S. M. (2019). Arabic digits and spoken number words: Timing modulates the cross-modal numerical distance effect. Quarterly Journal of Experimental Psychology, 1747021819854444. DOI: 10.1177/1747021819854444
Open DOI Search in Google Scholar Back to article
Lyle, H., Wylie, J., & Morsanyi, K. (2019). Cross-cultural differences in children’s mathematical development: Investigating the home numeracy environment. Ottawa, ON, Canada.
Search in Google Scholar Back to article
Mathur, M., & Reichling, D. (2018). Open-source software for mouse-tracking in Qualtrics to measure category competition. DOI: 10.31219/osf.io/ymxau
Open DOI Search in Google Scholar Back to article
Miller, K. F., & Stigler, J. W. (1987). Counting in Chinese: Cultural variation in a basic cognitive skill. Cognitive Development, 2(3), 279–305. DOI: 10.1016/S0885-2014(87)90091-8
Open DOI Search in Google Scholar Back to article
Moyer, R. S., & Landauer, T. K. (1967). Time required for judgements of numerical inequality. Nature, 215(5109), 1519–1520. DOI: 10.1038/2151519a0
Open DOI Search in Google Scholar Back to article
Nation, K. (2018). Online large-scale studies with children out of the lab: The promise and the challenge. London: Behavioral Science Online conference.
Search in Google Scholar Back to article
Neath, I., Earle, A., Hallett, D., & Surprenant, A. M. (2011). Response time accuracy in Apple Macintosh computers. Behavior Research Methods, 43(2), 353. DOI: 10.3758/s13428-011-0069-9
Open DOI Search in Google Scholar Back to article
Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016). Measuring the Prevalence of Problematic Respondent Behaviors among MTurk, Campus, and Community Participants. PLOS ONE, 11(6), e0157732. DOI: 10.1371/journal.pone.0157732
Open DOI Search in Google Scholar Back to article
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. DOI: 10.1073/pnas.1708274114
Open DOI Search in Google Scholar Back to article
Palan, S., & Schitter, C. (2018). Prolific.ac—A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27. DOI: 10.1016/j.jbef.2017.12.004
Open DOI Search in Google Scholar Back to article
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a Participant Pool. Current Directions in Psychological Science, 23(3), 184–188. DOI: 10.1177/0963721414531598
Open DOI Search in Google Scholar Back to article
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163. DOI: 10.1016/j.jesp.2017.01.006
Open DOI Search in Google Scholar Back to article
Peirce, J., Gray, J. R., Simpson, S., MacAskill, M., Höchenberger, R., Sogo, H., Lindeløv, J. K., et al. (2019). PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51(1), 195–203. DOI: 10.3758/s13428-018-01193-y
Open DOI Search in Google Scholar Back to article
Pinel, P., Piazza, M., Le Bihan, D., & Dehaene, S. (2004). Distributed and Overlapping Cerebral Representations of Number, Size, and Luminance during Comparative Judgments. Neuron, 41(6), 983–993. DOI: 10.1016/S0896-6273(04)00107-2
Open DOI Search in Google Scholar Back to article
Pitt, B., & Casasanto, D. (2016). Reading experience shapes the mental timeline but not the mental number line. In Proceedings of the 38th Annual Conference of the Cognitive Science Society (pp. 2753–2758). Austin, TX: Cognitive Science Society.
Search in Google Scholar Back to article
Plant, R. R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: New hardware, new problems? Behavior Research Methods, 41(3), 598–614. DOI: 10.3758/BRM.41.3.598
Open DOI Search in Google Scholar Back to article
Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 47(2), 309–327. DOI: 10.3758/s13428-014-0471-1
Open DOI Search in Google Scholar Back to article
Reimers, S., & Stewart, N. (2016). Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 48(3), 897–908. DOI: 10.3758/s13428-016-0758-5
Open DOI Search in Google Scholar Back to article
Santens, S., & Verguts, T. (2011). The size congruity effect: Is bigger always more? Cognition, 118(1), 94–110. DOI: 10.1016/j.cognition.2010.10.014
Open DOI Search in Google Scholar Back to article
Sekuler, R., Rubin, E., & Armstrong, R. (1971). Processing numerical information: A choice time analysis. Journal of Experimental Psychology, 90(1), 75–80. DOI: 10.1037/h0031366
Open DOI Search in Google Scholar Back to article
Semmelmann, K., Hönekopp, A., & Weigelt, S. (2017). Looking Tasks Online: Utilizing Webcams to Collect Video Data from Home. Frontiers in Psychology, 8. DOI: 10.3389/fpsyg.2017.01582
Open DOI Search in Google Scholar Back to article
Semmelmann, K., & Weigelt, S. (2017). Online psychophysics: Reaction time effects in cognitive experiments. Behavior Research Methods, 49(4), 1241–1260. DOI: 10.3758/s13428-016-0783-4
Open DOI Search in Google Scholar Back to article
Semmelmann, K., & Weigelt, S. (2018). Online webcam-based eye tracking in cognitive science: A first look. Behavior Research Methods, 50(2), 451–465. DOI: 10.3758/s13428-017-0913-7
Open DOI Search in Google Scholar Back to article
Shaki, S., Fischer, M. H., & Petrusic, W. M. (2009). Reading habits for both words and numbers contribute to the SNARC effect. Psychonomic Bulletin & Review, 16(2), 328–331. DOI: 10.3758/PBR.16.2.328
Open DOI Search in Google Scholar Back to article
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. DOI: 10.1177/0956797611417632
Open DOI Search in Google Scholar Back to article
Stewart, N., Chandler, J., & Paolacci, G. (2017). Crowdsourcing Samples in Cognitive Science. Trends in Cognitive Sciences, 21(10), 736–748. DOI: 10.1016/j.tics.2017.06.007
Open DOI Search in Google Scholar Back to article
Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10(5), 479–491.
Search in Google Scholar Back to article
Stoet, G. (2017). PsyToolkit: A Novel Web-Based Method for Running Online Questionnaires and Reaction-Time Experiments. Teaching of Psychology, 44(1), 24–31. DOI: 10.1177/0098628316677643
Open DOI Search in Google Scholar Back to article
Tzelgov, J., Meyer, J., & Henik, A. (1992). Automatic and intentional processing of numerical information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(1), 166–179. DOI: 10.1037/0278-7393.18.1.166
Open DOI Search in Google Scholar Back to article
Vakharia, D., & Lease, M. (2015). Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms. In iConference, 2015. Newport Beach, CA, USA.
Search in Google Scholar Back to article
Van Opstal, F., Gevers, W., De Moor, W., & Verguts, T. (2008). Dissecting the symbolic distance effect: Comparison and priming effects in numerical and nonnumerical orders. Psychonomic Bulletin & Review, 15(2), 419–425. DOI: 10.3758/PBR.15.2.419
Open DOI Search in Google Scholar Back to article
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An Agenda for Purely Confirmatory Research. Perspectives on Psychological Science, 7(6), 632–638. DOI: 10.1177/1745691612463078
Open DOI Search in Google Scholar Back to article
Weber, S. J., & Cook, T. D. (1972). Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference. Psychological Bulletin, 77(4), 273–295. DOI: 10.1037/h0032351
Open DOI Search in Google Scholar Back to article
Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015). Conducting perception research over the internet: A tutorial review. PeerJ, 3, e1058. DOI: 10.7717/peerj.1058
Open DOI Search in Google Scholar Back to article
Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (2018). Participant Nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review, 25(5), 1968–1972. DOI: 10.3758/s13423-017-1348-y
Open DOI Search in Google Scholar Back to article

Figures & tables

Articles in this issue

DOI: https://doi.org/10.5334/joc.85 | Journal eISSN: 2514-4820

Journal RSS Feed

Language: English

Submitted on: Mar 15, 2019

|

Accepted on: Sep 4, 2019

|

Published on: Sep 19, 2019

Published by: Ubiquity Press

In partnership with: Paradigm Publishing Services

Publication frequency: 1 issue per year

Keywords:

Numerical cognition,

Response speed,

Response accuracy

© 2019 Arnold R. Kochari, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.

Volume 2 (2019): Issue 1

Previous article