Conducting Web-Based Experiments for Numerical Cognition Research
References
- Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2019). Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods. DOI: 10.3758/s13428-019-01237-x
- Baker, D. H., Vilidaite, G., Lygo, F. A., Smith, A. K., Flack, T. R., Gouws, A. D., & Andrews, T. J. (2019). Power contours: Optimising sample size and precision in experimental psychology and human neuroscience. arXiv:1902.06122 [q-bio, stat].
- Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015). QRTEngine: An easy solution for running online reaction time experiments using Qualtrics. Behavior Research Methods, 47(4), 918–929. DOI: 10.3758/s13428-014-0530-7
- Besner, D., & Coltheart, M. (1979). Ideographic and alphabetic processing in skilled reading of English. Neuropsychologia, 17(5), 467–472. DOI: 10.1016/0028-3932(79)90053-8
- Birnbaum, M. H. (2000).
Introduction to psychological experiments on the internet . In: M. H. Birnbaum (Ed.), Psychological Experiments on the Internet (pp. xv–xx). San Diego: Academic Press. DOI: 10.1016/B978-012099980-4/50001-0 - Brysbaert, M. (2019). Power analysis and effect size in mixed effects models: A tutorial. DOI: 10.31234/osf.io/fahxc
- Brysbaert, M., & Stevens, M. (2018). Power Analysis and Effect Size in Mixed Effects Models: A Tutorial. Journal of Cognition, 1(1), 9. DOI: 10.5334/joc.10
- Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3–5. DOI: 10.1177/1745691610393980
- Buhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An Evaluation of Amazon’s Mechanical Turk, Its Rapid Rise, and Its Effective Use. Perspectives on Psychological Science, 13(2), 149–154. DOI: 10.1177/1745691617706516
- Callaway, E. (2013). Dyscalculia: Number games. Nature News, 493(7431), 150. DOI: 10.1038/493150a
- Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46(1), 112–130. DOI: 10.3758/s13428-013-0365-7
- Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using Nonnaive Participants Can Reduce Effect Sizes. Psychological Science, 26(7), 1131–1139. DOI: 10.1177/0956797615585115
- Cipora, K., Soltanlou, M., Reips, U.-D., & Nuerk, H.-C. (2019). The SNARC and MARC effects measured online: Large-scale assessment methods in flexible cognitive effects. Behavior Research Methods. DOI: 10.3758/s13428-019-01213-5
- Cipora, K., Willmes, K., Szwarc, A., & Nuerk, H.-C. (2018). Norms and Validation of the Online and Paper-and-Pencil Versions of the Abbreviated Math Anxiety Scale (AMAS) For Polish Adolescents and Adults. Journal of Numerical Cognition, 3(3), 667–693. DOI: 10.5964/jnc.v3i3.121
- Cohen Kadosh, R., & Henik, A. (2006). A Common Representation for Semantic and Physical Properties. Experimental Psychology, 53(2), 87–94. DOI: 10.1027/1618-3169.53.2.87
- Cohen Kadosh, R., Henik, A., & Rubinsten, O. (2008). Are Arabic and verbal numbers processed in different ways? Journal of Experimental Psychology: Learning, Memory, and Cognition, 34(6), 1377–1391. DOI: 10.1037/a0013413
- Cohen Kadosh, R., Lammertyn, J., & Izard, V. (2008). Are numbers special? An overview of chronometric, neuroimaging, developmental and comparative studies of magnitude representation. Progress in Neurobiology, 84(2), 132–147. DOI: 10.1016/j.pneurobio.2007.11.001
- Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349(6251),
aac4716 . DOI: 10.1126/science.aac4716 - Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLOS ONE, 8(3),
e57410 . DOI: 10.1371/journal.pone.0057410 - Dehaene, S., Dupoux, E., & Mehler, J. (1990). Is numerical comparison digital? Analogical and symbolic effects in two-digit number comparison. Journal of Experimental Psychology: Human Perception and Performance, 16(3), 626–641. DOI: 10.1037/0096-1523.16.3.626
- Dehaene, S., Naccache, L., Le Clec’H, G., Koechlin, E., Mueller, M., Dehaene-Lambertz, G., Le Bihan, D., et al. (1998). Imaging unconscious semantic priming. Nature, 395(6702), 597–600. DOI: 10.1038/26967
- de Leeuw, J. R. (2015). jsPsych: A JavaScript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47, 1–12. DOI: 10.3758/s13428-014-0458-y
- de Leeuw, J. R., & Motz, B. A. (2016). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods, 48(1), 1–12. DOI: 10.3758/s13428-015-0567-2
- Elze, T., & Tanner, T. G. (2012). Temporal Properties of Liquid Crystal Displays: Implications for Vision Science Experiments. PLOS ONE, 7(9),
e44048 . DOI: 10.1371/journal.pone.0044048 - Eriksson, K., & Lindskog, M. (2017). Encoding of numerical information in memory: Magnitude or nominal? DOI: 10.5964/jnc.v3i1.68
- Ferguson, A. M., Maloney, E. A., Fugelsang, J., & Risko, E. F. (2015). On the relation between math and spatial ability: The case of math anxiety. Learning and Individual Differences, 39, 1–12. DOI: 10.1016/j.lindif.2015.02.007
- Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon mechanical turk: Gold mine or coal mine? Computational Linguistics, 37(2), 413–420. DOI: 10.1162/COLI_a_00057
- Gleibs, I. H. (2017). Are all “research fields” equal? Rethinking practice for the use of data from crowdsourcing market places. Behavior Research Methods, 49(4), 1333–1342. DOI: 10.3758/s13428-016-0789-y
- Gökaydin, D., Brugger, P., & Loetscher, T. (2018). Sequential Effects in SNARC. Scientific Reports, 8(1),
10996 . DOI: 10.1038/s41598-018-29337-2 - Gosling, S. D., & Mason, W. (2015). Internet Research in Psychology. Annual Review of Psychology, 66(1), 877–902. DOI: 10.1146/annurev-psych-010814-015321
- Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should We Trust Web-Based Studies? A Comparative Analysis of Six Preconceptions About Internet Questionnaires. American Psychologist, 59(2), 93–104. DOI: 10.1037/0003-066X.59.2.93
- Halberda, J., Ly, R., Wilmer, J. B., Naiman, D. Q., & Germine, L. (2012). Number sense across the lifespan as revealed by a massive Internet-based sample. Proceedings of the National Academy of Sciences, 109(28), 11116–11120. DOI: 10.1073/pnas.1200196109
- Henik, A., & Tzelgov, J. (1982). Is three greater than five: The relation between physical and semantic size in comparison tasks. Memory & Cognition, 10(4), 389–395. DOI: 10.3758/BF03202431
- Henninger, F., Shevchenko, Y., Mertens, U. K., Kieslich, P. J., & Hilbig, B. E. (2019). Lab.js: A free, open, online study builder (Preprint). PsyArXiv. DOI: 10.31234/osf.io/fqr49
- Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. DOI: 10.1017/S0140525X0999152X
- Hilbig, B. E. (2016). Reaction time effects in lab- versus Web-based research: Experimental evidence. Behavior Research Methods, 48(4), 1718–1724. DOI: 10.3758/s13428-015-0678-9
- Hinrichs, J. V., Yurko, D. S., & Hu, J.-M. (1981). Two-digit number comparison: Use of place information. Journal of Experimental Psychology: Human Perception and Performance, 7(4), 890–901. DOI: 10.1037/0096-1523.7.4.890
- Huber, S., Nuerk, H.-C., Reips, U.-D., & Soltanlou, M. (2017). Individual differences influence two-digit number processing, but not their analog magnitude processing: A large-scale online study. Psychological Research. DOI: 10.1007/s00426-017-0964-5
- Irvine, A. (2018). The Unity game engine for a large-cohort, developmental study.
Behavioral Science Online conference , London. - Kaufmann, L., Koppelstaetter, F., Delazer, M., Siedentopf, C., Rhomberg, P., Golaszewski, S., Ischebeck, A., et al. (2005). Neural correlates of distance and congruity effects in a numerical Stroop task: An event-related fMRI study. NeuroImage, 25(3), 888–898. DOI: 10.1016/j.neuroimage.2004.12.041
- Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Nosek, B. A., et al. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. DOI: 10.1177/2515245918810225
- Kochari, A., & Schriefers, H. (in preparation). Processing symbolic magnitude information conveyed by number words and scalar adjectives: Parallel size-congruity effects.
- Landy, D., Silbert, N., & Goldin, A. (2013). Estimating Large Numbers. Cognitive Science, 37(5), 775–799. DOI: 10.1111/cogs.12028
- Lin, C.-Y., & Göbel, S. M. (2019). Arabic digits and spoken number words: Timing modulates the cross-modal numerical distance effect. Quarterly Journal of Experimental Psychology,
1747021819854444 . DOI: 10.1177/1747021819854444 - Lyle, H., Wylie, J., & Morsanyi, K. (2019). Cross-cultural differences in children’s mathematical development: Investigating the home numeracy environment. Ottawa, ON, Canada.
- Mathur, M., & Reichling, D. (2018). Open-source software for mouse-tracking in Qualtrics to measure category competition. DOI: 10.31219/osf.io/ymxau
- Miller, K. F., & Stigler, J. W. (1987). Counting in Chinese: Cultural variation in a basic cognitive skill. Cognitive Development, 2(3), 279–305. DOI: 10.1016/S0885-2014(87)90091-8
- Moyer, R. S., & Landauer, T. K. (1967). Time required for judgements of numerical inequality. Nature, 215(5109), 1519–1520. DOI: 10.1038/2151519a0
- Nation, K. (2018). Online large-scale studies with children out of the lab: The promise and the challenge. London: Behavioral Science Online conference.
- Neath, I., Earle, A., Hallett, D., & Surprenant, A. M. (2011). Response time accuracy in Apple Macintosh computers. Behavior Research Methods, 43(2), 353. DOI: 10.3758/s13428-011-0069-9
- Necka, E. A., Cacioppo, S., Norman, G. J., & Cacioppo, J. T. (2016). Measuring the Prevalence of Problematic Respondent Behaviors among MTurk, Campus, and Community Participants. PLOS ONE, 11(6),
e0157732 . DOI: 10.1371/journal.pone.0157732 - Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. DOI: 10.1073/pnas.1708274114
- Palan, S., & Schitter, C. (2018). Prolific.ac—A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27. DOI: 10.1016/j.jbef.2017.12.004
- Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a Participant Pool. Current Directions in Psychological Science, 23(3), 184–188. DOI: 10.1177/0963721414531598
- Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163. DOI: 10.1016/j.jesp.2017.01.006
- Peirce, J., Gray, J. R., Simpson, S., MacAskill, M., Höchenberger, R., Sogo, H., Lindeløv, J. K., et al. (2019). PsychoPy2: Experiments in behavior made easy. Behavior Research Methods, 51(1), 195–203. DOI: 10.3758/s13428-018-01193-y
- Pinel, P., Piazza, M., Le Bihan, D., & Dehaene, S. (2004). Distributed and Overlapping Cerebral Representations of Number, Size, and Luminance during Comparative Judgments. Neuron, 41(6), 983–993. DOI: 10.1016/S0896-6273(04)00107-2
- Pitt, B., & Casasanto, D. (2016). Reading experience shapes the mental timeline but not the mental number line. In Proceedings of the 38th Annual Conference of the Cognitive Science Society (pp. 2753–2758). Austin, TX:
Cognitive Science Society . - Plant, R. R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: New hardware, new problems? Behavior Research Methods, 41(3), 598–614. DOI: 10.3758/BRM.41.3.598
- Reimers, S., & Stewart, N. (2015). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 47(2), 309–327. DOI: 10.3758/s13428-014-0471-1
- Reimers, S., & Stewart, N. (2016). Auditory presentation and synchronization in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 48(3), 897–908. DOI: 10.3758/s13428-016-0758-5
- Santens, S., & Verguts, T. (2011). The size congruity effect: Is bigger always more? Cognition, 118(1), 94–110. DOI: 10.1016/j.cognition.2010.10.014
- Sekuler, R., Rubin, E., & Armstrong, R. (1971). Processing numerical information: A choice time analysis. Journal of Experimental Psychology, 90(1), 75–80. DOI: 10.1037/h0031366
- Semmelmann, K., Hönekopp, A., & Weigelt, S. (2017). Looking Tasks Online: Utilizing Webcams to Collect Video Data from Home. Frontiers in Psychology, 8. DOI: 10.3389/fpsyg.2017.01582
- Semmelmann, K., & Weigelt, S. (2017). Online psychophysics: Reaction time effects in cognitive experiments. Behavior Research Methods, 49(4), 1241–1260. DOI: 10.3758/s13428-016-0783-4
- Semmelmann, K., & Weigelt, S. (2018). Online webcam-based eye tracking in cognitive science: A first look. Behavior Research Methods, 50(2), 451–465. DOI: 10.3758/s13428-017-0913-7
- Shaki, S., Fischer, M. H., & Petrusic, W. M. (2009). Reading habits for both words and numbers contribute to the SNARC effect. Psychonomic Bulletin & Review, 16(2), 328–331. DOI: 10.3758/PBR.16.2.328
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. DOI: 10.1177/0956797611417632
- Stewart, N., Chandler, J., & Paolacci, G. (2017). Crowdsourcing Samples in Cognitive Science. Trends in Cognitive Sciences, 21(10), 736–748. DOI: 10.1016/j.tics.2017.06.007
- Stewart, N., Ungemach, C., Harris, A. J. L., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10(5), 479–491.
- Stoet, G. (2017). PsyToolkit: A Novel Web-Based Method for Running Online Questionnaires and Reaction-Time Experiments. Teaching of Psychology, 44(1), 24–31. DOI: 10.1177/0098628316677643
- Tzelgov, J., Meyer, J., & Henik, A. (1992). Automatic and intentional processing of numerical information. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(1), 166–179. DOI: 10.1037/0278-7393.18.1.166
- Vakharia, D., & Lease, M. (2015). Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms. In iConference, 2015.
Newport Beach , CA, USA. - Van Opstal, F., Gevers, W., De Moor, W., & Verguts, T. (2008). Dissecting the symbolic distance effect: Comparison and priming effects in numerical and nonnumerical orders. Psychonomic Bulletin & Review, 15(2), 419–425. DOI: 10.3758/PBR.15.2.419
- Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An Agenda for Purely Confirmatory Research. Perspectives on Psychological Science, 7(6), 632–638. DOI: 10.1177/1745691612463078
- Weber, S. J., & Cook, T. D. (1972). Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference. Psychological Bulletin, 77(4), 273–295. DOI: 10.1037/h0032351
- Woods, A. T., Velasco, C., Levitan, C. A., Wan, X., & Spence, C. (2015). Conducting perception research over the internet: A tutorial review. PeerJ, 3,
e1058 . DOI: 10.7717/peerj.1058 - Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (2018). Participant Nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review, 25(5), 1968–1972. DOI: 10.3758/s13423-017-1348-y
DOI: https://doi.org/10.5334/joc.85 | Journal eISSN: 2514-4820
Language: English
Submitted on: Mar 15, 2019
Accepted on: Sep 4, 2019
Published on: Sep 19, 2019
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Publication frequency: 1 issue per year
Keywords:
© 2019 Arnold R. Kochari, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.