The Danger of Testing by Selecting Controlled Subsets, with Applications to Spoken-Word Recognition
References
- Altieri, N., Gruenenfelder, T., & Pisoni, D. B. (2010). Clustering coefficients of lexical neighborhoods: Does neighborhood structure matter in spoken word recognition? Mental Lexicon, 5(1), 1–21. DOI: 10.1075/ml.5.1.01alt
- Anderson, S. F., & Maxwell, S. E. (2017, May). Addressing the “replication crisis”: Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52(3), 305–324. DOI: 10.1080/00273171.2017.1289361
- Arceneaux, K., Gerber, A. S., & Green, D. P. (2010). A cautionary note on the use of matching to estimate causal effects: An empirical example comparing matching estimates to an experimental benchmark. Sociological Methods & Research, 39(2), 256–282. DOI: 10.1177/0049124110378098
- Armstrong, B. C., Watson, C. E., & Plaut, D. C. (2012, September). SOS! an algorithm and software for the stochastic optimization of stimuli. Behavior Research Methods, 44(3), 675–705. DOI: 10.3758/s13428-011-0182-9
- Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Wicherts, J. M., et al. (2013, March). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119. DOI: 10.1002/per.1919
- Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424. DOI: 10.1080/00273171.2011.568786
- Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008, November). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. DOI: 10.1016/j.jml.2007.12.005
- Balota, D. A., Yap, M. J., Cortese, M. J., Hutchison, K. A., Kessler, B., Loftis, B., Treiman, R., et al. (2007). The English Lexicon Project. Behavior Research Methods, 39(3), 445–459. DOI: 10.3758/BF03193014
- Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013, April). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3). DOI: 10.1016/j.jml.2012.11.001
- Brysbaert, M., & New, B. (2009). Moving beyond Kucera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41(4), 977–990. DOI: 10.3758/BRM.41.4.977
- Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementaion of propensity score matching. Journal of Economic Surveys, 22(1), 31–72. DOI: 10.1111/j.1467-6419.2007.00527.x
- Carters, M. A., Rieger, E., & Bell, J. (2015, September). Reduced inhibition of return to food images in obese individuals. PLOS ONE, 10(9), 1–20. DOI: 10.1371/journal.pone.0137821
- Chan, K. Y., & Vitevitch, M. S. (2009, November). The influence of the phonological neighborhood clustering coefficient on spoken word recognition. Journal of Experimental Psychology: Human Perception & Performance, 35(6), 1934–1949. DOI: 10.1037/a0016902
- Clark, H. H. (1973, August). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335–359. DOI: 10.1016/S0022-5371(73)80014-3
- Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7(3), 249–253. DOI: 10.1177/014662168300700301
- Connine, C. M., Mullennix, J., Shernoff, E., & Yelen, J. (1990). Word familiarity and frequency in visual and auditory word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(6), 1084–1096. DOI: 10.1037/0278-7393.16.6.1084
- Connine, C. M., Titone, D., & Wang, J. (1993). Auditory word recognition: Extrinsic and intrinsic effects of word frequency. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19(1), 81–94. DOI: 10.1037/0278-7393.19.1.81
- Coupé, C. (2011). BALI: A tool to build experimental materials in psycholinguistics. In: Architectures and mechanisms of language processing.
- Cutler, A. (1981). Making up materials is a confounded nuisance, or: Will we able to run any psycholinguistic experiments at all in 1990? Cognition, 10, 65–70. DOI: 10.1016/0010-0277(81)90026-3
- Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006, June). The mini-IPIP scales: tiny-yet-effective measures of the big five factors of personality. Psychological Assessment, 18(2), 192–203. DOI: 10.1037/1040-3590.18.2.192
- D’Orazio, M., Zio, M. D., & Scanu, M. (2006). Statistical matching: Theory and practice (Wiley Series in Survey Methodology). John Wiley & Sons. DOI: 10.1002/0470023554
- Farrar, D. E., & Glauber, R. R. (1967). Multicollinearity in regression analysis: The problem revisited. The Review of Economics and Statistics, 49(1), 92–107. DOI: 10.2307/1937887
- Forster, K. (2000). The potential for experimenter bias effects in word recognition experiments. Memory & Cognition, 28(7), 1109–1115. DOI: 10.3758/BF03211812
- Garey, M. R., & Johnson, D. S. (1979). Computers and intractability: A guide to the theory of NP-completeness. W. H. Freeman & Co.
- Gelman, A., & Park, D. K. (2009). Splitting a predictor at the upper quarter or third and the lower quarter or third. The American Statistician, 63(1), 1–8. DOI: 10.1198/tast.2009.0001
- Goldinger, S. D., Luce, P. A., & Pisoni, D. B. (1989). Priming lexical neighbors of spoken words: Effects of competition and inhibition. Journal of Memory and Language, 28, 501–518. DOI: 10.1016/0749-596X(89)90009-0
- Gu, X. S., & Rosenbaum, P. R. (1993). Comparison of multivariate matching methods: Structures, distances, and algorithms. Journal of Computational and Graphical Statistics, 2(4), 405–420.
- Guasch, M., Haro, J., & Boada, R. (2017). Clustering words to match conditions: An algorithm for stimuli selection in factorial designs. Psicológica, 38(1), 111–131.
- Gurobi Optimization, Inc. (2015). Gurobi optimizer reference manual. Retrieved from:
http://www.gurobi.com . - Henrich, J., Heine, S. J., & Norenzayan, A. (2010, June). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. DOI: 10.1017/S0140525X0999152X
- Hettinger, V. E., Hutchinson, D. M., & Bosson, J. K. (2014, October). Influence of professional status on perceptions of romantic relationship dynamics. Psychology of Men & Masculinity, 15(4), 470–480. DOI: 10.1037/a0034034
- Hirano, K., Imbens, G. W., & Ridder, G. (2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71(4), 1161–1189. DOI: 10.1111/1468-0262.00442
- Huber, S., Dietrich, J. F., Nagengast, B., & Moeller, K. (2017, June 1). Using propensity score matching to construct experimental stimuli. Behavior Research Methods, 49(3), 1107–1119. DOI: 10.3758/s13428-016-0771-8
- Iacobucci, D., Posavac, S. S., Kardes, F. R., Schneider, M. J., & Popovich, D. L. (2015a). The median split: Robust, refined, and revived. J. Consumer Psychology, 25(4), 690–704. DOI: 10.1016/j.jcps.2015.06.014
- Iacobucci, D., Posavac, S. S., Kardes, F. R., Schneider, M. J., & Popovich, D. L. (2015b). Toward a more nuanced understanding of the statistical properties of a median split. J. Consumer Psychology, 25(4), 652–665. DOI: 10.1016/j.jcps.2014.12.002
- Ioannidis, J. P. A. (2005, August). Why most published research findings are false. PLoS Medicine, 2(8),
e124 . DOI: 10.1371/journal.pmed.0020124 - Jaeger, T. F. (2008, November). Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models. Journal of Memory and Language, 59(4), 434–446. DOI: 10.1016/j.jml.2007.11.007
- Jayachandran, S., De Laat, J., Lambin, E. F., Stanton, C. Y., Audy, R., & Thomas, N. E. (2017). Cash for carbon: A randomized trial of payments for ecosystem services to reduce deforestation. Science, 357(6348), 267–273. DOI: 10.1126/science.aan0568
- King, G., & Nielsen, R. (2016). Why propensity scores should not be used for matching. Working paper.
- Kleinberg, J., & Tardos, E. (2005). Algorithm design. Addison-Wesley.
- LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 604–620.
- Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584–585. DOI: 10.1126/science.aal3618
- Luce, P. A., & Pisoni, D. B. (1998). Recognizing spoken words: The Neighborhood Activation Model. Ear & Hearing, 19(1), 1–36. DOI: 10.1097/00003446-199802000-00001
- Marslen-Wilson, W., & Tyler, L. K. (1980). The temporal structure of spoken language understanding. Cognition, 8(1), 1–71. DOI: 10.1016/0010-0277(80)90015-3
- McClelland, G. H., Lynch, J. G.,
Jr. , Irwin, J. R., Spiller, S. A., & Fitzsimons, G. J. (2015). Median splits, type II errors, and false-positive consumer psychology: Don’t fight the power. J. Consumer Psychology, 25(4), 679–689. DOI: 10.1016/j.jcps.2015.05.006 - Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). DOI: 10.1126/science.aac4716
- Papadimitriou, C. H., & Steiglitz, K. (1982). Combinatorial optimization: Algorithms and complexity. Prentice-Hall, Inc.
- Pond, G. (2011). Statistical issues in the use of dynamic allocation methods for balancing baseline covariates. British Journal of Cancer, 104(11), 1711–1715. DOI: 10.1038/bjc.2011.157
- Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. DOI: 10.1093/biomet/70.1.41
- Rubin, D. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied Statistics, 808–840. DOI: 10.1214/08-AOAS187
- Rucker, D. D., McShane, B. B., & Preacher, K. J. (2015). A researcher’s guide to regression, discretization, and median splits of continuous variables. J. Consumer Psychology, 25(4), 666–678. DOI: 10.1016/j.jcps.2015.04.004
- Savin, H. B. (1963). Word-frequency effect and errors in the perception of speech. The Journal of the Acoustical Society of America, 35(2), 200–206. DOI: 10.1121/1.1918432
- Schmidt, K., Patnaik, P., & Kensinger, E. A. (2011, February). Emotion’s influence on memory for spatial and temporal context. Cognition and Emotion, 25(2), 229–243. DOI: 10.1080/02699931.2010.483123
- Sibley, D., Kello, C., & Seidenberg, M. (2009). Error, error everywhere: A look at megastudies of word reading. In: Proceedings of the 31st Annual Conference of the Cognitive Science Society.
- Siew, C. S. Q. (2016). The influence of 2-hop network density on spoken word recognition. Psychonomic Bulletin & Review, 496–502.
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. DOI: 10.1177/0956797611417632
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2017). False-positive citations. Perspectives on Psychological Science.
- Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017, November). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12(6), 1123–1128. DOI: 10.1177/1745691617708630
- Strand, J. F., & Sommers, M. S. (2011, September). Sizing up the competition: Quantifying the influence of the mental lexicon on auditory and visual spoken word recognition. Journal of the Acoustical Society of America, 130(3), 1663–1672. DOI: 10.1121/1.3613930
- Tucker, B. V., Brenner, D., Danielson, D. K., Kelley, M. C., Nenadić, F., & Sims, M. (2018, June). The massive auditory lexical decision (MALD) database. Behavior Research Methods. DOI: 10.3758/s13428-018-1056-1
- Van Casteren, M., & Davis, M. H. (2007, November). Match: A program to assist in matching the conditions of factorial experiments. Behavior Research Methods, 39(4), 973–978. DOI: 10.3758/BF03192992
- Varga, A., Picano, E., Dodi, C., Barbieri, A., Pratali, L., & Gaddi, O. (1999). Madness and method in stress echo reading. European Heart Journal, 20(17), 1271–1275. DOI: 10.1053/euhj.1999.1541
- Vitevitch, M. S. (2002). Influence of onset density on spoken-word recognition. Journal of Experimental Psychology: Human Perception and Performance, 28(2), 270–278. DOI: 10.1037/0096-1523.28.2.270
- Vitevitch, M. S. (2007). The spread of the phonological neighborhood influences spoken word recognition. Memory and Cognition, 35(1), 166–175. DOI: 10.3758/BF03195952
- Vitevitch, M. S., & Luce, P. A. (1998). When words compete: Levels of processing in perception of spoken words. Psychological Science, 9(4), 325–329. DOI: 10.1111/1467-9280.00064
- Vitevitch, M. S., Luce, P. A., Pisoni, D. B., & Auer, E. T. (1999). Phonotactics, neighborhood activation, and lexical access for spoken words. Brain and Language, 68(1), 306–311. DOI: 10.1006/brln.1999.2116
- Wicherts, J., Veldkamp, C., Augusteijn, H., Bakker, M., van Aert, R., & van Assen, M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. DOI: 10.3389/fpsyg.2016.01832
- Yates, M. (2013). How the clustering of phonological neighbors affects visual word recognition. J. Experimental Psychology: Learning, Memory, & Cognition, 39(5), 1649–1656. DOI: 10.1037/a0032422
- Zebrowitz, L., White, B., & Wieneke, K. (2008). Mere exposure and racial prejudice: Exposure to other-race faces increases liking for strangers of that race. Social Cognition, 26(3), 259–275. DOI: 10.1521/soco.2008.26.3.259
DOI: https://doi.org/10.5334/joc.51 | Journal eISSN: 2514-4820
Language: English
Submitted on: Feb 6, 2018
Accepted on: Dec 8, 2018
Published on: Jan 24, 2019
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Publication frequency: 1 issue per year
Keywords:
© 2019 David Liben-Nowell, Julia Strand, Alexa Sharp, Tom Wexler, Kevin Woods, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.