We are in the midst of a data explosion. Teams of researchers routinely collect data online and in-person for many different measures, methodologies, and (ideally) participants. These data inform increasingly complex theoretical questions. Yet, in my estimation, the analytical methods in the social and behavioral sciences have not kept up with the surge of data. For related reasons, we are simultaneously in the midst of a replication crisis, where foundational results fail to replicate.
Study preregistration and the adoption of open science practices have helped to address and mitigate the structural issues leading to the replication crisis. However, these practices may leave researchers feeling constrained in their ability to creatively explore unique and exciting questions at a time when more data are available than ever before. These issues mirror the growing pains experienced in machine learning at the onset of the “big data phenomenon” in industry. My research program is informed by modern practices from machine learning and data science. I employ and implement methods that reduce data dimensionality or capitalize on the multivariate nature of the problem space to identify the constructs that contribute the most signal while maintaining statistical accuracy and generalizability to new samples.
In my ongoing work, I am combining traditional inferential statistical methods and predictive machine learning methods to adjudicate between different models of executive control (Gullifer & Titone, 2021). The models differed in the extent to which features related to bilingualism impact (or interact to impact) executive control, thus mapping on to different theoretical perspectives. I found broad agreement between multiple model architectures (i.e., mixed-effects regression models and least absolute shrinkage selection models) that features indexing bilingual language experience better predicted executive control abilities vs. baseline models. Importantly, I confirmed that all models showed evidence of cross-validation: a keystone method in machine learning where model performance is tested against unmodeled data points to ensure reliability.
Below, you can watch a 15-minute talk on this topic (presented at Psychonomics 2020). You can also check out the project on OSF.
As part of this work, I implemented the computational methods for k-fold cross-validation of linear and logistic regression models (including mixed effects models). The approach I developed, combining inferential and predictive methods, allowed me to maximally interrogate the dataset without relying on problematic metrics like p-values and while providing evidence of reliability and model accuracy. Traditionally, inferential and predictive techniques have occupied distinct philosophical spaces in science, yet their combination holds high value for the future of the social and behavioral sciences where increasingly powerful analytical methods are being adopted. Following the initial publication in this line of work, Hofman and colleagues (2021) contributed a perspective piece in Nature: Human Behavior further outlining and advocating for a nearly identical approach that they termed “integrative modeling.”