Generalizability of Effect Sizes Within Aviation Research: More Samples Are Needed

It is often the case that researchers attempt to generalize findings from a single convenience sample to the population. They may also wish to make the claim that the sample effect sizes they discover are reasonably similar to the population parameters. The current study attempts to show that they can be mistaken in this assumption, and that different samples can vary dramatically in effect sizes due to myriad discrepancies, such as demographics, sample size, and random error, among other aspects of the samples. Seven hundred and eighty-one participants were recruited from Amazon’s Mechanical Turk, Florida Institute of Technology, Embry-Riddle Aeronautical University, and San Jose State University. The context of the study centered on participants’ willingness to fly on board completely autonomous (pilotless) commercial airliners. Participants were presented with one of two scenarios: a traditional configuration with two human pilots in control, and a flight piloted by an entirely autonomous autopilot system, with no human pilots in the cockpit. Not surprisingly, respondents in every sample were less inclined to fly without a human pilot. However, the differences in demographics and effect sizes between samples were significant; Cohen’s d effect sizes for the four samples were 2.10, 1.54, 1.51, and 1.76 for MTurk, FIT, ERAU and SJSU, respectively. These results indicate that, while a convenience sample may, to some degree, reflect the direction of the results one should expect to find in the general population, one such sample is not entirely sufficient to make claims of generalizability; rather, replications are necessary in order to produce a more precise population effect size.