Small numbers are an opportunity, not a problem

Keywords: mixed model, percentage of all non-overlapping data bayes, single case design, single case experimental design, time series

Abstract

Aims: outcomes of research in education and training are partly a function of the context in which that study takes place, the questions we ask, and what is feasible. Many questions are about learning, which involves repeated measurements in a particular time window, and the practical context is usually such that offering an intervention to some but not to all learners does not make sense or is unethical. For quality assurance and other purposes, education and training centers may have very locally oriented questions that they seek to answer, such as whether an intervention can be considered effective in their context of small numbers of learners. While the rationale behind the design and outcomes of this kind of studies may be of interest to a much wider community, for example to study the transferability of findings to other contexts, people are often discouraged to report on the outcomes of such studies at conferences or in educational research journals. The aim of this paper is to counter that discouragement and instead encourage people to see small numbers as an opportunity instead of as a problem.
Method: a worked example of a parametric and a non-parametric method for this type of situation, using simulated data in the zero-cost Open Source statistical program R version 4.0.5.
Results: contrary to the non-parametric method, the parametric method can provide estimates of intervention effectiveness for the individual participant, account for trends in different phases of a study. However, the non-parametric method provides a solution in several situations where the parametric method should be used.
Conclusion: Given the costs of research, the lessons to be learned from research, and statistical methods available, small numbers should be considered an opportunity, not a problem.

Downloads

Download data is not yet available.

Author Biography

Jimmie Leppink, University of York, York, North Yorkshire, United Kingdom.

PhD in Statistics Education, LLM in Forensics, Criminology and Law, and MSc in Psychology and Law from Maastricht University, the Netherlands; MSc in Statistics from Catholic University of Leuven, Belgium; currently Senior Lecturer in Medical Education and Director of Assessment at Hull York Medical School, University of York, United Kingdom.

References

Leppink J. The art of modelling the learning process: Uniting educational research and practice. Cham: Springer; 2020. https://doi.org/10.1007/978-3-030-43082-5

Michiels B, Heyvaert M, Meulders A, Onghena P. Confidence intervals for single-case effect size measures based on randomization test inversion. Behav Res Meth. 2017;49:363-81. https://doi.org/10.3758/s13428-016-0714-4

Michiels B, Onghena P. Randomized single-case AB phase designs: prospects and pitfalls. Behav Res Meth. 2018;51:2454-76. https://doi.org/10.3758/s13428-018-1084-x

Parker RI, Hagan-Burke S, Vannest KJ. Percentage of all non-overlapping data (PAND): An alternative to PND. J Spec Educ. 2007;40:194-204. https://doi.org/10.1177/00224669070400040101

Pérez-Fuster P, Sevilla J, Herrera G. Enhancing daily living skills in four adults with autism spectrum disorder through an embodied digital technology-mediated intervention. Res Aut Spect Dis. 2019;58:54-67. https://doi.org/10.1016/j.rasd.2018.08.006

Tanious R, De TK, Onghena P. A multiple randomization testing procedure for level, trend, variability, overlap, immediacy, and consistency in single-case phase designs. Behav Res Therap. 2019;119:103414. https://doi.org/10.1016/j.brat.2019.103414

Maric M, Van der Werff V. Single-case experimental designs in clinical intervention research. In: R Van de Schoot & M Milocević, Small sample size solutions: A guide for applied researchers and practitioners. OAPEN Home; 2020. p. 102-11. https://library.oapen.org/bitstream/handle/20.500.12657/22385/9780367221898_text%20(1).pdf?sequence=1#page=116

Pinheiro J, Bates D, DebRoy S, Sarkar D, Team RC. nlme: Linear and nonlinear mixed effects models. R Package Ver. 2013;3:111.

R Core Team. R: A language and environment for statistical computing [Internet]. Vienna: R Foundation for Statistical Computing (version 4.0.5); 2021 March 31 [cited 2021 May 6]. Available from: https://www.r-project.org

Viechtbauer W. Bias and efficiency of meta-analytic variance estimators in the random-effects model. J Educ Behav Stat. 2005;30:261-93. https://doi.org/10.3102/10769986030003261

Love J, Selker R, Marsman M, et al. JASP version 0.14.1.0 [Internet]; 2020 Dec 17 [cited 2021 May 6]. Available from: https://jasp-stats.org

Leppink J. Statistics for N = 1: A non-parametric Bayesian approach. Scientia Med. 2020;30:1-10. https://doi.org/10.15448/1980-6108.2020.1.38066

Parker RI, Hagan-Burke S, Vannest KJ. Percentage of all non-overlapping data (PAND): An alternative to PND. J Spec Educ. 2007;40:194-204. https://doi.org/10.1177/00224669070400040101

Published
2021-06-30
How to Cite
Leppink, J. (2021). Small numbers are an opportunity, not a problem. Scientia Medica, 31(1), e40128. https://doi.org/10.15448/1980-6108.2021.1.40128
Section
Education in Health Sciences