Nicolás Vizioli,
Alejandro Pagano
Universidad de Buenos Aires, Buenos Aires, Argentina
Abstract
The present study takes a tour of the concept of reliability as one of the fundamental psychometric properties in Classical Tests Theory. The concept and its different practical applications are developed to investigate the degree of reliability of a measuring instrument. Focusing on the calculation of internal consistency from alpha and omega as the most used coefficients and the importance of calculating these coefficients by using polychoric correlation matrices (PCM). The main objective is to present a guide in Spanish for the calculation of ordinal reliability coefficients using the R/Rstudio program. Providing an example at the empirical level that shows the relevance of calculating this type of coefficients for calculating the reliability of an instrument. Using a sample of 266 adults between the ages of 18 and 63 years (M = 31.91 SD = 11.50), the version adapted to Argentina of the Beck Anxiety Inventory and the Emotional Regulation Questionnaire were administered. In this way, coefficients are exposed to estimate the reliability of the instruments that account for their advantages and disadvantages, performing the calculation using MCP, Pearson’s correlation matrix and Pearson’s covariance matrix. From the results, it is evident that the calculation using MCP provided higher degrees of reliability compared to the calculation using the other two matrices. This document is expected to be of importance to researchers unfamiliar with R.
Keywords: Reliability, Single Administration, Alpha, Omega, Polychoric Correlations Matrix, R/Rstudio
References
Aguirre-Urreta, M. I., Rönkkö, M., & McIntosh, C. N. (2019). A cautionary note on the finite sample behavior of maximal reliability. Psychological Methods, 24(2), 236-252. https://psycnet.apa.org/doi/10.1037/met0000176
Armor, D. J. (1973). Theta reliability and factor scaling. Sociological Methodology, 5, 17-50. https://doi.org/10.2307/270831
Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuring clinical anxiety: Psychometric properties. Journal of Consulting and Clinical Psychology, 56(6), 893–897. https://doi.org/10.1037/0022-006X.56.6.893
Bernaards, C. A., & Jennrich, R. I. (2005). Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educational and Psychological Measurement, 65, 676-696. https://doi.org/10.1177/001316440427250
Browne, M. W., & Cudeck, R. (1993). Alternative ways ofassessing model fit. En K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Sage.
Catalán, H. E. N. (2019). Reliability, population classification and weighting in multidimensional poverty measurement: A Monte Carlo study. Social Indicators Research, 142(3), 887-910. https://doi.org/10.1007/s11205-018-1950-z
Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, (1), 98-104. https://psycnet.apa.org/doi/10.1037/0021-9010.78.1.98
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555
Deng, L., & Chan, W. (2017). Testing the difference between reliability coefficients alpha and omega. Educational and Psychological Measurement, 77(2), 185-203. https://doi.org/10.1177%2F0013164416658325
Domínguez-Lara, S. A. (2012). Propuesta para el cálculo del Alfa Ordinal y Theta de Armor. Revista de Investigación en Psicología, 15(1), 213-217. https://revistasinvestigacion.unmsm.edu.pe/index.php/psico/article/view/3684
Domínguez-Lara, S. A. (2016). Evaluación de la confiabilidad del constructo mediante el Coeficiente H: breve revisión conceptual y aplicaciones. Psychologia. Avances de la Disciplina, 10(2), 87-94. http://www.scielo.org.co/scielo.php?pid=S1900-23862016000200087&script=sci_abstract&tlng=en
Domínguez-Lara, S. (2018). Fiabilidad y alfa ordinal. Actas Urológicas Españolas, 42(2), 140-141. https://doi.org/10.1016/j.acuro.2017.07.002
Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problema of internal consistency estimation. British Journal of Psychology, 105(3), 399-412. https://doi.org/10.1111/bjop.12046
Elosua, P., & Zumbo, B. (2008). Coeficientes de fiabilidad para escalas de respuesta categórica ordenada. Psicothema, 20(4), 896-901. http://www.psicothema.com/pdf/3572.pdf
Flora, D. B. (2020). Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using R to obtain better reliability estimates. Advances in Methods and Practices in Psychological Science, 3(4), 484-501. https://doi.org/10.1177%2F2515245920951747
Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466-491. https://doi.org/10.1037/1082-989X.9.4.466
Fox, J., & Bouchet-Valat, M. (2019). Rcmdr: R Commander. R package version 2.5-2. https://cran.r-project.org/web/packages/Rcmdr/index.html
Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research, and Evaluation, 17(1), 3. https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1247&context=pare
Green, S. B., & Yang, Y. (2015). Evaluation of dimensionality in the assessment of internal consistency reliability: Coefficient alpha and omega coefficients. Educational Measurement: Issues and Practice, 34(4), 14-20.
Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85(2), 348-362. https://doi.org/10.1037/0022-3514.85.2.348
Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255-282. https://doi.org/10.1007/BF02288892
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2014). Multivariate data analysis (7th ed.). Pearson Education Limited Harlow.
Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. En R. Cudeck, S. du Toit, & D. Soerbom (Eds.), Structural equation modeling: Present and future—A festschrift in honor of Karl Jöreskog (pp. 195-216). Scientific Software International.
Jackson, P. H., & Agunwamba, C. C. (1977). Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I: Algebraic lower bounds. Psychometrika, 42(4), 567-578. https://doi.org/10.1007/BF02295979
Jöreskog, K. G. (1994). On the estimation of polychoric correlations and their asymptotic covariance matrix. Psychometrika, 59(3), 381-389. https://doi.org/10.1007/BF02296131
Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., & Rosseel, Y. (2020). semTools: Useful tools for structural equation modeling (R package Version 0.5-3). https://CRAN.R-project.org/package=semTools
Kelley, K., & Lai, K. (2012). MBESS: MBESS. R package version 3.3.2. http://CRAN.R-project.org/package=MBESS
Kline, R. B. (2011). Principles and practice of structural equation modeling. Guilford Press.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Addison Wesley.
Lorenzo-Seva, U., & Ferrando, P. (2020). Manual of the program FACTOR v. 10. http://psico.fcep.urv.es/
Manterola, C., Quiroz, G., Salazar, P., & García, N. (2019). Metodología de los tipos y diseños de estudio más frecuentemente utilizados en investigación clínica. Revista Médica Clínica Las Condes, 30(1), 36-49. https://doi.org/10.1016/j.rmclc.2018.11.005
McDonald, R. P. (1981). The dimensionality of tests and items. British Journal of mathematical and statistical Psychology, 34(1), 100-117. https://doi.org/10.1111/j.2044-8317.1981.tb00621.x
McDonald, R.P. (1999). Test theory: A unified treatment. Erlbaum.
McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412-433. https://doi.org/10.1037/met0000144
Mikulic, I. M., (2007). Construcción y adaptación de pruebas psicológicas [Manuscrito inédito]. Facultad de Psicología, Universidad de Buenos Aires.
Moltner, A., & Revelle, W. (2015). Find the Greatest Lower Bound to Reliability. http://personality-project.org/r/psych/help/glb.algebraic.html
Muñiz, J. (2010). Las teorías de los tests: teoría clásica y teoría de respuesta a los ítems. Papeles del Psicólogo, 31(1), 57-66. http://papelesdelpsicologo.es/pdf/1796.pdf
Pagano, A. E., & Vizioli, N. A. (en prensa). Adaptación del Cuestionario de Regulación Emocional (ERQ) en población adulta de la Ciudad Autónoma de Buenos Aires y el Conurbano Bonaerense. Psicodebate. Psicología, Cultura y Sociedad.
R Development Core Team. (2020). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. http://www.R-project.org
Racine, J. S. (2011). RStudio: A Platform-Independent IDE for R and Sweave. Journal of Applied Econometrics, 27(1), 167–172. https://doi.org/10.1002/jae.1278
Raykov, T. (1998). Coefficient alpha and composite reliability with interrelated nonhomogeneous items. Applied Psychological Measurement, 22(4), 375-385. https://doi.org/10.1177%2F014662169802200407
Raykov, T. (2012). Scale development using structural equation modeling. En Hoyle, R. (Ed.), Handbook of structural equation modeling (pp. 472-492). Guilford Press.
Raykov, T., & Marcoulides, G. A. (2019). Thanks coefficient alpha, we still need you! Educational and psychological measurement, 79(1), 200-210. https://doi.org/10.1177%2F0013164417725127
Revelle, W. (1979). Hierarchical cluster analysis and the internal structure of tests. Multivariate Behavioral Research, 14(1), 57-74. https://doi.org/10.1207/s15327906mbr1401_4
Revelle, W. (2011). An overview of the psych package. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=813c60673565b947fc406f480885e9f1d6694022
Revelle, W. (2020). Package ‘psych’. https://cran.r-project.org/web/packages/psych/psych.pdf
Revelle, W. (2021). How To: Use the psych package for Factor Analysis and data reduction. https://www.personality-project.org/r/psych/HowTo/factor.pdf
Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74(1), 145-154. https://doi.org/10.1007/s11336-008-9102-z
Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354. https://psycnet.apa.org/doi/10.1037/a0029315
Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. http://www.jstatsoft.org/v48/i02/
Rosseel, Y. (2020). The lavaan tutorial. https://www.lavaan.ugent.be/tutorial/tutorial.pdf
Savalei, V., Reise, S. P., Vazire, S., & Fried, E. (2019). Don’t Forget the Model in Your Model-based Reliability Coefficients: A Reply to McNeish (2018). Collabra: Psychology, 5(1). https://doi.org/10.1525/collabra.247
Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107. https://dx.doi.org/10.1007%2Fs11336-008-9101-0
Sijtsma, K., & van der Ark, L. A. (2015). Conceptions of reliability revisited and practical recommendations. Nursing Research, 64(2), 128-136. https://doi.org/10.1097/nnr.0000000000000077
Thurstone, L. L. (1928). Attitudes can be measured. American Journal of Sociology, 33(4), 529-554. https://www.gwern.net/docs/psychology/1928-thurstone.pdf
Trizano-Hermosilla, I., & Alvarado, J. M. (2016). Best alternatives to Cronbach’s alpha reliability in realistic conditions: congeneric and asymmetrical measurements. Frontiers in Psychology, 7, Article 769. https://doi.org/10.3389/fpsyg.2016.00769
Ventura-León, J. L. (2018). ¿Es el final del alfa de Cronbach? Adicciones, 31(1), 80-81. http://adicciones.es/index.php/adicciones/article/viewFile/1037/965
Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017). Un viaje alrededor de alfa y omega para estimar la fiabilidad de consistencia interna. Anales de Psicología, 33(3), 755-782. https://revistas.um.es/analesps/article/view/analesps.33.3.268401/215531
Vizioli, N. A., & Pagano, A. E. (2020). Adaptación del Inventario de Ansiedad de Beck en población de Buenos Aires. Interacciones, e171-e171. https://doi.org/10.24016/2020.v6n3.171
Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6(1), 4. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.890.6722&rep=rep1&type=pdf
Zumbo, B. D., & Kroc, E. (2019). A measurement is a choice and Stevens’ scales of measurement do not help make it: A response to Chalmers. Educational and Psychological Measurement, 79(6), 1184-1197. https://dx.doi.org/10.1177%2F0013164419844305