From Alpha to Omega: Estimation of Ordinal Reliability. A Practical Guide

Authors

  • Vizioli Vizioli Universidad de Buenos Aires
  • Alejandro Pagano Universidad de Buenos Aires

DOI:

https://doi.org/10.22544/rcps.v41i02.02

Keywords:

Reliability, Single Administration, Alpha, Omega, Polychoric Correlations Matrix, R/Rstudio

Abstract

The present study takes a tour of the concept of reliability as one of the fundamental psychometric properties in Classical Tests Theory. The concept and its different practical applications are developed to investigate the degree of reliability of a measuring instrument. Focusing on the calculation of internal consistency from alpha and omega as the most used coefficients and the importance of calculating these coefficients by using polychoric correlation matrices (PCM). The main objective is to present a guide in Spanish for the calculation of ordinal reliability coefficients using the R/Rstudio program. Providing an example at the empirical level that shows the relevance of calculating this type of coefficients for calculating the reliability of an instrument. Using a sample of 266 adults between the ages of 18 and 63 years (M = 31.91 SD = 11.50), the version adapted to Argentina of the Beck Anxiety Inventory and the Emotional Regulation Questionnaire were administered. In this way, coefficients are exposed to estimate the reliability of the instruments that account for their advantages and disadvantages, performing the calculation using MCP, Pearson’s correlation matrix and Pearson’s covariance matrix. From the results, it is evident that the calculation using MCP provided higher degrees of reliability compared to the calculation using the other two matrices. This document is expected to be of importance to researchers unfamiliar with R.

Author Biographies

Vizioli Vizioli, Universidad de Buenos Aires

magíster en Psicodiagnóstico y Evaluación Psicológica. Se desempeña como docentee investigador formado UBACyT en la Universidad de Buenos Aires.

Alejandro Pagano, Universidad de Buenos Aires

Magíster en Psicodiagnóstico y Evaluación Psicológica, docente e investigadorUBACyT en la Universidad de Buenos Aires

References

Aguirre-Urreta, M. I., Rönkkö, M., & McIntosh, C. N. (2019). A cautionary note on the finite sample behavior of maximal reliability. Psychological Methods, 24(2), 236-252. https://psycnet.apa.org/doi/10.1037/met0000176

Armor, D. J. (1973). Theta reliability and factor scaling. Sociological Methodology, 5, 17-50. https://doi.org/10.2307/270831

Beck, A. T., Epstein, N., Brown, G., & Steer, R. A. (1988). An inventory for measuring clinical anxiety: Psychometric properties. Journal of Consulting and Clinical Psychology, 56(6), 893–897. https://doi.org/10.1037/0022-006X.56.6.893

Bernaards, C. A., & Jennrich, R. I. (2005). Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educational and Psychological Measurement, 65, 676-696. https://doi.org/10.1177/001316440427250

Browne, M. W., & Cudeck, R. (1993). Alternative ways ofassessing model fit. En K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Sage.

Catalán, H. E. N. (2019). Reliability, population classification and weighting in multidimensional poverty measurement: A Monte Carlo study. Social Indicators Research, 142(3), 887-910. https://doi.org/10.1007/s11205-018-1950-z

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, (1), 98-104. https://psycnet.apa.org/doi/10.1037/0021-9010.78.1.98

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555

Deng, L., & Chan, W. (2017). Testing the difference between reliability coefficients alpha and omega. Educational and Psychological Measurement, 77(2), 185-203. https://doi.org/10.1177%2F0013164416658325

Domínguez-Lara, S. A. (2012). Propuesta para el cálculo del Alfa Ordinal y Theta de Armor. Revista de Investigación en Psicología, 15(1), 213-217. https://revistasinvestigacion.unmsm.edu.pe/index.php/psico/article/view/3684

Domínguez-Lara, S. A. (2016). Evaluación de la confiabilidad del constructo mediante el Coeficiente H: breve revisión conceptual y aplicaciones. Psychologia. Avances de la Disciplina, 10(2), 87-94. http://www.scielo.org.co/scielo.php?pid=S1900-23862016000200087&script=sci_abstract&tlng=en

Domínguez-Lara, S. (2018). Fiabilidad y alfa ordinal. Actas Urológicas Españolas, 42(2), 140-141. https://doi.org/10.1016/j.acuro.2017.07.002

Dunn, T. J., Baguley, T., & Brunsden, V. (2014). From alpha to omega: A practical solution to the pervasive problema of internal consistency estimation. British Journal of Psychology, 105(3), 399-412. https://doi.org/10.1111/bjop.12046

Elosua, P., & Zumbo, B. (2008). Coeficientes de fiabilidad para escalas de respuesta categórica ordenada. Psicothema, 20(4), 896-901. http://www.psicothema.com/pdf/3572.pdf

Flora, D. B. (2020). Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using R to obtain better reliability estimates. Advances in Methods and Practices in Psychological Science, 3(4), 484-501. https://doi.org/10.1177%2F2515245920951747

Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466-491. https://doi.org/10.1037/1082-989X.9.4.466

Fox, J., & Bouchet-Valat, M. (2019). Rcmdr: R Commander. R package version 2.5-2. https://cran.r-project.org/web/packages/Rcmdr/index.html

Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research, and Evaluation, 17(1), 3. https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1247&context=pare

Green, S. B., & Yang, Y. (2015). Evaluation of dimensionality in the assessment of internal consistency reliability: Coefficient alpha and omega coefficients. Educational Measurement: Issues and Practice, 34(4), 14-20.

Gross, J. J., & John, O. P. (2003). Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. Journal of Personality and Social Psychology, 85(2), 348-362. https://doi.org/10.1037/0022-3514.85.2.348

Guttman, L. (1945). A basis for analyzing test-retest reliability. Psychometrika, 10(4), 255-282. https://doi.org/10.1007/BF02288892

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2014). Multivariate data analysis (7th ed.). Pearson Education Limited Harlow.

Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent variable systems. En R. Cudeck, S. du Toit, & D. Soerbom (Eds.), Structural equation modeling: Present and future—A festschrift in honor of Karl Jöreskog (pp. 195-216). Scientific Software International.

Jackson, P. H., & Agunwamba, C. C. (1977). Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I: Algebraic lower bounds. Psychometrika, 42(4), 567-578. https://doi.org/10.1007/BF02295979

Jöreskog, K. G. (1994). On the estimation of polychoric correlations and their asymptotic covariance matrix. Psychometrika, 59(3), 381-389. https://doi.org/10.1007/BF02296131

Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., & Rosseel, Y. (2020). semTools: Useful tools for structural equation modeling (R package Version 0.5-3). https://CRAN.R-project.org/package=semTools

Kelley, K., & Lai, K. (2012). MBESS: MBESS. R package version 3.3.2. http://CRAN.R-project.org/package=MBESS

Kline, R. B. (2011). Principles and practice of structural equation modeling. Guilford Press.

Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Addison Wesley.

Lorenzo-Seva, U., & Ferrando, P. (2020). Manual of the program FACTOR v. 10. http://psico.fcep.urv.es/

Manterola, C., Quiroz, G., Salazar, P., & García, N. (2019). Metodología de los tipos y diseños de estudio más frecuentemente utilizados en investigación clínica. Revista Médica Clínica Las Condes, 30(1), 36-49. https://doi.org/10.1016/j.rmclc.2018.11.005

McDonald, R. P. (1981). The dimensionality of tests and items. British Journal of mathematical and statistical Psychology, 34(1), 100-117. https://doi.org/10.1111/j.2044-8317.1981.tb00621.x

McDonald, R.P. (1999). Test theory: A unified treatment. Erlbaum.

McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412-433. https://doi.org/10.1037/met0000144

Mikulic, I. M., (2007). Construcción y adaptación de pruebas psicológicas [Manuscrito inédito]. Facultad de Psicología, Universidad de Buenos Aires.

Moltner, A., & Revelle, W. (2015). Find the Greatest Lower Bound to Reliability. http://personality-project.org/r/psych/help/glb.algebraic.html

Muñiz, J. (2010). Las teorías de los tests: teoría clásica y teoría de respuesta a los ítems. Papeles del Psicólogo, 31(1), 57-66. http://papelesdelpsicologo.es/pdf/1796.pdf

Pagano, A. E., & Vizioli, N. A. (en prensa). Adaptación del Cuestionario de Regulación Emocional (ERQ) en población adulta de la Ciudad Autónoma de Buenos Aires y el Conurbano Bonaerense. Psicodebate. Psicología, Cultura y Sociedad.

R Development Core Team. (2020). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. http://www.R-project.org

Racine, J. S. (2011). RStudio: A Platform-Independent IDE for R and Sweave. Journal of Applied Econometrics, 27(1), 167–172. https://doi.org/10.1002/jae.1278

Raykov, T. (1998). Coefficient alpha and composite reliability with interrelated nonhomogeneous items. Applied Psychological Measurement, 22(4), 375-385. https://doi.org/10.1177%2F014662169802200407

Raykov, T. (2012). Scale development using structural equation modeling. In Hoyle, R. (Ed.), Handbook of structural equation modeling (pp. 472-492). Guilford Press.

Raykov, T., & Marcoulides, G. A. (2019). Thanks coefficient alpha, we still need you! Educational and psychological measurement, 79(1), 200-210. https://doi.org/10.1177%2F0013164417725127

Revelle, W. (1979). Hierarchical cluster analysis and the internal structure of tests. Multivariate Behavioral Research, 14(1), 57-74. https://doi.org/10.1207/s15327906mbr1401_4

Revelle, W. (2011). An overview of the psych package. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=813c60673565b947fc406f480885e9f1d6694022

Revelle, W. (2020). Package ‘psych’. https://cran.r-project.org/web/packages/psych/psych.pdf

Revelle, W. (2021). How To: Use the psych package for Factor Analysis and data reduction. https://www.personality-project.org/r/psych/HowTo/factor.pdf

Revelle, W., & Zinbarg, R. E. (2009). Coefficients alpha, beta, omega, and the glb: Comments on Sijtsma. Psychometrika, 74(1), 145-154. https://doi.org/10.1007/s11336-008-9102-z

Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354. https://psycnet.apa.org/doi/10.1037/a0029315

Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1–36. http://www.jstatsoft.org/v48/i02/

Rosseel, Y. (2020). The lavaan tutorial. https://www.lavaan.ugent.be/tutorial/tutorial.pdf

Savalei, V., Reise, S. P., Vazire, S., & Fried, E. (2019). Don’t Forget the Model in Your Model-based Reliability Coefficients: A Reply to McNeish (2018). Collabra: Psychology, 5(1). https://doi.org/10.1525/collabra.247

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107. https://dx.doi.org/10.1007%2Fs11336-008-9101-0

Sijtsma, K., & van der Ark, L. A. (2015). Conceptions of reliability revisited and practical recommendations. Nursing Research, 64(2), 128-136. https://doi.org/10.1097/nnr.0000000000000077

Thurstone, L. L. (1928). Attitudes can be measured. American Journal of Sociology, 33(4), 529-554. https://www.gwern.net/docs/psychology/1928-thurstone.pdf

Trizano-Hermosilla, I., & Alvarado, J. M. (2016). Best alternatives to Cronbach’s alpha reliability in realistic conditions: congeneric and asymmetrical measurements. Frontiers in Psychology, 7, Article 769. https://doi.org/10.3389/fpsyg.2016.00769

Ventura-León, J. L. (2018). ¿Es el final del alfa de Cronbach? Adicciones, 31(1), 80-81. http://adicciones.es/index.php/adicciones/article/viewFile/1037/965

Viladrich, C., Angulo-Brunet, A., & Doval, E. (2017). Un viaje alrededor de alfa y omega para estimar la fiabilidad de consistencia interna. Anales de Psicología, 33(3), 755-782. https://revistas.um.es/analesps/article/view/analesps.33.3.268401/215531

Vizioli, N. A., & Pagano, A. E. (2020). Adaptación del Inventario de Ansiedad de Beck en población de Buenos Aires. Interacciones, e171-e171. https://doi.org/10.24016/2020.v6n3.171

Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of Modern Applied Statistical Methods, 6(1), 4. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.890.6722&rep=rep1&type=pdf

Zumbo, B. D., & Kroc, E. (2019). A measurement is a choice and Stevens’ scales of measurement do not help make it: A response to Chalmers. Educational and Psychological Measurement, 79(6), 1184-1197. https://dx.doi.org/10.1177%2F0013164419844305

Published

2022-12-28

How to Cite

Vizioli, N. ., & Pagano, A. . (2022). From Alpha to Omega: Estimation of Ordinal Reliability. A Practical Guide. Costa Rican Journal of Psychology, 41(2), 119–136. https://doi.org/10.22544/rcps.v41i02.02