Education, Science, Technology, Innovation and Life
Open Access
Sign In
  1. Home
  2. Journals
  3. Journal of Language Testing & Assessment
  4. Vol 6, Issue 1, 2023

The Delivery of Speaking Tests in Traditional or Online Proctored Mode: A Comparability Study

Download as PDF

DOI: 10.23977/langta.2023.060101 | Downloads: 32 | Views: 637

Author(s)

Michael Milanovic 1, Tony Lee 1, David Coniam 1

Affiliation(s)

1 Language Cert, London, UK

Corresponding Author

David Coniam

ABSTRACT

This paper investigates the comparability of test scores recorded for high-stakes English language Speaking Tests administered face-to-face in either a traditional centre-based mode (TM) or in an online proctored mode (OLP). The data comprise a large sample of test takers taking English language Speaking Tests at four CEFR (the 'Common European Framework of Reference for Languages') levels – B1 to C2 – via TM or OLP. The data were analysed using descriptive statistics, effect size differences and equivalence tests. While a degree of difference in scores obtained between modes was apparent at C2 level, the differences were not found to be statistically significant. The paper concludes that whether Speaking Tests are delivered in online proctored mode or in traditional face-to-face mode, test takers receive similar scores. The study confirms that mode of test delivery does not significantly affect test taker scores.

KEYWORDS

Test score comparability, English language, Speaking tests, CEFR, online proctoring

CITE THIS PAPER

Michael Milanovic, Tony Lee, David Coniam, The Delivery of Speaking Tests in Traditional or Online Proctored Mode: A Comparability Study. Journal of Language Testing & Assessment (2023) Vol. 6: 1-10. DOI: http://dx.doi.org/10.23977/langta.2023.060101.

REFERENCES

[1] Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020).The difference between emergency remote teaching and online learning. EDUCAUSE Review.
[2] Lim, C. P., & Wang, L. (Eds.). (2016). Blended learning for quality higher education: Selected case studies on implementation from Asia-Pacific. Bangkok: UNESCO Bangkok Office.
[3] Todd, R. W. (2020). Teachers’ perceptions of the shift from the classroom to online teaching. International Journal of TESOL Studies, 2(2), 4-16.
[4] Coniam, D., Lampropoulou, L., & Cheilari, A. (2021). Online proctoring of high-stakes examinations: A survey of past test takers' attitudes and perceptions. English Language Teaching, 14(8), 58-72. doi.org/10.5539/elt.v14n8p58.
[5] Gardner, L. (2020). Covid-19 has forced higher ed to pivot to online learning. Here are 7 takeaways so far. The Chronicle of Higher Education, 20(5).
[6] Mays, T. J. (2021). Teaching the teachers. In Radical Solutions for Education in a Crisis Context (pp. 163-176). Springer, Singapore. doi.org/10.1007/978-981-15-7869-4_11.
[7] Sarrayrih, M. A., & Ilyas, M. (2013). Challenges of online exam, performances and problems for online university exam. International Journal of Computer Science Issues, 10(1), 439.
[8] Berrada, K., Ahmad, H. A. S., Margoum, S., EL Kharki, K., Machwate, S., Bendaoud, R., & Burgos, D. (2021). From the paper textbook to the online screen: A smart strategy to survive as an online learner. In Radical Solutions for Education in a Crisis Context (pp. 191-205). Singapore: Springer.
[9] Khan, R. A., & Jawaid, M. (2020). Technology enhanced assessment (TEA) in COVID 19 pandemic. Pakistan Journal of Medical Sciences, 36(19), 108-110. doi.org/10.12669/pjms.36.COVID19-S4.2795.
[10] García-Peñalvo, F. J., Corell, A., Abella-García, V., & Grande-de-Prado, M. (2021). Recommendations for mandatory online assessment in higher education during the COVID-19 pandemic. In Radical solutions for education in a crisis context (pp. 85-98). Singapore: Springer.
[11] Boekaerts, M., & Corno, L. (2005). Self‐regulation in the classroom: A perspective on assessment and intervention. Applied Psychology, 54(2), 199-231.
[12] Corrigan-Gibbs, H., Gupta, N., Northcutt, C., Cutrell, E., & Thies, W. (2015). Deterring cheating in online environments. ACM Transactions on Computer-Human Interaction, 22(6), 1-23. doi.org/10.1145/2810239.
[13] Foster, D., & Layman, H. (2013). Online proctoring systems compared. Webinar. http://www. slideshare.net/ caveonweb/caveon-webinar-series-online-proctoring-best-practicesoct-2013-slideshare-final.
[14] Watson, G., & Sottile, J. (2010). Cheating in the digital Age: Do students cheat more in on-line courses? Online Journal of Distance Learning Administration, 13(1).
[15] Rose, C. (2009). Virtual proctoring in distance education: An open-source solution. American Journal of Business Education, 2(2), 81-88. doi.org/10.19030/ajbe.v2i2.4039.
[16] Giller, P. (2021). E-proctoring in theory and practice: a review. Dublin, Ireland: Quality and Qualifications Ireland.
[17] Madsen, H. S. (1983). Techniques in testing. New York: Oxford University Press.
[18] Luoma, S. (2004). Assessing speaking. Cambridge University Press.
[19] Sujana, I. M. (2016). Assessing oral proficiency: Problems and suggestions for elicitation techniques. https://academia.edu.
[20] Fall, T., Adair‐Hauck, B., & Glisan, E. (2007). Assessing students' oral proficiency: A case for online testing. Foreign Language Annals, 40(3), 377-406. doi.org/10.1111/j.1944-9720.2007.tb02865.x.
[21] Ali, L., & Dmour, N. A. H. H. A. (2021). The shift to online assessment due to COVID-19: An empirical study of university students, behaviour and performance, in the region of UAE. International Journal of Information and Education Technology, 11(5), 220-228. doi.org/10.18178/ijiet. 2021.11.5.1515.
[22] Forrester, A. (2020). Addressing the challenges of group speaking assessments in the time of the Coronavirus. International Journal of TESOL Studies, 2(2), 74-88.
[23] Tippins, N. T. (2015). Technology and assessment in selection. Annual Review of Organizational Psychology and Organizational Behavior, 2, 551–582. https://doi.org/10.1146/annurev-orgpsych-031413-091317.
[24] Alessio, H. M., Malay, N., Maurer, K., Bailer, A. J., & Rubin, B. (2017). Examining the effect of proctoring on online test scores. Online Learning, 21(1), 146-161. doi.org/10.24059/olj.v21i1.885.
[25] Goedl, P. A., & Malla, G. B. (2020). A study of grade equivalency between proctored and unproctored exams in distance education. American Journal of Distance Education, 34(4), 280-289. doi.org/10.1080/08923647.2020.1796376.
[26] Reisenwitz, T. H. (2020). Examining the necessity of proctoring online exams. Journal of Higher Education Theory and Practice, 20(1), 118-124. doi.org/10.33423/jhetp.v20i1.2782.
[27] Castillo, M. S., & Doe, R. (2017). Mobile and nonmobile assessment in organizations: Does proctoring make a difference? Psychology, 8(06), 878. doi.org/10.4236/psych.2017.86057.
[28] Lee, J. W. (2020). Impact of proctoring environments on student performance: Online vs offline proctored exams. The Journal of Asian Finance, Economics, and Business, 7(8), 653-660. doi.org/10.13106/jafeb.2020.vol7.no8.653.
[29] Weiner, J. A., & Henderson, D. (2022). Online remote proctored delivery of high stakes tests: Issues and research. Journal of Applied Testing Technology, 23, 1-4.
[30] Weiner, J. A., & Hurtz, G. M. (2017). A comparative study of online remote proctored versus onsite proctored high-stakes exams. Journal of Applied Testing Technology, 18(1), 13-20.
[31] Hurtz, G. M., & Weiner, J. A. (2022). Comparability and integrity of online remote vs. onsite proctored credentialing exams. Journal of Applied Testing Technology, 23, 36-45.
[32] Wuthisatian, R. (2020). Student exam performance in different proctored environments: Evidence from an online economics course. International Review of Economics Education, 35, 100196. doi.org/10.1016/j.iree.2020.100196.
[33] Cherry, G., O'Leary, M., Naumenko, O., Kuan, L. A., & Waters, L. (2021). Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ? Computers and Education Open, 2, 100061. doi.org/10.1016/j.caeo.2021.100061.
[34] Morin, M., Alves, C., & De Champlain, A. (2021). The show must go on: Lessons learned from using remote proctoring in a high-stakes medical licensing exam program in response to severe disruption. Journal of Applied Testing Technology, 23, 15-35.
[35] Muckle, T. J., Meng, Y., & Johnson, S. (2022). A quantitative evaluation of a live remote proctoring pilot. Journal of Applied Testing Technology, 23, 46-53.
[36] Coniam, D. (2022). Online invigilation of English language examinations: A survey of past China test takers’ attitudes and perceptions. International Journal of TESOL Studies, 4(1), 21-31. doi.org/10.46451/ijts.2022.01.03.
[37] Papargyris, Y., & Yan, Z. (2022). Examiner quality and consistency across LanguageCert Writing Tests. International Journal of TESOL Studies, 4(1), 203-212. doi.org/10.46451/ijts.2022.01.13.
[38] Hayes, A. F., & Coutts, J. J. (2020). Use omega rather than Cronbach’s alpha for estimating reliability. But…. Communication Methods and Measures, 14(1), 1-24.
[39] Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social psychological and personality science, 8(4), 355-362. doi.org/10.1177/1948550617697177.
[40] Glen, S. (2021). Cohen's D: Definition, examples, formulas. https://www.statisticshowto.com/. 

All published work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright © 2016 - 2031 Clausius Scientific Press Inc. All Rights Reserved.