Author(s)

Okim Kang ¹, Xun Yan ²

Affiliation(s)

¹ Northern Arizona University, Flagstaff, Arizona, USA
² Department of Linguistics, University of Illinois at Urbana-Champaign, Urbana, USA

Corresponding Author

Okim Kang

ABSTRACT

A high-stakes speaking test needs to reflect a view of speaking ability that involves multiple competences by sampling features of language use (Davies, 2008). The current study examines linguistic features that distinguish examinee performance across Common European Framework of References (CEFR) levels in the Cambridge English Language Assessment. Using a quantitative/corpus-based approach, 1-minute long, mono-logic speech files of 106 candidates, at each of the CEFR levels were analyzed in various linguistic features. Dimension scores were subjected to correlational and MANOVA analyses. The findings suggest that there are distinctive differences in more linguistic dimensions between high and low CEFR speaking levels than between the adjacent levels. They also offer implications for the validation of the scoring criteria, and improvement of rater development and language pedagogy.

KEYWORDS

Speaking Assessment, Proficiency Level, Linguistic Analysis

CITE THIS PAPER

Okim, K., Xun, Y., Linguistic Features Distinguishing Examinees' Speaking Performances at Different Proficiency Levels. Journal of Language Testing & Assessment (2018) Vol. 1: 24-39.

REFERENCES

[1] American Council on the Teaching of Foreign Languages. (2012). ACTFL Proficiency Guidelines—Speaking. Retrieved October 12, 2012, from http://www.actfl.org/sites/default/files/pdfs/public/ACTFLProficiencyGuidelines2012_FINAL.pdf
[2] Anderson-Hsieh, J., Johnson, K., & Koehler, K. (1992). The relationship between native speaker judgments of non-native pronunciation and deviance in segmentals, prosody, and syllablestructure. Language Learning, 42, 529-555.
[3] Bachman, L.F. (1990). Fundamental considerations in language testing. Oxford, UK: Oxford University Press.
[4] Bailey, K. M. (2005). Practical English language teaching: Speaking. New York, NY: McGraw-Hill. 
[5] Biber, D., Johansson, S., Leech, G., Conrad, S., & Finnegan, E. (1999). Longman Grammar of Spoken and Written English. Essex: Pearson Education Limited. Educational Testing Service.
[6] Boersma, P., &Weenink, D. (2007). Praat, http://www.praat.org (Version 4.5.25).
[7] Brazil, D. (1997). The communicative value of intonation in English. Cambridge, UK: Cambridge University Press.
[8] Brown, A. (2006). Candidate discourse in the revised IELTS Speaking Test. In P. McGovern & S. Walsh (Eds.), IELTS research reports 2006 (pp.71-89). Canberra & Manchester: IELTS Australia and British Council.
[9] Brown, A., Iwashita, N., & McNamara, T. (2005). An examination of rater orientations and test taker performance on English-for-Academic-Purpose speaking tasks (TOEFL Monograph No. 29). Princeton, NJ: Educational Testing Service.
[10] Carroll, B. J. (1980). Testing communicative performance. Oxford, UK: Pergamon Press. 
[11] Cobb, T. (2002). The Web Vocabulary Profile. http://www.er.uqam.ca/nobel/r21270/textools/web_vp.html
[12] Davies, A. (2008). Assessing academic English: Testing English proficiency 1950-1989 – the IELTS solution, Studies in Language Testing 23, UCLES/Cambridge University Press, Cambridge
[13] de Jong, N.H., Steinel, M.P., Florijn, A.F., Schoonen, R. & Hulstijn, J. H. (2012). Facets of speaking proficiency. Studies in Second Language Acquisition, 34, 5-34.
[14] de Jong, N., Groenhout, R., Schoonen, R., and Hulstijn, J. H. (2012). L2 fluency: speaking style or proficiency? Correcting measures of L2 fluency for L1behavior, Applied Psycholinguistics. Retrieved from http://www.academia.edu/2240591/L2_fluency_speaking_style_or_proficiency_Correcting_measures_of_L2_fluency_for_L1_behavior.
[15] Douglas, D., & Selinker, L. (1993). Performance on a general versus a field-specific test of speaking proficiency by international teaching assistants. In D. Douglas, & C. Chapelle (Eds), A new decade of language testing research (pp. 235-256). Alexandria, VA: TESOL.
[16] Douglas, D., & Smith, J. (1997). Theoretical underpinnings of the Test of Spoken English revision project. Princeton, NJ: Educational Testing Service.
[17] Espada-Gustilo., L. (2011). Linguistic features that impact essay scores: A corpus linguistic analysis of ESL writing in three proficiency levels. The Southeast Asian Journal of English Language Studies, 17 (1), 55-64.
[18] Galaczi, E.D., & ffrench, A. (2011): Context validity of Cambridge ESOL speaking tests. In L. Taylor (Ed.), Examining speaking (Vol. 30). Cambridge, UK: Cambridge University Press.
[19] Galaczi, E. D., Post, B., Li, A., & Graham, C. (2011). Measuring L2 English phonological proficiency: Implications for language assessment, in Proceedings of the British Association of Applied Linguistics, 67-72.
[20] Ginther, A., Dimova, S., & Yang, R. (2010). Conceptual and empirical relationships between temporal measures of fluency and oral English proficiency with implications for automated scoring. Language Testing, 27(3), 379-399.
[21] Grant, L., & Ginther, A. (2000). Using computer-tagged linguistic features to describe L2 writing differences. Journal of Second Language Writing, 9 (2), 123-145.
[22] Hawkins, J. A., & Filipović, L. (2012). Criterial Features in L2 English: Specifying the reference levels of the Common European Framework, English Profile Studies volume 1, Cambridge: UCLES/Cambridge University Press.
[23] Hahn, L. D. (2004). Primary stress and intelligibility: Research to motivate the teaching of suprasegmentals. TESOL Quarterly, 38, 201–223.
[24] Hewings, M. (1995). Tone choice in the English intonation of non-native speakers. The International Review of Applied Linguistics, 33, 251-265.
[25] Hinkel, E. (2003). Simplicity without elegance: Features of sentences in L1 and L2 academic texts. TESOL Quarterly, 37(2), 275-301.
[26] International English Language Testing System (2012). IELTS Speaking band descriptors (public version). Retrieved October 15, 2012 from https://www.teachers.cambridgeesol.org/ts/digitalAssets/114292_IELTS_Speaking_Band_Descriptors.pdf
[27] Isaacs, T. (2008). Towards defining a valid assessment criterion of pronunciation proficiency in non-native English-speaking graduate students. The Canadian Modern Language Review, 64, 555–580.
[28] Iwashita, N., Brown, A., McNamara, T., & O’Hagan, S. (2008). Assessed levels of second language speaking proficiency: How distinct? Applied Linguistics, 29 (1), 24-49.
[29] Jamieson, J., & Poonpon, K. (2013). Developing analytic scoring guides for TOEFL iBT’s Speaking Measure. TOEFL Monograph Series. RR-13-13.
[30] Jin, T., & Mak, B. (2013). Distinguishing features in scoring L2 Chinese speaking performance: How do they work? Language Testing, 30 (1), 23-47.
[31] Jin, T., Mak, B., & Zhou, P. (2012). Confidence scoring of speaking performance: How does fuzziness become exact? Language Testing, 29(1), 43–65.
[32] Kang, O. (2010). Relative salience of suprasegmental features on judgments of L2 comprehensibility and accentedness. System, 38(2), 301-315.
[33] Kang, O., & Johnson, D. (2018). The roles of suprasegmental features in predicting English oral proficiency with an automated system. Language Assessment Quarterly, 15 (2), 150-168.
[34] Kang, O., & Moran, M. (2014). Functional loads of pronunciation features in non-native speakers' oral assessment. TESOL Quarterly , 48(1), 176-187.
[35] Kang, O., Rubin, D., Pickering, L. (2010). Suprasegmental measures of accentedness and judgments of language learner proficiency in oral English. Modern Language Journal, 94, 554-566.
[36] Kormos, J., & Dénes, M. (2004). Exploring measures and perceptions of fluency in the speech of second language learners. System, 32, 145–164.
[37] Lee, J-W., & Schallert, D. L. (1997). The relative contribution of L2 language proficiency and L1 reading ability to L2 reading performance: A test of the threshold hypothesis in an EFL context. TESOL Quarterly, 31, 713-739.
[38] Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal, 96 (2), 190-208.
[39] Malvern, D., & Richards, B. (2002). Investigating accommodation in language proficiency interviews using a new measure of lexical diversity. Language Testing, 19(1), 85–104.
[40] Munro, M. J., &Derwing, T. M. (2001). Modelling perceptions of the accentedness and comprehensibility of L2 speech: The role of speaking rate. Studies of Second Language Acquisition, 23, 451–468.
[41] Nakatsuhara, F. (2006). The impact of proficiency-level on conversational styles in paired speaking tests. Research Notes, 25, 15-20.
[42] Pearson. (2011). Versant English Test: Test description and validation summary. Pearson Knowledge Technologies, Palo Alto, CA. Retrieved from http://www.versanttest.com/technology/VersantEnglishTestValidation.pdf
[43] Pickering, L. (2001). The role of tone choice in improving ITA communication in the classroom. TESOL Quarterly 35, 233-255.
[44] Skehan, P., &Foster, P. (1999). The influence of task structure and processing conditions on narrative retellings. Language Learning, 49(1), 93-120.
[45] Staples, S., Laflair, G. T., & Egbert, J. (2017). Comparing language use in oral proficiency interviews to target domains: Conversational, academic, and professional discourse. The Modern Language Journal, 101(1), 194-213.
[46] Sun, Y. (2011). The influence of the social interactional context on test performance: A sociocultural view. The Canadian Journal of Applied Linguistics, 14 (1), 194-221.
[47] Syakur. (1987). Language Testing and Evaluation. Surakarta: SebelasMaret University Press
[48] Taylor, L. (2001). Revising the IELTS Speaking test: Developments in test format and task design. Research Notes, 5, 3-5.
[49] Taylor, L. (2011). Examining speaking: Research and practice in assessing second language speaking. Cambridge: Cambridge University Press. 
[50] UCLES (2011). Cambridge English: Proficiency specifications and sample papers. Retrieved October 16, 2012 fromhttp://www.cambridgeesol.org/assets/pdf/exams/cpe/cpe-spec-2013.pdf
[51] University of Cambridge ESOL Examination (2011). Using the CEFR: Principles of good practice. Retrieved from http://www.cambridgeenglish.org/research-and-validation/fitness-for-purpose/
[52] Wennerstrom, A. (1994). Intonational meaning in English discourse: A study of nonnative speakers. Applied Linguistics, 15, 399-421.
[53] Wennerstrom, A.(2000). The role of intonation in second language fluency. In Riggenbach, H. (Ed.), Perspectives on Fluency (pp.102-127). University of Michigan Press, Ann Arbor. 
[54] Yu, G. (2010). Lexical diversity in writing and speaking task performances. Applied Linguistics, 31(2), 236-259.

Sponsors, Associates, and Links

---

• Information and Knowledge Management
  
  
• Military and Armament Science
  
  
• Media and Communication Research
  
  
• Journal of Human Movement Science
  
  
• Art and Performance Letters
  
  
• Lecture Notes on History
  
  
• Lecture Notes on Language and Literature
  
  
• Philosophy Journal
  
  
• Science of Law Journal
  
  
• Journal of Political Science Research
  
  
• Journal of Sociology and Ethnology
  
  
• Advances in Broadcasting