Strengths and Limitations of SmallTalk2Me App in English Language Proficiency Evaluation

Laras Ayuningtyas Manggiasih; Yurike Risa Loreana; Abidatul Azizah; Nunung Nurjati

doi:10.30651/tell.v11i2.19560

Laras Ayuningtyas Manggiasih ⁽¹⁾, Yurike Risa Loreana ⁽²⁾, Abidatul Azizah ⁽³⁾, Nunung Nurjati ⁽⁴⁾

(1) PGRI Adibuana University, Indonesia,

(2) PGRI Adibuana University, Indonesia,

(3) PGRI Adibuana University,

(4) Graduate School, Universitas PGRI Adi Buana Surabaya

https://doi.org/10.30651/tell.v11i2.19560

Issue
Vol. 11 No. 2 (2023): September

Keywords:

SmallTalk2Me App, AI-driven Language Assessment, Strengths, Limitation

PDF

Abstract

This paper explores the strengths and limitations of the SmallTalk2Me App, an AI-driven language assessment tool, in evaluating English language proficiency. The study adopts a mixed-method approach, combining interviews with three experienced English teachers and a comprehensive literature review to provide a comprehensive analysis of the app's performance. The research begins with an exploration of the app's strengths, which include its objective and consistent evaluation metrics. The app's automated nature ensures that all test takers are assessed based on the same predefined criteria, reducing human bias and enhancing the reliability of evaluations. Also, it offers immediate feedback, allowing learners to identify their areas of improvement promptly and adapt their learning strategies accordingly. Conversely, the limitations of the SmallTalk2Me App are also discussed. One notable limitation is the challenge of replicating the complexity of real-life communication contexts. App-based assessments may not fully capture the intricacies of natural conversations. Additionally, the app's pronunciation assessment may struggle with accurately recognizing variations in accents and speech patterns, leading to potential inaccuracies in pronunciation evaluation. The insights from the interviews and literature review contribute to a comprehensive understanding of the app's performance, offering valuable implications for its effective use in language teaching and learning settings.

Full text article

Generated from XML file

References

Bajraktarevic, N., & Jang, J. (2019). Incorporating contextual aspects of language use in language assessment. TESOL Quarterly, 53(2), 496-503.

Brown, A. (2017). Personalized learning and artificial intelligence. Journal of Educational Technology, 45(2), 78–92.

Brown, A., & Hudson, R. (2021). Exploring the efficacy of AI-driven apps in assessing speaking proficiency. Educational Technology Research, 43(2), 145-163

Carless, D. (2009). Revisiting the quantitative-qualitative divide: Exploring the potential of mixed methods for language assessment research. Language Testing, 26(3), 327–350.

Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (2008). Building a validity argument for the test of English as a foreign language. New York: Routledge.

Chen, L. (2019). Addressing bias in artificial intelligence: the importance of diverse training data. Journal of AI Ethics, 3(1), 45-58

Dhara, Suvojit. Chatterjee, Sheshadari. Chaudhuri, Ranjan. Goswami, Adrijit. Kanti Ghosh, Soumya. (2022). Artificial intelligence in assessment of students' performance. 1–19. https://www.researchgate.net/publication/361861919_Artificial_Intelligence_in_Assessment_of_Students'_Performance

Fajri, M., & Indah, R. N. (2022). Students’ speaking problems in online learning: a systematic research review. Tell: Teaching of English Language and Literature Journal, 10(2), 99–111. https://doi.org/10.30651/tell.v10i2.13559

González-Calatayud, Victor. Prendes-Espinosa, Paz. Roig-Vila, R. (2021). Artificial intelligence for student assessment: a systematic review. https://www.mdpi.com/2076-3417/11/12/5467

Huhta, M., Vogt, K., Johnson, E., & Tulkki, H. (2013). Needs analysis for language course design: a holistic approach to ESP. Cambridge, UK: Cambridge University Press.

Jiang, Y., & Xiao, Y. (2022). App-based assessments might only partially account for cultural differences in communication styles. Language Testing, 39(1), 3-23. https://doi.org/10.1177/02655322211056312

Jones, S. (2016). Contextual understanding challenges in AI-based English language proficiency evaluation. International Journal of Applied Linguistics, 25(2), 89–104.

Kost, C. R., Ferguson, J. L., & Zlatev, J. J. (2019). The efficiency of automated evaluation in large-scale language assessments. Language Testing, 36(2), 209–231. https://doi.org/10.1177/0265532218789815

Lee, J. (2018). Limitations of AI in assessing higher-order skills in English language proficiency. Journal of Language Testing, 39(1), 56–72.

Lee, H., Warschauer, M., & Lee, J. H. (2019). The effects of corpus use on second language vocabulary learning: A multilevel meta-analysis. Applied Linguistics, 40(6), 1027-1052. https://doi.org/10.1093/applin/amz032

Lin, Peng, & Chen. (2019). Enhancing language assessment: the role of ai-integrated apps in evaluating speaking skills. Modern Language Education Journal, 28(1), 72-89

Li, Yang, & Wang. (2022). Assessing speaking ability through AI applications: a comprehensive review. Journal of Language Assessment, 15(3), 187–205.

Luoma, S. (2004). Assessing speaking. Cambridge University Press.

Lizasoain C, Andrea. Ortiz de Zárate F, Amalia. (2014). Alternative evaluation of a traditional oral skill assessment tool in an English teaching program. https://www.researchgate.net/publication/263660757

McNamara, T. F., & Roever, C. (2006). Language testing: The social dimension. Mouton de Gruyter.

Munirah, R. Refnaldi. (2020). An evaluation of assessment designed by English language education student teachers during teaching practice. https://ejournal.unp.ac.id/index.php/jelt

Nurdin, Hi. R., Zaim, M., & Refnaldi, R. (2019). Developing instruments for evaluating the implementation of authentic assessment for speaking skills at junior high school. Advances in Social Science, Education, and Humanities

Research, 276 (Icoelt, 2018), 106–111. https://doi.org/10.2991/icoelt-18.2019.17

Rintan, R., & Refnaldi, R. (2020). An evaluation of assessment designed by English language education student teachers during teaching practice. Journal of English Language Teaching and Linguistics, 5(2), 201-215. https://doi.org/10.24036/jelt.v9i1.108299

Seo, K., Tang, J., Roll, I., Fels, S., & Yoon, D. (2021). The impact of artificial intelligence on learners–instructor interaction in online learning. International Journal of Educational Technology in Higher Education, 18(1), 1–23. https://doi.org/10.1186/s41239-021-00292-9

Smith, T. (2015). Enhanced objectivity in an English language proficiency evaluation. Journal of Language Assessment, 28(3), 123–138.

Smith, A., & Jones, B. (2022). Challenges in AI-based language assessment tools for learners with non-standard accents or speech patterns. Language Testing, 39(3), 321–340. https://doi.org/10.1177/02655322211056312

Thompson, O., & Terkourafi. (2019). Beyond grammar and vocabulary: unveiling the potential of AI apps in assessing speaking competence. Applied Linguistics Review, 56(4), 301-318

Vogel, S., & Kasper, G. (2021). Challenges of assessing real-life communication skills in digital language assessment tools. Language Testing, 38(1), 3-23. https://doi.org/10.1177/0265532220967488

Weigle, S. C. (2014). Exploring multiple profiles of learner compositions. Journal of Second Language Writing.

Xu, J., & Recker, M. (2020). The importance of human interaction in AI-driven language assessment tools. Journal of Educational Technology, 45(3), 321-335.

Zhang, D., Cheng, L., & Wang, X. (2020). The application of AI chatbot-based evaluations in English language teaching. Journal of Educational Technology Development and Exchange, 13(1), 1-14. https://doi.org/10.11648/j.edu.20200501.11

Zhang, & Cheng. (2022). Challenges and opportunities of AI-driven speaking assessment applications in educational settings. Journal of Educational Technology and Applied Linguistics, 39(3), 208–225.

Authors

Laras Ayuningtyas Manggiasih

PGRI Adibuana University

larasayua@gmail.com (Primary Contact)

Yurike Risa Loreana

PGRI Adibuana University

Abidatul Azizah

PGRI Adibuana University

Nunung Nurjati

Graduate School, Universitas PGRI Adi Buana Surabaya

Manggiasih, L. A., Loreana, Y. R., Azizah, A., & Nurjati, N. (2023). Strengths and Limitations of SmallTalk2Me App in English Language Proficiency Evaluation. Tell : Teaching of English Language and Literature Journal, 11(2). https://doi.org/10.30651/tell.v11i2.19560

Download Citation

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Authors who publish with this journal agree to the following terms:

Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-NonCommercial 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work

Article Sidebar

Abstract

Full text article

References

Authors

Article Details