The Impact of ChatGPT on Academic Writing Skills and Knowledge: An Investigation of Its Use in Argumentative Essays
Article Sidebar
Main Article Content
Abstract
This study investigates the impact of ChatGPT on EFL students' argumentative writing development of 30 third-year English majors. The research examines writing quality improvements, student perceptions, and patterns of AI tool usage across a 16-week period. This study employed a mixed-methods design to investigate both students' writing development and their perceptions of AI integration. Writing development was assessed through comparative analysis of participants' first and fifth drafts as pre-test and post-test assessments, while perceptions data were collected through a validated 17-item questionnaire (Cronbach's α = 0.89) focusing on confidence and concerns in AI-assisted writing. Essays were evaluated using an adapted AIAS framework focusing on six dimensions: content, organization, language use, critical thinking, AI integration, and academic integrity. The quantitative analysis revealed significant enhancements in writing performance, with the most substantial improvements in academic integrity (+3.0 points. Furthermore, the thematic analysis of the draft comparisons demonstrated marked progression in argument construction, evidence integration, and academic voice. This research provides insights for developing pedagogical frameworks that optimize AI integration while fostering independent writing competencies.
Article Details
References
Alasgarova, R., & Rzayev, J. (2024). The role of artificial intelligence in shaping high school students’ motivation. International Journal of Technology in Education and Science, 8(2), 311–324. https://doi.org/10.46328/ijtes.553
Alharbi, W. (2023). AI in the foreign language classroom: A pedagogical overview of automated writing assistance tools. Education Research International, 2023, 1–15. https://doi.org/10.1155/2023/4253331
AL-Smadi, M. (2023). ChatGPT and beyond: The generative AI revolution in education (No. arXiv:2311.15198). arXiv. http://arxiv.org/abs/2311.15198
Bandura, A. (2006). Toward a psychology of human agency. Perspectives on Psychological Science, 1. https://doi.org/10.1111/j.1745-6916.2006.00011.x
Banihashem, S. K., Kerman, N. T., Noroozi, O., Moon, J., & Drachsler, H. (2024). Feedback sources in essay writing: Peer-generated or AI-generated feedback? International Journal of Educational Technology in Higher Education, 21(1). Scopus. https://doi.org/10.1186/s41239-024-00455-4
Bereiter, C., & Scardamalia, M. (Eds.). (2013). The psychology of written composition. Routledge. https://doi.org/10.4324/9780203812310
Braun, V., & Clarke, V. (2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. https://doi.org/10.1080/2159676X.2019.1628806
Brown, H. D., & Lee, H. (2015). Teaching by principles: An interactive approach to language pedagogy (4th edition). Pearson Education ESL.
Chala, P., & Chapetón, C. (2012). EFL argumentative essay writing as a situated-social practice: Review of concepts. Folios, 1, 23–36. https://doi.org/10.17227/01234870.36folios23.36
Chapelle, C., & Voss, E. (2016). 20 years of technology and language assessment in language learning & technology. Language, Learning and Technology, 20, 116–128.
Cohen, L., Manion, L., & Morrison, K. (2017). Research methods in education. Routledge. https://www.routledge.com/Research-Methods-in-Education/Cohen-Manion-Morrison/p/book/9781138209886?srsltid=AfmBOopQDBI4j6xzqa0MOWcK1O3ZK0ZxDIbv9zX9hxphIQONb45b_gW9
Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. SAGE Publications.
Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE Publications.
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024). Impact of AI assistance on student agency. Computers & Education, 210, 104967. https://doi.org/10.1016/j.compedu.2023.104967
Dashti, M., Londono, J., Ghasemi, S., & Moghaddasi, N. (2023). How much can we rely on artificial intelligence chatbots such as the ChatGPT software program to assist with scientific writing? Journal of Prosthetic Dentistry. Scopus. https://doi.org/10.1016/j.prosdent.2023.05.023
Dörnyei, Z., & Taguchi, T. (2009). Questionnaires in second language research: Construction, administration, and processing: Second edition. Questionnaires in second language research: Construction, administration, and processing: Second Edition. https://doi.org/10.4324/9780203864739
Eltahir, M., & Babiker, F. (2024). The influence of artificial intelligence tools on student performance in e-learning environments: Case study. Electronic Journal of E-Learning, 22, 91–110. https://doi.org/10.34190/ejel.22.9.3639
Etikan, I. (2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5, 1. https://doi.org/10.11648/j.ajtas.20160501.11
Ferris, D. (2010). Second language writing research and written corrective feedback in SLA: Intersections and practical applications. Studies in Second Language Acquisition, 32, 181–201. https://doi.org/10.1017/S0272263109990490
Field, A. (2009). Discovering statistics using IBM SPSS Statistics: And sex and drugs and rock’n’roll.
Flanagin, A., Bibbins-Domingo, K., Berkwits, M., & Christiansen, S. L. (2023). Nonhuman “authors” and implications for the integrity of scientific publication and medical knowledge. JAMA, 329(8), 637–639. PubMed. https://doi.org/10.1001/jama.2023.1344
Ghanbari, N., & Salari, M. (2022). Problematizing argumentative writing in an Iranian EFL undergraduate context. Frontiers in Psychology, 13. https://doi.org/10.3389/fpsyg.2022.862400
Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve writing of adolescents in middle and high schools. A report to carnegie corporation of new york. The Alliance for Excellent Education.
Guo, K., Wang, J., & Chu, S. K. W. (2022). Using chatbots to scaffold EFL students’ argumentative writing. Assessing Writing, 54, 100666. https://doi.org/10.1016/j.asw.2022.100666
Horta, H., & Santos, J. (2015). The impact of publishing during Ph.D. studies on career research publication, visibility, and collaborations. Research in Higher Education, 57. https://doi.org/10.1007/s11162-015-9380-0
Hyland, K. (2003). Second language writing. Cambridge University Press. https://doi.org/10.1017/cbo9780511667251
Jia, X.-H., & Tu, J.-C. (2024). Towards a new conceptual model of AI-enhanced learning for college students: The roles of artificial intelligence capabilities, general self-efficacy, learning motivation, and critical thinking awareness. Systems, 12, 74. https://doi.org/10.3390/systems12030074
Jonassen, D., & Kim, B. (2010). Arguing to learn and learning to argue: Design justifications and guidelines. Educational Technology Research and Development, 58, 439–457. https://doi.org/10.1007/s11423-009-9143-8
Kim, Y.-S., Park, C., & Park, Y. (2015). Dimensions of discourse level oral language skills and their relation to reading comprehension and written composition: An exploratory study. Reading and Writing, 28. https://doi.org/10.1007/s11145-015-9542-7
Liu, H., & Fan, J. (2024). AI-mediated communication in EFL classrooms: The role of technical and pedagogical stimuli and the mediating effects of AI literacy and enjoyment. European Journal of Education. https://doi.org/10.1111/ejed.12813
Liu, M., Zhang, L. J., & Biebricher, C. (2024). Investigating students’ cognitive processes in generative AI-assisted digital multimodal composing and traditional writing. Computers and Education, 211. Scopus. https://doi.org/10.1016/j.compedu.2023.104977
Liu, T., & Taresh, S. (2024). Balancing athletic and academic excellence: A quantitative study of student-athletes’ time management strategies. Journal of Ecohumanism, 3, 4004–4022. https://doi.org/10.62754/joe.v3i7.4520
Livberber, T. (2023). Toward non-human-centered design: Designing an academic article with ChatGPT. Profesional de La Informacion, 32(5). Scopus. https://doi.org/10.3145/epi.2023.sep.12
Livberber, T., & Ayvaz, S. (2023). The impact of Artificial Intelligence in academia: Views of Turkish academics on ChatGPT. Heliyon, 9(9), e19688. https://doi.org/10.1016/j.heliyon.2023.e19688
Lund, B. D., & Wang, T. (2023). Chatting about ChatGPT: How may AI and GPT impact academia and libraries? Library Hi Tech News, 40(3), 26–29. https://doi.org/10.1108/LHTN-01-2023-0009
Mahapatra, S. (2024). Impact of ChatGPT on ESL students’ academic writing skills: A mixed methods intervention study. Smart Learning Environments, 11(1), 9. https://doi.org/10.1186/s40561-024-00295-9
Mahmood Khan, W. (2024). Analyzing the AI tools’ impact on critical thinking in BS English students at Pakistani Universities. Journal of Applied Linguistics and TESOL, 7(4).
Marikyan, D., & Papagiannidis, S. (2024). Technology acceptance model: A review. In TheoryHub Book. https://open.ncl.ac.uk/theory-library/technology-acceptance-model.pdf
Marzuki, Widiati, U., Rusdin, D., Darwin, & Indrawati, I. (2023). The impact of AI writing tools on the content and organization of students’ writing: EFL teachers’ perspective. Cogent Education, 10(2). https://doi.org/10.1080/2331186x.2023.2236469
Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook, 2nd ed. (pp. xiv, 338). Sage Publications, Inc.
Nguyen, H. (2024). Harnessing AI-based tools for enhancing English speaking proficiency: Impacts, challenges, and long-term engagement. International Journal of AI in Language Education, 1, 18–29. https://doi.org/10.54855/ijaile.24122
Nguyen, T., Lai, N., & Nguyen, Q. (2024). Artificial intelligence (AI) in education: A case study on ChatGPT’s influence on student learning behaviors. Educational Process International Journal, 13, 105–121. https://doi.org/10.22521/edupij.2024.132.7
Patty, J. (2024). Addressing student writing challenges: A review of difficulties and effective strategies. Education Journal: Journal Educational Research and Development, 8, 369–392. https://doi.org/10.31537/ej.v8i2.1938
Perkins, M. (2023). Academic integrity considerations of AI large language models in the post-pandemic era: ChatGPT and beyond. Journal of University Teaching and Learning Practice, 20(2). Scopus. https://doi.org/10.53761/1.20.02.07
Perkins, M., Furze, L., Roe, J., & MacVaugh, J. (2024). The Artificial Intelligence Assessment Scale (AIAS): A framework for ethical integration of generative AI in educational assessment. Journal of University Teaching and Learning Practice, 21. https://doi.org/10.53761/q3azde36
Saldana, J. (2013). The coding manual for qualitative researchers (2nd ed.). SAGE Publications Ltd. https://emotrab.ufba.br/wp-content/uploads/2020/09/Saldana-2013-TheCodingManualforQualitativeResearchers.pdf
Shi, H., & Aryadoust, V. (2024). A systematic review of AI-based automated written feedback research. ReCALL, 36(2), 187–209. Cambridge Core. https://doi.org/10.1017/S0958344023000265
Slimi, Z., Alawai, F. A., Alyani, H. A., Abri, S. A., Al-Farsi, F. A., & Balushi, K. A. (2022). Writing issues in ESL and their potential solutions: Case study IMCO’s foundation students. Journal of Educational and Social Research, 12(6), 81–93. Scopus. https://doi.org/10.36941/jesr-2022-0146
Srinawati, W., & Alwi, R. (2020). Critical thinking ability in EFL students’ argumentative essay writing: The difficulties and the strategies. Jurnal Serambi Ilmu, 21, 200–210. https://doi.org/10.32672/si.v21i2.2194
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., Warschauer, M., & Olson, C. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, 101894. https://doi.org/10.1016/j.learninstruc.2024.101894
Su, Y., Lin, Y., & Lai, C. (2023). Collaborating with ChatGPT in argumentative writing classrooms. Assessing Writing, 57, 100752. https://doi.org/10.1016/j.asw.2023.100752
Uzun, L. (2023). ChatGPT and academic integrity concerns: Detecting artificial intelligence generated content. Language Education and Technology, 3, 45–54.
Wei, L. (2023). Artificial intelligence in language instruction: Impact on English learning achievement, L2 motivation, and self-regulated learning. Frontiers in Psychology, 14. https://doi.org/10.3389/fpsyg.2023.1261955
Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28(11), 13943–13967. Scopus. https://doi.org/10.1007/s10639-023-11742-4
Zou, B., & Lertlit, S. (2022). Oxford’s strategy inventory for language learning: English learning of Chinese students in Thai University. Language Education and Acquisition Research Network, 15(2), 705–723.