From Prompting to Proficiency: A Mixed-Methods Analysis of Prompting with ChatGPT Versus Lecturer Interaction in an EFL Classroom
Article Sidebar
Main Article Content
Abstract
The rapid advancement of artificial intelligence (AI) and extensive language models like ChatGPT have had profound implications for English as a Foreign Language (EFL) pedagogical practices. However, while its applications are widely explored, a notable gap persists in understanding the effectiveness of its core "prompting" feature for academic help-seeking compared to traditional lecturer interaction. This study aimed to address this gap by comparing the effectiveness of ChatGPT versus lecturers, exploring the advancement of student prompting strategies, and identifying associated challenges. A sequential explanatory mixed-methods design was employed over eight weeks with 60 Indonesian university students allocated to an experimental (ChatGPT) and a control (lecturer) group. Data collected via pre/post-tests (proficiency, writing, self-efficacy), interviews, and a Focus Group Discussion were analyzed using ANCOVA and thematic analysis. The quantitative findings conclusively demonstrated that the ChatGPT group significantly outperformed the control group in enhancing general English proficiency, writing competency, and self-efficacy (p < .001). Qualitatively, students’ prompting strategies evolved from simple, single-turn queries to sophisticated, multi-turn dialogic interactions, which were a key determinant of deeper learning. The most critical challenge identified was pedagogy, specifically the tendency for cognitive offloading alongside difficulties in vetting the AI’s accuracy and pervasive anxiety regarding academic integrity. These findings suggest that the effective use of Generative AI (GenAI) is not an innate skill but a learned competence, necessitating a pedagogical shift from merely providing technological access to the explicit instruction of AI literacy and strategic prompting to harness its full potential.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms: Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).References
An, R., Yang, Y., Yang, F., & Wang, S. (2023). Use prompt to differentiate text generated by ChatGPT and humans. Machine Learning with Applications, 14, Article 100497. https://doi.org/10.1016/j.mlwa.2023.100497
Bacon, E. D., & Kraus, H. (2025). Improving academic writing proficiency for EFL students: Leveraging ChatGPT using data-driven learning principles. REFLections, 32(1), 550–575. https://doi.org/10.61508/refl.v32i1.280410
Bai, L., Liu, X., & Su, J. (2023). ChatGPT: The cognitive effects on learning and memory. Brain‐X, 1(3), 1–9. https://doi.org/10.1002/brx2.30
Belda-Medina, J., & Kokošková, V. (2023). Integrating chatbots in education: Insights from the Chatbot-Human Interaction Satisfaction Model (CHISM). International Journal of Educational Technology in Higher Education, 20(1), 1–20. https://doi.org/10.1186/s41239-023-00432-3
Benson, P. (2011). Teaching and researching autonomy in language learning. Pearson Education.
Bowen, N. E. J. A., & Todd, R. W. (2025). Enhancing ChatGPT-based writing research through effective prompt use. Teaching English with Technology, 2025(1), 26–40. https://doi.org/10.56297/vaca6841/OZFH1876/SFKW4394
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Brinkmann, S., & Kvale, S. (2005). Confronting the ethics of qualitative research. Journal of Constructivist Psychology, 18(2), 157–181. https://doi.org/10.1080/10720530590914789
Brookhart, S. M. (2018). How to create and use rubrics for formative assessment and grading. ASCD.
Cain, W. (2024). Prompting change: Exploring prompt engineering in large language model AI and its potential to transform education. TechTrends, 68(1), 47–57. https://doi.org/10.1007/s11528-023-00896-0
Chen, C.-H., & Chang, C.-L. (2024). Effectiveness of AI-assisted game-based learning on science learning outcomes, intrinsic motivation, cognitive load, and learning behavior. Education and Information Technologies, 29, 18621-18642. https://doi.org/10.1007/s10639-024-12553-x
Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education. Routledge.
Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. Sage Publications.
Fan, L., & Cui, F. (2024). Mindfulness, self-efficacy, and self-regulation as predictors of psychological well-being in EFL learners. Frontiers in Psychology, 15, Article 1332002. https://doi.org/10.3389/fpsyg.2024.1332002
Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2025). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544
George, D., & Mallery, M. (2019). IBM SPSS statistics 26 step by step: A simple guide and reference. Routledge.
Gerlich, M. (2025). AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1), 1–28. https://doi.org/10.3390/soc15010006
Giray, L. (2023). Prompt engineering with ChatGPT: A guide for academic writers. Annals of Biomedical Engineering, 51(12), 2629–2633. https://doi.org/10.1007/s10439-023-03272-4
Hastomo, T., Sari, A. S., Widiati, U., Ivone, F. M., Zen, E. L., & Andianto, A. (2025). Exploring EFL teachers’ strategies in employing AI chatbots in writing instruction to enhance student engagement. World Journal of English Language, 15(7), 93–102. https://doi.org/10.5430/wjel.v15n7p93
Hwang, M., Lee, K.-H., & Lee, H.-K. (2025). A word to the wise: Crafting impactful prompts for ChatGPT. System, 133, Article 103756. https://doi.org/10.1016/j.system.2025.103756
Käser, T., & Schwartz, D. L. (2020). Modeling and analyzing inquiry strategies in open-ended learning environments. International Journal of Artificial Intelligence in Education, 30(3), 504–535. https://doi.org/10.1007/s40593-020-00199-y
Kerr, R. C., & Kim, H. (2025). From prompts to plans: A case study of pre-service EFL Teachers’ use of generative ai for lesson planning. English Teaching, 80(1), 95–118. https://doi.org/10.15858/engtea.80.1.202503.95
Kim, J., Yu, S., Lee, S.-S., & Detrick, R. (2025). Students’ prompt patterns and its effects in AI-assisted academic writing: Focusing on students’ level of AI literacy. Journal of Research on Technology in Education, 57, 1–18. https://doi.org/10.1080/15391523.2025.2456043
Knoth, N., Tolzin, A., Janson, A., & Leimeister, J. M. (2024). AI literacy and its implications for prompt engineering strategies. Computers and Education: Artificial Intelligence, 6, 100225. https://doi.org/10.1016/j.caeai.2024.100225
Krippendorff, K. (2018). Content analysis: An introduction to its methodology. Sage.
Krueger, R. A. (2014). Focus groups: A practical guide for applied research. Sage.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
Leung, C. H. (2024). Promoting optimal learning with ChatGPT: A comprehensive exploration of prompt engineering in education. Asian Journal of Contemporary Education, 8(2), 104–114. https://eric.ed.gov/?id=EJ1461939
Luther, T., Kimmerle, J., & Cress, U. (2024). Teaming up with an AI: Exploring human–AI collaboration in a writing scenario with ChatGPT. AI, 5(3), 1357–1376. https://doi.org/10.3390/ai5030065
Lynn, M. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–386. https://doi.org/10.1097/00006199-198611000-00017
Mahande, R. D., Fakhri, M. M., Suwahyu, I., & Sulaiman, D. R. A. (2025). Unveiling the impact of ChatGPT: Investigating self-efficacy, anxiety and motivation on student performance in blended learning environments. Journal of Applied Research in Higher Education. https://doi.org/10.1108/JARHE-07-2024-0372
Mohamed, A. M. (2024). Exploring the potential of an AI-based chatbot (ChatGPT) in enhancing English as a Foreign Language (EFL) teaching: Perceptions of EFL faculty members. Education and Information Technologies, 29(3), 3195–3217. https://doi.org/10.1007/s10639-023-11917-z
Mutanga, M. B., Msane, J., Mndaweni, T. N., Hlongwane, B. B., & Ngcobo, N. Z. (2025). Exploring the impact of LLM prompting on students’ learning. Trends in Higher Education, 4(3), 31. https://doi.org/10.3390/higheredu4030031
Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2, Article 100041. https://doi.org/10.1016/j.caeai.2021.100041
Nurchurifiani, E., Maximilian, A., Ajeng, G. D., Wiratno, P., Hastomo, T., & Wicaksono, A. (2025). Leveraging AI-powered tools in academic writing and research: Insights from English faculty members in Indonesia. International Journal of Information and Education Technology, 15(2), 312–322. https://doi.org/10.18178/ijiet.2025.15.2.2244
Oktarin, I. B., Saputri, M. E. E., Magdalena, B., Hastomo, T., & Maximilian, A. (2024). Leveraging ChatGPT to enhance students’ writing skills, engagement, and feedback literacy. Edelweiss Applied Science and Technology, 8(4), 2306–2319. https://doi.org/10.55214/25768484.v8i4.1600
Pallant, J. (2016). SPSS Survival Manual. Open University Press.
Perkins, M. (2023). Academic integrity considerations of AI large language models in the post-pandemic era: ChatGPT and beyond. Journal of University Teaching & Learning Practice, 20(2), Article 07. https://doi.org/10.53761/1.20.02.07
Sawalha, G., Taj, I., & Shoufan, A. (2024). Analyzing student prompts and their effect on ChatGPT’s performance. Cogent Education, 11(1). https://doi.org/10.1080/2331186X.2024.2397200
Slamet, J. (2024). Potential of ChatGPT as a digital language learning assistant: EFL teachers’ and students’ perceptions. Discover Artificial Intelligence, 4(1), Article 46. https://doi.org/10.1007/s44163-024-00143-2
van Woudenberg, R., Ranalli, C., & Bracker, D. (2024). Authorship and ChatGPT: A conservative view. Philosophy & Technology, 37(1), Article 34. https://doi.org/10.1007/s13347-024-00715-1
Vera, F. (2024). Student performance in writing prompts for text-based GenAI tools in a research methodology course. Revista Electrónica Transformar, 5(2), 71–75. https://revistatransformar.cl/index.php/transformar/article/view/129
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
Waziana, W., Andewi, W., Hastomo, T., & Hasbi, M. (2024). Students’ perceptions about the impact of AI chatbots on their vocabulary and grammar in EFL writing. Register Journal, 17(2), 328–362. https://doi.org/10.18326/register.v17i2.352-382
Woo, D. J., Guo, K., & Susanto, H. (2025). Exploring EFL students’ prompt engineering in human–AI story writing: An activity theory perspective. Interactive Learning Environments, 33(1), 863–882. https://doi.org/10.1080/10494820.2024.2361381
Yusuf, A., Pervin, N., & Román-González, M. (2024). Generative AI and the future of higher education: A threat to academic integrity or reformation? Evidence from multicultural perspectives. International Journal of Educational Technology in Higher Education, 21(1), Article 21. https://doi.org/10.1186/s41239-024-00453-6
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023). Why Johnny can’t prompt: How non-AI experts try (and fail) to design LLM prompts. In A. Schmidt, K. Vaananen, T. Goyal, P. O. Kristensson, A. Peter, s. Mueller, J. R. Williamson, & M. L. Wilson (Eds.), CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (pp. 1–21). Association for Computing Machinery. https://doi.org/10.1145/3544548.3581388
Zheng, Y., & Stewart, N. (2024). Improving EFL students’ cultural awareness: Reframing moral dilemmatic stories with ChatGPT. Computers and Education: Artificial Intelligence, 6, Article 100223. https://doi.org/10.1016/j.caeai.2024.100223
Zhu, S., Wang, Z., Zhuang, Y., Jiang, Y., Guo, M., Zhang, X., & Gao, Z. (2024). Exploring the impact of ChatGPT on art creation and collaboration: Benefits, challenges and ethical implications. Telematics and Informatics Reports, 14, Article 100138. https://doi.org/10.1016/j.teler.2024.100138
Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2), 64–70. https://doi.org/10.1207/s15430421tip4102_2
Zulianti, H., Hastuti, H., Nurchurifiani, E., Hastomo, T., Maximilian, A., & Ajeng, G. D. (2024). Enhancing novice EFL teachers’ competency in AI-powered tools through a TPACK-based professional development program. World Journal of English Language, 15(3), 117. https://doi.org/10.5430/wjel.v15n3p117
