Presentation Leveraging ChatGPT and Grammarly for Automated Text Analysis: A Validation Study more

Text analysis plays a crucial role in understanding language proficiency. Evaluating the complexity, accuracy, and fluency (CAF) of texts, for example, is a common method to assess and compare language learners' proficiency under various conditions. Traditionally, such analyses have required many hours of tedious manual labor on the part of researchers and their assistants. This study investigates the validity and reliability of incorporating the free online tools ChatGPT and Grammarly to fully or partially automate these tasks. The study involved 55 students at a national university in Japan and analyzed one-paragraph summaries they wrote of a textbook listening script. Strong, significant correlations emerged between human and AI-derived measures for syntactic accuracy and syntactic complexity, as well as content comprehensiveness. These results support the efficacy of AI-based technologies for text analyses that have hitherto required human raters. In addition to the results and their implications, the presentation will detail the methods utilized for the human and AI-based measures, enabling attendees to obtain similar results. The findings indicate a widening role for AI in language research and education.