Does ChatGPT Measure Up to Discourse Unit Segmentation?A Comparative Analysis Utilizing Zero-Shot Custom Prompts
157
by Kota Shamanth Ramanath Nayak
Published onMay 27, 2024
Does ChatGPT Measure Up to Discourse Unit Segmentation?A Comparative Analysis Utilizing Zero-Shot Custom Prompts
·
Abstract
In recent years, research in natural language processing has been disrupted by the emergence of large language models (LLMs) that demonstrate remarkable capabilities across a number of linguistic tasks. However, the question of whether these models can be effectively harnessed for discourse unit segmentation remains mainly underexplored. This paper investigates the usability of ChatGPT for zero-shot discourse unit segmentation. To evaluate the LLM's performance, we developed differently framed prompts to instruct the LLM to perform discourse segmentation on the GUM-RST English data. Results show that although 83% of ChatGPT's answers follow the correct format, only 5% are actually correct. This shows that the model is not capable of reaching the performance of smaller models trained specifically for the task, as it is mostly hallucinating the answers.
Creative Commons Attribution 4.0 International License (CC-BY 4.0)