When Truth Matters - Addressing Pragmatic Categories in Natural Language Inference (NLI) by Large Language Models (LLMs)
Journal
Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)
Type
conference paper
Date Issued
2023-07
Author(s)
Abstract
In this paper, we focus on the ability of large language models (LLMs) to accommodate different pragmatic sentence types, such as questions, commands, as well as sentence fragments for natural language inference (NLI). On the commonly used notion of logical inference, nothing can be inferred from a question, a command, or an incomprehensible sentence fragment. We find MNLI, arguably the most important NLI dataset, and hence models fine-tuned on this dataset, insensitive to this fact. Using a symbolic semantic parser, we develop and make publicly available, fine-tuning datasets designed specifically to address this issue, with promising results. We also make a first exploration of ChatGPT's concept of entailment.
Refereed
Yes
Official URL
File(s)![Thumbnail Image]()
Name
published_2023.starsem-1.4.pdf
Size
406.48 KB
Format
Adobe PDF
Checksum (MD5)
6ba84c38f378e7af8e092d8568f12d47