Is Small Really Beautiful for Central Bank Communication? Evaluating Language Models for Finance: Llama-3-70B, GPT-4, FinBERT-FOMC, FinBERT, and VADER
Journal
Proceedings of the 5th ACM International Conference on AI in Finance
ISBN
979-8-4007-1081-0/24/11
Type
conference paper
Date Issued
2024-11-14
Author(s)
Abstract
This study compares the sentiment detection capabilities of language models for the domain of central bank communication, particularly the official statements released by the U.S. Federal Open Market Committee (FOMC). While previous studies have explored FOMC communication, this work is one of the few studies that use a natural language processing-based approach. The analysis employs VADER, FinBERT, a fine-tuned FinBERT model (FinBERT-FOMC),
GPT-4, and Llama-3-70B.
Within the scope of our labeled dataset on FOMC minutes, Llama 3 is the most accurate model, followed by GPT-4, FinBERT-FOMC, FinBERT, and VADER. The FinBERT-FOMC model, which was fine-tuned on central bank communication and utilizes a text simplification pipeline, performs better than the original FinBERT model. Llama 3 and GPT-4 outperform at the expense of large model sizes. Unlike GPT-4, FinBERT and FinBERT-FOMC are open-source and can be deployed on consumer-grade hardware. Llama 3 requires substantial hardware investments to deploy.
The work thus finds that there is still a trade-off between model size and
performance, and that the notion that “small is beautiful” can still
hold for use cases where maximum accuracy is a lesser concern
than inference speed and cost.
Human performance is still significantly above all models, indicating that further improvements in language models and FOMC-specific prompting are possible. The labeled dataset for central bank communication we present in this paper is thus a challenging benchmark for future research.
GPT-4, and Llama-3-70B.
Within the scope of our labeled dataset on FOMC minutes, Llama 3 is the most accurate model, followed by GPT-4, FinBERT-FOMC, FinBERT, and VADER. The FinBERT-FOMC model, which was fine-tuned on central bank communication and utilizes a text simplification pipeline, performs better than the original FinBERT model. Llama 3 and GPT-4 outperform at the expense of large model sizes. Unlike GPT-4, FinBERT and FinBERT-FOMC are open-source and can be deployed on consumer-grade hardware. Llama 3 requires substantial hardware investments to deploy.
The work thus finds that there is still a trade-off between model size and
performance, and that the notion that “small is beautiful” can still
hold for use cases where maximum accuracy is a lesser concern
than inference speed and cost.
Human performance is still significantly above all models, indicating that further improvements in language models and FOMC-specific prompting are possible. The labeled dataset for central bank communication we present in this paper is thus a challenging benchmark for future research.
Language
English (United States)
Keywords
Computing methodologies → Natural language processing
• Information systems → Sentiment analysis.
HSG Classification
contribution to scientific community
Refereed
Yes
Book title
ICAIF ’24, November 14–17, 2024, Brooklyn, NY, USA
Publisher
ACM
Start page
626
End page
633
Pages
8
Event Location
New York City
Event Date
2024-11-14 - 17
Subject(s)
Division(s)
Contact Email Address
siegfried.handschuh@unisg.ch
File(s)![Thumbnail Image]()
Name
2024-11-14 FOMC Kim et al. (2024) - Final ACM ICAIF Proceedings Version.pdf
Size
1.09 MB
Format
Adobe PDF
Checksum (MD5)
c1dee17009157166c871cc19487b3298