/FinBERT

BERT for Finance : UC Berkeley MIDS w266 Final Project

Primary LanguageC++

FinBERT: Pre-Trained on SEC Filings for Financial NLP Tasks

Vinicio DeSola, Kevin Hanna, Pri Nonis

MOTIVATIONS

Goal 1 FinBERT-Prime_128MSL-500K+512MSL-10K vs BERT

  • Compare mask LM prediction accurracy on technical financial sentences
  • Compare analogy on financial relationships

Goal 2 FinBERT-Prime_128MSL-500K vs FinBERT-Pre2K_128MSL-500K

  • Compare mask LM prediction accuracy on financial news from 2019
  • Compare analogy on financial relationship, measure shift in understanding : risk vs climate in 1999 vs 2019

Goal 3 FinBERT-Prime_128MSL-500K vs FinBERT-Prime_128MSK-500K+512MSL-10K

  • Compare mask LM prediction accuracy on long financial sentences

Goal 4 FinBERT-Combo_128MSL-250K vs FinBERT-Prime_128MSL-500K+512MSL-10K

  • Compare mask LM prediction accuracy on financial sentences : can we get same accuracy with less training by building on original BERT weights.

TERMINOLOGY

  • Prime Pre-trained from scratch on 2017, 2018, 2019 SEC 10K dataset

  • Pre2K Pre-traind from scratch on 1998, 1999 SEC 10K dataset

  • Combo Pre-trained continued from original BERT on 2017, 2018, 2019 SEC 10K dataset

ANALYSIS