v0.3.0: Add OOD eval, custom fine-tuning, multilingual sections to README

522f497 verified 3 months ago

8.67 kB

library_name: transformers
license: apache-2.0
tags:
  - text-classification
  - sentiment-analysis
  - finance
  - finbert
  - roberta
  - climate
  - esg
  - macroeconomics
  - multilingual
datasets:
  - nickmuchi/financial-classification
  - zeroshot/twitter-financial-news-sentiment
  - FinanceInc/auditor_sentiment
  - pauri32/fiqa-2018
  - climatebert/climate_sentiment
metrics:
  - accuracy
  - f1
language:
  - en
  - ar
  - de
  - es
  - fr
  - hi
  - it
  - pt
pipeline_tag: text-classification
model-index:
  - name: macro-sentiment-finbert
    results:
      - task:
          type: text-classification
          name: Financial Sentiment Analysis
        metrics:
          - name: Accuracy (RoBERTa-Large)
            type: accuracy
            value: 0.913
          - name: F1 Macro (RoBERTa-Large)
            type: f1
            value: 0.902
          - name: Accuracy (FinBERT)
            type: accuracy
            value: 0.897
          - name: F1 Macro (FinBERT)
            type: f1
            value: 0.881

🏦 Macroeconomic Sentiment Analysis

What it does: Takes any financial or economic text and tells you whether the sentiment is positive, negative, or neutral — plus whether the tone is hawkish/dovish, whether it signals a crisis, and how much it relates to climate/ESG. Now supports multilingual text (v0.3.0).

Who it's for: Anyone analyzing news articles, central bank statements, earnings reports, financial tweets, or climate/ESG disclosures — in English or other languages.

The Simple Way (3 lines of Python)

from transformers import pipeline

classifier = pipeline("text-classification", model="peyterho/finbert-macro-sentiment")

classifier("Tesla shares surged 15% after beating earnings expectations.")
# → [{'label': 'positive', 'score': 0.998}]

classifier("Markets crashed amid recession fears and massive layoffs.")
# → [{'label': 'negative', 'score': 0.997}]

Which model should I pick?

Model	Size	Speed	Accuracy	Best for
`peyterho/finbert-macro-sentiment`	110M	Fast	89.7%	General financial text — good default choice
`peyterho/financial-roberta-large-macro-sentiment`	355M	Slower	91.3%	When accuracy matters most
`peyterho/climatebert-macro-sentiment`	82M	Fastest	88.9%	Climate/ESG text

Multilingual Support (v0.3.0)

The pipeline now auto-detects non-English text and routes it to an XLM-RoBERTa multilingual sentiment model.

Supported languages: English, Arabic, French, German, Hindi, Italian, Portuguese, Spanish — plus reasonable performance on many others.

from macro_sentiment import TransformerEnsemble

ensemble = TransformerEnsemble(device="cpu")

# German — auto-routed to multilingual head
result = ensemble.score_routed("EZB signalisiert Geduld bei Zinssenkungen.")
print(result["head_used"])           # "multilingual"
print(result["detected_language"])   # "de"
print(result["sentiment_score"])     # [-1, +1]

# French
result = ensemble.score_routed("La BCE maintient ses taux directeurs inchangés.")
print(result["head_used"])           # "multilingual"

# English — still routed to domain-specific heads as before
result = ensemble.score_routed("Fed raised rates by 75bps.")
print(result["head_used"])           # "policy" (RoBERTa-Large)

For better language detection accuracy, install langdetect:

pip install langdetect

To disable multilingual routing (v0.2.0 behavior):

ensemble = TransformerEnsemble(multilingual_model=None)

Fine-Tune on Your Own Data

Got your own labelled financial text? Fine-tune any head in one command:

python -m macro_sentiment.finetune \
    --data my_labels.csv \
    --text-column headline \
    --label-column sentiment \
    --base-model peyterho/finbert-macro-sentiment \
    --output my-org/my-custom-model \
    --push-to-hub

Your CSV/TSV/JSON/JSONL just needs two columns:

headline	sentiment
Company profits soared to record highs	positive
Stock prices crashed amid panic selling	negative
Revenue remained flat quarter over quarter	neutral

Labels accept: positive/negative/neutral (or bullish/bearish, pos/neg, risk/opportunity, or integers 0/1/2).

Options:

--base-model   Which model to start from (default: peyterho/finbert-macro-sentiment)
--epochs       Training epochs (default: 4)
--lr           Learning rate (default: 2e-5)
--batch-size   Batch size (default: 32)
--max-length   Max token length (default: 128)
--push-to-hub  Push result to Hugging Face Hub
--output       Local dir or Hub model ID

The script auto-detects label remapping from the model's config, applies class weighting for imbalanced data, and selects the best checkpoint by macro F1.

The Full Pipeline

For hawkish/dovish stance, crisis detection, climate exposure, and uncertainty scoring:

pip install transformers torch pysentiment2 scikit-learn numpy datasets huggingface_hub
python -c "from huggingface_hub import snapshot_download; snapshot_download('peyterho/macro-sentiment-finbert', local_dir='macro-sentiment-finbert')"
cd macro-sentiment-finbert

from macro_sentiment import MacroSentimentPipeline

pipe = MacroSentimentPipeline(device="cpu")

result = pipe("The Federal Reserve raised rates by 75bps citing persistent inflation.")
print(result.summary())
# → Sentiment: ... | Policy: Very Hawkish (+0.999) | Crisis: Normal | Domain: policy

result.macro_sentiment   # -1.0 to +1.0
result.policy_stance     # -1.0 (dovish) to +1.0 (hawkish)
result.crisis_signal     # 0.0 (calm) to 1.0 (crisis)
result.uncertainty       # 0.0 (certain) to 1.0 (uncertain)
result.climate_exposure  # 0.0 to 1.0

Accuracy

In-domain (4,333 test samples from training datasets)

Model	Accuracy	F1 Macro
RoBERTa-Large (fine-tuned)	91.3%	0.902
FinBERT (fine-tuned)	89.7%	0.881
ClimateBERT (fine-tuned)	88.9%	0.872

Out-of-domain (datasets NOT in training)

Model	Stock News (30K)	JB PhraseBank (785)
RoBERTa-Large	72.1% acc / 0.727 F1	94.1% acc / 0.936 F1
FinBERT	67.8% / 0.677	92.4% / 0.913
ClimateBERT	64.7% / 0.644	92.5% / 0.921

The Stock News OOD dataset (ic-fspml/stock_news_sentiment) uses 5-class labels mapped to 3-class, with different annotation conventions — the 20pp accuracy drop is expected and honest. The JB PhraseBank OOD dataset (Jean-Baptiste/financial_news_sentiment_mixte_with_phrasebank_75) is a similar domain and holds up well.

All Output Fields

Field	Range	What it means
`macro_sentiment`	-1 to +1	Overall sentiment
`policy_stance`	-1 to +1	Dovish ← 0 → Hawkish
`financial_sentiment`	-1 to +1	Raw sentiment from active head
`climate_sentiment`	-1 to +1	Climate risk ← 0 → opportunity
`crisis_signal`	0 to 1	Crisis intensity
`uncertainty`	0 to 1	Uncertainty level
`confidence`	0 to 1	Model confidence
`climate_exposure`	0 to 1	Climate topic density
`detected_domain`	text	policy / climate / financial_news / social
`head_used`	text	Which model was activated

File Structure

macro_sentiment/
├── __init__.py                 # Package entry point (v0.3.0)
├── dictionaries.py             # Four financial dictionaries
├── transformers_ensemble.py    # Four AI heads + language-aware router
├── pipeline.py                 # Full pipeline combining everything
├── finetune.py                 # Custom fine-tuning CLI
├── data_prep.py                # Dataset loading
└── train_meta.py               # Meta-classifier training
eval_results.json               # In-domain metrics
eval_ood_results.json           # Out-of-domain metrics

References

Araci (2019). FinBERT
Loughran & McDonald (2011). Journal of Finance
Henry (2008). Journal of Business Communication
Sautner et al. (2023). Review of Financial Studies
Barbieri et al. (2022). XLM-T: Multilingual Language Models for Twitter

License

Apache 2.0