v0.3.0: Add OOD eval, custom fine-tuning, multilingual sections to README

README.md

@@ -10,6 +10,7 @@ tags:
 - climate
 - esg
 - macroeconomics
+- multilingual
 datasets:
 - nickmuchi/financial-classification
 - zeroshot/twitter-financial-news-sentiment
@@ -21,6 +22,13 @@ metrics:
 - f1
 language:
 - en
+- ar
+- de
+- es
+- fr
+- hi
+- it
+- pt
 pipeline_tag: text-classification
 model-index:
 - name: macro-sentiment-finbert
@@ -41,181 +49,164 @@ model-index:
     - name: F1 Macro (FinBERT)
       type: f1
       value: 0.881
-    - name: Accuracy (ClimateBERT)
-      type: accuracy
-      value: 0.889
-    - name: F1 Macro (ClimateBERT)
-      type: f1
-      value: 0.872
 ---
 
 # 🏦 Macroeconomic Sentiment Analysis
 
-**What it does:** Takes any financial or economic text and tells you whether the sentiment is positive, negative, or neutral — plus whether the tone is hawkish/dovish, whether it signals a crisis, and how much it relates to climate/ESG.
+**What it does:** Takes any financial or economic text and tells you whether the sentiment is positive, negative, or neutral — plus whether the tone is hawkish/dovish, whether it signals a crisis, and how much it relates to climate/ESG. Now supports **multilingual text** (v0.3.0).
 
-**Who it's for:** Anyone analyzing news articles, central bank statements, earnings reports, financial tweets, or climate/ESG disclosures.
+**Who it's for:** Anyone analyzing news articles, central bank statements, earnings reports, financial tweets, or climate/ESG disclosures — in English or other languages.
 
 ---
 
 ## The Simple Way (3 lines of Python)
 
-If you just want to classify financial text as positive/negative/neutral, use the fine-tuned models directly. No cloning, no extra packages — just `pip install transformers torch` and go:
-
 ```python
 from transformers import pipeline
 
-# Pick one:
-classifier = pipeline("text-classification", model="peyterho/finbert-macro-sentiment")             # 110M, fast
-# classifier = pipeline("text-classification", model="peyterho/financial-roberta-large-macro-sentiment")  # 355M, most accurate
-# classifier = pipeline("text-classification", model="peyterho/climatebert-macro-sentiment")        # 82M, climate-focused
+classifier = pipeline("text-classification", model="peyterho/finbert-macro-sentiment")
 
-# Classify any financial text
 classifier("Tesla shares surged 15% after beating earnings expectations.")
 # → [{'label': 'positive', 'score': 0.998}]
 
 classifier("Markets crashed amid recession fears and massive layoffs.")
 # → [{'label': 'negative', 'score': 0.997}]
-
-classifier("The company reported quarterly revenue in line with expectations.")
-# → [{'label': 'neutral', 'score': 0.95}]
 ```
 
-That's it. Each model outputs one of three labels: **positive**, **negative**, or **neutral**, with a confidence score.
-
 ### Which model should I pick?
 
 | Model | Size | Speed | Accuracy | Best for |
 |-------|------|-------|----------|----------|
 | [`peyterho/finbert-macro-sentiment`](https://huggingface.co/peyterho/finbert-macro-sentiment) | 110M | Fast | 89.7% | General financial text — good default choice |
-| [`peyterho/financial-roberta-large-macro-sentiment`](https://huggingface.co/peyterho/financial-roberta-large-macro-sentiment) | 355M | Slower | **91.3%** | When accuracy matters most, policy/formal text |
-| [`peyterho/climatebert-macro-sentiment`](https://huggingface.co/peyterho/climatebert-macro-sentiment) | 82M | Fastest | 88.9% | Climate, ESG, and sustainability text |
+| [`peyterho/financial-roberta-large-macro-sentiment`](https://huggingface.co/peyterho/financial-roberta-large-macro-sentiment) | 355M | Slower | **91.3%** | When accuracy matters most |
+| [`peyterho/climatebert-macro-sentiment`](https://huggingface.co/peyterho/climatebert-macro-sentiment) | 82M | Fastest | 88.9% | Climate/ESG text |
+
+---
+
+## Multilingual Support (v0.3.0)
 
-### Classify a batch of texts
+The pipeline now auto-detects non-English text and routes it to an XLM-RoBERTa multilingual sentiment model.
+
+**Supported languages:** English, Arabic, French, German, Hindi, Italian, Portuguese, Spanish — plus reasonable performance on many others.
 
 ```python
-texts = [
-    "Fed raised rates by 75bps citing persistent inflation.",
-    "Renewable energy investments hit record highs.",
-    "Lehman Brothers filed for bankruptcy.",
-]
-results = classifier(texts)
-for text, result in zip(texts, results):
-    print(f"{result['label']:>8} ({result['score']:.2f})  {text}")
-# → neutral (0.91)  Fed raised rates by 75bps citing persistent inflation.
-#   positive (0.57)  Renewable energy investments hit record highs.
-#   negative (0.99)  Lehman Brothers filed for bankruptcy.
-```
+from macro_sentiment import TransformerEnsemble
 
----
+ensemble = TransformerEnsemble(device="cpu")
 
-## The Full Pipeline (more signals, more detail)
+# German — auto-routed to multilingual head
+result = ensemble.score_routed("EZB signalisiert Geduld bei Zinssenkungen.")
+print(result["head_used"])           # "multilingual"
+print(result["detected_language"])   # "de"
+print(result["sentiment_score"])     # [-1, +1]
 
-If you need more than just positive/negative/neutral — like **hawkish/dovish policy stance**, **crisis detection**, **climate exposure**, and **uncertainty scoring** — use the full pipeline. This combines three AI models with four financial dictionaries.
+# French
+result = ensemble.score_routed("La BCE maintient ses taux directeurs inchangés.")
+print(result["head_used"])           # "multilingual"
 
-### Setup
+# English — still routed to domain-specific heads as before
+result = ensemble.score_routed("Fed raised rates by 75bps.")
+print(result["head_used"])           # "policy" (RoBERTa-Large)
+```
 
+For better language detection accuracy, install `langdetect`:
 ```bash
-pip install transformers torch pysentiment2 scikit-learn numpy datasets
+pip install langdetect
+```
 
-# Download the pipeline code
-pip install huggingface_hub
-python -c "from huggingface_hub import snapshot_download; snapshot_download('peyterho/macro-sentiment-finbert', local_dir='macro-sentiment-finbert')"
-cd macro-sentiment-finbert
+To disable multilingual routing (v0.2.0 behavior):
+```python
+ensemble = TransformerEnsemble(multilingual_model=None)
 ```
 
-### Usage
+---
 
-```python
-from macro_sentiment import MacroSentimentPipeline
+## Fine-Tune on Your Own Data
 
-pipe = MacroSentimentPipeline(device="cpu")  # use "cuda:0" for GPU
+Got your own labelled financial text? Fine-tune any head in one command:
 
-result = pipe("The Federal Reserve raised rates by 75bps citing persistent inflation.")
-print(result.summary())
-# → Sentiment: Neutral | Policy: Very Hawkish (+0.999) | Crisis: Normal | Domain: policy
+```bash
+python -m macro_sentiment.finetune \
+    --data my_labels.csv \
+    --text-column headline \
+    --label-column sentiment \
+    --base-model peyterho/finbert-macro-sentiment \
+    --output my-org/my-custom-model \
+    --push-to-hub
 ```
 
-### What you get back
+**Your CSV/TSV/JSON/JSONL just needs two columns:**
 
-```python
-result.macro_sentiment   # -1.0 to +1.0  — overall sentiment
-result.policy_stance     # -1.0 (dovish) to +1.0 (hawkish)
-result.crisis_signal     # 0.0 (calm) to 1.0 (crisis)
-result.uncertainty       # 0.0 (certain) to 1.0 (uncertain)
-result.climate_exposure  # 0.0 (no climate topic) to 1.0 (climate-heavy)
-result.detected_domain   # "policy", "climate", "financial_news", or "social"
-result.confidence        # 0.0 to 1.0 — how confident the model is
-```
+| headline | sentiment |
+|----------|-----------|
+| Company profits soared to record highs | positive |
+| Stock prices crashed amid panic selling | negative |
+| Revenue remained flat quarter over quarter | neutral |
 
-### How it works
+Labels accept: `positive`/`negative`/`neutral` (or `bullish`/`bearish`, `pos`/`neg`, `risk`/`opportunity`, or integers `0`/`1`/`2`).
 
-When you feed in a text, the pipeline:
+**Options:**
+```
+--base-model   Which model to start from (default: peyterho/finbert-macro-sentiment)
+--epochs       Training epochs (default: 4)
+--lr           Learning rate (default: 2e-5)
+--batch-size   Batch size (default: 32)
+--max-length   Max token length (default: 128)
+--push-to-hub  Push result to Hugging Face Hub
+--output       Local dir or Hub model ID
+```
+
+The script auto-detects label remapping from the model's config, applies class weighting for imbalanced data, and selects the best checkpoint by macro F1.
 
-1. **Routes it** to the right AI model based on keywords (central bank language → RoBERTa, climate terms → ClimateBERT, everything else → FinBERT)
-2. **Scores it** with four financial dictionaries (Loughran-McDonald, Henry, climate exposure, macro policy)
-3. **Combines** both signals with crisis-adaptive weighting — when crisis terms are detected, dictionary signals get more weight (AI models can struggle with novel crisis patterns)
+---
 
-### Dictionary-only mode (instant, no GPU)
+## The Full Pipeline
 
-If you don't need AI models — just dictionary-based scoring:
+For hawkish/dovish stance, crisis detection, climate exposure, and uncertainty scoring:
 
-```python
-pipe = MacroSentimentPipeline(load_transformers=False)
-result = pipe("Markets crashed amid recession fears.", mode="dict_only")
-print(result.crisis_signal)    # 0.8 — high crisis
-print(result.policy_stance)    # -0.5 — dovish
+```bash
+pip install transformers torch pysentiment2 scikit-learn numpy datasets huggingface_hub
+python -c "from huggingface_hub import snapshot_download; snapshot_download('peyterho/macro-sentiment-finbert', local_dir='macro-sentiment-finbert')"
+cd macro-sentiment-finbert
 ```
 
----
+```python
+from macro_sentiment import MacroSentimentPipeline
 
-## Accuracy
+pipe = MacroSentimentPipeline(device="cpu")
 
-All models were trained on 20,000 financial texts from 5 datasets (news headlines, tweets, audit reports, financial Q&A, climate reports) and tested on 4,333 held-out samples.
+result = pipe("The Federal Reserve raised rates by 75bps citing persistent inflation.")
+print(result.summary())
+# → Sentiment: ... | Policy: Very Hawkish (+0.999) | Crisis: Normal | Domain: policy
 
-| Model | Accuracy | F1 (macro) | Negative recall | Positive recall |
-|-------|----------|------------|-----------------|-----------------|
-| RoBERTa-Large (fine-tuned) | **91.3%** | **0.902** | — | — |
-| FinBERT (fine-tuned) | 89.7% | 0.881 | 89% | 91% |
-| ClimateBERT (fine-tuned) | 88.9% | 0.872 | — | — |
-| *Previous version (dict-only meta-classifier)* | *66.9%* | *0.578* | *47%* | *35%* |
+result.macro_sentiment   # -1.0 to +1.0
+result.policy_stance     # -1.0 (dovish) to +1.0 (hawkish)
+result.crisis_signal     # 0.0 (calm) to 1.0 (crisis)
+result.uncertainty       # 0.0 (certain) to 1.0 (uncertain)
+result.climate_exposure  # 0.0 to 1.0
+```
 
 ---
 
-## Advanced: Customizing the Ensemble
+## Accuracy
 
-```python
-from macro_sentiment import TransformerEnsemble
+### In-domain (4,333 test samples from training datasets)
 
-# Default: uses all three fine-tuned models
-ensemble = TransformerEnsemble(device="cpu")
+| Model | Accuracy | F1 Macro |
+|-------|----------|----------|
+| RoBERTa-Large (fine-tuned) | **91.3%** | **0.902** |
+| FinBERT (fine-tuned) | 89.7% | 0.881 |
+| ClimateBERT (fine-tuned) | 88.9% | 0.872 |
 
-# Or pick specific models for each head
-ensemble = TransformerEnsemble(
-    finbert_model="peyterho/finbert-macro-sentiment",
-    roberta_model="peyterho/financial-roberta-large-macro-sentiment",
-    climatebert_model="peyterho/climatebert-macro-sentiment",
-)
-
-# Route automatically to the best head for the input
-result = ensemble.score_routed("ECB signals patience on rate cuts.")
-print(result["head_used"])        # "policy"
-print(result["sentiment_score"])  # [-1, +1]
-
-# Or score with all three heads at once
-result = ensemble.score_all("Markets tumbled on trade war fears.")
-print(result["ensemble_mean"])
-```
+### Out-of-domain (datasets NOT in training)
 
-To revert to the original off-the-shelf models:
-```python
-from macro_sentiment import ORIGINAL_FINBERT, ORIGINAL_ROBERTA, ORIGINAL_CLIMATEBERT
+| Model | Stock News (30K) | JB PhraseBank (785) |
+|-------|------------------|---------------------|
+| RoBERTa-Large | **72.1% acc / 0.727 F1** | **94.1% acc / 0.936 F1** |
+| FinBERT | 67.8% / 0.677 | 92.4% / 0.913 |
+| ClimateBERT | 64.7% / 0.644 | 92.5% / 0.921 |
 
-ensemble = TransformerEnsemble(
-    finbert_model=ORIGINAL_FINBERT,
-    roberta_model=ORIGINAL_ROBERTA,
-    climatebert_model=ORIGINAL_CLIMATEBERT,
-)
-```
+The Stock News OOD dataset ([ic-fspml/stock_news_sentiment](https://huggingface.co/datasets/ic-fspml/stock_news_sentiment)) uses 5-class labels mapped to 3-class, with different annotation conventions — the 20pp accuracy drop is expected and honest. The JB PhraseBank OOD dataset ([Jean-Baptiste/financial_news_sentiment_mixte_with_phrasebank_75](https://huggingface.co/datasets/Jean-Baptiste/financial_news_sentiment_mixte_with_phrasebank_75)) is a similar domain and holds up well.
 
 ---
 
@@ -223,51 +214,39 @@ ensemble = TransformerEnsemble(
 
 | Field | Range | What it means |
 |-------|-------|---------------|
-| `macro_sentiment` | -1 to +1 | Overall sentiment (negative ← 0 → positive) |
-| `policy_stance` | -1 to +1 | Dovish (rate cuts, easing) ← 0 → Hawkish (rate hikes, tightening) |
-| `financial_sentiment` | -1 to +1 | Raw financial sentiment from whichever AI model was used |
-| `climate_sentiment` | -1 to +1 | Climate risk ← 0 → Climate opportunity |
-| `crisis_signal` | 0 to 1 | 0 = calm, 1 = severe crisis language detected |
-| `uncertainty` | 0 to 1 | 0 = certain, 1 = highly uncertain |
-| `confidence` | 0 to 1 | How confident the pipeline is in its output |
-| `climate_exposure` | 0 to 1 | How much the text relates to climate/ESG topics |
-| `detected_domain` | text | `policy` / `climate` / `financial_news` / `social` |
-| `head_used` | text | Which AI model was activated |
-| `lm_polarity` | -1 to +1 | Loughran-McDonald dictionary score |
-| `henry_polarity` | -1 to +1 | Henry (2008) earnings tone score |
-
----
-
-## Training Data
-
-~20,000 training samples from 5 public datasets:
-
-| Dataset | Samples | What it covers |
-|---------|---------|---------------|
-| [Financial PhraseBank](https://huggingface.co/datasets/nickmuchi/financial-classification) | 5,057 | News headlines about companies |
-| [Twitter Financial News](https://huggingface.co/datasets/zeroshot/twitter-financial-news-sentiment) | 11,931 | Financial tweets with $cashtags |
-| [Auditor Sentiment](https://huggingface.co/datasets/FinanceInc/auditor_sentiment) | 4,846 | Audit and accounting reports |
-| [FiQA](https://huggingface.co/datasets/pauri32/fiqa-2018) | 1,213 | Financial questions and opinions |
-| [Climate Sentiment](https://huggingface.co/datasets/climatebert/climate_sentiment) | 1,320 | Climate and ESG disclosures |
+| `macro_sentiment` | -1 to +1 | Overall sentiment |
+| `policy_stance` | -1 to +1 | Dovish ← 0 → Hawkish |
+| `financial_sentiment` | -1 to +1 | Raw sentiment from active head |
+| `climate_sentiment` | -1 to +1 | Climate risk ← 0 → opportunity |
+| `crisis_signal` | 0 to 1 | Crisis intensity |
+| `uncertainty` | 0 to 1 | Uncertainty level |
+| `confidence` | 0 to 1 | Model confidence |
+| `climate_exposure` | 0 to 1 | Climate topic density |
+| `detected_domain` | text | policy / climate / financial_news / social |
+| `head_used` | text | Which model was activated |
 
 ## File Structure
 
 ```
 macro_sentiment/
-├── __init__.py                 # Package entry point
+├── __init__.py                 # Package entry point (v0.3.0)
 ├── dictionaries.py             # Four financial dictionaries
-├── transformers_ensemble.py    # Three AI model heads + topic router
+├── transformers_ensemble.py    # Four AI heads + language-aware router
 ├── pipeline.py                 # Full pipeline combining everything
+├── finetune.py                 # Custom fine-tuning CLI
 ├── data_prep.py                # Dataset loading
 └── train_meta.py               # Meta-classifier training
+eval_results.json               # In-domain metrics
+eval_ood_results.json           # Out-of-domain metrics
 ```
 
 ## References
 
-- Araci (2019). [FinBERT](https://arxiv.org/abs/1908.10063) — Financial Sentiment Analysis with Pre-Trained Language Models
-- Loughran & McDonald (2011). "When Is a Liability Not a Liability?" *Journal of Finance*
-- Henry (2008). "Are Investors Influenced by How Earnings Press Releases Are Written?" *Journal of Business Communication*
-- Sautner et al. (2023). "Firm-Level Climate Change Exposure." *Review of Financial Studies*
+- Araci (2019). [FinBERT](https://arxiv.org/abs/1908.10063)
+- Loughran & McDonald (2011). *Journal of Finance*
+- Henry (2008). *Journal of Business Communication*
+- Sautner et al. (2023). *Review of Financial Studies*
+- Barbieri et al. (2022). [XLM-T: Multilingual Language Models for Twitter](https://arxiv.org/abs/2104.12250)
 
 ## License
 Apache 2.0