sentiment-fiction-seq

A RoBERTa-large model finetuned for 3-class sentiment classification (negative / neutral / positive) on literary and fictional text, with complete narrative sequences held out from training to enable evaluation of detrended sentiment arcs.

This is a variant of fpianz/sentiment-fiction. The two models share the same architecture, base model, and training procedure. They differ only in their training splits: this model excludes complete sequential texts (three Andersen fairy tales and the final section of Hemingway's The Old Man and the Sea) to allow uncontaminated evaluation of narrative arc dynamics. Users should validate both models on their own data to determine which best fits their use case.

Model description

This model is a finetuned version of j-hartmann/sentiment-roberta-large-english-3-classes (RoBERTa-large, 355M parameters). It was trained on a combined corpus of human-annotated fiction sentences using class-weighted cross-entropy loss to handle label imbalance.

Training data

Only human-annotated texts. Compared to sentiment-fiction, this model excludes all Andersen fairy tale sentences and 400 contiguous Hemingway sentences from training.

Source	n (train)	Label type
Project Gutenberg and Wattpad excerpts	6,646	Nine emotions labels → binned to 3 classes
EmoBank Fiction (American National Corpus)	2,164	Continuous valence → binned to 3 classes
Fiction4 Hymns (translated from Danish)	1,620	Continuous valence → binned to 3 classes
Fiction4 Poetry (Plath)	1,263	Continuous valence → binned to 3 classes
Hemingway — The Old Man and the Sea (first 1,236 sentences)	1,236	Continuous 1–10 valence → binned to 3 classes
Total	12,929

Continuous valence scores were binned using the thresholds: ≤4 → negative, (4, 6] → neutral, >6 → positive on a 0–10 scale.

Intended use

This model is intended for research on literary sentiment, narrative emotion arcs, and computational literary studies. It can be used for:

• Sentence-level sentiment classification of fiction and literary prose
• Generating continuous sentiment arcs by converting class probabilities to a valence score: valence = p(positive) - p(negative)
• Studying detrended sentiment dynamics in sequential narrative text

Evaluation

Sentence-level (raw) correlation

Spearman ρ between model-predicted continuous valence and human annotations, on sequential held-out texts. Continuous valence for correlation is computed as p(positive) − p(negative) from the model's softmax probabilities, yielding a score in approximately [−1, +1] rather than a discrete class label. Accuracy is computed on the 3-class prediction (argmax over negative/neutral/positive) against human valence binned with the same thresholds used for training (≤4 → negative, (4, 6] → neutral, >6 → positive). Note that literary texts are heavily neutral-skewed, where always predicting "neutral" would do better. For this reason, the continuous valence correlation (Spearman ρ) is the more meaningful metric here.

Eval set	n	Spearman ρ (Tr)	Spearman ρ (Sy)	Accuracy	Majority Baseline
Hemingway — The Old Man and the Sea	400	0.712	0.465	0.818	0.688
Andersen — The Ugly Duckling	211	0.600	0.469	0.668	0.692
Andersen — The Little Mermaid	293	0.654	0.523	0.614	0.474
Andersen — The Shadow	267	0.734	0.456	0.704	0.742

Tr = Transformer (this model), Sy = Syuzhet lexicon baseline (Jockers, 2015).

Detrended arc correlation

Detrending follows Hu et al. (2021): the sentiment arc is integrated into a random walk, a nonlinear adaptive filter extracts the global trend, and the residuals capture local narrative dynamics. Spearman ρ is computed between the detrended model arc and the detrended human annotation arc, at window size L/8.

Eval set	n	Raw Spearman ρ (Tr)	Detrended Spearman ρ (Tr)	Δ (Tr)	Raw Spearman ρ (Sy)	Detrended Spearman ρ (Sy)
Hemingway	400	0.712	0.781	+0.069	0.465	0.335
The Ugly Duckling	211	0.600	0.741	+0.141	0.469	0.584
The Little Mermaid	293	0.654	0.754	+0.100	0.523	0.624
The Shadow	267	0.734	0.796	+0.062	0.456	0.657

Detrending consistently improves the transformer's correlation with human annotations, indicating that the model captures arc-level narrative dynamics beyond sentence-level sentiment. The Hemingway inter-annotator agreement (Spearman ρ between two human annotators) is 0.613 on this subset.

Usage

from transformers import pipeline

classifier = pipeline("text-classification", model="fpianz/sentiment-fiction-seq")
result = classifier("The old man was thin and gaunt with deep wrinkles in the back of his neck.")
print(result)
# [{'label': 'negative', 'score': 0.82}]

For continuous sentiment arcs:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("fpianz/sentiment-fiction-seq")
model = AutoModelForSequenceClassification.from_pretrained("fpianz/sentiment-fiction-seq")

def valence(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
    with torch.no_grad():
        logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]
    return (probs[2] - probs[0]).item()  # p(positive) - p(negative)

score = valence("He was an old man who fished alone in a skiff in the Gulf Stream.")
print(f"Valence: {score:.3f}")  # range approx [-1, +1]

Training details

• Base model: j-hartmann/sentiment-roberta-large-english-3-classes
• Architecture: RoBERTa-large (355M parameters)
• Loss: Class-weighted cross-entropy (weights: negative=0.99, neutral=0.74, positive=1.56)
• Epochs: 5 (with early stopping, patience=3)
• Learning rate: 2e-5
• Batch size: 16
• Max sequence length: 512
• Optimizer: AdamW (weight decay=0.01, warmup ratio=0.1)
• Precision: FP16
• Hardware: NVIDIA A100 (University of Groningen Habrok HPC)

Limitations

• The detrended arc evaluation is limited to three Andersen fairy tales (translated from Danish) and one section of a Hemingway novella. These results may not generalize to other genres, periods, or languages.
• Fiction4 texts are Google-translated from Danish (Feldkamp et al., 2024); translation artifacts may affect evaluation scores for the fairy tales.
• The 3-class label scheme (negative/neutral/positive) collapses the valence spectrum. The continuous valence conversion (p(pos) - p(neg)) provides finer granularity but is an approximation.
• This model has slightly less training data than sentiment-fiction (12,929 vs. 13,864 sentences). For sentence-level classification where arc evaluation is not needed, sentiment-fiction may be preferable.

References

• Sentiment Below the Surface: Omissive and Evocative Strategies in Literature and Beyond (Feldkamp et al., CHR 2024)
• DENS: A Dataset for Multi-class Emotion Analysis (Liu et al., EMNLP-IJCNLP 2019)
• Comparing Tools for Sentiment Analysis of Danish Literature from Hymns to Fairy Tales: Low-Resource Language and Domain Challenges (Feldkamp et al., WASSA 2024)
• Dynamic evolution of sentiments in Never Let Me Go: Insights from multifractal theory and its implications for literary analysis (Hu et al., DSH 2021)

Citation

Paper under review — citation will be added upon publication.