Hugging Face's logo

BFTree
/
rwgbench-citation-frame-classifier

Text Classification
Transformers
Safetensors
deberta-v2
citation-function-classification
scholarly-positioning
related-work-generation
rwgbench
multicite
text-embeddings-inference
Model card Files
xet
 Community
BFTree commited on
Commit
075922d
·
verified ·
1 Parent(s): c2b753f

Add RWGBench citation frame classifier

Browse files
Files changed (12) hide show
  1. README.md +111 -0
  2. added_tokens.json +3 -0
  3. config.json +54 -0
  4. label_map.json +29 -0
  5. metrics.json +197 -0
  6. model.safetensors +3 -0
  7. rwgb_classifier_config.json +31 -0
  8. special_tokens_map.json +15 -0
  9. spm.model +3 -0
  10. thresholds.json +9 -0
  11. tokenizer.json +0 -0
  12. tokenizer_config.json +58 -0
README.md CHANGED
@@ -1,3 +1,114 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ library_name: transformers
4
+ pipeline_tag: text-classification
5
+ base_model: microsoft/deberta-v3-large
6
+ tags:
7
+ - citation-function-classification
8
+ - scholarly-positioning
9
+ - related-work-generation
10
+ - rwgbench
11
+ - multicite
12
+ datasets:
13
+ - multicite
14
+ metrics:
15
+ - f1
16
  ---
17
+
18
+ # RWGBench Citation Frame Classifier
19
+
20
+ This model is the citation-frame classifier used by RWGBench to compute
21
+ Citation Frame Alignment (CFA). CFA evaluates whether a generated related work
22
+ section uses citations with a rhetorical frame distribution similar to the
23
+ author-written reference section.
24
+
25
+ The classifier is a DeBERTa-v3-large sequence-classification model fine-tuned
26
+ for multi-label citation-function prediction. It predicts seven
27
+ MultiCite-derived labels:
28
+
29
+ | Label | Intended citation function |
30
+ |---|---|
31
+ | `motivation` | motivates the target problem or research direction |
32
+ | `background` | provides general background or prior findings |
33
+ | `uses` | describes methods, data, tools, or results used by the target work |
34
+ | `extends` | indicates that the target work builds on or extends prior work |
35
+ | `similarities` | marks similarity or close relationship to prior work |
36
+ | `differences` | marks contrast, limitation, gap, or distinction |
37
+ | `future_work` | discusses future directions |
38
+
39
+ ## Intended Use
40
+
41
+ The model is intended for automatic evaluation and analysis of related work
42
+ generation systems. In RWGBench, citation contexts are extracted from generated
43
+ and reference related work sections, classified with this model, aggregated
44
+ into frame-count profiles, and compared with macro count F1.
45
+
46
+ It is not intended as a general-purpose judge of paper quality or citation
47
+ correctness. It should be used together with citation-selection and
48
+ citation-support metrics.
49
+
50
+ ## Files
51
+
52
+ | File | Description |
53
+ |---|---|
54
+ | `config.json`, `model.safetensors` | Transformers sequence-classification checkpoint |
55
+ | `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, `spm.model`, `added_tokens.json` | Tokenizer files |
56
+ | `thresholds.json` | Per-label development thresholds used for multi-label decoding |
57
+ | `metrics.json` | Development and held-out MultiCite evaluation metrics |
58
+ | `label_map.json` | Label mapping used during training |
59
+ | `rwgb_classifier_config.json` | RWGBench-specific usage metadata |
60
+
61
+ ## Evaluation
62
+
63
+ Held-out MultiCite test performance:
64
+
65
+ | Metric | Score |
66
+ |---|---:|
67
+ | Macro F1 | 0.5585 |
68
+ | Micro F1 | 0.6751 |
69
+ | Samples F1 | 0.6933 |
70
+ | Weighted F1 | 0.6750 |
71
+ | Subset accuracy | 0.5119 |
72
+
73
+ Per-label held-out F1:
74
+
75
+ | Label | F1 |
76
+ |---|---:|
77
+ | `motivation` | 0.3523 |
78
+ | `background` | 0.7756 |
79
+ | `uses` | 0.7372 |
80
+ | `extends` | 0.4450 |
81
+ | `similarities` | 0.5508 |
82
+ | `differences` | 0.6274 |
83
+ | `future_work` | 0.4211 |
84
+
85
+ The thresholds in `thresholds.json` are applied after sigmoid probabilities.
86
+ If no label passes its threshold, RWGBench assigns the highest-probability label
87
+ so that every citation context contributes at least one frame.
88
+
89
+ ## Usage With RWGBench
90
+
91
+ Download the model into the RWGBench code repository:
92
+
93
+ ```text
94
+ models/citation_frame_multicite/
95
+ config.json
96
+ model.safetensors
97
+ tokenizer.json
98
+ thresholds.json
99
+ ...
100
+ ```
101
+
102
+ run:
103
+
104
+ ```bash
105
+ python src/evaluation/evaluate_generated_results.py \
106
+ --input results/your_results.json \
107
+ --output results/your_eval.json \
108
+ --gold data/gold100_papers.json \
109
+ --corpus data/corpus.json \
110
+ --citation-frame-model-path models/citation_frame_multicite
111
+ ```
112
+
113
+ RWGBench automatically marks numbered citations with `<cite>...</cite>` before
114
+ classification, matching the input format used during training.
added_tokens.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {
2
+ "[MASK]": 128000
3
+ }
config.json ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "microsoft/deberta-v3-large",
3
+ "architectures": [
4
+ "DebertaV2ForSequenceClassification"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "hidden_act": "gelu",
8
+ "hidden_dropout_prob": 0.1,
9
+ "hidden_size": 1024,
10
+ "id2label": {
11
+ "0": "motivation",
12
+ "1": "background",
13
+ "2": "uses",
14
+ "3": "extends",
15
+ "4": "similarities",
16
+ "5": "differences",
17
+ "6": "future_work"
18
+ },
19
+ "initializer_range": 0.02,
20
+ "intermediate_size": 4096,
21
+ "label2id": {
22
+ "background": 1,
23
+ "differences": 5,
24
+ "extends": 3,
25
+ "future_work": 6,
26
+ "motivation": 0,
27
+ "similarities": 4,
28
+ "uses": 2
29
+ },
30
+ "layer_norm_eps": 1e-07,
31
+ "max_position_embeddings": 512,
32
+ "max_relative_positions": -1,
33
+ "model_type": "deberta-v2",
34
+ "norm_rel_ebd": "layer_norm",
35
+ "num_attention_heads": 16,
36
+ "num_hidden_layers": 24,
37
+ "pad_token_id": 0,
38
+ "pooler_dropout": 0,
39
+ "pooler_hidden_act": "gelu",
40
+ "pooler_hidden_size": 1024,
41
+ "pos_att_type": [
42
+ "p2c",
43
+ "c2p"
44
+ ],
45
+ "position_biased_input": false,
46
+ "position_buckets": 256,
47
+ "problem_type": "multi_label_classification",
48
+ "relative_attention": true,
49
+ "share_att_key": true,
50
+ "torch_dtype": "float32",
51
+ "transformers_version": "4.43.1",
52
+ "type_vocab_size": 0,
53
+ "vocab_size": 128100
54
+ }
label_map.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "labels": [
3
+ "motivation",
4
+ "background",
5
+ "uses",
6
+ "extends",
7
+ "similarities",
8
+ "differences",
9
+ "future_work"
10
+ ],
11
+ "id2label": {
12
+ "0": "motivation",
13
+ "1": "background",
14
+ "2": "uses",
15
+ "3": "extends",
16
+ "4": "similarities",
17
+ "5": "differences",
18
+ "6": "future_work"
19
+ },
20
+ "label2id": {
21
+ "motivation": 0,
22
+ "background": 1,
23
+ "uses": 2,
24
+ "extends": 3,
25
+ "similarities": 4,
26
+ "differences": 5,
27
+ "future_work": 6
28
+ }
29
+ }
metrics.json ADDED
@@ -0,0 +1,197 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_name": "microsoft/deberta-v3-large",
3
+ "max_length": 384,
4
+ "epochs": 8,
5
+ "batch_size": 4,
6
+ "grad_accum": 4,
7
+ "learning_rate": 1e-05,
8
+ "weight_decay": 0.01,
9
+ "seed": 42,
10
+ "history": [
11
+ {
12
+ "epoch": 1,
13
+ "train_loss": 1.0055836753218346,
14
+ "dev_loss": 0.9146218588149625,
15
+ "dev_subset_accuracy": 0.2214957090314671,
16
+ "dev_micro_f1": 0.5491691637156089,
17
+ "dev_macro_f1": 0.4234808296833363,
18
+ "dev_samples_f1": 0.5578122871543386,
19
+ "dev_weighted_f1": 0.5740849903216618,
20
+ "dev_hamming_loss": 0.19323953529102691,
21
+ "dev_lrap": 0.7145651299615349
22
+ },
23
+ {
24
+ "epoch": 2,
25
+ "train_loss": 0.8151864616196258,
26
+ "dev_loss": 0.7926584045871411,
27
+ "dev_subset_accuracy": 0.3902738046587658,
28
+ "dev_micro_f1": 0.6228679722046747,
29
+ "dev_macro_f1": 0.5072227399300326,
30
+ "dev_samples_f1": 0.6281569268492031,
31
+ "dev_weighted_f1": 0.6312166518801912,
32
+ "dev_hamming_loss": 0.1394126919259735,
33
+ "dev_lrap": 0.7435317622485574
34
+ },
35
+ {
36
+ "epoch": 3,
37
+ "train_loss": 0.6799942377984567,
38
+ "dev_loss": 0.7320666873304088,
39
+ "dev_subset_accuracy": 0.4139762975071516,
40
+ "dev_micro_f1": 0.6458755652580696,
41
+ "dev_macro_f1": 0.5516055100096154,
42
+ "dev_samples_f1": 0.6623484538891159,
43
+ "dev_weighted_f1": 0.6542220786376058,
44
+ "dev_hamming_loss": 0.13258217058789187,
45
+ "dev_lrap": 0.749611445177444
46
+ },
47
+ {
48
+ "epoch": 4,
49
+ "train_loss": 0.5624804687018127,
50
+ "dev_loss": 0.7597886227996521,
51
+ "dev_subset_accuracy": 0.4176542705353494,
52
+ "dev_micro_f1": 0.651979664150362,
53
+ "dev_macro_f1": 0.5712473553205266,
54
+ "dev_samples_f1": 0.6739817463560823,
55
+ "dev_weighted_f1": 0.657612341205038,
56
+ "dev_hamming_loss": 0.1318816042968066,
57
+ "dev_lrap": 0.7790767444424961
58
+ },
59
+ {
60
+ "epoch": 5,
61
+ "train_loss": 0.48175797338522186,
62
+ "dev_loss": 0.8293445085120552,
63
+ "dev_subset_accuracy": 0.44707805476093176,
64
+ "dev_micro_f1": 0.6561595368285622,
65
+ "dev_macro_f1": 0.5679052613344295,
66
+ "dev_samples_f1": 0.67219724833129,
67
+ "dev_weighted_f1": 0.6576940876277884,
68
+ "dev_hamming_loss": 0.12481756086169654,
69
+ "dev_lrap": 0.780261706916793
70
+ },
71
+ {
72
+ "epoch": 6,
73
+ "train_loss": 0.40624411452699993,
74
+ "dev_loss": 0.8781815413659548,
75
+ "dev_subset_accuracy": 0.47609317531671436,
76
+ "dev_micro_f1": 0.6613861386138614,
77
+ "dev_macro_f1": 0.574749358378983,
78
+ "dev_samples_f1": 0.6784906688462062,
79
+ "dev_weighted_f1": 0.6597610710894909,
80
+ "dev_hamming_loss": 0.11979683577558527,
81
+ "dev_lrap": 0.7896338892456563
82
+ },
83
+ {
84
+ "epoch": 7,
85
+ "train_loss": 0.35122106560334315,
86
+ "dev_loss": 0.9226574831903954,
87
+ "dev_subset_accuracy": 0.43890478136493666,
88
+ "dev_micro_f1": 0.6547694251421352,
89
+ "dev_macro_f1": 0.5651438218029249,
90
+ "dev_samples_f1": 0.6769105026563139,
91
+ "dev_weighted_f1": 0.6555867703017909,
92
+ "dev_hamming_loss": 0.12761982602603772,
93
+ "dev_lrap": 0.791656774411166
94
+ },
95
+ {
96
+ "epoch": 8,
97
+ "train_loss": 0.30937692145996737,
98
+ "dev_loss": 0.9469064347890321,
99
+ "dev_subset_accuracy": 0.46710257458111976,
100
+ "dev_micro_f1": 0.6534165571616294,
101
+ "dev_macro_f1": 0.5651322221205731,
102
+ "dev_samples_f1": 0.670358261817191,
103
+ "dev_weighted_f1": 0.656064167967615,
104
+ "dev_hamming_loss": 0.12318290618249751,
105
+ "dev_lrap": 0.7881818358728842
106
+ }
107
+ ],
108
+ "best_dev_macro_f1": 0.574749358378983,
109
+ "best_thresholds": {
110
+ "motivation": 0.6339556574821472,
111
+ "background": 0.45784226059913635,
112
+ "uses": 0.3992602229118347,
113
+ "extends": 0.8947582244873047,
114
+ "similarities": 0.7605156898498535,
115
+ "differences": 0.8103693723678589,
116
+ "future_work": 0.873489499092102
117
+ },
118
+ "dev_metrics": {
119
+ "dev_loss": 0.8781815413659548,
120
+ "dev_subset_accuracy": 0.47609317531671436,
121
+ "dev_micro_f1": 0.6613861386138614,
122
+ "dev_macro_f1": 0.574749358378983,
123
+ "dev_samples_f1": 0.6784906688462062,
124
+ "dev_weighted_f1": 0.6597610710894909,
125
+ "dev_hamming_loss": 0.11979683577558527,
126
+ "dev_lrap": 0.7896338892456563
127
+ },
128
+ "test_metrics": {
129
+ "test_subset_accuracy": 0.5119227286447329,
130
+ "test_micro_f1": 0.6750780762023735,
131
+ "test_macro_f1": 0.5584698142469106,
132
+ "test_samples_f1": 0.6932588791628935,
133
+ "test_weighted_f1": 0.6750260332760398,
134
+ "test_hamming_loss": 0.11215557759475658,
135
+ "test_lrap": 0.8042620700559133,
136
+ "label_metrics": {
137
+ "motivation": {
138
+ "precision": 0.2882882882882883,
139
+ "recall": 0.4528301886792453,
140
+ "f1": 0.3522935779816514,
141
+ "support": 212,
142
+ "threshold": 0.6339556574821472
143
+ },
144
+ "background": {
145
+ "precision": 0.7161803713527851,
146
+ "recall": 0.845732184808144,
147
+ "f1": 0.7755834829443446,
148
+ "support": 1277,
149
+ "threshold": 0.45784226059913635
150
+ },
151
+ "uses": {
152
+ "precision": 0.6863496932515337,
153
+ "recall": 0.7962633451957295,
154
+ "f1": 0.7372322899505767,
155
+ "support": 1124,
156
+ "threshold": 0.3992602229118347
157
+ },
158
+ "extends": {
159
+ "precision": 0.44907407407407407,
160
+ "recall": 0.4409090909090909,
161
+ "f1": 0.44495412844036697,
162
+ "support": 220,
163
+ "threshold": 0.8947582244873047
164
+ },
165
+ "similarities": {
166
+ "precision": 0.5564102564102564,
167
+ "recall": 0.5452261306532663,
168
+ "f1": 0.550761421319797,
169
+ "support": 398,
170
+ "threshold": 0.7605156898498535
171
+ },
172
+ "differences": {
173
+ "precision": 0.6424116424116424,
174
+ "recall": 0.6130952380952381,
175
+ "f1": 0.6274111675126903,
176
+ "support": 504,
177
+ "threshold": 0.8103693723678589
178
+ },
179
+ "future_work": {
180
+ "precision": 0.5333333333333333,
181
+ "recall": 0.34782608695652173,
182
+ "f1": 0.42105263157894735,
183
+ "support": 23,
184
+ "threshold": 0.873489499092102
185
+ }
186
+ }
187
+ },
188
+ "labels": [
189
+ "motivation",
190
+ "background",
191
+ "uses",
192
+ "extends",
193
+ "similarities",
194
+ "differences",
195
+ "future_work"
196
+ ]
197
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:160da0f3208c04ee39c6193eec76f858cc65d3494b095ca21e634b026dcf4192
3
+ size 1740324948
rwgb_classifier_config.json ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "RWGBench Citation Frame Classifier",
3
+ "base_model": "microsoft/deberta-v3-large",
4
+ "task": "multi-label citation-function classification",
5
+ "max_length": 512,
6
+ "labels": [
7
+ "motivation",
8
+ "background",
9
+ "uses",
10
+ "extends",
11
+ "similarities",
12
+ "differences",
13
+ "future_work"
14
+ ],
15
+ "decoding": {
16
+ "threshold_file": "thresholds.json",
17
+ "min_one_label": true,
18
+ "max_labels": null
19
+ },
20
+ "rwgb_metric_use": {
21
+ "metric": "citation_frame_alignment",
22
+ "description": "Predict MultiCite-derived citation-function labels for citation contexts, count frame occurrences in generated and reference related work sections, and compare the two frame-count profiles using macro count F1."
23
+ },
24
+ "heldout_multicite_test": {
25
+ "macro_f1": 0.5584698142469106,
26
+ "micro_f1": 0.6750780762023735,
27
+ "subset_accuracy": 0.5119227286447329,
28
+ "samples_f1": 0.6932588791628935,
29
+ "weighted_f1": 0.6750260332760398
30
+ }
31
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "[CLS]",
3
+ "cls_token": "[CLS]",
4
+ "eos_token": "[SEP]",
5
+ "mask_token": "[MASK]",
6
+ "pad_token": "[PAD]",
7
+ "sep_token": "[SEP]",
8
+ "unk_token": {
9
+ "content": "[UNK]",
10
+ "lstrip": false,
11
+ "normalized": true,
12
+ "rstrip": false,
13
+ "single_word": false
14
+ }
15
+ }
spm.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd
3
+ size 2464616
thresholds.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "motivation": 0.6339556574821472,
3
+ "background": 0.45784226059913635,
4
+ "uses": 0.3992602229118347,
5
+ "extends": 0.8947582244873047,
6
+ "similarities": 0.7605156898498535,
7
+ "differences": 0.8103693723678589,
8
+ "future_work": 0.873489499092102
9
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "[PAD]",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "[CLS]",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "[SEP]",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "[UNK]",
29
+ "lstrip": false,
30
+ "normalized": true,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "128000": {
36
+ "content": "[MASK]",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "bos_token": "[CLS]",
45
+ "clean_up_tokenization_spaces": true,
46
+ "cls_token": "[CLS]",
47
+ "do_lower_case": false,
48
+ "eos_token": "[SEP]",
49
+ "mask_token": "[MASK]",
50
+ "model_max_length": 1000000000000000019884624838656,
51
+ "pad_token": "[PAD]",
52
+ "sep_token": "[SEP]",
53
+ "sp_model_kwargs": {},
54
+ "split_by_punct": false,
55
+ "tokenizer_class": "DebertaV2Tokenizer",
56
+ "unk_token": "[UNK]",
57
+ "vocab_type": "spm"
58
+ }