| --- |
| datasets: |
| - tner/tweetner7 |
| metrics: |
| - f1 |
| - precision |
| - recall |
| model-index: |
| - name: tner/roberta-base-tweetner7-random |
| results: |
| - task: |
| name: Token Classification |
| type: token-classification |
| dataset: |
| name: tner/tweetner7 |
| type: tner/tweetner7 |
| args: tner/tweetner7 |
| metrics: |
| - name: F1 (test_2021) |
| type: f1 |
| value: 0.6404223573969929 |
| - name: Precision (test_2021) |
| type: precision |
| value: 0.6356801093643198 |
| - name: Recall (test_2021) |
| type: recall |
| value: 0.6452358926919519 |
| - name: Macro F1 (test_2021) |
| type: f1_macro |
| value: 0.5923474605228576 |
| - name: Macro Precision (test_2021) |
| type: precision_macro |
| value: 0.5882611136070073 |
| - name: Macro Recall (test_2021) |
| type: recall_macro |
| value: 0.6009576961702408 |
| - name: Entity Span F1 (test_2021) |
| type: f1_entity_span |
| value: 0.7803730272596844 |
| - name: Entity Span Precision (test_2020) |
| type: precision_entity_span |
| value: 0.7745500113921167 |
| - name: Entity Span Recall (test_2021) |
| type: recall_entity_span |
| value: 0.7862842604371458 |
| - name: F1 (test_2020) |
| type: f1 |
| value: 0.6413755458515283 |
| - name: Precision (test_2020) |
| type: precision |
| value: 0.6764536557282671 |
| - name: Recall (test_2020) |
| type: recall |
| value: 0.6097560975609756 |
| - name: Macro F1 (test_2020) |
| type: f1_macro |
| value: 0.5978359372811374 |
| - name: Macro Precision (test_2020) |
| type: precision_macro |
| value: 0.6331635922016912 |
| - name: Macro Recall (test_2020) |
| type: recall_macro |
| value: 0.5696483348245823 |
| - name: Entity Span F1 (test_2020) |
| type: f1_entity_span |
| value: 0.7425607425607427 |
| - name: Entity Span Precision (test_2020) |
| type: precision_entity_span |
| value: 0.783410138248848 |
| - name: Entity Span Recall (test_2020) |
| type: recall_entity_span |
| value: 0.7057602490918526 |
|
|
| pipeline_tag: token-classification |
| widget: |
| - text: "Get the all-analog Classic Vinyl Edition of `Takin' Off` Album from {@herbiehancock@} via {@bluenoterecords@} link below: {{URL}}" |
| example_title: "NER Example 1" |
| --- |
| # tner/roberta-base-tweetner7-random |
|
|
| This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the |
| [tner/tweetner7](https://huggingface.co/datasets/tner/tweetner7) dataset (`train_random` split). |
| Model fine-tuning is done via [T-NER](https://github.com/asahi417/tner)'s hyper-parameter search (see the repository |
| for more detail). It achieves the following results on the test set of 2021: |
| - F1 (micro): 0.6404223573969929 |
| - Precision (micro): 0.6356801093643198 |
| - Recall (micro): 0.6452358926919519 |
| - F1 (macro): 0.5923474605228576 |
| - Precision (macro): 0.5882611136070073 |
| - Recall (macro): 0.6009576961702408 |
|
|
|
|
|
|
| The per-entity breakdown of the F1 score on the test set are below: |
| - corporation: 0.5073313782991202 |
| - creative_work: 0.4235127478753541 |
| - event: 0.4598470363288719 |
| - group: 0.5972820682797481 |
| - location: 0.6794871794871795 |
| - person: 0.8231629100238226 |
| - product: 0.6558089033659067 |
| |
| For F1 scores, the confidence interval is obtained by bootstrap as below: |
| - F1 (micro): |
| - 90%: [0.6318442200239734, 0.6492710692925945] |
| - 95%: [0.6304569517689761, 0.6507742568199575] |
| - F1 (macro): |
| - 90%: [0.6318442200239734, 0.6492710692925945] |
| - 95%: [0.6304569517689761, 0.6507742568199575] |
| |
| Full evaluation can be found at [metric file of NER](https://huggingface.co/tner/roberta-base-tweetner7-random/raw/main/eval/metric.json) |
| and [metric file of entity span](https://huggingface.co/tner/roberta-base-tweetner7-random/raw/main/eval/metric_span.json). |
| |
| ### Usage |
| This model can be used through the [tner library](https://github.com/asahi417/tner). Install the library via pip. |
| ```shell |
| pip install tner |
| ``` |
| [TweetNER7](https://huggingface.co/datasets/tner/tweetner7) pre-processed tweets where the account name and URLs are |
| converted into special formats (see the dataset page for more detail), so we process tweets accordingly and then run the model prediction as below. |
| |
| ```python |
| import re |
| from urlextract import URLExtract |
| from tner import TransformersNER |
| |
| extractor = URLExtract() |
| |
| def format_tweet(tweet): |
| # mask web urls |
| urls = extractor.find_urls(tweet) |
| for url in urls: |
| tweet = tweet.replace(url, "{{URL}}") |
| # format twitter account |
| tweet = re.sub(r"\b(\s*)(@[\S]+)\b", r'\1{\2@}', tweet) |
| return tweet |
| |
|
|
| text = "Get the all-analog Classic Vinyl Edition of `Takin' Off` Album from @herbiehancock via @bluenoterecords link below: http://bluenote.lnk.to/AlbumOfTheWeek" |
| text_format = format_tweet(text) |
| model = TransformersNER("tner/roberta-base-tweetner7-random") |
| model.predict([text_format]) |
| ``` |
| It can be used via transformers library but it is not recommended as CRF layer is not supported at the moment. |
| |
| ### Training hyperparameters |
| |
| The following hyperparameters were used during training: |
| - dataset: ['tner/tweetner7'] |
| - dataset_split: train_random |
| - dataset_name: None |
| - local_dataset: None |
| - model: roberta-base |
| - crf: True |
| - max_length: 128 |
| - epoch: 30 |
| - batch_size: 32 |
| - lr: 1e-05 |
| - random_seed: 0 |
| - gradient_accumulation_steps: 1 |
| - weight_decay: 1e-07 |
| - lr_warmup_step_ratio: 0.15 |
| - max_grad_norm: 1 |
|
|
| The full configuration can be found at [fine-tuning parameter file](https://huggingface.co/tner/roberta-base-tweetner7-random/raw/main/trainer_config.json). |
|
|
| ### Reference |
| If you use the model, please cite T-NER paper and TweetNER7 paper. |
| - T-NER |
| ``` |
| |
| @inproceedings{ushio-camacho-collados-2021-ner, |
| title = "{T}-{NER}: An All-Round Python Library for Transformer-based Named Entity Recognition", |
| author = "Ushio, Asahi and |
| Camacho-Collados, Jose", |
| booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations", |
| month = apr, |
| year = "2021", |
| address = "Online", |
| publisher = "Association for Computational Linguistics", |
| url = "https://aclanthology.org/2021.eacl-demos.7", |
| doi = "10.18653/v1/2021.eacl-demos.7", |
| pages = "53--62", |
| abstract = "Language model (LM) pretraining has led to consistent improvements in many NLP downstream tasks, including named entity recognition (NER). In this paper, we present T-NER (Transformer-based Named Entity Recognition), a Python library for NER LM finetuning. In addition to its practical utility, T-NER facilitates the study and investigation of the cross-domain and cross-lingual generalization ability of LMs finetuned on NER. Our library also provides a web app where users can get model predictions interactively for arbitrary text, which facilitates qualitative model evaluation for non-expert programmers. We show the potential of the library by compiling nine public NER datasets into a unified format and evaluating the cross-domain and cross- lingual performance across the datasets. The results from our initial experiments show that in-domain performance is generally competitive across datasets. However, cross-domain generalization is challenging even with a large pretrained LM, which has nevertheless capacity to learn domain-specific features if fine- tuned on a combined dataset. To facilitate future research, we also release all our LM checkpoints via the Hugging Face model hub.", |
| } |
| |
| ``` |
| - TweetNER7 |
| ``` |
| |
| @inproceedings{ushio-etal-2022-tweet, |
| title = "{N}amed {E}ntity {R}ecognition in {T}witter: {A} {D}ataset and {A}nalysis on {S}hort-{T}erm {T}emporal {S}hifts", |
| author = "Ushio, Asahi and |
| Neves, Leonardo and |
| Silva, Vitor and |
| Barbieri, Francesco. and |
| Camacho-Collados, Jose", |
| booktitle = "The 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing", |
| month = nov, |
| year = "2022", |
| address = "Online", |
| publisher = "Association for Computational Linguistics", |
| } |
| |
| ``` |
|
|
|
|
|
|
