| --- |
| library_name: transformers |
| datasets: |
| - WebOrganizer/TopicAnnotations-Llama-3.1-8B |
| - WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8 |
| base_model: |
| - answerdotai/ModernBERT-base |
| --- |
| # wissamantoun/WebOrganizer-TopicClassifier-ModernBERT |
|
|
| [[Paper](https://arxiv.org/abs/2502.10341)] [[Website](https://weborganizer.allenai.org)] [[GitHub](https://github.com/CodeCreator/WebOrganizer)] |
|
|
| *All credit goes to the original authors of the model and dataset. This is a retraining of the original model with a different base model* |
|
|
| The TopicClassifier organizes web content into 17 categories based on the URL and text contents of web pages. |
| The model is a [ModernBERT-base](answerdotai/ModernBERT-base) with 140M parameters fine-tuned on the following training data: |
|
|
| 1. [WebOrganizer/TopicAnnotations-Llama-3.1-8B](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-8B): 1M documents annotated by Llama-3.1-8B (first-stage training) |
| 2. [WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8](https://huggingface.co/datasets/WebOrganizer/TopicAnnotations-Llama-3.1-405B-FP8): 100K documents annotated by Llama-3.1-405B-FP8 (second-stage training) |
|
|
| #### All Domain Classifiers |
| - [wissamantoun/WebOrganizer-FormatClassifier-ModernBERT](https://huggingface.co/wissamantoun/WebOrganizer-FormatClassifier-ModernBERT) |
| - [wissamantoun/WebOrganizer-TopicClassifier-ModernBERT](https://huggingface.co/wissamantoun/WebOrganizer-TopicClassifier-ModernBERT) *← you are here!* |
|
|
| ## Usage |
|
|
| This classifier expects input in the following input format: |
| ``` |
| {url} |
| |
| {text} |
| ``` |
|
|
| Example: |
| ```python |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification |
| |
| tokenizer = AutoTokenizer.from_pretrained("wissamantoun/wissamantoun/WebOrganizer-TopicClassifier-ModernBERT") |
| model = AutoModelForSequenceClassification.from_pretrained( |
| "wissamantoun/wissamantoun/WebOrganizer-TopicClassifier-ModernBERT", |
| trust_remote_code=True, |
| use_memory_efficient_attention=False) |
| |
| web_page = """http://www.example.com |
| |
| How to build a computer from scratch? Here are the components you need...""" |
| |
| inputs = tokenizer([web_page], return_tensors="pt") |
| outputs = model(**inputs) |
| |
| probs = outputs.logits.softmax(dim=-1) |
| print(probs.argmax(dim=-1)) |
| # -> 5 ("Hardware" topic) |
| ``` |
|
|
| You can convert the `logits` of the model with a softmax to obtain a probability distribution over the following 24 categories (in order of labels, also see `id2label` and `label2id` in the model config): |
| 1. Adult |
| 2. Art & Design |
| 3. Software Dev. |
| 4. Crime & Law |
| 5. Education & Jobs |
| 6. Hardware |
| 7. Entertainment |
| 8. Social Life |
| 9. Fashion & Beauty |
| 10. Finance & Business |
| 11. Food & Dining |
| 12. Games |
| 13. Health |
| 14. History |
| 15. Home & Hobbies |
| 16. Industrial |
| 17. Literature |
| 18. Politics |
| 19. Religion |
| 20. Science & Tech. |
| 21. Software |
| 22. Sports & Fitness |
| 23. Transportation |
| 24. Travel |
|
|
| The full definitions of the categories can be found in the [taxonomy config](https://github.com/CodeCreator/WebOrganizer/blob/main/define_domains/taxonomies/topics.yaml). |
|
|
| # Scores |
| ``` |
| ***** pred metrics ***** |
| test_accuracy = 0.8585 |
| test_accuracy__0 = 0.9346 |
| test_accuracy__1 = 0.7317 |
| test_accuracy__10 = 0.9148 |
| test_accuracy__11 = 0.8927 |
| test_accuracy__12 = 0.8687 |
| test_accuracy__13 = 0.814 |
| test_accuracy__14 = 0.8616 |
| test_accuracy__15 = 0.7179 |
| test_accuracy__16 = 0.855 |
| test_accuracy__17 = 0.8246 |
| test_accuracy__18 = 0.907 |
| test_accuracy__19 = 0.8333 |
| test_accuracy__2 = 0.866 |
| test_accuracy__20 = 0.8294 |
| test_accuracy__21 = 0.9441 |
| test_accuracy__22 = 0.8788 |
| test_accuracy__23 = 0.9 |
| test_accuracy__3 = 0.847 |
| test_accuracy__4 = 0.8442 |
| test_accuracy__5 = 0.8189 |
| test_accuracy__6 = 0.8997 |
| test_accuracy__7 = 0.7295 |
| test_accuracy__8 = 0.8937 |
| test_accuracy__9 = 0.8665 |
| test_accuracy_conf50 = 0.8674 |
| test_accuracy_conf50__0 = 0.9434 |
| test_accuracy_conf50__1 = 0.7453 |
| test_accuracy_conf50__10 = 0.93 |
| test_accuracy_conf50__11 = 0.8958 |
| test_accuracy_conf50__12 = 0.8768 |
| test_accuracy_conf50__13 = 0.8193 |
| test_accuracy_conf50__14 = 0.8691 |
| test_accuracy_conf50__15 = 0.7237 |
| test_accuracy_conf50__16 = 0.864 |
| test_accuracy_conf50__17 = 0.8358 |
| test_accuracy_conf50__18 = 0.91 |
| test_accuracy_conf50__19 = 0.8481 |
| test_accuracy_conf50__2 = 0.8768 |
| test_accuracy_conf50__20 = 0.8434 |
| test_accuracy_conf50__21 = 0.9505 |
| test_accuracy_conf50__22 = 0.8844 |
| test_accuracy_conf50__23 = 0.9028 |
| test_accuracy_conf50__3 = 0.8571 |
| test_accuracy_conf50__4 = 0.851 |
| test_accuracy_conf50__5 = 0.8206 |
| test_accuracy_conf50__6 = 0.9071 |
| test_accuracy_conf50__7 = 0.7442 |
| test_accuracy_conf50__8 = 0.9006 |
| test_accuracy_conf50__9 = 0.8761 |
| test_accuracy_conf75 = 0.9178 <--- Metric from the paper |
| test_accuracy_conf75__0 = 0.95 |
| test_accuracy_conf75__1 = 0.8413 |
| test_accuracy_conf75__10 = 0.9556 |
| test_accuracy_conf75__11 = 0.9298 |
| test_accuracy_conf75__12 = 0.9299 |
| test_accuracy_conf75__13 = 0.8788 |
| test_accuracy_conf75__14 = 0.9126 |
| test_accuracy_conf75__15 = 0.8253 |
| test_accuracy_conf75__16 = 0.8885 |
| test_accuracy_conf75__17 = 0.8968 |
| test_accuracy_conf75__18 = 0.938 |
| test_accuracy_conf75__19 = 0.9113 |
| test_accuracy_conf75__2 = 0.9029 |
| test_accuracy_conf75__20 = 0.8966 |
| test_accuracy_conf75__21 = 0.968 |
| test_accuracy_conf75__22 = 0.9225 |
| test_accuracy_conf75__23 = 0.9444 |
| test_accuracy_conf75__3 = 0.9319 |
| test_accuracy_conf75__4 = 0.8976 |
| test_accuracy_conf75__5 = 0.9167 |
| test_accuracy_conf75__6 = 0.9483 |
| test_accuracy_conf75__7 = 0.804 |
| test_accuracy_conf75__8 = 0.9448 |
| test_accuracy_conf75__9 = 0.932 |
| test_accuracy_label_average = 0.8531 |
| test_accuracy_label_average_conf50 = 0.8615 |
| test_accuracy_label_average_conf75 = 0.9111 |
| test_accuracy_label_min = 0.7179 |
| test_accuracy_label_min_conf50 = 0.7237 |
| test_accuracy_label_min_conf75 = 0.804 <--- Metric from the paper |
| test_loss = 0.4694 |
| test_proportion_conf50 = 0.9811 |
| test_proportion_conf75 = 0.8535 |
| test_runtime = 0:00:08.39 |
| test_samples_per_second = 1191.144 |
| test_steps_per_second = 37.283 |
| ``` |
|
|
|
|
|
|
| ## Citation |
| ```bibtex |
| @article{wettig2025organize, |
| title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation}, |
| author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini}, |
| journal={arXiv preprint arXiv:2502.10341}, |
| year={2025} |
| } |
| ``` |
