Hugging Face's logo

clarin-pl
/
combo-seg-xlm-roberta-base-telugu-mtg-ud2.17

Token Classification
PyTorch
Telugu
segmentation
tokenization
combo-seg
universal-dependencies
Model card Files
xet
 Community
combo-seg-xlm-roberta-base-telugu-mtg-ud2.17 / README.md
Michal's picture
Michal
Regenerate model card
8db8799 verified
preview code
|
Raw
History Blame
3.08 kB
metadata 
license: cc-by-sa-4.0
language:
  - te
tags:
  - segmentation
  - tokenization
  - combo-seg
  - universal-dependencies
datasets:
  - universal_dependencies
model-name: combo-seg-xlm-roberta-base-telugu-mtg-ud2.17
pipeline_tag: token-classification

COMBO-SEG Model for Telugu

Model Description

This is a Telugu-language character-level segmentation model based on COMBO-SEG, an open-source text segmentation system. It performs:

  • sentence segmentation
  • tokenisation (including multi-word token detection)

The Telugu model uses FacebookAI/xlm-roberta-base as its base encoder and is trained on UD_Telugu-MTG (UD v2.17).

Evaluation

Metric Tokens Words Sentences
F1 99.77 99.40 97.71

Usage

Install the library from PyPI:

pip install combo-seg

from combo_seg import ComboSeg

# Load a pre-trained model
nlp = ComboSeg("Telugu")

# Segment raw text — returns Document with hierarchy: Document -> Turn -> Sentence -> Token
doc = nlp("వేగవంతమైన గోధుమ రంగు నక్క బద్ధకపు కుక్క మీదుగా దూకుతుంది.")

# Inspect results
for turn in doc.turns:
    for sentence in turn.sentences:
        print(f"Sentence: {sentence.text}")
        for token in sentence.tokens:
            if token.is_multi_word:
                print(f"  MWT: {token.text} -> {token.subwords}")
            else:
                print(f"  Token: {token.text}")

Or load directly from HuggingFace:

from combo_seg import ComboSeg

nlp = ComboSeg.from_pretrained("clarin-pl/combo-seg-xlm-roberta-base-telugu-mtg-ud2.17")
doc = nlp("వేగవంతమైన గోధుమ రంగు నక్క బద్ధకపు కుక్క మీదుగా దూకుతుంది.")

License

The training data license: cc-by-sa-4.0 is derived from the Universal Dependencies treebank. For the full license terms of each treebank, please refer to the corresponding LICENSE.txt file in the treebank repository:

Citation

If you use this model, please cite:

Ulewicz, M., & Wróblewska, A. (2026). COMBO-SEG Models Trained on UD v2.17. https://doi.org/10.5281/zenodo.19651441

@software{combo_seg_2026,
  author    = {Ulewicz, Michał and Wróblewska, Alina},
  title     = {{COMBO-SEG} Models Trained on {UD} v2.17},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.19651441},
  url       = {https://doi.org/10.5281/zenodo.19651441}
}

Resources