README.md

---
language:
- en
library_name: gliformer
pipeline_tag: token-classification
tags:
- gliformer
- deberta
- named-entity-recognition
- text-classification
- relation-extraction
- structured-extraction
- feature-extraction
- document-understanding
license: apache-2.0
---

# GLiFormer Large v1

**One encoder for PDF layout processing, entity recognition, classification, relation extraction, structured records, and text embeddings.**

![GLiFormer supported tasks](gliformer-tasks.gif)

`knowledgator/gliformer-large-v1` is the 575.6M-parameter large release of
[GLiFormer](https://github.com/Knowledgator/GLiFormer). It accepts task labels and
extraction schemas at inference time, with task heads sharing a DeBERTa encoder.
The layout-aware architecture supports text and document-layout inputs; the usage
examples and quality results below focus on text tasks.

## Usage

```bash
pip install gliformer -U
```

Or install the GLiFormer framework in a Python 3.10+ environment from the source:

```bash
git clone https://github.com/Knowledgator/GLiFormer.git
cd GLiFormer
pip install -e .
```

Optional CUDA attention kernels are available with `pip install -e ".[flash]"`.
CPU inference uses eager attention. The following examples reuse `model`:

```python
import torch
from gliformer import GLiFormer

model = GLiFormer.from_pretrained(
    "knowledgator/gliformer-large-v1",
    load_tokenizer=True,
)
model = model.to("cuda" if torch.cuda.is_available() else "cpu").eval()
```

For a local copy, replace the model ID with the checkpoint directory.

### Named entity recognition

Specify the entity types at inference time:

```python
text = "Alice works at Acme in London."
entities = model.predict_entities(
    text,
    ["person", "organization", "location"],
    threshold=0.5,
)
for entity in entities:
    print(entity["text"], entity["label"], entity["score"])
```

Each entity includes `text`, `label`, `start`, `end`, and `score`.
Offsets are character positions with an exclusive `end`.
Pass a list of texts and `batch_size=8` for batched extraction.

### Text classification

```python
predictions = model.classify(
    "The new search feature is fast and easy to use.",
    ["positive", "negative", "neutral"],
    threshold=0.5,
)
print(predictions)  # Label dictionaries containing class_name and score.
```

Named groups are also supported, for example
`{"sentiment": ["positive", "negative"], "topic": ["product", "support"]}`.

### Joint relation extraction

Supply entity and relation labels together to use this checkpoint's joint relation head:

```python
results = model.inference(
    "Alice works at Acme.",
    joint_relations={
        "employment": {
            "entities": ["person", "organization"],
            "relations": ["works_at"],
        }
    },
    threshold=0.5,
)
for relation in results["joint_relex"][0]:
    print(relation["head"]["text"], relation["relation"], relation["tail"]["text"])
```

`inference` returns a dictionary of task outputs, each containing one result per input text.
The separate `predict_relations` convenience method requires an open relation head;
use `joint_relations` for this model.

### Structured extraction

Extract records directly into a Python dictionary:

```python
records = model.structure(
    "Alice works at Acme.",
    {"employee": ["name", "company"]},
)
print(records)
# {'employee': [{'name': 'Alice', 'company': 'Acme'}]}
```

Nested Pydantic schemas support multilevel records:

```python
from pydantic import BaseModel

class Employee(BaseModel):
    name: str
    role: str

class Department(BaseModel):
    name: str
    employees: list[Employee]

class Company(BaseModel):
    name: str
    departments: list[Department]

records = model.structure(
    "At Acme, Engineering includes Alice, a software engineer, and Bob, "
    "a designer. Sales includes Carol, an account manager.",
    {"company": Company},
    validate_output=True,
)
print(records)
```

The decoder assembles source-grounded fields and parent–child relationships into nested
records. Predictions depend on the schema, input, and thresholds; Pydantic validation
checks the output schema, not factual correctness.

### Multiple tasks in one call

```python
results = model.inference(
    "Alice joined Acme as a software engineer.",
    entities=["person", "organization"],
    classes=["business", "sports", "technology"],
    structures={"employee": ["name", "company"]},
)
print(results["ner"][0])
print(results["classification"][0])
print(results["structuring"][0])
```

### Text embeddings

```python
import torch.nn.functional as F

embeddings = model.embed_text([
    "A scientist works in a laboratory.",
    "A researcher conducts an experiment.",
])
print(embeddings.shape)  # torch.Size([2, 1024])
print(F.cosine_similarity(embeddings[0:1], embeddings[1:2]).item())
```

## Reported evaluation results


| Task | Metric | Score |
| --- | --- | ---: |
| NER, 26 datasets / 131,156 examples | Mean dataset strict entity F1 | 50.91 |
| CrossNER, 5 domains / 2,505 examples | Mean domain strict entity F1 | 64.35 |
| Classification, 13 datasets / 79,828 examples | Mean dataset macro-F1 | 75.03 |
| Multilevel structuring, 500 examples | Order-free, boundary-tolerant JSON F1 | 91.10 |

Dataset means weight datasets equally. NER requires both the entity span and type
to match. Classification macro-F1 averages class F1 scores within each dataset.
Structuring compares flattened JSON value paths after aligning records without
requiring their original order and allowing the evaluator's limited boundary repairs;
it is not exact JSON match. The manuscript corrects the structuring evaluation size
to 500; historical notes contain obsolete bucket counts totaling 300.

### Named entity recognition

| Dataset | Examples | Precision | Recall | F1 |
| --- | ---: | ---: | ---: | ---: |
| ACE 2004 | 812 | 51.67 | 29.00 | 37.15 |
| ACE 2005 | 1,060 | 44.88 | 22.42 | 29.90 |
| AnatEM | 3,830 | 26.18 | 31.74 | 28.70 |
| bc2gm | 5,000 | 45.46 | 51.53 | 48.30 |
| bc4chemd | 26,364 | 40.74 | 67.58 | 50.84 |
| bc5cdr | 4,797 | 61.92 | 72.05 | 66.60 |
| Broad Tweet Corpus | 2,000 | 55.55 | 70.99 | 62.33 |
| CoNLL 2003 | 3,453 | 58.96 | 72.82 | 65.16 |
| CrossNER_AI | 431 | 51.69 | 51.83 | 51.76 |
| CrossNER_literature | 416 | 63.11 | 60.10 | 61.57 |
| CrossNER_music | 465 | 67.48 | 66.17 | 66.82 |
| CrossNER_politics | 650 | 70.46 | 72.52 | 71.48 |
| CrossNER_science | 543 | 71.63 | 68.69 | 70.13 |
| FabNER | 2,064 | 27.17 | 18.47 | 21.99 |
| FindVehicle | 20,777 | 42.11 | 51.21 | 46.21 |
| GENIA_NER | 1,854 | 48.42 | 57.23 | 52.46 |
| HarveyNER | 1,303 | 9.95 | 22.54 | 13.81 |
| mit-movie | 2,442 | 63.10 | 52.98 | 57.60 |
| mit-restaurant | 1,520 | 38.88 | 30.25 | 34.03 |
| MultiNERD | 10,000 | 54.96 | 91.91 | 68.79 |
| ncbi | 940 | 47.63 | 64.05 | 54.63 |
| Ontonotes | 8,262 | 28.20 | 42.18 | 33.80 |
| PolyglotNER | 10,000 | 35.52 | 70.85 | 47.31 |
| TweetNER7 | 576 | 43.46 | 48.62 | 45.90 |
| WikiANN en | 10,000 | 54.85 | 57.74 | 56.26 |
| WikiNeural | 11,597 | 73.93 | 87.53 | 80.16 |

### Text classification

| Dataset | Examples | Accuracy | Macro-F1 | Weighted F1 |
| --- | ---: | ---: | ---: | ---: |
| SetFit/CR | 376 | 91.22 | 90.45 | 91.20 |
| SetFit/sst2 | 1,821 | 92.97 | 92.97 | 92.97 |
| SetFit/sst5 | 2,210 | 44.34 | 40.33 | 43.39 |
| stanfordnlp/imdb | 25,000 | 93.94 | 93.93 | 93.93 |
| SetFit/20_newsgroups | 7,532 | 57.94 | 57.18 | 58.70 |
| SetFit/enron_spam | 2,000 | 97.95 | 97.95 | 97.95 |
| AmazonScience/massive | 2,974 | 71.32 | 69.98 | 71.93 |
| PolyAI/banking77 | 3,080 | 70.97 | 70.55 | 70.55 |
| mteb/financial_phrasebank | 1,129 | 97.25 | 96.77 | 97.24 |
| SetFit/ag_news | 7,600 | 82.07 | 81.53 | 81.53 |
| dair-ai/emotion | 2,000 | 54.75 | 48.07 | 55.59 |
| MoritzLaurer/cap_sotu | 23,040 | 51.74 | 49.00 | 51.15 |
| cornell-movie-review-data/rotten_tomatoes | 1,066 | 86.68 | 86.68 | 86.68 |

Micro-F1 equals accuracy in these single-label runs. Reported prediction coverage
is 97.15% for 20 Newsgroups and 100% for the other datasets. Summary scores retain
the original reports' precision; means of the rounded rows can differ by 0.01.

### Joint relation extraction

These runs use predicted entities. Gold counts are relation instances, not documents.
Base and large were evaluated on different-sized subsets, so their relation scores
are not a controlled comparison on identical examples.

| Dataset | Gold relations | Precision | Recall | Micro-F1 | Macro-F1 |
| --- | ---: | ---: | ---: | ---: | ---: |
| DocRED | 6,003 | 29.90 | 8.13 | 12.78 | 3.64 |
| CrossRE | 1,926 | 22.76 | 8.72 | 12.61 | 11.72 |
| FewRel | 500 | 21.86 | 26.80 | 24.08 | 21.40 |
| CoNLL04 zero-shot | 677 | 38.47 | 33.53 | 35.83 | 35.32 |

CoNLL04 zero-shot typed F1, which also checks endpoint entity types, is 34.73%.

### Multilevel structuring

| Gold JSON depth | Order-free, boundary-tolerant F1 |
| --- | ---: |
| 3 | 89.94 |
| 4 | 95.11 |
| 5 | 91.69 |
| 6+ | 92.80 |

### Evaluation provenance and reproduction

You can find more information on the evaluation methodology here: https://www.knowledgator.com/research

Evaluation entry points are in
[`gliformer_eval`](https://github.com/Knowledgator/GLiFormer/tree/main/gliformer_eval).
For example, after preparing the CrossNER files, run from the framework repository:

```bash
python gliformer_eval/eval_ner.py \
  --model knowledgator/gliformer-large-v1 \
  --data data/NER \
  --datasets CrossNER_AI CrossNER_literature CrossNER_music CrossNER_politics CrossNER_science \
  --output eval_results/gliformer_large_v1_ner.json
```

Each dataset directory must contain `labels.json` and `test.json`. The other task
entry points are `eval_classification.py`, `eval_relex.py`, and `eval_structuring.py`;
use `--help` for data paths and inference settings. Reproduction requires matching
the original data subsets, schema labels, thresholds, and decoding settings.

## Training and intended use

The checkpoint uses the backbone listed above with supervised task heads for
information extraction, classification, structuring, and embeddings. See the
manuscript for the documented multitask training mixtures. Full checkpoint-specific
training provenance is not recorded in the saved evaluation reports.

Use this model for extracting labeled mentions, candidate classes, relations, and
structured records from text, and for producing text similarity vectors. The available
results cover English tasks. Quality on other languages, document-layout inputs,
and embedding benchmarks is not established by the tables above.

## Limitations

- Labels, schema wording, domain, input length, and thresholds affect predictions.
- Extraction can omit information, choose incorrect spans, or attach records to the wrong parent.
- Reported NER transfer groups do not establish that every evaluated domain was absent from training.
- Fixed record anchors and the configured span width constrain extraction capacity.
- This checkpoint has no dedicated vision, audio, or open relation head.