OpceanAI
/

PASITA

@@ -3,335 +3,91 @@ license: apache-2.0
 language:
 - es
 - en
 ---
-PASITA v1
-PASITA is a compact generative language model specialized in plain-text to Markdown (GFM) transformation with an emphasis on preserving the information contained in the input.
-It is a decoder-only Transformer based on the Llama architecture, trained entirely from scratch without a pretrained base model. PASITA is designed as a specialized text transformation model rather than a general-purpose conversational or instruction-following language model.
-Overview
-PASITA targets the following transformation:
-Plain text
-    |
-    v
-PASITA
-    |
-    v
-Markdown (GFM)
-The primary objective is to introduce Markdown structure while minimizing changes to the source content.
-PASITA is intended for use cases where preserving the original information is more important than aggressively formatting the input.
-Model Specifications
-Property| Value
-Model| PASITA v1
-Task| Plain text → Markdown (GFM)
-Architecture| LlamaForCausalLM
-Parameters| 88,099,584 (~88M)
-Hidden size| 768
-Layers| 12
-Intermediate size| 2048
-Attention heads| 12
-Key/value heads| 4
-Attention| Grouped Query Attention (GQA)
-Head dimension| 64
-Activation| SiLU / SwiGLU
-Normalization| RMSNorm
-RMSNorm epsilon| 1e-5
-Positional encoding| RoPE
-RoPE theta| 100,000
-Maximum context| 2,048 tokens
-Vocabulary| 16,384 tokens
-Word embeddings| Tied
-Attention bias| Disabled
-MLP bias| Disabled
-Parameter dtype| bfloat16
-Weight format| safetensors
-Number of tensors| 110
-Architecture
-PASITA uses a 12-layer decoder-only Transformer with a hidden dimension of 768 and an intermediate dimension of 2048.
-The attention mechanism uses Grouped Query Attention with 12 query heads and 4 key/value heads. This reduces the number of key/value projections and associated cache requirements while retaining multiple query heads.
-The model uses:
-- RMSNorm
-- SiLU activation
-- SwiGLU feed-forward blocks
-- Rotary Position Embeddings (RoPE)
-- A maximum sequence length of 2,048 tokens
-- Tied input and output embeddings
-- No attention or MLP biases
-The model is implemented using "LlamaForCausalLM".
-Tokenizer
-PASITA uses a custom Byte-Level BPE tokenizer with a vocabulary of 16,384 pieces.
-The tokenizer includes:
-<pad>
-<s>
-</s>
-<unk>
-<think>
-</think>
-The ByteLevel pre-tokenizer and decoder have been verified using exact round-trip tests.
-Measured tokenization efficiency is approximately:
-- 4.0 characters per token for Spanish and English
-- 2.0 characters per token for code
-Training Data
-PASITA was trained on approximately 80 million BPE tokens.
-The underlying corpus consists of human-produced Markdown obtained from public sources, including:
-- GoodWiki English
-- Spanish Wikipedia dumps
-- StackExchange Q&A
-- WikiHow
-- Cosmopedia
-The Markdown documents were deterministically degraded to produce noisy plain-text inputs.
-The resulting training data contains approximately 37% Spanish and 63% English.
-Dataset Splits
-Split| Tokens / prompts| Description
-SFT| 58M / 47k| Supervised fine-tuning
-DPO| 16M / 9k pairs| Preference optimization
-GRPO| 6M / 10.4k prompts| Group-relative optimization
-Benchmark| 2.5M / 2k| Held-out evaluation
-The benchmark split is disjoint from the training data by hash.
-Dataset quality controls included Markdown validity checks, hash and ROUGE-based deduplication, context-length validation, and multi-judge verification.
-Training
-PASITA v1 was trained using bfloat16 on NVIDIA B300 and A100 hardware.
-The training pipeline consisted of three stages.
-1. Supervised Fine-Tuning
-SFT was performed for two epochs.
-Reported final metrics:
-- Loss: approximately 0.09
-- Token accuracy: 98.4%
-2. Direct Preference Optimization
-DPO was performed for one epoch with:
-β = 0.1
-The reported final preference margin was approximately 4.2.
-The original DPO run contained partially spurious signals in some rejected examples. A subsequent DPO-v2 dataset was developed with more minimal edits and harder negatives.
-3. Group Relative Policy Optimization
-GRPO was performed for 550 steps using groups of four generations.
-The reward design focused primarily on structural correctness and content fidelity.
-During the original training process, the fidelity component of the reward exhibited noise related to the decoder configuration, while structural rewards were learned more consistently.
-Evaluation
-Evaluation was performed greedily on held-out data.
-Bench-500
-Metric| Score
-GFM validity| 0.958
-Faithfulness| 0.930
-Semantic faithfulness| 0.880
-Table correctness| 1.000
-Bench-1000
-Metric| Score
-GFM validity| 0.956
-Faithfulness| 0.927
-Semantic faithfulness| 0.888
-Table correctness| 1.000
-Domain Results
-Domain| Score
-Code| 0.965
-OCR / transcripts| 0.953
-Documents| 0.946
-Mathematics| 0.981
-Tables| 0.945
-HTML| 1.000
-Control| 0.190
-The control category is the principal known weakness of the current version.
-Inference
-PASITA expects a prompt beginning with "CONVIERTE A MARKDOWN", followed by the source text and the "### Markdown:" separator.
-Example:
-prompt = (
-    "CONVIERTE A MARKDOWN:\n"
-    + text
-    + "\n\n### Markdown:\n"
-)
-A recommended deterministic generation configuration is:
-generate(
-    do_sample=False,
-    max_new_tokens=int(input_length * 1.3 + 128)
-)
-The model is intended for deterministic transformation rather than conversational generation.
-Model Size and Runtime
-The model contains approximately 88 million parameters.
-The stored model weights occupy approximately:
-176,211,400 bytes
-or approximately 168 MB.
-The weights are stored natively in the safetensors format.
-The model uses bfloat16 parameters, while CPU inference can be performed using a float32-loaded model when required by the runtime configuration.
-Approximate memory usage for the weights is around 180 MB, excluding runtime allocations and KV-cache requirements.
-The model is therefore small enough to be deployed on CPU systems and on GPUs with at least 2 GB of memory, subject to runtime overhead and the selected inference configuration.
-Weight Integrity
-The primary model file is:
-final_model/model.safetensors
-SHA-256 prefix:
-664665e7ee2b0002
-Full hashes should be obtained from the corresponding release artifacts when verifying a downloaded model.
-Intended Use
-PASITA is designed for applications that require deterministic or near-deterministic conversion of unstructured text into Markdown.
-Potential applications include:
-- Plain-text document formatting
-- Markdown normalization pipelines
-- Documentation preprocessing
-- Text-to-Markdown conversion
-- Local document transformation
-- CPU-based Markdown processing
-PASITA is not designed or evaluated as a general-purpose conversational assistant.
-Known Limitations
-PASITA v1 has several known limitations.
-Content Preservation
-Although faithfulness is substantially stronger than its Markdown validity score alone would indicate, the model can still:
-- Truncate numerical values
-- Omit secondary information
-- Echo headings unnecessarily
-- Continue generating after completing the transformation
-For example, numerical content such as "23.3" may occasionally be reduced to "23".
-Short Inputs
-The current model is less reliable on very short inputs, particularly inputs consisting of only one to three lines.
-Strict Formatting Instructions
-The control benchmark remains substantially weaker than the other evaluated domains. PASITA should therefore not be assumed to reliably satisfy arbitrary, highly specific formatting constraints.
-Generation Termination
-Under greedy generation, PASITA can occasionally continue generating after the correct Markdown transformation has already been produced.
-Applications should therefore use an appropriate generation limit and explicitly handle end-of-sequence behavior.
-License
-PASITA v1 is released under the Apache License 2.0.
-Copyright © 2026 OpceanAI.
-The Apache License 2.0 permits use, reproduction, modification, distribution, and creation of derivative works subject to the terms and conditions of the license.
-The complete license text is provided in the ""LICENSE"" (LICENSE) file.
-The Apache License 2.0 applies to the PASITA project and its released model artifacts. Third-party datasets and source material used during training may be subject to separate licenses, attribution requirements, or other terms.
-Users intending to redistribute or commercially deploy systems trained on the underlying datasets should independently review the applicable licenses and attribution requirements, including the requirements associated with Wikipedia-derived material.
-Data Provenance
-The model weights were produced from a random initialization and were not initialized from a pretrained language model.
-Training data was derived from publicly available sources, including GoodWiki, Wikipedia, StackExchange, WikiHow, and Cosmopedia.
-The licensing status of the resulting model does not supersede the licensing requirements applicable to third-party source material.
-Project Status
-PASITA v1 should be considered an experimental specialized language model.
-The current results demonstrate that a compact model trained specifically for plain-text-to-Markdown transformation can achieve high content-fidelity scores on held-out evaluation data while remaining small enough for practical local inference.
-Future development is focused on improving:
-- Termination behavior
-- Control and instruction-following performance
-- Content preservation
-- Robustness on short inputs
-- Markdown structural correctness
-- Preference and reinforcement-learning data quality
-Summary
-PASITA v1 is an approximately 88M-parameter decoder-only Transformer trained from scratch specifically for plain-text-to-Markdown transformation.
-Its current Bench-1000 results are:
-GFM validity       95.6%
-Faithfulness       92.7%
-Semantic faith.    88.8%
-Table correctness 100.0%
-The model's primary strength is its ability to perform structured Markdown transformation while preserving most of the source information.
-Its primary weaknesses are strict control tasks, very short inputs, and occasional post-completion generation.
-PASITA is therefore best understood as a small, specialized text-transformation model, rather than a general-purpose LLM.

 language:
 - es
 - en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- markdown
+- text-to-markdown
+- faithful-generation
+- llama
+- from-scratch
+- spanish
+- english
+- ocr-postprocessing
+- rag
+- document-understanding
+- tiny-llm
+model-index:
+- name: PASITA
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      name: PASITA-bench-1000 (held-out, private)
+      type: custom
+    metrics:
+    - name: gfm_validity
+      type: accuracy
+      value: 0.956
+    - name: faithfulness
+      type: faithfulness
+      value: 0.927
+    - name: semantic_faithfulness
+      type: faithfulness
+      value: 0.888
+    - name: table_fidelity
+      type: accuracy
+      value: 1.0
 ---
+# PASITA v1 — plain text → Markdown (fiel)
+Modelo de lenguaje **decoder-only entrenado 100% desde cero** (sin modelo base), especializado en una sola tarea: convertir **texto plano en Markdown válido preservando la información**.
+> Comportamiento de compilador, no de chatbot: añade estructura, no inventa contenido.
+## Arquitectura (laboratorio)
+| Parámetro | Valor |
+|---|---|
+| Clase | `LlamaForCausalLM` (decoder-only, denso, sin MoE) |
+| Parámetros | **88,099,584** (~88M) en `bfloat16` |
+| Capas / hidden / FFN | 12 / 768 / 2048 (SwiGLU) |
+| Atención | GQA 12Q/4KV, head_dim 64, sin bias |
+| Posiciones | RoPE θ=100000, ctx 2048, RMSNorm ε=1e-5 |
+| Embeddings | atados (ahorra ~12.6M) |
+| Archivo | `model.safetensors` (176 MB, 110 tensores, sha `664665e7…`) |
+| Tokenizer | BPE byte-level propio 16k, decoder ByteLevel verificado (~4.0 chars/tok ES/EN) |
+| Especiales | `<pad> <s> </s> <unk> <think> </think>` |
+## Entrenamiento
+1. **SFT** 58M toks ×2 epochs — loss 0.09, acc 98.4%
+2. **DPO** β=0.1 — margen de preferencia 4.2
+3. **GRPO** 550+150 steps, G=4, rewards verificables (formato + fidelidad numérica + anti-overformat)
+4. Datos: 80M oro humano real degradado (Wikipedia ES/EN, StackExchange, WikiHow) + corpus v4 hasta 625M
+## Benchmark (held-out n=1000, greedy)
+| Global | GFM 0.956 · faith 0.927 · sem 0.888 · tablas 1.0 |
+|---|---|
+| code / ocr / docs / math / tables / html | 0.93 – 1.00 |
+| control (instrucciones estrictas) | 0.19 (limitación conocida) |
+## Uso
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+tok = AutoTokenizer.from_pretrained("OpceanAI/PASITA")
+model = AutoModelForCausalLM.from_pretrained("OpceanAI/PASITA", dtype="auto")
+prompt = "CONVIERTE A MARKDOWN:\n" + texto + "\n\n### Markdown:\n"
+ids = tok(prompt, return_tensors="pt", truncation=True, max_length=1024)
+out = model.generate(**ids, max_new_tokens=512, do_sample=False)
+print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
+```
+Régimen válido: documentos medios/largos (OCR, HTML pegado, actas, tutoriales). Frágil en inputs de 1-3 líneas.
+## Limitaciones
+Puede truncar cifras, omitir datos secundarios, emitir H1 ecoicos o continuar tras terminar. Recomendado: `max_new_tokens` adaptativo + beam + rerank por fidelidad.