README.md

---
title: Vernacular
emoji: 😻
colorFrom: gray
colorTo: purple
sdk: gradio
sdk_version: 6.16.0
python_version: '3.10'
app_file: app.py
pinned: false
license: mit
short_description: Translate games. Keep every character's voice
---

# Vernacular

Vernacular is a Gradio review tool and character-aware translation pipeline for
*An Elmwood Trail*, a narrative mystery mobile game, developed by our friends at **Techyonic** <https://www.techyonic.co/>. It translates source strings,
uses character voice wikis to rewrite dialogue in the speaker's style, and lets
a reviewer approve, reject, or edit every final line.

The current app workflow also supports direct uploads: users can upload `.docx`,
`.xlsx`, `.json`, `.txt`, or `.csv` files, build/update character wikis, download
or re-upload the wiki JSON bundle, then press **Refresh** to send the uploaded
file through the normal translation and tone workflow.

```
Uploaded doc/xlsx ─► converter/docs.py or converter/sheets.py ─► English_JSON/Uploaded/
Uploaded wiki JSON ───────────────────────────────────────────► character_wikis/*.md
Uploaded source ─► wiki builder ─► character_wikis/*.md ─► character_wikis.json

English_JSON/Uploaded/ ─► TranslateGemma ─► Gemma tone pass + character wiki
                                               │
                              translations/<lang>/ review records
                                               │
                         app.py review: approve / reject / edit
```

## Setup

Install the runtime dependencies:

```bash
pip install -r requirements.txt
```

The app uses Hugging Face Transformers in-process, not local llama.cpp or Ollama
servers. The model IDs are configured in `config.py`:

```python
TRANSLATE_MODEL_ID = "google/translategemma-12b-it"
TONE_MODEL_ID = "google/gemma-4-12B-it"
```

For Hugging Face Spaces, add an `HF_TOKEN` secret for reliable downloads and for
access to gated Google Gemma models. The account behind that token must have
accepted the model terms.

Run the app locally:

```bash
python app.py
```

## App Workflow

1. Open the Gradio app.
2. In **Character wiki builder**, upload one or more source files.
3. Optionally upload an existing `character_wikis.json`.
4. Click **Build / update wiki JSON**.
5. Download the generated wiki JSON if you want to keep or reuse it later.
6. Click the existing **Refresh** button.
7. Select the uploaded file from the file dropdown and review its translated
   strings as usual.

For `.docx` and `.xlsx` uploads, the app uses the same converter modules as the
original pack conversion:

| Upload type | Conversion path |
|---|---|
| `.docx` | `converter/docs.py` via `readLocalDoc()` |
| `.xlsx` | `converter/sheets.py` via `readLocalSheet()` |
| `.json`, `.txt`, `.csv` | Plain-text fallback wrapped as uploaded chat rows |

If a converted doc/sheet produces no translatable strings, the app falls back to
plain extracted text so the upload still enters the workflow.

## What Uploads Create

Uploaded files are stored in app-managed paths:

| Path | Purpose |
|---|---|
| `uploaded_wiki_sources/` | Plain text JSON used to build/update character wikis |
| `English_JSON/Uploaded/Filler Chats/` | Converted source JSON used by translation/review |
| `character_data/` | Character index entries pointing to uploaded files |
| `character_wikis/*.md` | Per-character markdown wikis used by the tone pass |
| `character_wikis/character_wikis.json` | Downloadable/uploadable wiki bundle |
| `translations/<lang>/Uploaded/Filler Chats/` | Review records produced after Refresh |

## Existing Batch Workflow

The original full-pack workflow is still available:

```bash
python convert.py
python build_character_data.py
python build_character_wikis.py
python -m pipeline.build_file_context
python -m pipeline.translate_pack
python app.py
python -m pipeline.export_pack
```

`pipeline.translate_pack` remains resumable and supports targeted runs:

```bash
python -m pipeline.translate_pack --dry-run
python -m pipeline.translate_pack --filter Initial/
python -m pipeline.translate_pack --stage translate
python -m pipeline.translate_pack --stage tone
```

Note: `build_character_wikis.py` is the older batch wiki builder and still uses
its original local Ollama path. The Gradio upload workflow builds wikis through
the in-process Hugging Face Gemma client instead.

## Changing The Target Language

Edit the target values in `config.py`:

```python
TARGET_LANG_CODE = "de"
TARGET_LANG_NAME = "German"
PACK_NAME = "German_JSON"
```

Then rerun the translation/review/export workflow. TranslateGemma supports many
language pairs, but the selected Hugging Face model and token access must be
available in the runtime.

## Repository Map

| Path | Purpose |
|---|---|
| `app.py` | Gradio app: wiki upload/build, refresh, review UI |
| `requirements.txt` | Space/local Python runtime dependencies |
| `config.py` | Target language, paths, Hugging Face model settings |
| `converter/` | `.docx` / `.xlsx` to structured JSON conversion |
| `English_JSON/` | Source-of-truth JSON files, including uploaded files |
| `character_data/` | Per-character file indexes |
| `character_wikis/` | Markdown wikis and downloadable wiki JSON bundle |
| `pipeline/clients.py` | In-process TranslateGemma and Gemma tone inference |
| `pipeline/translate_pack.py` | Batch translation/tone driver |
| `translations/<lang>/` | Review records used by the app |
| `pipeline/export_pack.py` | Exports approved/edited translations to final pack |