Update README.md

README.md

@@ -1,10 +1,11 @@
 ---
 license: gemma
+base_model: Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA-GGUF
+library_name: peft
+pipeline_tag: text-generation
 language:
 - en
 - ru
-base_model:
-- Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA-GGUF
 tags:
 - lora
 - roleplay
@@ -13,50 +14,115 @@ tags:
 - russian
 - gemma4
 - moe
-pipeline_tag: text-generation
-library_name: peft
+- conversational
 ---
 
-Atmospheric genre LoRA adapter primarily for English-language roleplay (also works in Russian). Adds a darker, more literary tone to the narrative. Designed exclusively to adjust style and response structure in "Dark Fantasy" RP. The adapter does not contain any plots or characters from the training data.
+# Dark-Goetia-26B-A4B-LoRA-v2
+
+Atmospheric genre LoRA adapter primarily for English-language roleplay (also works in
+Russian). Adds a darker, more literary tone to the narrative. Designed exclusively to
+adjust style and response structure in Dark Fantasy RP. The adapter contains no plots or
+characters from the training data.
 
-## GGUF Version
+**This is the general-purpose release.** Two experimental variants exist that split this
+adapter's target set in half — see [Related adapters](#related-adapters) at the bottom.
 
-Two GGUF files are provided:
+## Adapter versions
 
-- `goetia-book-lora-v2-main.gguf` — **recommended** — 2 epochs → stronger influence → use the recommended scales as listed above
-- `goetia-book-lora-v2-chk177.gguf` — 1 epoch → the influence is weaker and less pronounceduse → **1.3×** the scale from the table
+- `goetia-book-lora-v2-main` — 2 epochs → use the recommended scales below
+- `goetia-book-lora-v2-chk177` — 1 epoch → use **1.5×** the scale from the table
 
-Example: if you run `main` at 0.55, run `chk177` at 0.71.
+Example: if you run main at 0.55, run chk177 at 1.1.
 
-# Recommended Scale Settings (main ver.)
-| Scale | Effect (Test setup: SillyTavern + [Preset](https://github.com/SpicyMarinara/SillyTavern-Settings/blob/main/Marinara%27s%20Essentials/Preset/Marinara%27s%20Spaghetti%20Recipe.json) + 2000+ token character card) |
-|---|---|
+## Recommended scale settings (main ver.)
+
+Test setup: SillyTavern + [Marinara's preset](https://github.com/SpicyMarinara/SillyTavern-Settings/blob/main/Marinara%27s%20Essentials/Preset/Marinara%27s%20Spaghetti%20Recipe.json) + 2000+ token character card.
+
+**The right ceiling depends on what your card asks for.** Free-form prose RP tolerates
+much more than a card that demands structured output blocks (status trackers, World State
+sections, inner-thought blocks). Both regimes below are measured, not guessed.
+
+### Free-form prose RP — no strict formatting requirements
+
+| Scale | Effect |
+| --- | --- |
 | 0.1–0.2 | Effect barely noticeable |
-| 0.3–0.5 | Recommended (stable, from the least to the greatest influence) |
-| 0.7+ | Very strong influence (can start overpowering the base model) |
+| 0.2–0.3 | Minimal / partial influence |
+| **0.3–0.55** | **Recommended** — stable, controlled style |
+| 0.55+ | Very strong influence, can start overpowering the base model |
+
+Performs reliably in all RP scenarios at scales above 0.5. The darker literary tone
+becomes most pronounced in Dark Fantasy settings. Best used in full roleplay sessions
+(character card + system prompt); isolated short prompts show a weaker effect.
 
-**Note**: Performs reliably in all RP scenarios at scales above 0.5. The darker literary tone becomes most pronounced in Dark Fantasy settings. Best used in full roleplay sessions (SillyTavern with character card + system prompt). Isolated short prompts may show a weaker effect.
+### Cards with strict structured output
+
+If your card requires the model to emit formatted blocks every turn, the ceiling is
+considerably lower, and it differs by language:
+
+| Language | Safe | First observed failure |
+| --- | --- | --- |
+| English | up to **0.40** | above 0.43 (estimated) |
+| Russian | up to **0.37** | 0.40 |
+
+Above these, the model keeps the style but starts silently dropping the formatted blocks.
+Russian is measurably more fragile than English at identical adapter strength — if you
+run bilingual sessions, tune to the Russian number.
+
+**Note on scaling:** `alpha/r = 2` is stored in the GGUF metadata, and llama.cpp multiplies
+by it *on top of* your scale. All numbers above assume a loader that respects that. If
+yours ignores the metadata, halve everything.
+
+## What's actually in the adapter
+
+For anyone merging this or building on it — the trained delta isn't distributed evenly
+across attention. Frobenius norms of the effective delta `ΔW = (B·A)·(alpha/r)`, aggregated
+over layers:
+
+| Projection | ‖ΔW‖ |
+| --- | --- |
+| `o_proj` | 1.563 |
+| `q_proj` | 1.314 |
+| `v_proj` | 1.049 |
+| `k_proj` | 0.828 |
+| **OV total** | **1.882** |
+| **QK total** | **1.553** |
+| Full delta | 2.440 |
+
+So roughly **61% OV / 39% QK** by squared norm. That split turns out to matter: the OV
+half carries almost all of the style, while both halves independently contribute to
+formatting degradation — OV about 3× more readily per unit of weight change. The
+structured-output ceilings in the table above are set by the OV component alone.
 
 ## Training
 
 - **Base:** Naphula/Goetia-26B-A4B-v1.3-Absolute-Heretic-ARA (Gemma 4 MoE, abliterated)
-- **Method:** QLoRA, attention-only (`q/k/v/o_proj`, 30 layers of the text tower), r=32, alpha=64, 2 epochs, completion-only loss
-- **Data:** Novel broken down into scenes → scenes rewritten by a neural network into a synthetic RP dataset (actions and internal thoughts preserved, original author’s prose style removed). Dataset contains both English and Russian samples, with primary focus on English.
+- **Method:** QLoRA 4-bit, attention-only (`q/k/v/o_proj`), all 30 layers of the text tower,
+  r=32, alpha=64, 2 epochs, completion-only loss
+- **Scope:** **115** attention projections, not 120 — layers 5, 11, 17, 23 and 29 are
+  global-attention layers with no `v_proj` at all. 22,978,560 trainable params (0.089%).
+- **Data:** Novel broken down into scenes → scenes rewritten by a neural network into a
+  synthetic RP dataset (actions and internal thoughts preserved, original author's prose
+  style removed). Contains both English and Russian samples, English-primary.
 
 ## Limitations
 
 - **18+.** Base model is abliterated, training data includes adult and dark content.
-- Primarily optimized for English. Works in Russian as well, but the effect is generally weaker.
+- Primarily optimized for English. Works in Russian, but the style effect is generally
+  weaker *and* instruction-following is more fragile there — see the scale tables.
+- Structured-output thresholds were measured with one generation per setting at a single
+  context depth. Treat them as calibrated starting points, not hard constants.
 - Inherits Gemma license terms.
 
-## ⚠️ Current Limitations & Known Issues (v2)
-
-> **Note:** I am using this project as a hands-on learning experience, so trial and error are an inevitable part of the process!
+## Related adapters
 
-This adapter was trained across **all Transformer layers (0–31)** using **Attention-only targets** (`q_proj`, `k_proj`, `v_proj`, `o_proj`), omitting FFN/MLP blocks. Evaluation and user testing revealed the following trade-offs:
+v3 splits this adapter's 115 targets into two non-overlapping halves, trained separately
+on identical data with identical hyperparameters and seed:
 
-* **Lower-Layer Interference:** Fine-tuning initial syntax layers (0–11) forces rigid structural constraints, occasionally stripping away the base model's native linguistic fluidity and stylistic richness.
-* **Lack of FFN/MLP Adaptation:** Because the associative Feed-Forward blocks (`gate_proj`, `up_proj`, `down_proj`) were frozen, the model relies solely on attention routing to convey style. This can lead to slightly dry, repetitive phrasing or overly generic vocabulary choices in complex roleplay/narrative scenarios.
-* **Over-Formatting Bias:** While XML tag adherence and structural tracking are near-perfect, the narrative prose itself can feel overly sanitized unless the LoRA weight is scaled down (0.4 - 0.6).
+- [**v3-A**](https://huggingface.co/SubMaroon/Dark-Goetia-26B-A4B-LoRA-v3-A) — `v_proj` + `o_proj`
+  (55 modules). The style channel on its own, without the QK component. Experimental;
+  useful if your card demands strict formatting and v2 keeps dropping it.
+- [**v3-B**](https://huggingface.co/SubMaroon/Dark-Goetia-26B-A4B-LoRA-v3-B) — `q_proj` + `k_proj`
+  (60 modules). Research artifact — weak style, published for reproducibility.
 
-> **Planned Fix for v3:** Restricting LoRA targeting exclusively to mid-to-upper layers (12–31) to preserve prose depth while maintaining structural stability.
+A + B = exactly this adapter's target set and parameter budget.