# EVR-1 Maano-8b-Instruct: Benchmark Results

**Date:** 2026-03-09
**Model:** Meta Llama 3.1 8B Instruct, EVR-1 quantized
**Hardware:** RTX 6000 Ada (GPU offload, -ngl 99)

## Model Tested

| Model | File | Size | Notes |
|-------|------|------|-------|
| EVR-1 Instruct | evr-llama-3.1-8b-instruct.gguf | 3.93 GiB | EVR-1 3-bit compression |

## Test 1: Coherence (5 continuation-style prompts, 500 + 1000 tokens)

**Setup:** 5 continuation-style prompts, generated at both 500 and 1000 tokens. Temperature 0, no repeat penalty, `--ignore-eos`. Repetition measured by 4-gram repetition rate (rep4). Lower is better.

### Summary Table

| Model | Size | rep4 @ 500 | rep4 @ 1000 |
|-------|------|-----------|-------------|
| **EVR-1 Instruct** | **3.93 GiB** | **2.77%** | **9.66%** |

### Per-Prompt Results (EVR-1 Instruct)

| Prompt | rep4 @ 500 | rep4 @ 1000 |
|--------|-----------|-------------|
| Nuclear fusion in stars | 1.17% | 9.44% |
| Global financial crisis of 2008 | 5.08% | 12.14% |
| General relativity | 0.49% | 0.75% |
| Human immune system | 0.80% | 6.89% |
| Quantum computing | 6.30% | 19.09% |

### Perplexity

| Model | Size | Perplexity (wikitext-2, ctx=512) |
|-------|------|--------------------------------|
| **EVR-1 Instruct** | **3.93 GiB** | **7.37 ± 0.05** |

### Sample Outputs (Coherence Tests)

#### "Albert Einstein's theory of general relativity describes" (500 tokens, 0.49% rep4)
```
Albert Einstein's theory of general relativity describes gravity as the curvature of
spacetime caused by the presence of mass and energy. According to this theory, the
curvature of space and time around a massive object such as the Earth causes objects
to fall towards it, which we experience on our planet.

The curvature caused by a massive object is not limited to its surface. The curvature
extends into space and time, creating a region around it known as its gravitational
field. The strength and direction of this field depend on how much mass is present
in that region.

The gravitational field is not just limited to objects with mass; it also affects light.
According to general relativity, light follows curved paths in spacetime around massive
objects.
```
*(continues coherently for full 500 tokens - discusses gravitational lensing, Einstein rings, arcs, multiple images)*

#### "The process of nuclear fusion in stars begins when" (500 tokens, 1.17% rep4)
```
The process of nuclear fusion in stars begins when hydrogen atoms in the core of the
star are heated to extremely high temperatures. At these temperatures, the protons in
the hydrogen nuclei are stripped of their electrons, leaving only protons. These protons
then begin to fuse together to form helium nuclei. This process releases a tremendous
amount of energy, which is what powers a star.

The process begins when hydrogen nuclei are heated to a temperature at which they can
overcome their mutual repulsion and fuse together. This temperature is known as the
"fusion temperature." Once this temperature is reached, nuclear fusion can begin.
```
*(continues for full 500 tokens - transitions to discussing nuclear fission, which is a topic drift but not repetitive)*

---

## Test 2: Chat Quality Samples

### Prompt: "hello, are you there?"
```
Hello! Yes, I'm here. How can I assist you today?
```

### Prompt: "What are the main causes of the French Revolution?"
```
The French Revolution, which lasted from 1789 to 1792 and 1799 to 1804, was a complex
event with multiple causes. However, some of the main contributing factors include:

1. Financial Crisis: France was deeply in debt from its involvement in American
Revolutionary War and other wars. The burden of taxation fell on the common people,
who were already facing economic hardship. The financial crisis exacerbated social
and economic issues.

2. Social Inequality: The French nobility held a significant amount of power over
commoners through their control over land and government positions. This created a
sense of discontent among the lower classes who felt they were being oppressed.

3. Enlightenment Ideas: The ideas presented by Enlightenment thinkers such as
Rousseau, Voltaire, Diderot encouraged people to think critically about their
government and seek change.

4. Food Shortages: A series of poor harvests led to widespread hunger and starvation.
This further exacerbated social unrest, as people began to blame the government for
the shortages.

5. Absolute Monarchy: The rule of King Louis XVI was seen as tyrannical by many.
His attempts to reform the tax system and other aspects of government were met with
resistance from those who saw his efforts as an attempt to tighten his control rather
than alleviate their suffering.

6. The American Revolution's Influence: The American Revolution inspired the French
people with its ideas of liberty, equality, and fraternity.
```

### Prompt: "Explain quantum computing in detail"
```
Quantum computing is a type of computing that uses the principles of quantum mechanics
to perform calculations on data. It's a new paradigm for computation that's fundamentally
different from classical computers.

Qubits: A qubit is a two-state quantum system that represents information in the form
of a quantum bit. Qubits can exist in states like 0, 1, or both simultaneously, thanks
to superposition.

Superposition: In quantum mechanics and computing superposition means that a qubit
exists at both states simultaneously (0 and/or 1) until it's observed or measured.

Entanglement: Entanglement is when two qubits become connected in such a way they can't
exist independently. When something happens to one entangled particle, the other particle
instantly responds, no matter how far apart they are.

Quantum Gates: Quantum gates are the quantum equivalent of logic gates in digital
computing. Types include: Hadamard Gate, Pauli X and Z Gates, CNOT Gate, Swap Gates.

Current Challenges: Quantum Noise, Scalability, Quantum Error Correction, Quantifying
Quantum Advantage.
```
*(Chat-formatted prompt, 1000 tokens. Note: The coherence test above used a different methodology - continuation-style prompts with `--ignore-eos` - which produces different repetition rates.)*

---

## Performance

| Metric | Value |
|--------|-------|
| Prompt eval | ~37 tokens/sec |
| Generation | ~34 tokens/sec |
| Model load | ~7-16 sec |
| VRAM usage | ~3.7 GiB (model) + KV cache |

*RTX 6000 Ada, CUDA 12, -ngl 99*

### Apple Silicon (Mac Mini M4, Metal)

| Metric | Value |
|--------|-------|
| Prompt eval | ~9 tokens/sec |
| Generation | ~8 tokens/sec |
| Model load | ~6 sec |
| Memory usage | ~4.0 GiB (fits in 16 GB unified memory) |

*Apple M4, Metal backend, ARM64 static binary with LTO*