README.md

---
license: mit
---

<div align="center">
  <img src="https://raw.githubusercontent.com/BGI-HangzhouAI/Genos/main/images/Genos_model.png" width="100%" />
</div>

# Genos

Genos, as a foundational model in the field of human genomics, trained on hundreds of high-quality genome reference data, has achieved the ability to contextually model human genome sequences up to millions of base pairs. Through single-base resolution learning, this model possesses the capability to identify hidden deep sequence patterns and functional features within genomes, providing scientists with a new research method that connects genetic information with life activities.

For instructions, details, and examples, please refer to the [Genos GitHub](https://github.com/BGI-HangzhouAI/Genos).

Below are the data volume of our model training and related parameters.

<table align="center">
  <tr>
    <th>Model Specification</th>
    <th>Genos 1.2B</th>
    <th>Genos 10B</th>
  </tr>

  <!-- Model Scale category title - span 3 columns -->
  <tr>
    <td colspan="3" align="center"><b>Model Scale</b></td>
  </tr>
  <tr>
    <td>Total Parameters</td>
    <td>1.2B</td>
    <td>10B</td>
  </tr>
  <tr>
    <td>Activated Parameters</td>
    <td>0.33B</td>
    <td>2.87B</td>
  </tr>
  <tr>
    <td>Trained Tokens</td>
    <td>1600 B</td>
    <td>2200 B</td>
  </tr>

  <!-- Architecture category title - span 3 columns -->
  <tr>
    <td colspan="3" align="center"><b>Architecture</b></td>
  </tr>
  <tr>
    <td>Architecture Type</td>
    <td>MoE</td>
    <td>MoE</td>
  </tr>
  <tr>
    <td>Number of Experts</td>
    <td>8</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Selected Experts per Token</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Number of Layers</td>
    <td>12</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Attention Hidden Dimension</td>
    <td>1024</td>
    <td>4096</td>
  </tr>
  <tr>
    <td>Number of Attention Heads</td>
    <td>16</td>
    <td>16</td>
  </tr>
  <tr>
    <td>MoE Hidden Dimension (per Expert)</td>
    <td>4096</td>
    <td>8192</td>
  </tr>
  <tr>
    <td>Vocabulary Size</td>
    <td>128 (padded)</td>
    <td>256 (padded)</td>
  </tr>
  <tr>
    <td>Context Length</td>
    <td>up to 1M</td>
    <td>up to 1M</td>
  </tr>
</table>


Genos 1.2B and 10B checkpoints are available here:

- [Genos-1.2B](https://huggingface.co/BGI-HangzhouAI/Genos-1.2B)
- [Genos-10B](https://huggingface.co/BGI-HangzhouAI/Genos-10B) 

We also provide checkpoints trained under the [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) framework:

- [Genos-Megatron-1.2B](https://huggingface.co/BGI-HangzhouAI/Genos-Megatron-1.2B) 
- [Genos-Megatron-10B](https://huggingface.co/BGI-HangzhouAI/Genos-Megatron-10B)