---
license: apache-2.0
datasets:
- Shuu12121/python-treesitter-filtered-datasetsV2
- Shuu12121/javascript-treesitter-filtered-datasetsV2
- Shuu12121/ruby-treesitter-filtered-datasetsV2
- Shuu12121/go-treesitter-dedupe_doc-filtered-dataset
- Shuu12121/java-treesitter-dedupe_doc-filtered-dataset
- Shuu12121/rust-treesitter-filtered-datasetsV2
- Shuu12121/php-treesitter-filtered-datasetsV2
- Shuu12121/typescript-treesitter-filtered-datasetsV2
pipeline_tag: fill-mask
tags:
- code
- python
- java
- javascript
- typescript
- go
- ruby
- rust
- php
language:
- en
base_model:
- Shuu12121/CodeModernBERT-Crow-v1-Pre
---
# CodeModernBERT-Crow-v1.1🐦‍⬛

## Model Details

* **Model type**: Bi-encoder architecture based on ModernBERT
* **Architecture**:
  * Hidden size: 768
  * Layers: 12
  * Attention heads: 12
  * Intermediate size: 3,072
  * Max position embeddings: 8,192
  * Local attention window size: 128
  * RoPE positional encoding: θ = 160,000
  * Local RoPE positional encoding: θ = 10,000
* **Sequence length**: up to 2,048 tokens for code and docstring inputs during pretraining

## Pretraining

* **Tokenizer**: Custom BPE tokenizer trained for code and docstring pairs.
* **Data**: Functions and natural language descriptions extracted from GitHub repositories.
* **Masking strategy**: Two-phase pretraining.
  * **Phase 1: Random Masked Language Modeling (MLM)**  
    30% of tokens in code functions are randomly masked and predicted using standard MLM.
  * **Phase 2: Line-level Span Masking**  
    Inspired by SpanBERT, continued pretraining on the same data with span masking at line granularity:  
      1. Convert input tokens back to strings.  
      2. Detect newline tokens with regex and segment inputs by line.  
      3. Exclude whitespace-only tokens from masking.  
      4. Apply padding to align sequence lengths.  
      5. Randomly mask 30% of tokens in each line segment and predict them.

* **Pretraining hyperparameters**:
  * Batch size: 16
  * Gradient accumulation steps: 16
  * Effective batch size: 256
  * Optimizer: AdamW
  * Learning rate: 5e-5
  * Scheduler: Cosine
  * Epochs: 3
  * Precision: Mixed precision (fp16) using `transformers`