--- language: - en license: apache-2.0 tags: - secret-detection - token-classification - ner - security - api-keys - credentials - distilbert library_name: transformers pipeline_tag: token-classification datasets: - custom metrics: - f1 - precision - recall model-index: - name: distilbert-secret-masker results: - task: type: token-classification name: Secret Detection dataset: name: SecretMask v2 (600 test examples) type: custom metrics: - type: f1 value: 0.52 name: F1 Score verified: true - type: precision value: 0.82 name: Precision verified: true - type: recall value: 0.38 name: Recall verified: true base_model: distilbert-base-uncased --- > **New recommended model:** [`AndrewAndrewsen/distilbert-secret-masker-v3.3a-rs`](https://huggingface.co/AndrewAndrewsen/distilbert-secret-masker-v3.3a-rs) > This repository is the previous **v2** release, kept for reproducibility. v3.3a-RS has > different training, a stronger real-code evaluation, and a frozen sliding-window inference > path — see its card for benchmarks. # DistilBERT Secret Masker (Fast Expert) [![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![Model Size](https://img.shields.io/badge/Size-265MB-green.svg)](https://huggingface.co/AndrewAndrewsen/distilbert-secret-masker) [![Inference Speed](https://img.shields.io/badge/Latency-6ms-brightgreen.svg)](https://huggingface.co/AndrewAndrewsen/distilbert-secret-masker) **Fast Expert** model for SecMask MoE system - specialized for rapid secret detection in short to medium-length texts (≤512 tokens). --- ## 🎯 Overview Fine-tuned DistilBERT model for detecting and classifying secrets (API keys, tokens, credentials) in text using Named Entity Recognition (NER). Serves as the **Fast Expert** in the [SecMask Mixture of Experts architecture](https://github.com/AndrewAndrewsen/secmask), handling **92.7%** of inference requests with ~6ms latency. ### Key Features ✅ **High Speed**: 11ms P50 latency on CPU ✅ **High Precision**: 82% (NER-only), **92.3% with filters** ✅ **Production Ready**: Handles 92.7% of real-world cases ✅ **Lightweight**: 265MB (66M parameters) ✅ **Multi-Secret Types**: AWS keys, GitHub tokens, JWTs, API keys, PEM blocks, K8s secrets > **Production Performance**: When combined with post-processing filters (PEM blocks, K8s secrets, pattern matching), achieves **92.3% precision, 80% recall, F1: 0.857**. The NER model alone achieves 82% precision and 38% recall. See [comprehensive benchmarks](https://github.com/AndrewAndrewsen/secmask/blob/main/BENCHMARK_RESULTS.md) for details. > **Recommended Configuration**: Fast Expert + Filters (this model with post-processing) is the recommended production setup, outperforming Full MoE configurations. See [Configuration Guide](https://github.com/AndrewAndrewsen/secmask/blob/main/CONFIGURATION_GUIDE.md) for usage recommendations. ### Detected Secret Types | Secret Type | Example Pattern | F1 Score | | ----------------------------- | --------------------------------- | -------- | | **AWS Access Keys** | `AKIA...` | 0.92 | | **GitHub Personal Tokens** | `ghp_...`, `gho_...` | 0.88 | | **JWT Tokens** | `eyJ0eXAiOiJKV1QiLCJhbGc...` | 0.85 | | **Generic API Keys** | `sk-proj-...`, `api_key=...` | 0.79 | | **PEM Certificate Blocks** | `-----BEGIN PRIVATE KEY-----` | 0.95 | | **Kubernetes Secrets (data)** | `kind: Secret` → `data:` values | 0.81 | | **Database Credentials** | Connection strings with passwords | 0.74 | --- ## 🚀 Quick Start ### Installation ```bash pip install transformers torch ``` ### Basic Usage **Standalone (Direct):** ```python from transformers import pipeline # Load model classifier = pipeline( "token-classification", model="AndrewAndrewsen/distilbert-secret-masker", aggregation_strategy="simple" ) # Detect secrets text = "My API key is sk-proj-1234567890abcdefghijklmnopqrstuvwxyz" results = classifier(text) print(results) # [{'entity_group': 'SECRET', 'score': 0.95, 'word': 'sk-proj-1234567890abcdefghijklmnopqrstuvwxyz', ...}] ``` **Recommended (via SecMask MoE):** ```python # Clone SecMask repo # git clone https://github.com/AndrewAndrewsen/secmask.git from infer_moe import mask_text_moe masked = mask_text_moe( "My GitHub token is ghp_1234567890abcdefghijklmnopqrstuvwxyz", fast_model_dir="AndrewAndrewsen/distilbert-secret-masker", tau=0.80, routing_mode="heuristic" ) print(masked) # "My GitHub token is [SECRET]" ``` ### Command Line (via SecMask) ```bash # Clone repo git clone https://github.com/AndrewAndrewsen/secmask.git cd secmask # Mask secrets python infer_moe.py \ --text "AWS key: AKIAIOSFODNN7EXAMPLE" \ --fast-model AndrewAndrewsen/distilbert-secret-masker \ --routing-mode heuristic \ --tau 0.80 # Output: AWS key: [SECRET] ``` --- ## 📊 Performance ### Secret Detection Metrics | Metric | NER Only | With Filters (Recommended) | | --------------- | -------- | -------------------------- | | **F1 Score** | 0.52 | **0.857** | | **Precision** | 82% | **92.3%** | | **Recall** | 38% | **80.0%** | | **P50 Latency** | 11ms | 11ms | | **P90 Latency** | 14ms | 14ms | | **P99 Latency** | 17ms | 17ms | | **Throughput** | 84 req/s | 84 req/s (CPU) | > **Note**: NER-only metrics measured at τ=0.80. Production systems combine NER with post-processing filters (PEM blocks, K8s secrets, pattern matching) to achieve 92.3% precision and 80% recall. Post-processing adds no latency overhead. See [BENCHMARK_RESULTS.md](https://github.com/AndrewAndrewsen/secmask/blob/main/BENCHMARK_RESULTS.md) for comprehensive benchmarks. ### When This Model Is Used (MoE Routing) The router selects this **Fast Expert** when: - Token count ≤ 512 - No multi-line structures (PEM blocks, K8s YAML) - Simple text patterns - **Coverage: 92.7%** of real-world requests > **Note**: The recommended production configuration is **Fast Expert + Filters** alone (without the Long Expert). This achieves better results than Full MoE. See [Configuration Guide](https://github.com/AndrewAndrewsen/secmask/blob/main/CONFIGURATION_GUIDE.md) for details. --- ## 🏗️ Model Details ### Architecture - **Base Model**: [`distilbert-base-uncased`](https://huggingface.co/distilbert-base-uncased) (66M params, Apache 2.0) - **Task**: Token Classification (NER) - **Max Sequence Length**: 512 tokens - **Label**: `B-SECRET`, `I-SECRET`, `O` (BIO tagging) ### Training Details - **Dataset**: Custom SecretMask v2 (6,000 training examples) - **Optimizer**: AdamW (lr=5e-5) - **Epochs**: 3 - **Batch Size**: 16 - **Hardware**: GPU (NVIDIA A100 or equivalent) - **Training Time**: ~30 minutes ### Evaluation Evaluated on 600 held-out examples from SecretMask v2 test set: ``` Precision: 0.82 Recall: 0.38 F1: 0.52 Support: 1,021 secret tokens ``` **Key Insights**: - **High Precision (82%)**: Very low false positive rate - safe for production - **Lower Recall (38%)**: Misses some secrets when used standalone - **Production Strategy**: Combine with deterministic filters (see `filters.py`) for PEM blocks, K8s secrets, and AWS patterns to achieve >90% coverage - **Threshold Tuning**: Lower τ from 0.80 to 0.50 for higher recall (trade-off: more false positives) --- ## 💡 Use Cases ### Production Applications 1. **Pre-Commit Hooks** - Prevent secrets in git commits 2. **CI/CD Pipelines** - Scan code before deployment 3. **Log Sanitization** - Remove secrets from application logs 4. **API Response Filtering** - Mask secrets in debug output 5. **Documentation Cleanup** - Sanitize before open-sourcing 6. **Security Audits** - Scan codebases for exposed credentials ### Example: Pre-Commit Hook ```python # .git/hooks/pre-commit from transformers import pipeline classifier = pipeline("token-classification", model="AndrewAndrewsen/distilbert-secret-masker") for file in staged_files: content = read_file(file) secrets = classifier(content) if secrets: print(f"❌ Secret detected in {file}!") exit(1) ``` See [SecMask Examples](https://github.com/AndrewAndrewsen/secmask/blob/main/EXAMPLES.md) for more. --- ## ⚠️ Limitations ### Known Issues 1. **Token Limit**: Cannot handle texts >512 tokens (use Longformer expert) 2. **English Only**: Trained on English text 3. **False Negatives**: ~25% recall means some secrets may be missed 4. **Context Sensitivity**: May struggle with unusual formatting 5. **Novel Patterns**: May miss new secret types not in training data ### Not Suitable For ❌ Non-English text ❌ Binary data or encrypted content ❌ Images/PDFs (extract text first) ❌ Very long documents (use [longformer-secret-masker](https://huggingface.co/AndrewAndrewsen/longformer-secret-masker)) ❌ Real-time streaming (consider batching) ### Recommended Mitigations - **Combine with filters**: Use deterministic filters for PEM blocks, K8s secrets (see [SecMask filters](https://github.com/AndrewAndrewsen/secmask/blob/main/filters.py)) - **Adjust threshold**: Lower `tau` for higher recall (more false positives) - **Use MoE system**: Automatic routing to appropriate expert - **Add regex patterns**: Supplement with custom patterns for your use case --- ## 📜 License & Attribution ### Model License **Apache 2.0** (inherited from `distilbert-base-uncased`) ### Base Model Attribution This model is fine-tuned from: - **Model**: [`distilbert-base-uncased`](https://huggingface.co/distilbert-base-uncased) - **Authors**: Hugging Face - **License**: Apache 2.0 - **Citation**: ``` @inproceedings{sanh2019distilbert, title={DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter}, author={Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas}, booktitle={NeurIPS EMC^2 Workshop}, year={2019} } ``` ### SecMask Code License The SecMask inference code and training scripts are licensed under **MIT**. See [GitHub repo](https://github.com/AndrewAndrewsen/secmask/blob/main/LICENSE). --- ## 🔗 Related Models | Model | Size | Max Tokens | Latency | Use Case | | ------------------------------------------------------------------------------------------------------------ | ----- | ---------- | ------- | ------------------------- | | **[distilbert-secret-masker](https://huggingface.co/AndrewAndrewsen/distilbert-secret-masker)** (this model) | 265MB | 512 | 6ms | Short texts, fast routing | | **[longformer-secret-masker](https://huggingface.co/AndrewAndrewsen/longformer-secret-masker)** | 592MB | 2048 | 12ms | Long documents, configs | | **[secretmask-gate](https://huggingface.co/AndrewAndrewsen/secretmask-gate)** | 12KB | N/A | +0.2ms | Learned MoE routing | --- ## 📚 Resources - **GitHub Repository**: [AndrewAndrewsen/secmask](https://github.com/AndrewAndrewsen/secmask) - **Documentation**: [README](https://github.com/AndrewAndrewsen/secmask/blob/main/README.md) - **Benchmarks**: [BENCHMARKS.md](https://github.com/AndrewAndrewsen/secmask/blob/main/BENCHMARKS.md) - **Examples**: [EXAMPLES.md](https://github.com/AndrewAndrewsen/secmask/blob/main/EXAMPLES.md) - **Deployment**: [DEPLOYMENT.md](https://github.com/AndrewAndrewsen/secmask/blob/main/DEPLOYMENT.md) --- ## 🤝 Contributing Issues and contributions welcome! See [CONTRIBUTING.md](https://github.com/AndrewAndrewsen/secmask/blob/main/CONTRIBUTING.md). --- **Developed by**: Anders Andersson ([@AndrewAndrewsen](https://huggingface.co/AndrewAndrewsen)) **Part of**: [SecMask MoE System](https://github.com/AndrewAndrewsen/secmask)