pii-en-65m / tokenizer_config.json

Stage DataFog PII v0.1.0: epoch-2 checkpoint, frozen MCP bundle, provenance and licenses (part 2)

4b51830 verified 7 days ago

350 Bytes

	{
	"backend": "tokenizers",
	"cls_token": "[CLS]",
	"do_lower_case": false,
	"is_local": true,
	"local_files_only": true,
	"mask_token": "[MASK]",
	"model_max_length": 512,
	"pad_token": "[PAD]",
	"sep_token": "[SEP]",
	"strip_accents": null,
	"tokenize_chinese_chars": true,
	"tokenizer_class": "BertTokenizer",
	"unk_token": "[UNK]"
	}