---
datasets:
- phonemetransformers/IPA-CHILDES
language:
- en
- eu
- zh
- da
- nl
- hr
- es
- et
- fa
- fr
- de
- hu
- is
- id
- ga
- it
- ja
- ko
- 'no'
- pt
- pl
- qu
- ro
- sr
- sv
- tr
- cy
---

# CHILDES IPA Tokenizers

Tokenizers for each language in [IPA-CHILDES](https://huggingface.co/datasets/phonemetransformers/IPA-CHILDES) used to train cross-lingual phoneme LLMs in our papers:
- [IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modeling](https://arxiv.org/abs/2504.03036)
- [BabyLM's First Words: Word Segmentation as a Phonological Probing Task](https://arxiv.org/abs/2504.03338)

Scripts for creating the tokenizers can be found [here](https://github.com/codebyzeb/childes-processor).
Scripts for training models using these tokenizers can be found [here](https://github.com/codebyzeb/PhonemeTransformers).

To load a tokenizer:
```python
from transformers import AutoTokenizer
dutch_tokenizer = AutoTokenizer.from_pretrained('phonemetransformers/ipa-childes-tokenizers', subfolder='Dutch')
```