Buckets:

hf-doc-build/doc-dev / hub /pr_2549 /en /datasets-polars-file-formats.md
HuggingFaceDocBuilder's picture
|
download
raw
1.23 kB
# File formats
Polars supports the following file formats when reading from Hugging Face:
- [Parquet](https://docs.pola.rs/api/python/stable/reference/api/polars.read_parquet.html)
- [CSV](https://docs.pola.rs/api/python/stable/reference/api/polars.read_csv.html)
- [JSON Lines](https://docs.pola.rs/api/python/stable/reference/api/polars.read_ndjson.html)
The examples below show the default settings only. Use the links above to view all available parameters in the API reference guide.
# Parquet
Parquet is the preferred file format as it stores the schema with type information within the file. This avoids any ambiguity with parsing and speeds up reading. To read a Parquet file in Polars, use the `read_parquet` function:
```python
pl.read_parquet("hf://datasets/roneneldan/TinyStories/data/train-00000-of-00004-2d5a1467fff1081b.parquet")
```
# CSV
The `read_csv` function can be used to read a CSV file:
```python
pl.read_csv("hf://datasets/lhoestq/demo1/data/train.csv")
```
# JSON
Polars supports reading new line delimited JSON — also known as [json lines](https://jsonlines.org/) — with the `read_ndjson` function:
```python
pl.read_ndjson("hf://datasets/proj-persona/PersonaHub/persona.jsonl")
```

Xet Storage Details

Size:
1.23 kB
·
Xet hash:
e408c153d285c713d43aeeb427914cabeed6cf23a7b0f4d19ea2190307187914

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.