File Formats

Formats are contracts. CSV lies, JSON flexes, Parquet tells the truth — choose per workload.

▶ Watch this reel

What you'll learn

CSV pitfalls
JSON & JSONL
Parquet
Parquet partitioning
Arrow / PyArrow
Parquet vs DB vs Vector

Remember this

CSV: explicit encoding/dtypes always; JSON for APIs, JSONL for pipelines (streamable, corrupt-line-safe)
Parquet: columnar + typed + compressed; partition by filtered columns; Arrow = zero-copy memory glue
Three layers: DB for mutating truth, Parquet for bulk analytics, vector store for meaning

CSV pitfalls

No types, encoding landmines (utf-8 vs latin-1), delimiter variance, quoting traps.
Prod rule: explicit encoding, dtype, parse_dates — never infer silently.

JSON & JSONL

JSON: APIs, config, documents (nested, flexible).
JSONL: one JSON object per line — streamable, appendable, corrupt-line-safe.
Compress text with gzip (.jsonl.gz).

Parquet

Columnar + typed schema + compressed (snappy/zstd). ~10x smaller, 10-50x faster analytics vs CSV.
df.to_parquet(...) / pd.read_parquet(...).

Partitioning

Folder tree encodes column values (year=2026/month=03/) → engine prunes folders.
Partition by filtered low-cardinality columns; avoid the small-files problem (target ~100MB-1GB files).

Arrow / PyArrow

Standard in-memory columnar layout → zero-copy between pandas/Polars/DuckDB/Spark.
Powers pandas' Parquet reader; pa.Table.from_pandas(df) converts in μs.

Parquet vs DB vs Vector

Job	Tool
Mutating transactional truth	Postgres / SQL Server
Bulk immutable analytics	Parquet (+ DuckDB)
Semantic similarity at scale	Vector store (Stage 4)

Code: Formats in practice — CSV in, Parquet out, partitioned

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

# --- CSV: parse explicitly (no silent inference in prod) ----------
dtypes = {"amount": "float64", "status": "category"}
df = pd.read_csv("events.csv", encoding="utf-8", dtype=dtypes,
                 parse_dates=["created"])

# --- Parquet: typed + compressed -----------------------------------
df.to_parquet("events.parquet", index=False, compression="snappy")
# Typical result: 10x smaller than CSV, schema embedded, 10-50x faster reads

# --- Partitioned dataset: folders as an index -----------------------
table = pa.Table.from_pandas(df)
pq.write_to_dataset(
    table,
    root_path="events_lake",
    partition_cols=["status"],       # year/month for time series
)
# → events_lake/status=P1/0000.parquet  events_lake/status=P2/...
# DuckDB/pandas read only the partitions a filter touches

# --- JSONL: the pipeline interchange --------------------------------
with open("embeddings.jsonl", "w", encoding="utf-8") as f:
    for row in df.itertuples():
        f.write(json.dumps({"id": row.id, "text": row.text}) + "\n")
# Read back with constant memory:
# for line in open("embeddings.jsonl"): rec = json.loads(line)