Understanding Parquet File Structure: Row Groups, Columns, and Metadata

Apache Parquet’s performance does not come from magic — it comes from a carefully designed physical layout that enables query engines to read the minimum amount of data necessary. Understanding this layout helps you write faster queries, choose better compression settings, and debug issues when they arise.

This article dissects the internal structure of a Parquet file from the top level down to individual pages and encoding schemes — and shows how you can inspect every layer yourself.

High-Level Structure

Every Parquet file follows the same layout:

+-----------------------------+
|        Magic Number         |  4 bytes: "PAR1"
+-----------------------------+
|        Row Group 0          |
|  Column Chunk 0.0           |
|  Column Chunk 0.1           |
|  Column Chunk 0.2           |
|  ...                        |
+-----------------------------+
|        Row Group 1          |
|  Column Chunk 1.0           |
|  Column Chunk 1.1           |
|  Column Chunk 1.2           |
|  ...                        |
+-----------------------------+
|           ...               |
+-----------------------------+
|       File Metadata         |
+-----------------------------+
|    Footer Length (4 bytes)   |
+-----------------------------+
|        Magic Number         |  4 bytes: "PAR1"
+-----------------------------+

The file starts and ends with the magic bytes PAR1. The actual data lives in row groups in the middle. The file metadata (the footer) sits at the end, right before the closing magic number.

A reader always starts by reading the last 8 bytes of the file to find the footer length, then reads the footer to learn the schema, row group locations, and column statistics. Only then does it read the actual data — and only the parts it needs.

Row Groups

A row group is a horizontal partition of the data. If your file contains 10 million rows and uses a row group size of 1 million, the file will have 10 row groups.

Purpose

Row groups serve several critical functions:

1. Parallelism: Multiple threads or cores can process different row groups simultaneously.
2. Memory management: A query engine can process one row group at a time, keeping memory usage bounded.
3. Predicate pushdown: Each row group carries column statistics (min, max, null count). If a query filters on WHERE date > '2025-06-01' and a row group’s max date is 2025-03-15, the engine skips that entire row group without reading any data.

Seeing Row Groups in Practice

You do not have to take row groups on faith. In Parquet Explorer, the metadata inspector shows you the row group breakdown for any file you load: how many row groups exist, how many rows each contains, and the compression codec and column-level statistics (min/max values, null counts, distinct counts) for each column chunk. This makes it tangible — you can see exactly which row groups a query would skip based on your filter conditions.

Sizing

The default row group size varies by writer:

• Apache Spark: 128 MB (compressed)
• PyArrow: 64 MB or by row count
• DuckDB: Tuned automatically

Trade-offs:

• Smaller row groups: Better predicate pushdown granularity, lower memory per group, but more metadata overhead and slightly worse compression.
• Larger row groups: Better compression ratios and less metadata overhead, but higher memory requirements and coarser predicate pushdown.

For most workloads, the default is fine. If you are optimizing, target row groups between 50 MB and 256 MB.

Column Chunks

Within each row group, data is stored by column. A column chunk contains all the values for one column within one row group.

For example, in a table with columns (id, name, age, city) and 2 row groups, the file contains 8 column chunks:

Row Group 0: [id_chunk, name_chunk, age_chunk, city_chunk]
Row Group 1: [id_chunk, name_chunk, age_chunk, city_chunk]

Each column chunk is stored contiguously on disk. This is the key to column pruning — if your query only references id and age, the engine reads only those column chunks and skips name and city entirely.

Each column chunk has its own compression codec. While most writers use the same codec for all columns, it is technically possible to use different codecs per column (e.g., Snappy for large string columns, Zstd for numeric columns).

Pages

Column chunks are further divided into pages, the smallest unit of I/O in Parquet. Pages are typically around 1 MB. There are three types:

Data Pages

Data pages contain the actual column values, encoded and optionally compressed. Each data page includes:

• Repetition and definition levels: Used for nested data (lists, maps, structs). For flat schemas, these are trivial.
• Encoded values: The column data, encoded using one of several schemes (see Encodings below).
• Page header: Metadata including uncompressed and compressed sizes, encoding type, and value count.

There are two versions: Data Page V1 and Data Page V2. V2 separates the repetition/definition levels from the data and allows the levels to remain uncompressed while the data is compressed, enabling more efficient reading.

Dictionary Pages

When dictionary encoding is used (the default for most writers), the first page in a column chunk is a dictionary page. It contains the unique values for that column, and subsequent data pages reference the dictionary by index.

For example, a country column with 1 million rows but only 50 unique countries would store a 50-entry dictionary page, and each data page would contain integer indices (0-49) rather than full country name strings.

Index Pages

Parquet supports optional column index and offset index pages that enable fine-grained filtering within a column chunk. The column index stores min/max values per page, allowing the engine to skip individual pages — not just entire row groups.

Encodings

Parquet uses several encoding schemes to represent column values efficiently. The writer chooses the encoding based on the column type and data characteristics.

Plain Encoding

Values are stored as-is in their native format. This is the fallback when no other encoding is beneficial.

Dictionary Encoding (PLAIN_DICTIONARY / RLE_DICTIONARY)

The most common encoding. A dictionary of unique values is built, and each value is replaced with an integer index. The indices are then run-length or bit-packed encoded.

Dictionary encoding is extremely effective for columns with low to moderate cardinality — status codes, country names, categories, boolean flags. It is less effective (and may be disabled) for high-cardinality columns like UUIDs or free-text fields.

Most writers start with dictionary encoding and fall back to plain encoding if the dictionary exceeds a size threshold (typically 1 MB or when the number of unique values exceeds a fraction of the page size).

Run-Length Encoding (RLE)

Consecutive repeated values are stored as a (value, count) pair. Effective for sorted data or columns with many repeated values. Parquet uses a hybrid RLE/bit-packing scheme that handles both repeated and non-repeated values efficiently.

Delta Encoding

Stores the difference between consecutive values rather than the values themselves. Highly effective for sorted integer columns (timestamps, auto-incrementing IDs). Variants include:

• DELTA_BINARY_PACKED: For integers.
• DELTA_LENGTH_BYTE_ARRAY: For variable-length strings (encodes the lengths with delta encoding).
• DELTA_BYTE_ARRAY: For strings with common prefixes.

Byte Stream Split

A newer encoding for floating-point data. It rearranges the bytes of float/double values to group similar bytes together (all first bytes, then all second bytes, etc.), which compresses much better with general-purpose codecs.

Compression

After encoding, each page can be compressed with a general-purpose compression codec:

Codec	Speed	Ratio	Notes
Snappy	Very fast	Moderate	Default in many tools. Good for interactive workloads.
Zstd	Fast	High	Best overall balance. Increasingly the recommended default.
Gzip	Slow	High	Legacy choice. Zstd is better in almost all cases.
LZ4	Very fast	Lower	Fast but lower ratio than Snappy in practice.
Brotli	Slow	Very high	Rarely used for Parquet. Better for web content.
Uncompressed	N/A	None	Useful for debugging or pre-compressed data.

Compression is applied per page, after encoding. The encoding step (dictionary, RLE, delta) does the heavy lifting of reducing data size, and the compression codec handles the remaining redundancy.

When working with Parquet files in Parquet Explorer, you can choose between Snappy, Zstd, and Gzip when exporting or converting files — so you can match the codec to your use case without needing to write code.

The Metadata Footer

The footer is the most important part of a Parquet file for query planning. It is a Thrift-serialized structure containing:

File Metadata

• Version: Parquet format version.
• Schema: The full column schema, including nested types, with logical types (DATE, TIMESTAMP, DECIMAL, etc.) annotated on top of physical types.
• Number of rows: Total row count across all row groups.
• Row group metadata: For each row group, the metadata for each column chunk.
• Key-value metadata: Arbitrary user-defined metadata (e.g., Spark schema, Arrow schema, pandas metadata, or application-specific tags).

Column Chunk Metadata (per row group, per column)

• File offset and size: Where the column chunk lives in the file.
• Compression codec: Which codec was used.
• Encodings: Which encodings were applied.
• Number of values: Including nulls.
• Statistics: Min value, max value, null count, distinct count (optional). These power predicate pushdown.
• Size information: Compressed and uncompressed sizes.

Why the Footer Matters

When a query engine opens a Parquet file, it reads the footer first. From the footer alone, without reading any data, it can:

1. Determine the schema and validate it against the query.
2. Identify which row groups to skip based on column statistics.
3. Identify which column chunks to read based on the columns referenced in the query.
4. Plan parallel reads across row groups.

This is why Parquet queries can start returning results almost instantly, even on multi-gigabyte files.

Inspecting Parquet Structure Yourself

Understanding the theory is useful, but seeing it in your own files makes it concrete. Here are the best ways to inspect Parquet internals.

Parquet Explorer

At parquetexplorer.com, load any Parquet file to get a comprehensive view of its structure:

• Schema tree view: Browse all column types, including nested STRUCT, LIST, and MAP hierarchies, rendered as an expandable tree.
• Metadata inspector: See row group count, rows per group, compression codecs, and per-column statistics (min, max, null count, distinct count).
• Data profiler: Go beyond raw metadata — get per-column histograms, semantic type detection (emails, URLs, UUIDs, IPs, phone numbers), and a data quality score.
• SQL queries: Immediately query the data to validate what the metadata tells you.

All of this runs in your browser with zero installation. Your files stay on your machine.

PyArrow

import pyarrow.parquet as pq

meta = pq.read_metadata("data.parquet")
print(f"Rows: {meta.num_rows}")
print(f"Row groups: {meta.num_row_groups}")
print(f"Columns: {meta.num_columns}")

for i in range(meta.num_row_groups):
    rg = meta.row_group(i)
    print(f"\nRow Group {i}: {rg.num_rows} rows")
    for j in range(rg.num_columns):
        col = rg.column(j)
        print(f"  {col.path_in_schema}: {col.compression} "
              f"({col.total_compressed_size}/{col.total_uncompressed_size} bytes)")

DuckDB

SELECT * FROM parquet_metadata('data.parquet');
SELECT * FROM parquet_schema('data.parquet');

Practical Implications

Understanding the file structure leads to concrete optimizations:

1. Sort your data before writing. Sorted columns have tighter min/max ranges per row group, which improves predicate pushdown. Sorting also boosts RLE and delta encoding effectiveness.

2. Choose column order thoughtfully. Columns queried together should be close in the schema (though modern readers handle non-contiguous reads well).

3. Monitor dictionary encoding fallback. If a column falls back from dictionary to plain encoding, its compression ratio drops. The data profiler in Parquet Explorer can help you spot high-cardinality columns where this might happen — look for columns with distinct counts approaching the total row count.

4. Right-size your row groups. The default is usually fine, but if you are optimizing for very selective queries (reading 0.1% of rows), smaller row groups help. For full-scan workloads, larger row groups are better.

5. Use Zstd compression. Unless you have a specific reason for Snappy or Gzip, Zstd offers the best ratio-to-speed trade-off in 2026.

Conclusion

Parquet’s performance comes from its layered architecture: row groups enable parallelism and pruning, column chunks enable projection pushdown, pages enable fine-grained I/O, encodings minimize data size at the logical level, and compression reduces it further at the physical level. The metadata footer ties it all together by giving query engines a complete map of the file before reading a single data byte.

This design is why a query over 3 columns of a 50-column, 10 GB Parquet file can complete in under a second — the engine reads only the relevant column chunks from the relevant row groups and skips everything else.

To explore the structure of your own Parquet files hands-on, try parquetexplorer.com. Load a file and browse the schema tree, inspect row group metadata, run the data profiler, and query the data — all in one place, right in your browser.