Skip to content

fix(file): normalize fixed-size struct list row counts - #9999

Merged
Xuanwo merged 4 commits into
lance-format:mainfrom
vera-rubin:fix/struct-list-row-counts
Oct 30, 2026
Merged

Xuanwo merged 4 commits into
lance-format:mainfrom
vera-rubin:fix/struct-list-row-counts

Conversation

@vera-rubin

Copy link
Copy Markdown
Contributor

A valid table with a scalar column and a two-item FixedSizeList<Struct> writes successfully in the default 2.2 format, but to_table() rejects it with cannot read columns of differing lengths together (id=1, points=2). We expect the original one-row table, as the read guide describes.

import lance
import pyarrow as pa

points = pa.list_(pa.struct([pa.field("x", pa.int32())]), 2)
table = pa.table({"id": [1], "points": pa.array([[{"x": 10}, {"x": 20}]], points)})
assert lance.write_dataset(table, "data.lance").to_table().equals(table)

We scale dense page counts by the structural list dimensions during projection validation and preserve sparse-page normalization.

The Python reproducer and all 260 Rust file tests pass, including dense 2.2/2.3 and sparse 2.3 regression cases.

Workspace Clippy is unverified; crate Clippy without --no-deps failed on existing lance-io dead-code warnings.

— ChipLab

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer bug Something isn't working labels Oct 15, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 15, 2026
@wjones127 wjones127 added the critical-fix Bugs that cause crashes, security vulnerabilities, or incorrect data. label Oct 19, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 30, 2026
@Xuanwo
Xuanwo requested a review from lancedb-robot October 30, 2026 15:40
@lance-gatekeeper
lance-gatekeeper Bot removed the request for review from lancedb-robot October 30, 2026 15:40

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Gate recommendation: approve.

Scalar and FixedSizeList<Struct> columns now round-trip together without changing stored encodings. Dense normalization rejects non-divisible row counts, while sparse normalization retains its existing semantics. The file crate suite and additional nested/null, multi-page projection checks pass.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Oct 30, 2026

@Xuanwo Xuanwo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the current revision after merging main. The fixed-size-list row scale now covers stable 2.2 Structural pages and dense 2.3 Planar pages while preserving sparse page-local normalization. Non-divisible dense counts return an invalid-input error, and the Python round trip plus full current-head CI pass.

@Xuanwo
Xuanwo merged commit e575f50 into lance-format:main Oct 30, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer bug Something isn't working critical-fix Bugs that cause crashes, security vulnerabilities, or incorrect data. K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants