[shahwazreza]

Which issue does this PR close?

• Closes repeat on LargeUtf8 aborts the process on a result size no allocation can satisfy #26948.

Rationale for this change

repeat on a LargeUtf8 string with a huge count, e.g. repeat(arrow_cast('a', 'LargeUtf8'), 1152921504606846976), aborts the process (scalar case) or panics (array case) instead of returning an error. Utf8 and Utf8View already error above i32::MAX bytes, but LargeUtf8 was capped at i64::MAX, which is way past anything that can actually be allocated.

What changes are included in this PR?

LargeUtf8 now uses a fixed cap of u32::MAX bytes (LARGE_UTF8_MAX_SIZE). In the array path the cap applies to each value and to the total size of the output, since the builder allocates the whole output up front. My first version only checked each value, so lots of rows could still add up to an allocation that panics. The second commit fixes that.

The issue asked whether to use a cap or fallible allocation. I went with a cap because the builder has no fallible way to allocate, and changing that would be a bigger separate change. A cap also behaves the same way Utf8 already does.

u32::MAX matches DuckDB's string size limit and still lets LargeUtf8 go past the Utf8 limit. Happy to change it, e.g. to Postgres' 1 GB, if it is preferred.

This doesn't help when an allocation under the cap succeeds and then fails later because of overcommit, same as the issue says.

What is the testing strategy for this PR?

Unit tests in repeat.rs:

• scalar LargeUtf8 with a count of 1 << 60 (used to abort the test process)
• array LargeUtf8 with a count of 1 << 60 (used to panic)
• array with two rows of ~3 GB each: fine on their own, over the cap together

Are there any user-facing changes?

repeat returns an error for LargeUtf8 results over u32::MAX bytes instead of trying to allocate them.

[mbutrovich]

Thanks @shahwazreza for the fix, and for catching the total-size case in the second commit.

[mbutrovich]

Thanks for the PR @shahwazreza! Approved pending CI.

[shahwazreza]

CI failed on the new total test. Looking into it. Please give me a few minutes!

[shahwazreza]

Hey, @mbutrovich. CI fails on the new total test. CI runs the tests with the backtrace feature on, so the error string gets a backtrace appended after the message, and the exact assert_eq! doesn't match (the message itself is right). What's the preferred way to approach this?

[mbutrovich]

Thanks @shahwazreza for folding in the changes.

[mbutrovich]

What's the preferred way to approach this?

Sorry, that one came from my suggestion. DataFusionError::strip_backtrace() returns the message without the backtrace, and other function tests compare errors that way, for example in date_bin.rs. I left a suggestion on the line.

[mbutrovich]

Approved pending CI.

[shahwazreza]

Thank you very much for you help!