Skip to content

[format] Check value encoding before reusing Avro string-map bytes - #11334

Open
zhangyue19921010 wants to merge 3 commits into
apache:masterfrom
zhangyue19921010:contrib/fix-avro-string-map-layout
Open

zhangyue19921010 wants to merge 3 commits into
apache:masterfrom
zhangyue19921010:contrib/fix-avro-string-map-layout

Conversation

@zhangyue19921010

Copy link
Copy Markdown
Contributor

Purpose

Fixes #11333.

Fix reading external Avro string maps whose value encoding differs from Paimon's requested nullability or null-first union layout. Also avoid copying an encoded map into a destination with different value nullability, which produces invalid Avro output.

Use the existing schema-aware reader for incompatible layouts and retain raw-byte reuse when the encoded value nullability matches the destination.

Tests

Added a real-file read/rewrite matrix using Avro's generic writer and reader as independent producers and consumers. It covers required values, nullable unions in both orders, nulls, empty maps, following fields and both directions of value-nullability changes. Four of six cases fail on the unchanged base; all six pass with the fix.

Final verification: 85 tests passed, with no skips, using mvn -pl paimon-format -am -DfailIfNoTests=false -DwildcardSuites=none -Dtest=AvroStringMapReadTest,AvroFileFormatTest,AvroFormatReadWriteTest,AvroBytesStringMapTest,AvroBlockReaderTest,AvroSimpleStatsExtractorLeakTest test on JDK 11. Checkstyle, Spotless and Enforcer passed without fast-build; the repository-wide license check passed with mvn -N apache-rat:check.

zhangyue19921010 and others added 3 commits November 22, 2026 08:06
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decoding an encoded string map turns a null into a zeroed slot, which the
required string writer would silently write as an empty string.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Avro string-map byte reuse assumes matching value encodings

1 participant