Skip to content

[python] Support native batch overwrite commits - #10097

Merged
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/python-native-overwrite
Sep 22, 2026
Merged

JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/python-native-overwrite

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #10081. Enable batch overwrite through the existing opt-in commit.native.enabled route, using apache/paimon-rust#916 on top of the merged apache/paimon-rust#915.

  • Match Java's batch/stream API split: move overwrite() and its configuration from WriteBuilder to BatchWriteBuilder. StreamWriteBuilder no longer exposes overwrite; stream writer/committer constructors accept only the table and commit user.
  • Submit native overwrite through BatchTableCommit, preserving the Python writer's identity via the internal batch builder bridge. Remove the proposed stream overwrite route and its tests. No isinstance mode guard or older-Rust API compatibility probes are needed.
  • Preserve static/dynamic partition overwrite, null/default partitions, imported fixed-bucket messages, the batch identifier, and one-shot lifecycle. Batch and stream append retain their existing native path.
  • Match Java for static/table overwrite on empty tables or unmatched specs: record an OVERWRITE snapshot in both native and Python paths. Dynamic empty overwrite remains a no-op.
  • Normalize Python boolean spellings such as off and 0 for native overwrite/ignore-empty options.
  • The optional runtime can be absent; callbacks, snapshot properties, unsupported environments, and row-id recovery keep the Python route. Native mutation failures propagate without Python retry or automatic abort. Truncate APIs retain their existing route.

Usage

native_table = table.copy({
    "commit.native.enabled": "true",
    "dynamic-partition-overwrite": "false",
})
builder = native_table.new_batch_write_builder().overwrite({"pt": "2026-09-22"})
writer, commit = builder.new_write(), builder.new_commit()
try:
    writer.write_arrow(data)
    commit.commit(writer.prepare_commit())
finally:
    writer.close()
    commit.close()

The existing dynamic overwrite default is unchanged; set dynamic-partition-overwrite=false for static partition matching as in this example. Overwrite is available only through the batch API, following Java.

Verification

  • Built the Rust Add listPartitions to FlinkCatalog [WIP] #916 wheel; its Python commit binding suite passed 43 tests.
  • Against that wheel, ran native_commit_test.py, table_commit_test.py, file_store_commit_test.py, commit_message_serializer_test.py, overwrite_commit_conflict_test.py, write/commit_callback_test.py, write/changelog_producer_test.py, and write/table_write_test.py: 214 passed, 8 subtests passed. The suite was run outside the sandbox because the ORC reader needs local sysctl hardware information.
  • Tests protect the batch-only API, stream constructors rejecting overwrite configuration, actual Python writer/native batch commit/Python reader roundtrips, snapshot identity and identifier, empty input/tables/unmatched specs, Boolean normalization, abort, response loss after publication, and preflight fallback preserving other partitions.
  • Updated the old stream-overwrite assertion in ray_sink_test.py. That Ray-specific test could not be collected locally because ray is not installed; the builder API boundary is covered by the passing native commit suite.
  • Flake8 on all changed Python files and git diff --check passed.

@JingsongLi JingsongLi changed the title [python] Support native batch overwrite commits [python] Support native overwrite commits Sep 22, 2026
@JingsongLi JingsongLi changed the title [python] Support native overwrite commits [python] Support native batch overwrite commits Sep 22, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Reviewed the batch-only overwrite API, preserved commit identity, static/dynamic/empty/null-partition overwrite semantics, Python option normalization, and the preflight-only fallback boundary. Native mutation failures remain outside the fallback path, so they do not trigger a second Python commit or automatic file cleanup.

Validated against a freshly built apache/paimon-rust#916 runtime:

  • Nine focused commit/write/serialization/conflict/callback/deletion-vector suites: 294 passed, 51 skipped.
  • Ray sink suite: 16 passed.
  • Independent Python/native overwrite-then-append checks for bucket=-1 and bucket=2: passed, with matching visible rows and snapshot record counts.
  • Full PyPaimon flake8 and license checks: passed.

No blocking findings. GitHub CI is still pending; this approval is based on code review and the local validation above.

@JingsongLi
JingsongLi merged commit 5fa4a33 into apache:master Sep 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants