Skip to content

[FLINK-39380][table] Add test snapshotting and restoration to ProcessTableFunctionTestHarness - #29345

Open
SEPURI-SAI-KRISHNA wants to merge 1 commit into
apache:masterfrom
SEPURI-SAI-KRISHNA:FLINK-39380-ptf-test-snapshot
Open

SEPURI-SAI-KRISHNA wants to merge 1 commit into
apache:masterfrom
SEPURI-SAI-KRISHNA:FLINK-39380-ptf-test-snapshot

Conversation

@SEPURI-SAI-KRISHNA

@SEPURI-SAI-KRISHNA SEPURI-SAI-KRISHNA commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change

FLIP-567 lists test snapshotting and restoration as part of the PTF test harness, so that a test
setup can be built once and reused by several cases. This adds it:
ProcessTableFunctionTestHarness.snapshot() captures a test at a point in time and the static
ProcessTableFunctionTestHarness.restoreFromSnapshot(TestSnapshot) builds a new harness that
continues from there.

Two points settled with @autophagy on the ticket:

  • Only the static restoreFromSnapshot is added. The Builder.restoreFromSnapshot variant the FLIP
    also showed was dropped as unintuitive, and has since been removed from the FLIP page.
  • State and timers are deep copied when the snapshot is taken, so that mutating them afterwards
    cannot change the snapshot.

The snapshot holds the configuration the harness was built with, the state of every partition, the
pending and the fired timers, the watermark of every table argument, and the collected output. The
system clock the FLIP also mentions is not covered, because the harness has no system clock yet
(withInitialSystemClock is not implemented).

Rebased on top of FLINK-40708, which added ValueView state after this was opened. That brought a
fifth StateConverter, so ValueViewStateConverter implements copyInternal here too. Rebased
again on top of FLINK-40577: HarnessSpec now holds the configured TableArguments, so a snapshot
keeps each table argument's changelog mode and upsert keys.

Brief change log

  • ProcessTableFunctionTestHarness: snapshot(), restoreFromSnapshot(TestSnapshot) and the
    TestSnapshot<OUT> holder. The harness holds an immutable HarnessSpec built at build() time
    rather than the Builder itself, so a snapshot can rebuild an equivalently configured harness and
    further configuration of the original builder cannot change what a snapshot restores.
  • A private Builder.build(TestSnapshot) installs state, timers and output while the harness is
    being constructed, before the function is opened, rather than overwriting them afterwards. There
    is therefore no point at which a failure can leave an opened harness behind.
  • Everything a snapshot holds is deep copied:
    • output rows and the values collected by the PTF, through ExternalSerializer of the derived
      output type and of the PTF output type. Row#copy copies only the field array, so a nested
      value such as an ARRAY or a POJO field would otherwise be shared. A PTF may collect(null),
      which the serializer cannot convert, so both copies skip nulls.
    • partition keys, through an ExternalSerializer of the partition-key row type owned by
      PartitionKeyInfo. This matters more than the output rows: Row compares and hashes its fields
      deeply, so a shared array inside a key also changes that key's hash and can make the snapshot's
      own map entry unreachable. An unpartitioned harness keys everything by an empty row, so it keeps
      Row#copy.
    • state, through StateConverter.copyInternal. State is stored in its internal representation
      (ArrayData / MapData / RowData) even for POJO state, since every write goes through
      StateConverter.toInternal first, so the copy never instantiates or touches a user-facing state
      object. StructuredTypeStateConverter and ValueViewStateConverter take the state DataType
      instead of only a converter so they can build that serializer, and ListViewStateConverter uses
      an ArrayDataSerializer of the element type. ValueViewStateConverter is the one case where the
      internal value can be null, since an empty view is represented that way, so its copyInternal
      is null-safe.
  • TestHarnessStateManager: snapshot() / restore(...), both deep copying. restore installs
    state into a fresh manager and asserts it has not already been populated, rather than clearing
    first.
  • TestHarnessTimerManager: snapshot() / restore(...) covering pending timers, fired timers,
    watermarkByTable and globalWatermark, with the same fresh-manager precondition.
    globalWatermark matters on its own: it is what TimeContext.currentWatermark() reads, and it is
    what rejects a table whose first watermark arrives after the restore and would pull the global
    watermark backwards.
  • Timer gains a constructor taking the fired state, and copy(UnaryOperator<Row>) so the key is
    copied the same way the managers copy their keys.
  • Documentation: a new "Snapshotting and Restoring a Test" section in ptfs.md (en and zh).

Verifying this change

This change added tests and can be verified as follows:

  • ProcessTableFunctionTestHarnessSnapshotTest (new, 23 tests) covers snapshot and restore of
    output for all three output kinds including atomic, POJO / Row / ListView / MapView /
    ValueView state, pending and fired timers and watermarks; that a snapshot is unaffected by later
    elements, state changes, timer firings, builder configuration and by mutating a nested output
    value, a collected POJO or a partition key; that a collected null is tolerated; that a snapshot can
    be restored more than once into harnesses that share no state, timers or output rows with each
    other; that a restore uses a fresh function instance; that a restore keeps a table argument's
    changelog mode and upsert key; and that state cleared before the snapshot does not come back from
    the builder's initial state.
  • StateConverterTest (new, 7 tests) covers copyInternal for all five converters, including that
    mutating a nested element, map value or row after the copy is not visible through it, and that an
    empty ValueView stays empty.
  • TestHarnessTimerManagerTest (30 tests) covers the timer manager's snapshot and restore directly,
    including that restoring onto a populated manager is rejected.
  • Every production change was checked red/green by reverting it in isolation and confirming a named
    test fails: no value copy in each converter, the per-key state map shared instead of copied, the
    partition-key copy reverted to Row#copy (the snapshot then comes back keyed [-1, 2]), the
    output-row and collected-value copies reverted, the null guards removed, Timer's key copy,
    globalWatermark not restored, watermarkByTable not restored, pending timer sets shared, and
    each of state / timers / output / function output not copied on the restore side.
  • After the rebase, all 205 tests of flink-table/flink-table-test-utils pass on JDK 17, and
    formatting and checkstyle are clean.
  • The examples in the documentation were run as written before being added.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): yes - two new
    methods and one new nested class on ProcessTableFunctionTestHarness, which is
    @PublicEvolving, and package-private changes on Timer. All additive.
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs and JavaDocs

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5)

@flinkbot

flinkbot commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@SEPURI-SAI-KRISHNA

Copy link
Copy Markdown
Contributor Author

@autophagy this is the snapshotting subtask from FLINK-39380, ready when you have time.

Two things I would like your call on:

  1. TestSnapshot does not implement Serializable. Nothing in FLIP-567 needs it, so I left it out, but if you want a snapshot to be shareable beyond a single test run I can look at what that would take.

  2. The FLIP page still lists Builder.restoreFromSnapshot. Happy to update the page myself if you would rather not.

The system clock the FLIP mentions is not in the snapshot because withInitialSystemClock does not exist yet. I can extend it when that lands.

@SEPURI-SAI-KRISHNA
SEPURI-SAI-KRISHNA force-pushed the FLINK-39380-ptf-test-snapshot branch from 94c8bbd to 2cfb931 Compare October 1, 2026 09:14
@autophagy

Copy link
Copy Markdown
Contributor

Thanks @SEPURI-SAI-KRISHNA ! I will take a look and review when I can. On your two points:

  1. I think forgoing Serializable for now is fine. If we find a nice use case in the future that needs it, we can always revisit.
  2. Thanks for the heads up - have removed the Builder's restoration function from the FLIP.

@SEPURI-SAI-KRISHNA

Copy link
Copy Markdown
Contributor Author

Sounds good, leaving Serializable out. Thanks for updating the FLIP.

@autophagy autophagy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for submitting! I have a few comments

}

private static List<Row> copyRows(List<Row> rows) {
return rows.stream().map(Row::copy).collect(Collectors.toList());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the doc for Row.copy, it doesn't perform a deep copy. In a situation with a PTF with a nested output type, like "ROW<arr ARRAY<INT>>", this means you could modify an output row from the harness and it would modify the output row in the snapshot (or vice verse), like:

final TestSnapshot<Row> snapshot = harness.snapshot();

final Integer[] liveArray = (Integer[]) harness.getOutput().get(0).getField(0);
liveArray[0] = -1;

If you then did ProcessTableFunctionTestHarness.restoreFromSnapshot(snapshot), the restored harness' getOutput().get(0).getField(0)[0] would contain -1, and not whatever the value was when the snapshot was taken.

It's a bit contrived but worth keeping in mind, I'm not sure of the best way of doing a deep copy of a row is (other than roundtripping it through something like ExternalSerializer).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, and your snippet is the test: testSnapshotIsNotAffectedByMutatingANestedOutputValue. It applied to three more places I had written: the state map keys, the pending timer map keys and Timer's own key. Those are worse, because Row hashes deeply, so a shared array inside a key changes that key's hash and can make the snapshot's own entry unreachable. PARTITION BY on an ARRAY<INT> column is accepted, so it is reachable. Everything deep copies through ExternalSerializer now, as you suggested.

Comment on lines +254 to +255
* The one part that is not copied is the values collected by the PTF itself (see {@link
* #getFunctionOutput()}), since those are the instances the function emitted.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should also copy the values collected by the PTF itself, since I think this means that output from a pojo-typed ptf would leak across restore/snapshot.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right, a POJO-typed PTF leaked. functionOutput is copied now. One wrinkle: a PTF may collect(null), which the serializer cannot convert, so both copies skip nulls.

Comment on lines +252 to +253
* watermark or mutating state afterwards does not change what the snapshot holds. State is
* copied in its internal representation, which means no user-facing state object is touched.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure stating how the snapshotting is done is very relevant to the user, I think detail like this description of how state is snapshotted can be omitted.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trimmed to what it guarantees rather than how.


@Override
public Object copyInternal(Object internal) {
final ArrayData arrayData = (ArrayData) internal;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if you replaced the elementSerializer with a ArrayDataSerializer of arrayType.getElementType(), this function could just use its existing copy rather than re-implementing it:

public Object copyInternal(Object internal) {
    return arrayDataSerializer.copy((ArrayData) internal);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, same field name as your snippet. I did not do the same for MapViewStateConverter: MapDataSerializer.copy returns BinaryMapData while toInternal and createNewInternalState produce GenericMapData. ArrayDataSerializer.copy keeps GenericArrayData, so the list case has no such mismatch. Happy to switch both if you prefer the consistency.

harness.stateManager.restoreState(snapshot.state);
harness.timerManager.restore(snapshot.timers);
harness.functionOutput.addAll(snapshot.functionOutput);
harness.output.addAll(copyRows(snapshot.output));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess here what we are doing is building the harness from its initial builder again (with initial conditions like initial state), creating the state/timer manager from scratch and the updating all of the harness properties with the restoration state. I suppose this is why we also test for testRestoreDropsStateThatWasClearedBeforeTheSnapshot.

I wonder if it would be easier & clearer to have a private buildFromSnapshot(snapshot) method on the builder itself that handle this. By this i mean that instead of setting up the initial state and then overwriting it on the harness' internal state/timer managers, the builder itself could construct the right initial state from the snapshot and initialise the harness with it directly?

It would also make it easier to handle close()ing the opened harness too on restoration failure (since in this current construction I dont think this happens if restoreFromSnapshot fails.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. A private build(@Nullable TestSnapshot) seeds the state manager and the timer manager before the harness is constructed, and restoreFromSnapshot is now snapshot.spec.toBuilder().build(snapshot). I used an overload rather than a separate buildFromSnapshot so there is one construction path, happy to use your name instead. On close(): the restore work all happens before openFunction(), so a failure while copying cannot leave an opened function behind. openFunction() itself does function.open(..) then setCollector(..), so a failure in that second step would still leak one, but that is the same for an ordinary build() and unchanged here.

Comment on lines +1184 to +1197
private Builder(Builder<OUT> other) {
this.functionClass = other.functionClass;
this.scalarArgs.putAll(other.scalarArgs);
this.tableArgs.putAll(other.tableArgs);
this.partitionConfigs.putAll(other.partitionConfigs);
other.stateArgs.forEach((name, config) -> this.stateArgs.put(name, config.copy()));
this.onTimeColumnName = other.onTimeColumnName;
}

/** Creates a copy of this builder, detached from any further configuration of it. */
private Builder<OUT> copy() {
return new Builder<>(this);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As stated above, if we instead have the builder construct and pass in a frozen immutable representation of its build state/initial conditions to the harness, we wouldn't need this constructor or the copy.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gone, with Builder.copy() and StateArgumentConfiguration.copy().

Comment on lines +210 to +213
void restoreState(Map<Row, Map<String, Object>> snapshot) {
stateByKey.clear();
stateByKey.putAll(copyStateByKey(snapshot));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if we tackle the restoration of state/timers as commented in https://github.com/apache/flink/pull/29345/changes#r4246665648 then I think this pattern of clearing and then putAll would not be necessary, as we'd be able to initialise the state manager with the correct state.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gone, and the timer manager had the identical pattern, which I had missed. Both now assert they are populated once instead of clearing first. testRestoreReplacesLiveTimers asserted replace semantics the new construction cannot reach, so it is now testRestoreRequiresAFreshManager.

* Returns a deep copy of all state, keyed by partition key. Later mutations of the live state
* are not visible through the returned map.
*/
Map<Row, Map<String, Object>> snapshotState() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nit, but I think just snapshot and restore would be fine to use here. its clear from them being in the state manager what we are snapshotting and restoring.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed, and it matches the timer manager, which already had restore.

Comment on lines +85 to +86
Timer copy() {
final Timer copy = new Timer(timestamp, name, Row.copy(partitionKey));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be more simpler to have a Timer constructor that also takes in the fired state too, rather than marking it after construction.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added, and copy(..) uses it, so the construct-then-mark here is gone. markFired() stays because fireEligibleTimers flips a timer already in the pending set. One call for you: copy(..) now has a single caller and could be inlined as new Timer(t.timestamp, t.name, copyKey(t.partitionKey), t.hasFired()). I kept the method because fired is private and a copy() next to the fields is what gets updated if a field is ever added. Tell me which you prefer and I will do it, here or as a follow-up.

}

@Test
void testSnapshotIsNotAffectedByLaterOutput() throws Exception {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this test could be folded into testSnapshotAndRestoreOutput

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Folded, keeping the clearOutput() assertion.

@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label Oct 12, 2026
@SEPURI-SAI-KRISHNA
SEPURI-SAI-KRISHNA force-pushed the FLINK-39380-ptf-test-snapshot branch from 2cfb931 to 5be5c98 Compare October 14, 2026 17:07
@SEPURI-SAI-KRISHNA

Copy link
Copy Markdown
Contributor Author

Thanks, all twelve are addressed and I have replied on each thread. Two were real bugs and both have a test that fails without the fix.

The Row.copy one went further than the output rows. The state map keys, the pending timer map keys and Timer's own key had it too, and there it is worse because Row hashes deeply. Everything deep copies through ExternalSerializer now.

Checking the isolation claim in the other direction turned up a gap of my own. Every test mutated the live harness and asserted the snapshot held, and nothing asserted the reverse. Restoring one snapshot twice shared the per-key state maps, the Timer objects and the rows in the output lists. Deleting the copies on the restore side left the whole suite green, so none of it was covered. Four tests cover it now.

One change you did not ask for, so flagging it rather than leaving it in the diff: snapshot() now requires the harness to be open, matching process and getOutput, so snapshotting a closed harness is illegal where it previously worked. Say if you would rather it stayed permissive.

Left alone on purpose: getOutput() and getFunctionOutput() use List.copyOf, which throws on a collected null, so collect(null) plus either getter fails on master today, independent of snapshotting. Happy to file that separately.

Open in the threads: MapViewStateConverter, cloning the arrays in HarnessSpec, the build(TestSnapshot) name, and inlining Timer.copy. Happy to take any of them as follow-up.

154 tests pass, spotless and checkstyle clean.

…TableFunctionTestHarness

Generated-by: Claude Code (Opus 5)
@SEPURI-SAI-KRISHNA
SEPURI-SAI-KRISHNA force-pushed the FLINK-39380-ptf-test-snapshot branch from 5be5c98 to a966594 Compare November 21, 2026 09:26
@SEPURI-SAI-KRISHNA

Copy link
Copy Markdown
Contributor Author

@autophagy I rebased this onto master, on top of FLINK-40577. A snapshot now also keeps each table argument's changelog mode and upsert keys, with a test for it. One test changed: on master, a PTF with ROW output that collects null fails in the harness before a snapshot can be taken, so the null-collect test uses a STRING output instead. Could you take another look when you have time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants