Repository navigation
[FLINK-39380][table] Add test snapshotting and restoration to ProcessTableFunctionTestHarness - #29345
SEPURI-SAI-KRISHNA wants to merge 1 commit into
Conversation
|
@autophagy this is the snapshotting subtask from FLINK-39380, ready when you have time. Two things I would like your call on:
The system clock the FLIP mentions is not in the snapshot because |
94c8bbd to
2cfb931
Compare
|
Thanks @SEPURI-SAI-KRISHNA ! I will take a look and review when I can. On your two points:
|
|
Sounds good, leaving |
autophagy
left a comment
There was a problem hiding this comment.
Thank you for submitting! I have a few comments
| } | ||
|
|
||
| private static List<Row> copyRows(List<Row> rows) { | ||
| return rows.stream().map(Row::copy).collect(Collectors.toList()); |
There was a problem hiding this comment.
According to the doc for Row.copy, it doesn't perform a deep copy. In a situation with a PTF with a nested output type, like "ROW<arr ARRAY<INT>>", this means you could modify an output row from the harness and it would modify the output row in the snapshot (or vice verse), like:
final TestSnapshot<Row> snapshot = harness.snapshot();
final Integer[] liveArray = (Integer[]) harness.getOutput().get(0).getField(0);
liveArray[0] = -1;
If you then did ProcessTableFunctionTestHarness.restoreFromSnapshot(snapshot), the restored harness' getOutput().get(0).getField(0)[0] would contain -1, and not whatever the value was when the snapshot was taken.
It's a bit contrived but worth keeping in mind, I'm not sure of the best way of doing a deep copy of a row is (other than roundtripping it through something like ExternalSerializer).
There was a problem hiding this comment.
Fixed, and your snippet is the test: testSnapshotIsNotAffectedByMutatingANestedOutputValue. It applied to three more places I had written: the state map keys, the pending timer map keys and Timer's own key. Those are worse, because Row hashes deeply, so a shared array inside a key changes that key's hash and can make the snapshot's own entry unreachable. PARTITION BY on an ARRAY<INT> column is accepted, so it is reachable. Everything deep copies through ExternalSerializer now, as you suggested.
| * The one part that is not copied is the values collected by the PTF itself (see {@link | ||
| * #getFunctionOutput()}), since those are the instances the function emitted. |
There was a problem hiding this comment.
I wonder if we should also copy the values collected by the PTF itself, since I think this means that output from a pojo-typed ptf would leak across restore/snapshot.
There was a problem hiding this comment.
You were right, a POJO-typed PTF leaked. functionOutput is copied now. One wrinkle: a PTF may collect(null), which the serializer cannot convert, so both copies skip nulls.
| * watermark or mutating state afterwards does not change what the snapshot holds. State is | ||
| * copied in its internal representation, which means no user-facing state object is touched. |
There was a problem hiding this comment.
I'm not sure stating how the snapshotting is done is very relevant to the user, I think detail like this description of how state is snapshotted can be omitted.
There was a problem hiding this comment.
Trimmed to what it guarantees rather than how.
|
|
||
| @Override | ||
| public Object copyInternal(Object internal) { | ||
| final ArrayData arrayData = (ArrayData) internal; |
There was a problem hiding this comment.
I think if you replaced the elementSerializer with a ArrayDataSerializer of arrayType.getElementType(), this function could just use its existing copy rather than re-implementing it:
public Object copyInternal(Object internal) {
return arrayDataSerializer.copy((ArrayData) internal);
}
There was a problem hiding this comment.
Done, same field name as your snippet. I did not do the same for MapViewStateConverter: MapDataSerializer.copy returns BinaryMapData while toInternal and createNewInternalState produce GenericMapData. ArrayDataSerializer.copy keeps GenericArrayData, so the list case has no such mismatch. Happy to switch both if you prefer the consistency.
| harness.stateManager.restoreState(snapshot.state); | ||
| harness.timerManager.restore(snapshot.timers); | ||
| harness.functionOutput.addAll(snapshot.functionOutput); | ||
| harness.output.addAll(copyRows(snapshot.output)); |
There was a problem hiding this comment.
I guess here what we are doing is building the harness from its initial builder again (with initial conditions like initial state), creating the state/timer manager from scratch and the updating all of the harness properties with the restoration state. I suppose this is why we also test for testRestoreDropsStateThatWasClearedBeforeTheSnapshot.
I wonder if it would be easier & clearer to have a private buildFromSnapshot(snapshot) method on the builder itself that handle this. By this i mean that instead of setting up the initial state and then overwriting it on the harness' internal state/timer managers, the builder itself could construct the right initial state from the snapshot and initialise the harness with it directly?
It would also make it easier to handle close()ing the opened harness too on restoration failure (since in this current construction I dont think this happens if restoreFromSnapshot fails.
There was a problem hiding this comment.
Done. A private build(@Nullable TestSnapshot) seeds the state manager and the timer manager before the harness is constructed, and restoreFromSnapshot is now snapshot.spec.toBuilder().build(snapshot). I used an overload rather than a separate buildFromSnapshot so there is one construction path, happy to use your name instead. On close(): the restore work all happens before openFunction(), so a failure while copying cannot leave an opened function behind. openFunction() itself does function.open(..) then setCollector(..), so a failure in that second step would still leak one, but that is the same for an ordinary build() and unchanged here.
| private Builder(Builder<OUT> other) { | ||
| this.functionClass = other.functionClass; | ||
| this.scalarArgs.putAll(other.scalarArgs); | ||
| this.tableArgs.putAll(other.tableArgs); | ||
| this.partitionConfigs.putAll(other.partitionConfigs); | ||
| other.stateArgs.forEach((name, config) -> this.stateArgs.put(name, config.copy())); | ||
| this.onTimeColumnName = other.onTimeColumnName; | ||
| } | ||
|
|
||
| /** Creates a copy of this builder, detached from any further configuration of it. */ | ||
| private Builder<OUT> copy() { | ||
| return new Builder<>(this); | ||
| } | ||
|
|
There was a problem hiding this comment.
As stated above, if we instead have the builder construct and pass in a frozen immutable representation of its build state/initial conditions to the harness, we wouldn't need this constructor or the copy.
There was a problem hiding this comment.
Gone, with Builder.copy() and StateArgumentConfiguration.copy().
| void restoreState(Map<Row, Map<String, Object>> snapshot) { | ||
| stateByKey.clear(); | ||
| stateByKey.putAll(copyStateByKey(snapshot)); | ||
| } |
There was a problem hiding this comment.
I think if we tackle the restoration of state/timers as commented in https://github.com/apache/flink/pull/29345/changes#r4246665648 then I think this pattern of clearing and then putAll would not be necessary, as we'd be able to initialise the state manager with the correct state.
There was a problem hiding this comment.
Gone, and the timer manager had the identical pattern, which I had missed. Both now assert they are populated once instead of clearing first. testRestoreReplacesLiveTimers asserted replace semantics the new construction cannot reach, so it is now testRestoreRequiresAFreshManager.
| * Returns a deep copy of all state, keyed by partition key. Later mutations of the live state | ||
| * are not visible through the returned map. | ||
| */ | ||
| Map<Row, Map<String, Object>> snapshotState() { |
There was a problem hiding this comment.
Small nit, but I think just snapshot and restore would be fine to use here. its clear from them being in the state manager what we are snapshotting and restoring.
There was a problem hiding this comment.
Renamed, and it matches the timer manager, which already had restore.
| Timer copy() { | ||
| final Timer copy = new Timer(timestamp, name, Row.copy(partitionKey)); |
There was a problem hiding this comment.
Might be more simpler to have a Timer constructor that also takes in the fired state too, rather than marking it after construction.
There was a problem hiding this comment.
Added, and copy(..) uses it, so the construct-then-mark here is gone. markFired() stays because fireEligibleTimers flips a timer already in the pending set. One call for you: copy(..) now has a single caller and could be inlined as new Timer(t.timestamp, t.name, copyKey(t.partitionKey), t.hasFired()). I kept the method because fired is private and a copy() next to the fields is what gets updated if a field is ever added. Tell me which you prefer and I will do it, here or as a follow-up.
| } | ||
|
|
||
| @Test | ||
| void testSnapshotIsNotAffectedByLaterOutput() throws Exception { |
There was a problem hiding this comment.
I think this test could be folded into testSnapshotAndRestoreOutput
There was a problem hiding this comment.
Folded, keeping the clearOutput() assertion.
2cfb931 to
5be5c98
Compare
|
Thanks, all twelve are addressed and I have replied on each thread. Two were real bugs and both have a test that fails without the fix. The Checking the isolation claim in the other direction turned up a gap of my own. Every test mutated the live harness and asserted the snapshot held, and nothing asserted the reverse. Restoring one snapshot twice shared the per-key state maps, the One change you did not ask for, so flagging it rather than leaving it in the diff: Left alone on purpose: Open in the threads: 154 tests pass, spotless and checkstyle clean. |
…TableFunctionTestHarness Generated-by: Claude Code (Opus 5)
5be5c98 to
a966594
Compare
|
@autophagy I rebased this onto master, on top of FLINK-40577. A snapshot now also keeps each table argument's changelog mode and upsert keys, with a test for it. One test changed: on master, a PTF with ROW output that collects null fails in the harness before a snapshot can be taken, so the null-collect test uses a STRING output instead. Could you take another look when you have time? |
What is the purpose of the change
FLIP-567 lists test snapshotting and restoration as part of the PTF test harness, so that a test
setup can be built once and reused by several cases. This adds it:
ProcessTableFunctionTestHarness.snapshot()captures a test at a point in time and the staticProcessTableFunctionTestHarness.restoreFromSnapshot(TestSnapshot)builds a new harness thatcontinues from there.
Two points settled with @autophagy on the ticket:
restoreFromSnapshotis added. TheBuilder.restoreFromSnapshotvariant the FLIPalso showed was dropped as unintuitive, and has since been removed from the FLIP page.
cannot change the snapshot.
The snapshot holds the configuration the harness was built with, the state of every partition, the
pending and the fired timers, the watermark of every table argument, and the collected output. The
system clock the FLIP also mentions is not covered, because the harness has no system clock yet
(
withInitialSystemClockis not implemented).Rebased on top of FLINK-40708, which added
ValueViewstate after this was opened. That brought afifth
StateConverter, soValueViewStateConverterimplementscopyInternalhere too. Rebasedagain on top of FLINK-40577:
HarnessSpecnow holds the configuredTableArguments, so a snapshotkeeps each table argument's changelog mode and upsert keys.
Brief change log
ProcessTableFunctionTestHarness:snapshot(),restoreFromSnapshot(TestSnapshot)and theTestSnapshot<OUT>holder. The harness holds an immutableHarnessSpecbuilt atbuild()timerather than the
Builderitself, so a snapshot can rebuild an equivalently configured harness andfurther configuration of the original builder cannot change what a snapshot restores.
Builder.build(TestSnapshot)installs state, timers and output while the harness isbeing constructed, before the function is opened, rather than overwriting them afterwards. There
is therefore no point at which a failure can leave an opened harness behind.
ExternalSerializerof the derivedoutput type and of the PTF output type.
Row#copycopies only the field array, so a nestedvalue such as an
ARRAYor a POJO field would otherwise be shared. A PTF maycollect(null),which the serializer cannot convert, so both copies skip nulls.
ExternalSerializerof the partition-key row type owned byPartitionKeyInfo. This matters more than the output rows:Rowcompares and hashes its fieldsdeeply, so a shared array inside a key also changes that key's hash and can make the snapshot's
own map entry unreachable. An unpartitioned harness keys everything by an empty row, so it keeps
Row#copy.StateConverter.copyInternal. State is stored in its internal representation(
ArrayData/MapData/RowData) even for POJO state, since every write goes throughStateConverter.toInternalfirst, so the copy never instantiates or touches a user-facing stateobject.
StructuredTypeStateConverterandValueViewStateConvertertake the stateDataTypeinstead of only a converter so they can build that serializer, and
ListViewStateConverterusesan
ArrayDataSerializerof the element type.ValueViewStateConverteris the one case where theinternal value can be null, since an empty view is represented that way, so its
copyInternalis null-safe.
TestHarnessStateManager:snapshot()/restore(...), both deep copying.restoreinstallsstate into a fresh manager and asserts it has not already been populated, rather than clearing
first.
TestHarnessTimerManager:snapshot()/restore(...)covering pending timers, fired timers,watermarkByTableandglobalWatermark, with the same fresh-manager precondition.globalWatermarkmatters on its own: it is whatTimeContext.currentWatermark()reads, and it iswhat rejects a table whose first watermark arrives after the restore and would pull the global
watermark backwards.
Timergains a constructor taking the fired state, andcopy(UnaryOperator<Row>)so the key iscopied the same way the managers copy their keys.
ptfs.md(en and zh).Verifying this change
This change added tests and can be verified as follows:
ProcessTableFunctionTestHarnessSnapshotTest(new, 23 tests) covers snapshot and restore ofoutput for all three output kinds including atomic, POJO /
Row/ListView/MapView/ValueViewstate, pending and fired timers and watermarks; that a snapshot is unaffected by laterelements, state changes, timer firings, builder configuration and by mutating a nested output
value, a collected POJO or a partition key; that a collected null is tolerated; that a snapshot can
be restored more than once into harnesses that share no state, timers or output rows with each
other; that a restore uses a fresh function instance; that a restore keeps a table argument's
changelog mode and upsert key; and that state cleared before the snapshot does not come back from
the builder's initial state.
StateConverterTest(new, 7 tests) coverscopyInternalfor all five converters, including thatmutating a nested element, map value or row after the copy is not visible through it, and that an
empty
ValueViewstays empty.TestHarnessTimerManagerTest(30 tests) covers the timer manager's snapshot and restore directly,including that restoring onto a populated manager is rejected.
test fails: no value copy in each converter, the per-key state map shared instead of copied, the
partition-key copy reverted to
Row#copy(the snapshot then comes back keyed[-1, 2]), theoutput-row and collected-value copies reverted, the null guards removed,
Timer's key copy,globalWatermarknot restored,watermarkByTablenot restored, pending timer sets shared, andeach of state / timers / output / function output not copied on the restore side.
flink-table/flink-table-test-utilspass on JDK 17, andformatting and checkstyle are clean.
Does this pull request potentially affect one of the following parts:
@Public(Evolving): yes - two newmethods and one new nested class on
ProcessTableFunctionTestHarness, which is@PublicEvolving, and package-private changes onTimer. All additive.Documentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5)