Repository navigation
Conversation
Collaborator
…uns inside an operator Code generated for constant folding, partition pruning, pushed-down watermark generators and SpecializedFunction expression evaluators is opened without a runtime context. Add CodeGeneratorContext #supportsUdfMetrics, which is false for these contexts and inherited by child contexts, and generate UDF metrics only where it is true.
UDF metrics are documented for user-defined functions. Pass no metric name for a BuiltInFunctionDefinition, which also covers the internal functions used for UNNEST and INTERSECT ALL.
…erator Several generated classes of one operator can call the same function and register its metrics on the same operator metric group, which keeps one metric per name. UdfMetrics.register now returns the handle already registered on the same udf.<name> group, so all classes record into one histogram and one counter. Handles are cached with weak identity keys, so a finished task's group is not kept reachable.
weiqingy
force-pushed
the
FLINK-41585-udf-metrics-registration
branch
from
November 20, 2026 19:30
47282b5 to
6bafa12
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The test can falsely report a cache leak because its short System.gc() polling window does not guarantee collection of the weakly reachable object.
1 open issue
What changed in this PR
This PR refines Flink Table/SQL UDF metrics so registration is limited to user-defined functions running with an operator runtime context, with shared handles across generated classes in the same operator.
Changes:
- Skips metrics for built-ins and contexts without a runtime context.
- Shares metric handles per function and operator using a weak-key cache.
- Adds regression coverage for planning, runtime registration, restarts, and metric-group retention.
| File | Description |
|---|---|
| UdfMetricsTest.java | Tests restart isolation and metric-group collection. |
| UdfMetrics.java | Caches shared handles using weak metric-group keys. |
| PartitionableSourceTest.scala | Tests partition pruning with UDF metrics enabled. |
| ExpressionReductionRulesTest.scala | Tests constant folding with UDF metrics enabled. |
| PartitionableSourceTest.xml | Adds expected partition-pruning plans. |
| ExpressionReductionRulesTest.xml | Adds the expected constant-folding plan. |
| UdfMetricsITCase.java | Covers metric exclusions and shared runtime handles. |
| WatermarkGeneratorCodeGenerator.scala | Disables metrics for source-pushed watermark generators. |
| ExpressionReducer.scala | Disables metrics in planning-time contexts. |
| CodeGeneratorContext.scala | Adds inherited UDF-metrics capability. |
| BridgingSqlFunctionCallGen.scala | Excludes built-in functions from instrumentation. |
| BridgingFunctionGenUtil.scala | Checks context capability and disables evaluator metrics. |
Review effort: Balanced
… and streaming UDF metrics are reported in batch mode as well, so label both options with Batch and Streaming in the configuration docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What is the purpose of the change
This pull request refines where UDF metrics are registered when
table.exec.udf-metric-enabledis true:SpecializedFunctionexpression evaluators is opened without a runtime context and skips registration, so these paths work with metrics enabled.UNNESTandINTERSECT ALL, do not getudf.*groups.DEFINEandMEASURESofMATCH_RECOGNIZE) registers once per operator, and all parts share one handle, so every call is reported.Brief change log
CodeGeneratorContext#supportsUdfMetricsis true by default and false for the contexts used for constant folding and partition pruning, for pushed-down watermark generators and forSpecializedFunctionexpression evaluators. Child contexts inherit it.BridgingFunctionGenUtil#udfMetricsTermIfEnabledgenerates no metrics where it is false.BridgingSqlFunctionCallGenpasses no metric name when the function definition is aBuiltInFunctionDefinition. This also covers the internal UNNEST and REPLICATE_ROWS functions.UdfMetrics.registerreturns the handle already registered on the sameudf.<name>group, so all generated classes of one operator record into one histogram and one counter. Handles are cached with weak keys compared by identity, so the group of a finished task is not kept reachable, and a restarted task registers on a new group and gets a new handle.@Documentation.TableOption(execMode = BATCH_STREAMING), andexecution_config_configuration.htmlis regenerated. The metrics docs already say only user-defined functions are metered, so they are unchanged.Verifying this change
This change added tests and can be verified as follows:
ExpressionReductionRulesTest#testExpressionReductionWithUDFAndUdfMetricsEnabled: a UDF call with constant arguments is folded with metrics on.PartitionableSourceTest#testWithUdfAndUdfMetricsEnabled: a UDF in a partition filter plans and prunes with metrics on.UdfMetricsITCase#testFunctionCallingExpressionEvaluator: aSpecializedFunctionwhose evaluator calls another UDF runs with metrics on.UdfMetricsITCase#testFunctionInPushedDownWatermark: a user-defined function in aWATERMARKexpression pushed into the source runs with metrics on.UdfMetricsITCase#testBuiltInFunctionsNotMetered:TYPEOFandUNNESTregister no group, while a user-defined function in the same query does.UdfMetricsITCase#testCorrelateInvocationAndConditionShareOneHandleand#testLookupJoinPreFilterAndConditionShareOneHandle: every call is sampled and the exception is counted.UdfMetricsTest#restartedOperatorGetsNewHandleand#doesNotRetainOperatorMetricGroup: the shared handle follows the operator group and does not keep it reachable.udf.*metrics with metrics enabled and none with metrics disabled.UdfMetricsITCase,AsyncCalcITCase,AsyncCorrelateITCase, the lookup join, MATCH_RECOGNIZE and watermark push-down ITCases pass.Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code 2.1.322 (Claude Opus 5.5)