Skip to content

[FLINK-41585][table] Refine UDF metrics registration - #30044

Open
weiqingy wants to merge 5 commits into
apache:masterfrom
weiqingy:FLINK-41585-udf-metrics-registration
Open

weiqingy wants to merge 5 commits into
apache:masterfrom
weiqingy:FLINK-41585-udf-metrics-registration

Conversation

@weiqingy

@weiqingy weiqingy commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This pull request refines where UDF metrics are registered when table.exec.udf-metric-enabled is true:

  • Only code that runs inside an operator registers metrics. Code generated for constant folding, partition pruning, watermark generators pushed into a source and SpecializedFunction expression evaluators is opened without a runtime context and skips registration, so these paths work with metrics enabled.
  • Only user-defined functions are metered, as documented. Built-in functions, including the internal functions used for UNNEST and INTERSECT ALL, do not get udf.* groups.
  • A function used in several parts of one operator (for example a correlate's arguments and condition, a lookup join's pre-filter and join condition, or DEFINE and MEASURES of MATCH_RECOGNIZE) registers once per operator, and all parts share one handle, so every call is reported.
  • The configuration docs list both UDF metric options for batch and streaming, since the metrics are reported in both modes.

Brief change log

  • CodeGeneratorContext#supportsUdfMetrics is true by default and false for the contexts used for constant folding and partition pruning, for pushed-down watermark generators and for SpecializedFunction expression evaluators. Child contexts inherit it. BridgingFunctionGenUtil#udfMetricsTermIfEnabled generates no metrics where it is false.
  • BridgingSqlFunctionCallGen passes no metric name when the function definition is a BuiltInFunctionDefinition. This also covers the internal UNNEST and REPLICATE_ROWS functions.
  • UdfMetrics.register returns the handle already registered on the same udf.<name> group, so all generated classes of one operator record into one histogram and one counter. Handles are cached with weak keys compared by identity, so the group of a finished task is not kept reachable, and a restarted task registers on a new group and gets a new handle.
  • Both UDF metric options use @Documentation.TableOption(execMode = BATCH_STREAMING), and execution_config_configuration.html is regenerated. The metrics docs already say only user-defined functions are metered, so they are unchanged.

Verifying this change

This change added tests and can be verified as follows:

  • ExpressionReductionRulesTest#testExpressionReductionWithUDFAndUdfMetricsEnabled: a UDF call with constant arguments is folded with metrics on.
  • PartitionableSourceTest#testWithUdfAndUdfMetricsEnabled: a UDF in a partition filter plans and prunes with metrics on.
  • UdfMetricsITCase#testFunctionCallingExpressionEvaluator: a SpecializedFunction whose evaluator calls another UDF runs with metrics on.
  • UdfMetricsITCase#testFunctionInPushedDownWatermark: a user-defined function in a WATERMARK expression pushed into the source runs with metrics on.
  • UdfMetricsITCase#testBuiltInFunctionsNotMetered: TYPEOF and UNNEST register no group, while a user-defined function in the same query does.
  • UdfMetricsITCase#testCorrelateInvocationAndConditionShareOneHandle and #testLookupJoinPreFilterAndConditionShareOneHandle: every call is sampled and the exception is counted.
  • UdfMetricsTest#restartedOperatorGetsNewHandle and #doesNotRetainOperatorMetricGroup: the shared handle follows the operator group and does not keep it reachable.
  • The docs change has no runtime effect. A batch job with a scalar and a table function reports udf.* metrics with metrics enabled and none with metrics disabled.
  • Each new test was also run without its part of the change, to confirm it covers that part. The planner plan tests and UdfMetricsITCase, AsyncCalcITCase, AsyncCorrelateITCase, the lookup join, MATCH_RECOGNIZE and watermark push-down ITCases pass.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code 2.1.322 (Claude Opus 5.5)

@flinkbot

flinkbot commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

…uns inside an operator

Code generated for constant folding, partition pruning, pushed-down
watermark generators and SpecializedFunction expression evaluators is
opened without a runtime context. Add CodeGeneratorContext
#supportsUdfMetrics, which is false for these contexts and inherited by
child contexts, and generate UDF metrics only where it is true.
UDF metrics are documented for user-defined functions. Pass no metric
name for a BuiltInFunctionDefinition, which also covers the internal
functions used for UNNEST and INTERSECT ALL.
…erator

Several generated classes of one operator can call the same function and
register its metrics on the same operator metric group, which keeps one
metric per name. UdfMetrics.register now returns the handle already
registered on the same udf.<name> group, so all classes record into one
histogram and one counter. Handles are cached with weak identity keys,
so a finished task's group is not kept reachable.
@weiqingy
weiqingy force-pushed the FLINK-41585-udf-metrics-registration branch from 47282b5 to 6bafa12 Compare November 20, 2026 19:30
@weiqingy
weiqingy requested a balanced review from Copilot November 21, 2026 01:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The test can falsely report a cache leak because its short System.gc() polling window does not guarantee collection of the weakly reachable object.

1 open issue
What changed in this PR

This PR refines Flink Table/SQL UDF metrics so registration is limited to user-defined functions running with an operator runtime context, with shared handles across generated classes in the same operator.

Changes:

  • Skips metrics for built-ins and contexts without a runtime context.
  • Shares metric handles per function and operator using a weak-key cache.
  • Adds regression coverage for planning, runtime registration, restarts, and metric-group retention.
File Description
UdfMetricsTest.java Tests restart isolation and metric-group collection.
UdfMetrics.java Caches shared handles using weak metric-group keys.
PartitionableSourceTest.scala Tests partition pruning with UDF metrics enabled.
ExpressionReductionRulesTest.scala Tests constant folding with UDF metrics enabled.
PartitionableSourceTest.xml Adds expected partition-pruning plans.
ExpressionReductionRulesTest.xml Adds the expected constant-folding plan.
UdfMetricsITCase.java Covers metric exclusions and shared runtime handles.
WatermarkGeneratorCodeGenerator.scala Disables metrics for source-pushed watermark generators.
ExpressionReducer.scala Disables metrics in planning-time contexts.
CodeGeneratorContext.scala Adds inherited UDF-metrics capability.
BridgingSqlFunctionCallGen.scala Excludes built-in functions from instrumentation.
BridgingFunctionGenUtil.scala Checks context capability and disables evaluator metrics.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the UDF metrics changes.

1 open issue

Review effort: Balanced

@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label Nov 21, 2026
… and streaming

UDF metrics are reported in batch mode as well, so label both options
with Batch and Streaming in the configuration docs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants