Skip to content

[core] Fix silent decimal overflow in sum aggregation - #11308

Open
sundapeng wants to merge 4 commits into
apache:masterfrom
sundapeng:decimal-sum-agg-overflow
Open

sundapeng wants to merge 4 commits into
apache:masterfrom
sundapeng:decimal-sum-agg-overflow

Conversation

@sundapeng

@sundapeng sundapeng commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Purpose

On a table with merge-engine=aggregation and fields.<col>.aggregate-function='sum', a DECIMAL column can overflow its declared precision in two silent ways:

  1. The compact fast path in DecimalUtils.add/subtract checked only long overflow (Math.addExact) but not the declared precision, so 99999 + 1 on DECIMAL(5,0) returned Decimal.fromUnscaledLong(100000, 5, 0): a value violating its own precision invariant. The corrupt value is then written to data files — ParquetRowDataWriter.Int32Writer casts unscaledLong to int and stores the out-of-range unscaled value under the column's declared precision.
  2. The non-compact path returns null from Decimal.fromBigDecimal when the result precision overflows; FieldSumAgg did not check it, so the aggregated column silently became NULL and the whole accumulated value was lost — the next merge restarted the sum from scratch.

This adds a precision bounds check to the compact fast path, so both paths now return null on overflow instead of a corrupt value. FieldSumAgg#agg/#retract honor fields.<field>.sum.fail-on-overflow (introduced for the integral types in #9874): throw ArithmeticException when the option is true, produce NULL when it is false (the default) — the same "overflows are ignored by default" contract as the integral types, and the same result FieldProductAgg already produces on a DECIMAL overflow. Honoring the option matters because the aggregation runs during LSM compaction and merge-on-read over already-committed files, where an unconditional throw could put compaction and readers into an unrecoverable failure loop (review feedback).

The compact fast path now also requires scale == v1.scale(): a target scale different from the operands' scale rescales via fromBigDecimal instead of reusing the unscaled value under a new scale (previously guarded only by an assert, i.e. silently wrong in production with asserts disabled).

Compatibility consequence: with the default fail-on-overflow = false, jobs that today write corrupt decimals now write NULL for the overflowing aggregate; setting the option to true turns the overflow into a loud ArithmeticException.

Tests

  • DecimalTest#testAddSubtractPrecisionOverflow (paimon-common): compact and non-compact add/subtract return null instead of an out-of-range Decimal, for both DECIMAL(5,0) and DECIMAL(5,2) (the bound is 10^precision, not 10^(precision+scale)); a target scale different from the operands' scale rescales correctly (add(10, 15, 5, 2) is 25.00, not 0.25).
  • FieldAggregatorTest#testFieldSumDecimalOverflow / #testFieldSumDecimalRetractOverflow / #testFieldSumDecimalOverflowWithScale (compact, DECIMAL(5,0) / (5,2)) and #testFieldSumDecimalOverflowNonCompact (DECIMAL(38,2)), all with fail-on-overflow = true: agg/retract throw ArithmeticException on overflow.
  • FieldAggregatorTest#testFieldSumDecimalOverflowDisabled (default option): agg/retract return NULL on overflow, compact and non-compact.
  • Existing DecimalTest cases exercising add() and the full FieldAggregatorTest pass unchanged.

API and Format

No.

Documentation

Yes: docs/docs/primary-key-table/merge-engine/aggregation.mdx now states that for DECIMAL a precision overflow makes the aggregated value NULL by default, and that fields.<field-name>.sum.fail-on-overflow applies to DECIMAL as well.

… aggregation

On a table with merge-engine=aggregation and
fields.<col>.aggregate-function=sum, a DECIMAL column could overflow its
declared precision in two silent ways:

- The compact fast path in DecimalUtils.add/subtract checked only long
  overflow (Math.addExact) but not the declared precision, so 99999 + 1 on
  DECIMAL(5,0) produced Decimal.fromUnscaledLong(100000, 5, 0): a value
  violating its own precision invariant, later written to data files as an
  out-of-range decimal (ParquetRowDataWriter casts unscaledLong to int).
- The non-compact path returns null from Decimal.fromBigDecimal when the
  result precision overflows; FieldSumAgg did not check it, so the
  aggregated column silently became NULL and the accumulated value was lost.

This adds a precision bounds check to the compact fast path (falling
through to fromBigDecimal, which returns null on overflow) and throws
ArithmeticException on the null result in FieldSumAgg agg/retract, following
the integral overflow precedent of apache#7906 and apache#7992. DECIMAL is not covered
by the fields.<f>.sum.fail-on-overflow option added in apache#9874 (integral types
only): a decimal precision overflow has no wrap-around semantics, so silent
null or corruption is the only alternative to failing.

Jobs that today succeed while writing corrupt decimals will now fail, the
same compatibility consequence apache#7906/apache#7992 accepted for integral types.
Add DECIMAL(5,2) cases to DecimalTest#testAddSubtractPrecisionOverflow
and a new FieldAggregatorTest#testFieldSumDecimalOverflowWithScale so
the compact fast-path bound is pinned at 10^precision rather than
10^(precision + scale): add(999.99, 0.01) and subtract(-999.99, 0.01)
return null, and agg/retract throw ArithmeticException. Both fail
against the pre-fix compact path.
inFieldDD.toBigDecimal(),
mergeFieldDD.precision(),
mergeFieldDD.scale()));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #9874, fields.<field-name>.sum.fail-on-overflow (default false) was introduced after #7906 and #7992 because FieldSumAgg executes during LSM compaction and merge-on-read over already-committed Level-0 SST files. If individual batches write valid DECIMAL(p, s) values that only overflow when merged across sorted runs during compaction, throwing an unconditional ArithmeticException (ignoring failOnOverflow) can put compaction and readers into an unrecoverable failure loop with no table option to bypass it.

Also note that before this change, non-compact DECIMAL (precision > 18) returned null on overflow in FieldSumAgg (and FieldProductAgg still returns null on DECIMAL overflow).

  1. Should this check respect failOnOverflow (i.e. if (failOnOverflow && sum == null)), or is there a fallback strategy when failOnOverflow == false?
  2. If DECIMAL is intentionally meant to always throw on overflow regardless of fields.<field-name>.sum.fail-on-overflow, could we also update docs/docs/primary-key-table/merge-engine/aggregation.mdx under ### sum to clarify that DECIMAL always throws ArithmeticException on precision overflow?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point — the unconditional throw ignored fields.<field>.sum.fail-on-overflow and could indeed put compaction / merge-on-read over committed files into an unrecoverable failure loop. 35e753d makes the DECIMAL branch honor the option, mirroring the integral types:

  • fail-on-overflow = true → throw ArithmeticException (as before);
  • fail-on-overflow = false (the default) → the aggregated value becomes NULL, the same result FieldProductAgg already produces on a DECIMAL overflow, and the same "overflows are ignored by default" contract as TINYINT/SMALLINT/INTEGER/BIGINT.

The default is still a strict improvement over master: with the DecimalUtils bounds check, both the compact and the non-compact path now return NULL, whereas the compact path used to return a Decimal whose unscaled value violated the declared precision and was written to the data files.

Docs are updated in the same commit (aggregation.mdx: a DECIMAL precision overflow makes the aggregated value NULL by default), and testFieldSumDecimalOverflowDisabled pins the default mode for compact and non-compact, agg and retract.

v1.toUnscaledLong(), v2.toUnscaledLong()); // checks overflow
return fromUnscaledLong(ls, precision, scale);
if (ls < POW10[precision] && ls > -POW10[precision]) {
return fromUnscaledLong(ls, precision, scale);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this compact branch (v1.isCompact() && v2.isCompact() && v1.scale() == v2.scale() && Decimal.isCompact(precision) with scale == v1.scale()), ls is already the exact unscaled result at the target scale. When |ls| >= POW10[precision] (or Math.addExact / Math.subtractExact throws ArithmeticException), falling through to v1.toBigDecimal().add(v2.toBigDecimal()) and fromBigDecimal(bd, precision, scale) is guaranteed to have bd.precision() > precision and return null, after unnecessarily allocating three BigDecimal objects.

We can return null directly from the compact branch on overflow (and similarly in subtract below):

                if (ls < POW10[precision] && ls > -POW10[precision]) {
                    return Decimal.fromUnscaledLong(ls, precision, scale);
                }
            } catch (ArithmeticException ignored) {
            }
            return null;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 35e753d, with one refinement: the fast path now also requires scale == v1.scale() before returning null directly. Without that, a call whose target scale differs from the operands' scale — previously guarded only by assert scale == v1.scale(), i.e. silently wrong in production with asserts disabled — would return null instead of rescaling through fromBigDecimal. That case is now pinned by new assertions in DecimalTest#testAddSubtractPrecisionOverflow (add(10, 15, 5, 2) must be 25.00, not 0.25).

With the condition in place, ls is the exact unscaled result at the target scale, so on a bound miss or an addExact/subtractExact overflow the BigDecimal path is guaranteed to return null (|ls| >= 10^precision implies bd.precision() > precision; a long overflow implies |sum| >= 2^63 > 10^18 >= 10^precision) — returning null directly skips the three BigDecimal allocations.

…fast path

Review feedback on the decimal overflow fix:

- FieldSumAgg threw ArithmeticException on decimal precision overflow
  unconditionally, ignoring fields.<field>.sum.fail-on-overflow
  (default false). Since the aggregation runs during LSM compaction and
  merge-on-read over committed files, an unconditional throw could put
  compaction and readers into an unrecoverable failure loop. The option
  now decides: throw when true, NULL result when false, consistent with
  the integral types' 'overflows are ignored by default' contract and
  with FieldProductAgg's DECIMAL overflow behavior. Document the DECIMAL
  default in the aggregation merge-engine docs.

- DecimalUtils.add/subtract fell through to BigDecimal on a compact
  overflow only to re-derive a result that fromBigDecimal is guaranteed
  to reject; return null directly from the fast path instead. The fast
  path now also requires scale == v1.scale(), so a target scale
  different from the operands' scale rescales via BigDecimal instead of
  reusing the unscaled value under a new scale, which was previously
  guarded only by an assert.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants