Repository navigation
Conversation
… aggregation On a table with merge-engine=aggregation and fields.<col>.aggregate-function=sum, a DECIMAL column could overflow its declared precision in two silent ways: - The compact fast path in DecimalUtils.add/subtract checked only long overflow (Math.addExact) but not the declared precision, so 99999 + 1 on DECIMAL(5,0) produced Decimal.fromUnscaledLong(100000, 5, 0): a value violating its own precision invariant, later written to data files as an out-of-range decimal (ParquetRowDataWriter casts unscaledLong to int). - The non-compact path returns null from Decimal.fromBigDecimal when the result precision overflows; FieldSumAgg did not check it, so the aggregated column silently became NULL and the accumulated value was lost. This adds a precision bounds check to the compact fast path (falling through to fromBigDecimal, which returns null on overflow) and throws ArithmeticException on the null result in FieldSumAgg agg/retract, following the integral overflow precedent of apache#7906 and apache#7992. DECIMAL is not covered by the fields.<f>.sum.fail-on-overflow option added in apache#9874 (integral types only): a decimal precision overflow has no wrap-around semantics, so silent null or corruption is the only alternative to failing. Jobs that today succeed while writing corrupt decimals will now fail, the same compatibility consequence apache#7906/apache#7992 accepted for integral types.
Add DECIMAL(5,2) cases to DecimalTest#testAddSubtractPrecisionOverflow and a new FieldAggregatorTest#testFieldSumDecimalOverflowWithScale so the compact fast-path bound is pinned at 10^precision rather than 10^(precision + scale): add(999.99, 0.01) and subtract(-999.99, 0.01) return null, and agg/retract throw ArithmeticException. Both fail against the pre-fix compact path.
| inFieldDD.toBigDecimal(), | ||
| mergeFieldDD.precision(), | ||
| mergeFieldDD.scale())); | ||
| } |
There was a problem hiding this comment.
In #9874, fields.<field-name>.sum.fail-on-overflow (default false) was introduced after #7906 and #7992 because FieldSumAgg executes during LSM compaction and merge-on-read over already-committed Level-0 SST files. If individual batches write valid DECIMAL(p, s) values that only overflow when merged across sorted runs during compaction, throwing an unconditional ArithmeticException (ignoring failOnOverflow) can put compaction and readers into an unrecoverable failure loop with no table option to bypass it.
Also note that before this change, non-compact DECIMAL (precision > 18) returned null on overflow in FieldSumAgg (and FieldProductAgg still returns null on DECIMAL overflow).
- Should this check respect
failOnOverflow(i.e.if (failOnOverflow && sum == null)), or is there a fallback strategy whenfailOnOverflow == false? - If
DECIMALis intentionally meant to always throw on overflow regardless offields.<field-name>.sum.fail-on-overflow, could we also updatedocs/docs/primary-key-table/merge-engine/aggregation.mdxunder### sumto clarify thatDECIMALalways throwsArithmeticExceptionon precision overflow?
There was a problem hiding this comment.
Good point — the unconditional throw ignored fields.<field>.sum.fail-on-overflow and could indeed put compaction / merge-on-read over committed files into an unrecoverable failure loop. 35e753d makes the DECIMAL branch honor the option, mirroring the integral types:
fail-on-overflow = true→ throwArithmeticException(as before);fail-on-overflow = false(the default) → the aggregated value becomes NULL, the same resultFieldProductAggalready produces on a DECIMAL overflow, and the same "overflows are ignored by default" contract as TINYINT/SMALLINT/INTEGER/BIGINT.
The default is still a strict improvement over master: with the DecimalUtils bounds check, both the compact and the non-compact path now return NULL, whereas the compact path used to return a Decimal whose unscaled value violated the declared precision and was written to the data files.
Docs are updated in the same commit (aggregation.mdx: a DECIMAL precision overflow makes the aggregated value NULL by default), and testFieldSumDecimalOverflowDisabled pins the default mode for compact and non-compact, agg and retract.
| v1.toUnscaledLong(), v2.toUnscaledLong()); // checks overflow | ||
| return fromUnscaledLong(ls, precision, scale); | ||
| if (ls < POW10[precision] && ls > -POW10[precision]) { | ||
| return fromUnscaledLong(ls, precision, scale); |
There was a problem hiding this comment.
In this compact branch (v1.isCompact() && v2.isCompact() && v1.scale() == v2.scale() && Decimal.isCompact(precision) with scale == v1.scale()), ls is already the exact unscaled result at the target scale. When |ls| >= POW10[precision] (or Math.addExact / Math.subtractExact throws ArithmeticException), falling through to v1.toBigDecimal().add(v2.toBigDecimal()) and fromBigDecimal(bd, precision, scale) is guaranteed to have bd.precision() > precision and return null, after unnecessarily allocating three BigDecimal objects.
We can return null directly from the compact branch on overflow (and similarly in subtract below):
if (ls < POW10[precision] && ls > -POW10[precision]) {
return Decimal.fromUnscaledLong(ls, precision, scale);
}
} catch (ArithmeticException ignored) {
}
return null;There was a problem hiding this comment.
Applied in 35e753d, with one refinement: the fast path now also requires scale == v1.scale() before returning null directly. Without that, a call whose target scale differs from the operands' scale — previously guarded only by assert scale == v1.scale(), i.e. silently wrong in production with asserts disabled — would return null instead of rescaling through fromBigDecimal. That case is now pinned by new assertions in DecimalTest#testAddSubtractPrecisionOverflow (add(10, 15, 5, 2) must be 25.00, not 0.25).
With the condition in place, ls is the exact unscaled result at the target scale, so on a bound miss or an addExact/subtractExact overflow the BigDecimal path is guaranteed to return null (|ls| >= 10^precision implies bd.precision() > precision; a long overflow implies |sum| >= 2^63 > 10^18 >= 10^precision) — returning null directly skips the three BigDecimal allocations.
…fast path Review feedback on the decimal overflow fix: - FieldSumAgg threw ArithmeticException on decimal precision overflow unconditionally, ignoring fields.<field>.sum.fail-on-overflow (default false). Since the aggregation runs during LSM compaction and merge-on-read over committed files, an unconditional throw could put compaction and readers into an unrecoverable failure loop. The option now decides: throw when true, NULL result when false, consistent with the integral types' 'overflows are ignored by default' contract and with FieldProductAgg's DECIMAL overflow behavior. Document the DECIMAL default in the aggregation merge-engine docs. - DecimalUtils.add/subtract fell through to BigDecimal on a compact overflow only to re-derive a result that fromBigDecimal is guaranteed to reject; return null directly from the fast path instead. The fast path now also requires scale == v1.scale(), so a target scale different from the operands' scale rescales via BigDecimal instead of reusing the unscaled value under a new scale, which was previously guarded only by an assert.
Purpose
On a table with
merge-engine=aggregationandfields.<col>.aggregate-function='sum', a DECIMAL column can overflow its declared precision in two silent ways:DecimalUtils.add/subtractchecked only long overflow (Math.addExact) but not the declared precision, so99999 + 1onDECIMAL(5,0)returnedDecimal.fromUnscaledLong(100000, 5, 0): a value violating its own precision invariant. The corrupt value is then written to data files —ParquetRowDataWriter.Int32WritercastsunscaledLongtointand stores the out-of-range unscaled value under the column's declared precision.Decimal.fromBigDecimalwhen the result precision overflows;FieldSumAggdid not check it, so the aggregated column silently became NULL and the whole accumulated value was lost — the next merge restarted the sum from scratch.This adds a precision bounds check to the compact fast path, so both paths now return null on overflow instead of a corrupt value.
FieldSumAgg#agg/#retracthonorfields.<field>.sum.fail-on-overflow(introduced for the integral types in #9874): throwArithmeticExceptionwhen the option istrue, produce NULL when it isfalse(the default) — the same "overflows are ignored by default" contract as the integral types, and the same resultFieldProductAggalready produces on a DECIMAL overflow. Honoring the option matters because the aggregation runs during LSM compaction and merge-on-read over already-committed files, where an unconditional throw could put compaction and readers into an unrecoverable failure loop (review feedback).The compact fast path now also requires
scale == v1.scale(): a target scale different from the operands' scale rescales viafromBigDecimalinstead of reusing the unscaled value under a new scale (previously guarded only by an assert, i.e. silently wrong in production with asserts disabled).Compatibility consequence: with the default
fail-on-overflow = false, jobs that today write corrupt decimals now write NULL for the overflowing aggregate; setting the option totrueturns the overflow into a loudArithmeticException.Tests
DecimalTest#testAddSubtractPrecisionOverflow(paimon-common): compact and non-compactadd/subtractreturn null instead of an out-of-rangeDecimal, for bothDECIMAL(5,0)andDECIMAL(5,2)(the bound is10^precision, not10^(precision+scale)); a target scale different from the operands' scale rescales correctly (add(10, 15, 5, 2)is25.00, not0.25).FieldAggregatorTest#testFieldSumDecimalOverflow/#testFieldSumDecimalRetractOverflow/#testFieldSumDecimalOverflowWithScale(compact,DECIMAL(5,0)/(5,2)) and#testFieldSumDecimalOverflowNonCompact(DECIMAL(38,2)), all withfail-on-overflow = true:agg/retractthrowArithmeticExceptionon overflow.FieldAggregatorTest#testFieldSumDecimalOverflowDisabled(default option):agg/retractreturn NULL on overflow, compact and non-compact.DecimalTestcases exercisingadd()and the fullFieldAggregatorTestpass unchanged.API and Format
No.
Documentation
Yes:
docs/docs/primary-key-table/merge-engine/aggregation.mdxnow states that for DECIMAL a precision overflow makes the aggregated value NULL by default, and thatfields.<field-name>.sum.fail-on-overflowapplies to DECIMAL as well.