Repository navigation
Description
problem
Domain resource-count recalculation fails with a NullPointerException when a tagged domain counter is absent and recalculated usage is nonzero. This affects both updateResourceCount and the periodic ResourceCountChecker.
The issue was investigated using production logs, read-only SQL queries and source-code inspection. No database repair or destructive reproduction was performed during diagnosis.
The API returns HTTP 530. The relevant stack trace is:
java.lang.NullPointerException: Cannot invoke "com.cloud.configuration.ResourceCountVO.setCount(long)" because "domainRC" is null
at com.cloud.resourcelimit.ResourceLimitManagerImpl.lambda$recalculateDomainResourceCount$2(ResourceLimitManagerImpl.java:1221)
at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
at com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateDomainResourceCount(ResourceLimitManagerImpl.java:1184)
at com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateDomainTaggedResourceCount(ResourceLimitManagerImpl.java:1063)
at com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateResourceCount(ResourceLimitManagerImpl.java:1128)
at org.apache.cloudstack.api.command.user.resource.UpdateResourceCountCmd.execute(UpdateResourceCountCmd.java:130)
Read-only inspection confirmed:
- Positive account-level
volume/SSDcounters with corresponding domain counters absent. - A missing domain
primary_storage/SSDcounter with positive account-level usage. - A disabled but non-removed account contributing positive tagged usage in a failing domain.
- Successful recalculation for zero-usage domains even though their tagged domain counters remain absent.
- Parent-domain failure during recursive child-domain recalculation.
- Repeated failures of the periodic checker instead of automatic recovery.
- No duplicate counters, negative values or invalid owners in the diagnostic SELECT results. Unique indexes on
(type, tag, domain_id)and(type, tag, account_id)are present.
The failure does not require child domains or an explicitly configured tagged limit on the affected domain.
Potential impact is incomplete domain resource accounting. An actual end-to-end quota bypass has not been demonstrated; account and untagged limits may still reject allocations.
Disclosure: This report was generated with ChatGPT using operator-provided SQL results, inspected production logs and public Apache CloudStack source code. Confirmed observations and unverified hypotheses are identified separately. Internal addresses, customer names, UUIDs and authentication parameters have been omitted.
versions
- Apache CloudStack management package: 4.20.3.1 on all three management servers.
- Management operating system: Ubuntu; exact release not included in this report.
- Hypervisor: KVM.
- Database: separate MySQL cluster; exact version not yet collected.
Relevant global settings:
resource.limit.storage.tags = SSD,HDD
resource.limit.host.tags = <empty>
resourcecount.check.interval = 300
Runtime observations are from 4.20.3.1. No runtime reproduction on another CloudStack version is claimed.
The steps to reproduce the bug
Observed preconditions and action:
- A storage tag is configured in
resource.limit.storage.tags. - A non-removed account has positive resource usage for that tag and resource type.
- The corresponding domain row in
cloud.resource_countis absent. - Call
updateResourceCountfor the domain without anaccountparameter, or let the periodic resource-count checker run. - Observe the NullPointerException and failed domain recalculation.
Expected: CloudStack safely initializes the missing domain counter and records the recalculated usage.
Actual: The API returns HTTP 530, or the periodic checker logs an exception. Repeated periodic attempts can fail again.
For zero usage, recalculation can return successfully while leaving the domain counter absent.
A possible clean-lab scenario is to create domains and storage-tagged resources before enabling those tags for resource-limit accounting, then enable the tags and recalculate domain usage. This scenario has not been independently reproduced. The exact historical action that originally left the production rows absent has not been established.
What to do about it?
Source inspection identifies the failing path in ResourceLimitManagerImpl.java, version 4.20.3.1:
- Accounts and child domains are recalculated.
- Existing counter rows are selected and locked.
- If the current domain/type/tag row is absent,
domainRCremainsnullandoldResourceCountremains zero. - A nonzero
newResourceCountreachesdomainRC.setCount(newResourceCount)at line 1221. - With zero usage, that branch is skipped and the missing row is not created.
By contrast, account recalculation can persist a missing account counter. ResourceCountDaoImpl also contains create-on-miss logic for tagged domain counters, but the domain recalculation path does not use it.
Potential regression lead: PR #8903 removed lockDomainRows(), which called listAllRowsToUpdate(), and introduced selection of existing rows. This conclusion is based on diff inspection; it has not been verified by reproducing the scenario against the parent revision.
PR #11138 fixes a different failure, Unable to find account null; the corresponding guard is already present in 4.20.3.1.
A fix should initialize missing domain counters safely and calculate correct usage, including:
- Positive and zero usage.
- Disabled but non-removed accounts.
- Nested domains.
- Repeated recalculation.
- Concurrent creation of the same counter by multiple management servers.
Simply skipping a null counter would hide the exception while leaving accounting incomplete.
Is there an existing fix or recommended backport for this missing-domain-counter path?
Activity