Skip to content

[4.20.3.1] Tagged domain resource-count recalculation fails when domain counter is missing #14729

Open

Description

@SviridoffA

problem

Domain resource-count recalculation fails with a NullPointerException when a tagged domain counter is absent and recalculated usage is nonzero. This affects both updateResourceCount and the periodic ResourceCountChecker.

The issue was investigated using production logs, read-only SQL queries and source-code inspection. No database repair or destructive reproduction was performed during diagnosis.

The API returns HTTP 530. The relevant stack trace is:

java.lang.NullPointerException: Cannot invoke "com.cloud.configuration.ResourceCountVO.setCount(long)" because "domainRC" is null
    at com.cloud.resourcelimit.ResourceLimitManagerImpl.lambda$recalculateDomainResourceCount$2(ResourceLimitManagerImpl.java:1221)
    at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:48)
    at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
    at com.cloud.utils.db.Transaction.execute(Transaction.java:45)
    at com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateDomainResourceCount(ResourceLimitManagerImpl.java:1184)
    at com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateDomainTaggedResourceCount(ResourceLimitManagerImpl.java:1063)
    at com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateResourceCount(ResourceLimitManagerImpl.java:1128)
    at org.apache.cloudstack.api.command.user.resource.UpdateResourceCountCmd.execute(UpdateResourceCountCmd.java:130)

Read-only inspection confirmed:

  • Positive account-level volume/SSD counters with corresponding domain counters absent.
  • A missing domain primary_storage/SSD counter with positive account-level usage.
  • A disabled but non-removed account contributing positive tagged usage in a failing domain.
  • Successful recalculation for zero-usage domains even though their tagged domain counters remain absent.
  • Parent-domain failure during recursive child-domain recalculation.
  • Repeated failures of the periodic checker instead of automatic recovery.
  • No duplicate counters, negative values or invalid owners in the diagnostic SELECT results. Unique indexes on (type, tag, domain_id) and (type, tag, account_id) are present.

The failure does not require child domains or an explicitly configured tagged limit on the affected domain.

Potential impact is incomplete domain resource accounting. An actual end-to-end quota bypass has not been demonstrated; account and untagged limits may still reject allocations.

Disclosure: This report was generated with ChatGPT using operator-provided SQL results, inspected production logs and public Apache CloudStack source code. Confirmed observations and unverified hypotheses are identified separately. Internal addresses, customer names, UUIDs and authentication parameters have been omitted.

versions

  • Apache CloudStack management package: 4.20.3.1 on all three management servers.
  • Management operating system: Ubuntu; exact release not included in this report.
  • Hypervisor: KVM.
  • Database: separate MySQL cluster; exact version not yet collected.

Relevant global settings:

resource.limit.storage.tags = SSD,HDD
resource.limit.host.tags = <empty>
resourcecount.check.interval = 300

Runtime observations are from 4.20.3.1. No runtime reproduction on another CloudStack version is claimed.

The steps to reproduce the bug

Observed preconditions and action:

  1. A storage tag is configured in resource.limit.storage.tags.
  2. A non-removed account has positive resource usage for that tag and resource type.
  3. The corresponding domain row in cloud.resource_count is absent.
  4. Call updateResourceCount for the domain without an account parameter, or let the periodic resource-count checker run.
  5. Observe the NullPointerException and failed domain recalculation.

Expected: CloudStack safely initializes the missing domain counter and records the recalculated usage.

Actual: The API returns HTTP 530, or the periodic checker logs an exception. Repeated periodic attempts can fail again.

For zero usage, recalculation can return successfully while leaving the domain counter absent.

A possible clean-lab scenario is to create domains and storage-tagged resources before enabling those tags for resource-limit accounting, then enable the tags and recalculate domain usage. This scenario has not been independently reproduced. The exact historical action that originally left the production rows absent has not been established.

What to do about it?

Source inspection identifies the failing path in ResourceLimitManagerImpl.java, version 4.20.3.1:

  1. Accounts and child domains are recalculated.
  2. Existing counter rows are selected and locked.
  3. If the current domain/type/tag row is absent, domainRC remains null and oldResourceCount remains zero.
  4. A nonzero newResourceCount reaches domainRC.setCount(newResourceCount) at line 1221.
  5. With zero usage, that branch is skipped and the missing row is not created.

By contrast, account recalculation can persist a missing account counter. ResourceCountDaoImpl also contains create-on-miss logic for tagged domain counters, but the domain recalculation path does not use it.

Potential regression lead: PR #8903 removed lockDomainRows(), which called listAllRowsToUpdate(), and introduced selection of existing rows. This conclusion is based on diff inspection; it has not been verified by reproducing the scenario against the parent revision.

PR #11138 fixes a different failure, Unable to find account null; the corresponding guard is already present in 4.20.3.1.

A fix should initialize missing domain counters safely and calculate correct usage, including:

  • Positive and zero usage.
  • Disabled but non-removed accounts.
  • Nested domains.
  • Repeated recalculation.
  • Concurrent creation of the same counter by multiple management servers.

Simply skipping a null counter would hide the exception while leaving accounting incomplete.

Is there an existing fix or recommended backport for this missing-domain-counter path?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions