Repository navigation
feat(aws-marketplace): bill a listing's LLM usage to AWS hourly through a metering ledger - #9921
waleedlatif1 wants to merge 1 commit into
Conversation
…gh a metering ledger
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
 [Critical impact] The PR adds an automated path that reports customer usage charges to AWS Marketplace. Fix the missed billing paths and satisfy the shared response contract requirement before merging. Findings
SummaryAdds hourly AWS Marketplace billing for billed model usage.
Before merging, fix skipped usage for existing links, renewal boundaries, later cost increases, and deprovisioned licenses without an expiry. The new cron route also needs the required shared response contract.
|
| and( | ||
| inArray(awsMarketplaceLicense.productCode, [...products.keys()]), | ||
| isNotNull(awsMarketplaceLicense.organizationId), | ||
| isNotNull(awsMarketplaceLicense.organizationLinkedAt), |
There was a problem hiding this comment.
Migration 0456 leaves organizationLinkedAt null on existing linked licenses, and this filter excludes them. Neither verification nor linking the same organization again fills the stamp. Their future model usage therefore never reaches AWS. Set a safe billing start for existing links during rollout.
| const from = Math.max( | ||
| license.organizationLinkedAt.getTime(), | ||
| floorHour(now - COMPUTE_WINDOW_MS) + HOUR_MS, | ||
| last ? last.hourStart.getTime() + HOUR_MS : 0 |
There was a problem hiding this comment.
If the old license expires at 10:20, it writes a partial row for 10:00. The successor reads that organization-wide row and starts at 11:00, permanently skipping usage from 10:20 to 11:00. Preserve the exact billed-through time and account for the remaining usage rather than treating every row as a complete hour.
| const from = Math.max( | ||
| license.organizationLinkedAt.getTime(), | ||
| floorHour(now - COMPUTE_WINDOW_MS) + HOUR_MS, | ||
| last ? last.hourStart.getTime() + HOUR_MS : 0 | ||
| ) |
There was a problem hiding this comment.
recordCumulativeUsage increases an existing row's cost without changing its createdAt. /api/billing/update-cost uses that path for partial, terminal, and retry callbacks. If a run or retry adds cost after its original hour has been metered, this cursor never reads that increase. Record newly added costs separately or provide another durable way to capture changes to previously read rows.
| and( | ||
| eq(awsMarketplaceLicense.status, 'inactive'), | ||
| gte(awsMarketplaceLicense.expiresAt, windowStart) |
There was a problem hiding this comment.
An active entitlement may have no ExpirationDate. When AWS later returns no entitlement, verification marks the license inactive but leaves expiresAt null. This comparison excludes that license entirely, so usage since its last report is never flushed. Store an effective end time for this path and include these licenses when recording final usage.
| return NextResponse.json( | ||
| { success: true, message: 'AWS Marketplace metering started', status: 'started' }, | ||
| { status: 202 } | ||
| ) |
There was a problem hiding this comment.
The new route defines its JSON responses locally. The repository requires every API route's request and response shapes to live under apps/sim/lib/api/contracts/** through defineRouteContract. Having no request input does not remove the response requirement. Add a shared response contract before merging instead of relying only on the audit allowlist.
Context Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
5 issues found across 13 files
Confidence score: 2/5
route.tsis not scheduled by the Helm chart, so Marketplace usage is never metered in the standard deployment. Add the intended 15-minute cronjob.0456_aws_marketplace_usage_report.sqlleaves existing organization-linked licenses withNULL, which excludes their usage from metering. Backfill those linked rows during the migration.metering.tsignoresexpiresAtwhile a license is still locally marked active, so usage after entitlement expiry can be billed to the old license. Make the expiry bound take precedence.metering.tscan leave usage unbilled between a license expiring mid-hour and the next license starting at the following hour boundary. Make the handoff cover that interval.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/lib/billing/aws-marketplace/metering.ts">
<violation number="1" location="apps/sim/lib/billing/aws-marketplace/metering.ts:147">
P2: `billableUntil` ignores a stored `expiresAt` once the license is still locally `active`. If re-verification has not yet changed the status after entitlement expiry, settled usage after expiry is billed to the old license; cap the end time at `expiresAt` whenever it has passed.</violation>
<violation number="2" location="apps/sim/lib/billing/aws-marketplace/metering.ts:184">
P2: A mid-hour license handoff drops usage. When the lapsed license's final row covers only up to its expiry, the next license starts at the following hour boundary, so usage between expiry and that boundary is never billed. Start the successor's chain at the lapsed license's expiry instead of `last.hourStart + HOUR_MS`, or carry the exact boundary forward.</violation>
</file>
<file name="apps/sim/app/api/cron/aws-marketplace-meter/route.ts">
<violation number="1" location="apps/sim/app/api/cron/aws-marketplace-meter/route.ts:25">
P1: This handler is not scheduled by the Helm chart, so Marketplace usage metering never runs in the standard deployment. Add a `cronjobs.jobs` entry for `/api/cron/aws-marketplace-meter` with the intended 15-minute schedule.</violation>
</file>
<file name="packages/db/migrations/0456_aws_marketplace_usage_report.sql">
<violation number="1" location="packages/db/migrations/0456_aws_marketplace_usage_report.sql:19">
P1: Existing organization-linked licenses receive `NULL` here and are then excluded by the metering query, so their organization’s model usage is never billed. Backfill linked rows to the migration time while leaving unlinked rows `NULL`.</violation>
</file>
<file name="apps/sim/lib/billing/aws-marketplace/metering.integration.ts">
<violation number="1" location="apps/sim/lib/billing/aws-marketplace/metering.integration.ts:153">
P3: This cleanup leaves every fixture license behind because `organizationId` uses `ON DELETE SET NULL`. Delete the licenses by organization before deleting the organizations, otherwise repeated integration runs accumulate orphaned rows.</violation>
</file>
| * runs from overlapping, and the ledger makes a replayed run harmless. A no-op on deployments | ||
| * where no configured listing bills usage. | ||
| */ | ||
| export const GET = withRouteHandler(async (request: NextRequest) => { |
There was a problem hiding this comment.
P1: This handler is not scheduled by the Helm chart, so Marketplace usage metering never runs in the standard deployment. Add a cronjobs.jobs entry for /api/cron/aws-marketplace-meter with the intended 15-minute schedule.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/sim/app/api/cron/aws-marketplace-meter/route.ts, line 25:
<comment>This handler is not scheduled by the Helm chart, so Marketplace usage metering never runs in the standard deployment. Add a `cronjobs.jobs` entry for `/api/cron/aws-marketplace-meter` with the intended 15-minute schedule.</comment>
<file context>
@@ -0,0 +1,67 @@
+ * runs from overlapping, and the ledger makes a replayed run harmless. A no-op on deployments
+ * where no configured listing bills usage.
+ */
+export const GET = withRouteHandler(async (request: NextRequest) => {
+ const authError = verifyCronAuth(request, 'AWS Marketplace metering')
+ if (authError) {
</file context>
| "updated_at" timestamp DEFAULT now() NOT NULL | ||
| ); | ||
| --> statement-breakpoint | ||
| ALTER TABLE "aws_marketplace_license" ADD COLUMN "organization_linked_at" timestamp;--> statement-breakpoint |
There was a problem hiding this comment.
P1: Existing organization-linked licenses receive NULL here and are then excluded by the metering query, so their organization’s model usage is never billed. Backfill linked rows to the migration time while leaving unlinked rows NULL.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/db/migrations/0456_aws_marketplace_usage_report.sql, line 19:
<comment>Existing organization-linked licenses receive `NULL` here and are then excluded by the metering query, so their organization’s model usage is never billed. Backfill linked rows to the migration time while leaving unlinked rows `NULL`.</comment>
<file context>
@@ -0,0 +1,24 @@
+ "updated_at" timestamp DEFAULT now() NOT NULL
+);
+--> statement-breakpoint
+ALTER TABLE "aws_marketplace_license" ADD COLUMN "organization_linked_at" timestamp;--> statement-breakpoint
+ALTER TABLE "aws_marketplace_usage_report" ADD CONSTRAINT "aws_marketplace_usage_report_organization_id_organization_id_fk" FOREIGN KEY ("organization_id") REFERENCES "public"."organization"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
+ALTER TABLE "aws_marketplace_usage_report" ADD CONSTRAINT "aws_marketplace_usage_report_license_id_aws_marketplace_license_id_fk" FOREIGN KEY ("license_id") REFERENCES "public"."aws_marketplace_license"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
</file context>
| ALTER TABLE "aws_marketplace_license" ADD COLUMN "organization_linked_at" timestamp;--> statement-breakpoint | |
| ALTER TABLE "aws_marketplace_license" ADD COLUMN "organization_linked_at" timestamp;--> statement-breakpoint | |
| UPDATE "aws_marketplace_license" SET "organization_linked_at" = now() WHERE "organization_id" IS NOT NULL;--> statement-breakpoint |
| const from = Math.max( | ||
| license.organizationLinkedAt.getTime(), | ||
| floorHour(now - COMPUTE_WINDOW_MS) + HOUR_MS, | ||
| last ? last.hourStart.getTime() + HOUR_MS : 0 |
There was a problem hiding this comment.
P2: A mid-hour license handoff drops usage. When the lapsed license's final row covers only up to its expiry, the next license starts at the following hour boundary, so usage between expiry and that boundary is never billed. Start the successor's chain at the lapsed license's expiry instead of last.hourStart + HOUR_MS, or carry the exact boundary forward.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/sim/lib/billing/aws-marketplace/metering.ts, line 184:
<comment>A mid-hour license handoff drops usage. When the lapsed license's final row covers only up to its expiry, the next license starts at the following hour boundary, so usage between expiry and that boundary is never billed. Start the successor's chain at the lapsed license's expiry instead of `last.hourStart + HOUR_MS`, or carry the exact boundary forward.</comment>
<file context>
@@ -0,0 +1,403 @@
+ const from = Math.max(
+ license.organizationLinkedAt.getTime(),
+ floorHour(now - COMPUTE_WINDOW_MS) + HOUR_MS,
+ last ? last.hourStart.getTime() + HOUR_MS : 0
+ )
+ const until = billableUntil(license, now)
</file context>
| function billableUntil(license: MeteredLicense, now: number): number { | ||
| const settled = now - SETTLE_MS | ||
| const expiry = license.expiresAt?.getTime() | ||
| if (license.status !== 'active' && expiry !== undefined && expiry <= settled) return expiry |
There was a problem hiding this comment.
P2: billableUntil ignores a stored expiresAt once the license is still locally active. If re-verification has not yet changed the status after entitlement expiry, settled usage after expiry is billed to the old license; cap the end time at expiresAt whenever it has passed.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/sim/lib/billing/aws-marketplace/metering.ts, line 147:
<comment>`billableUntil` ignores a stored `expiresAt` once the license is still locally `active`. If re-verification has not yet changed the status after entitlement expiry, settled usage after expiry is billed to the old license; cap the end time at `expiresAt` whenever it has passed.</comment>
<file context>
@@ -0,0 +1,403 @@
+function billableUntil(license: MeteredLicense, now: number): number {
+ const settled = now - SETTLE_MS
+ const expiry = license.expiresAt?.getTime()
+ if (license.status !== 'active' && expiry !== undefined && expiry <= settled) return expiry
+ return floorHour(settled)
+}
</file context>
|
|
||
| afterEach(async () => { | ||
| if (organizationIds.length) { | ||
| await db.delete(organization).where(inArray(organization.id, organizationIds.splice(0))) |
There was a problem hiding this comment.
P3: This cleanup leaves every fixture license behind because organizationId uses ON DELETE SET NULL. Delete the licenses by organization before deleting the organizations, otherwise repeated integration runs accumulate orphaned rows.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At apps/sim/lib/billing/aws-marketplace/metering.integration.ts, line 153:
<comment>This cleanup leaves every fixture license behind because `organizationId` uses `ON DELETE SET NULL`. Delete the licenses by organization before deleting the organizations, otherwise repeated integration runs accumulate orphaned rows.</comment>
<file context>
@@ -0,0 +1,348 @@
+
+afterEach(async () => {
+ if (organizationIds.length) {
+ await db.delete(organization).where(inArray(organization.id, organizationIds.splice(0)))
+ }
+ if (userIds.length) await db.delete(user).where(inArray(user.id, userIds.splice(0)))
</file context>
Summary
platform-with-usagebills the organization's billedmodelusage (BYOK ismodel_unbilledand never billed) asllm_creditsof $0.01;contractbills nothing beyond the contract. Repricing or changing what counts is a plan editaws_marketplace_usage_reportledger, one row per organization, dimension and closed hour, each billed to one license. Its unique key keeps an hour from ever being billed twice (reruns, overlapping licenses, renewals); running totals carry fractions of a credit forward exactly (Decimal)meterAwsMarketplaceUsage(cron/api/cron/aws-marketplace-meter, every 15 min) writes rows for newly settled hours under a per-organization advisory lock, then sends pending rows withBatchMeterUsagebyLicenseArn+CustomerAWSAccountId(noProductCode), 25 per call, grouped per licenseaws_marketplace_license.organization_linked_at, stamped wherever a license is linked to an organization (buyer link, team link, renewal inheritance); usage counts from that momentType of Change
Testing
metering.integration.ts(9, real Postgres; only the AWS metering client is mocked) + a new link-time test inlifecycle.integration.ts; marketplace integration suites 86/86, unit 10/10bun run lint,tsc,check:migrations,check:audits(the only local failure is the gitignored docs.contentregistry CI generates), drizzle-kit generate reports no driftChecklist
test-auditauthoring gate)