fix(jobs): requeue PostgreSQL lanes whose durable work has no River job by ThomasK33 · Pull Request #726 · coder/openstorage

ThomasK33 · 2026-11-06T09:26:45Z

Why

Every accepted write to a forward_on_push repository must reach the upstream, and no crash window may silently lose a forward or its mirror-back pull. On PostgreSQL, River keeps each sync and push request alive through the repository's unique job, and the job's completion gates keep it while a claim holds a consumed request (#679 for push, #716 for pull). River discards that job after its third crashed attempt (syncRescueAttempts), an operator can delete job rows, and a process can die or a job insert can fail right after pull-upstream, push-upstream or the forward poke marked the request. The upstream row then keeps push_pending, sync_pending or an expired claim that holds a consumed request, and nothing turns it into a run until an unrelated later write or API call. SQLite and S3-WAL do not have this gap: their sweep reads the rows (ADR-50).

Fix

Each PostgreSQL serve replica scans the served organization's live repositories every five minutes, the first time 15 seconds after boot (jobs.River.StartLaneScan).

List. meta.PGListLaneWork returns lanes whose claim guard holds now: a pending request with no claim, or an expired claim. These are the guards of TryBeginPush and TryBeginSync. A lapsed legacy sync claim counts only with OPENSTORAGE_SYNC_STEAL_LEGACY. Soft-deleted repos and other organizations' repos are left out.
Insert. For each lane the scan inserts the job that EnqueuePush or EnqueueSync inserts. River's uniqueness makes the insert a no-op while a live job covers the lane, so every replica can scan.
Re-check. A new job is kept only if the lane still has work after the insert, in the same transaction (meta.PGLaneWork). The list reads without locks, and a run can consume the work before the insert. After an insert that was not a duplicate, the new job is the lane's only live job, and it cannot run before the transaction commits. So the re-check sees every claim that consumed the listed work, and such a lane gets no second run with its own events.
Bounds. A pass reads up to 1,000 lanes per query in repository-ID order and ends within 30 seconds. A pass that runs out of time keeps its cursor, and the next pass continues after its last lane. A lane whose requeue fails, or waits more than five seconds on a locked job row, is logged and skipped. DrainSyncRuns and Close stop the scan before River stops. A stop that cuts a pass short logs no failure, and a later StartLaneScan fails its assertion.

Each requeue logs lane scan: requeued a lane whose work had no job with org, repo and lane. The production deployment guide lists it under "What to monitor".

Rolling upgrade. No schema change, no new index, no new job kind or queue, and the same args as EnqueuePush and EnqueueSync. Older replicas do not scan, and they run the jobs that newer replicas insert. The scan also recovers claims that an older binary's gate completed its job over.

Cost. No index covers the lane guards, so each list query costs time in proportion to the number of linked repositories, whatever the page size. A verifier measured it on PostgreSQL 18.6 with 300,000 repositories: 22 to 61 ms per query with one organization, and up to about 1 s when the served organization held 165,000 of them. A replica runs one such query per pass when few lanes have work, and passes are five minutes apart. Each lane with work also costs one insert, which is a no-op upsert on the live job when one exists. No new index is needed at this size.

Tests

Test	Package	Result on main (`d42b6c7`)
`TestServeRequeuesLanesWithoutJob`	`cmd/openstorage`, PostgreSQL	fails: `stranded lanes got no job: 0 push jobs, 0 sync jobs`
`TestRiverLaneScanRequeuesLanesWithoutJob`	`internal/jobs`, PostgreSQL	new API
`TestRiverLaneScanRechecksAfterInsert`	`internal/jobs`, PostgreSQL	new API
`TestRiverLaneScanRunsStrandedLanes`	`internal/jobs`, PostgreSQL	new API
`TestRiverLaneScanResumesAfterTimeout`	`internal/jobs`, PostgreSQL	new API
`TestRiverLaneScanSkipsFailingLane`	`internal/jobs`, PostgreSQL	new API
`TestRiverLaneScanLogsOutOfTimeAtInfo`	`internal/jobs`, PostgreSQL	new API
`TestRiverLaneScanPassSchedule`	`internal/jobs`, PostgreSQL	new API
`TestRiverLaneScanStopEndsPassQuietly`	`internal/jobs`, PostgreSQL	new API
`TestPGLaneWorkMatchesClaimGuards`	`internal/meta`, PostgreSQL	new API

The table test covers discarded push and pull jobs, expired push and sync claims, live claims, a lapsed legacy claim without the opt-in, a lane with a live job, a deleted repository, another organization and an idle lane, over several pages, and checks that the scan's jobs equal the ones EnqueuePush and EnqueueSync insert. TestPGLaneWorkMatchesClaimGuards compares both guards with what TryBeginSync and TryBeginPush do for every holder, expiry and pending combination, with the legacy steal opt-in off and on. Twenty-four source mutations, one per guard (each predicate branch, the org and deleted filters, the duplicate check, the re-check, pagination, the cursor, the pass and lane timeouts, the log level, the stops in DrainSyncRuns and Close, and the serve wiring), each turn at least one of these tests red. Ten more mutations turn the two new tests or the requeue-log check red: no wait = interval, a first pass that ignores the delay or waits an interval, no requeue warning, other text, no lane attribute or the Info level for it, no shutdown case in the pass outcome switch, and no stop marker or a nil one. TestRiverLaneScanRequeuesLanesWithoutJob pins the requeue warning's text and its org, repo and lane attributes. Left untested on purpose: the wait in stopLaneScan (a mutation that drops it stays green) and the scan's cadence constants, which only change timing.

TestRiverLaneScanLogsOutOfTimeAtInfo and TestRiverLaneScanStopEndsPassQuietly lock a job row while River's workers run. Each inserts that job an hour ahead (schedulePush), so no worker can fetch it before the lock. Before this change, a 500 ms sleep before the lock failed the log test in 3 of 3 runs (lock the push job of repo_…: 0 rows). With the change, both tests pass 3 of 3 runs with that sleep, and 40 of 40 runs each without it.

StartLaneScan now holds its lock while it starts the pass goroutine. So a stop always waits for the pass that the start began, and a failed assertion does not leave the lock held.

The verify-lane-schedule model covers only the local schedule. buildServeEventing changed only on its PostgreSQL branch, where it starts the scan, and riverLaneScanInterval is new, so the model still describes the local code. check.sh passes, and check.sh --record recorded the new sync commit.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: xhigh

…h-lane schedule Model-checks the ADR-50 push lane (SQLite and S3-WAL) against git pushes, push-upstream calls, runs, steals, retries, restores, crashes and shutdown. Nine code facts tie the model to both backends and the worker; fingerprints in the lane-schedule-fingerprints CI job flag covered-code changes. The NoCompletionAssumption probe records two findings: a push that dies after its authority CAS can lose its forward. Change-Id: I07baac11eba4d51333064aa0fb5e90dd84e48653 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…e model variables Change-Id: Iafc903e2e9a284d33bd0cb076b93f5c621837eb2 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…print the push-hold path Codex round 1 on #675: fingerprint the push-hold lifecycle, both refspec vars and LocalScheduler.Start; model pushes that record no intent; state that ref deletions are not forwarded by design. The model now counts a write as owed at its CAS, adds REST writes (OtherWriters.cfg) and a HeldRefsFresh probe that fails with no fault. Change-Id: I863914ac2567ec75ba1b8f92a9b68663b3816c6d Signed-off-by: Thomas Kosiewski <tk@coder.com>

On PostgreSQL a River push run on the node that took the push can snapshot the cache before receive-pack publishes the pushed ref, and a hook that loses its CAS outcome after the commit records no forward intent. Both tests fail on this commit: the upstream never gets the accepted main. Change-Id: Ia41ff2be8d3b15ff159e7531f8955a667355b237 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…pshots A git push's River job exists from its authority CAS on, before receive-pack publishes the cache ref. A push run on the same node could claim it, snapshot the cache while the push's hold kept the old value, forward that, and end done: the accepted push never reached the upstream. River push runs now call the publication wait after the claim, like local runs. A failed wait hands the request back with RestorePush and fails the run, so the job's completion gate snoozes it. Fixes TestPGHookForwardBeforeCachePublication. Change-Id: I5c7fc02af849c5fe27d90e121156e1b29d6d4c4d Signed-off-by: Thomas Kosiewski <tk@coder.com>

…EST CAS On PostgreSQL a REST write records forward intent only after its ref CAS commits. A serve process that dies between the two leaves the accepted write with no push job. The test fails on this commit. Change-Id: I4f1721fdef341cf8834e00c5a79a87f484afa930 Signed-off-by: Thomas Kosiewski <tk@coder.com>

On PostgreSQL a write to a forward_on_push repository recorded its forward intent (push_pending and the River push job) only after the ref CAS committed. A hook or serve process that died in between, or a failed RequestPush or EnqueuePush, left the accepted write with no intent, and nothing on PostgreSQL scans for it. Metadata and refs share one PostgreSQL database, so the CAS now writes the intent in its own transaction: ref.WithCASTxHook runs a hook after a won swap statement and before COMMIT, and the serve and hook wiring set it for every write the sync poke forwards. The pending level, the push job and the ref commit together or not at all. The after-CAS poke stays, for old replicas during a rolling upgrade and for wins recovered from another caller's identical write. Fixes TestPGHookForwardAfterLostCASOutcome and TestServeRecordsForwardIntentWithTheCAS. Change-Id: I18b0962ee614728adffac868277ab8164c8ce97a Signed-off-by: Thomas Kosiewski <tk@coder.com>

…saction Change-Id: Ib7dd1c7257f42c1f6a65197cfb066cec9616b9a9 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…efore the poke The CAS transaction now inserts the push job, so a River run can claim it before the decorator's after-CAS sync poke runs. The poke then marks push_pending again, and the job's completion schedules a second forward run with its own repo.push_upstream events. The new subtest fails on this commit (PushPending:1 after the claim). Change-Id: Id72e26deed6ab966bbc467120b2295901d8efae2 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…e job first The sync poke after a PostgreSQL CAS re-marked a forward that the CAS transaction had already recorded. When a run claimed the new job before the poke, that scheduled a second forward run. The CAS hook now reports through ref.CASTxHook.Committed that its committed transaction recorded the intent, and the poke skips the forward then. A win that ref.IdempotentCAS recovered still gets the poke. RecordForwardTx now inserts the push job before it marks push_pending. It used to hold the upstream row while it waited for a job row held by the job's completion transaction, which reads that row through another pool. With that pool full of statements waiting on the row, neither side could move, and PostgreSQL cannot see that cycle. Fixes TestServeRecordsForwardIntentWithTheCAS/claimed_before_poke; adds TestRiverRecordForwardTxWaitsForJobBeforeUpstreamRow. Change-Id: Id5c9241b5b84457fbe41988cc62e57a2a30bd9ef Signed-off-by: Thomas Kosiewski <tk@coder.com>

… River publication wait A River push run whose publication wait the shutdown drain cancelled restored its request through the 30-second detached finish instead of the short shutdown restore that a cancelled transfer uses. It now uses restoreForShutdown and returns ErrShutdown, so River keeps the job. TestRiverPushRestoredRunSnoozesForRetry pins that the job of a run that restored its request after a failed wait snoozes and runs again. Change-Id: I4c97e34e77aa6812b2949a34cee0d96358cc6867 Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: I008f14e5ef848e5505344bebcf6288b8e3bec719 Signed-off-by: Thomas Kosiewski <tk@coder.com>

… caller's write Change-Id: Id162f732f5606bf1f0157e29aa2a6425c5039415 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…inish Two River push runs end with their claim held and the request they consumed: a run whose publication-wait restore the shutdown drain cancels returns an error without ErrShutdown, and a run whose finish or restore write fails returns a plain error. The push job's completion gate then completes the job, and nothing forwards the request. Both new tests fail on this commit. Change-Id: Id0fdef5b68c69bfebd9e5177657e4523f9035505 Signed-off-by: Thomas Kosiewski <tk@coder.com>

… held A River push run whose finish or restore write failed returned a plain error with its claim still held and the request it had consumed, and the job's completion gate completed the job. The gate now snoozes such a job until the claim's lease runs out; the re-run steals the claim and its force level, and forwards. A run that the shutdown drain cancels during its publication-wait restore now also returns ErrShutdown, so River keeps that job too. Fixes TestPushRunShutdownDuringFailedWaitRestoreKeepsJob and TestRiverPushFailedFinishKeepsJobUntilClaimExpires. Change-Id: I7eb270392d85832ea7b510eb3dd331bca0ee46d0 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…ead or completion The completion gate keeps a push job whose failed run left its claim held only when that same Work call saw the run fail and read the row before the run. When the pre-run read fails, or when the first completion fails and the re-run meets the unexpired claim, the gate completes the job and the consumed request is lost. Both new subtests fail on this commit. Change-Id: I7e7b8b8a6012adf537cd91c54e2d8fdffdd4c70e Signed-off-by: Thomas Kosiewski <tk@coder.com>

Only the push job runs a repository's push lane, so a claim still held at completion belongs to one of the job's own runs whose finish or restore failed, or that died, and it holds the request it consumed. The completion gate now snoozes the job until the lease can be stolen whatever this Work call saw, instead of only when it saw the run fail and knew the run count. Fixes TestRiverPushFailedFinishKeepsJobUntilClaimExpires subtests pre_run_read_fails and completion_fails_once. Change-Id: I0b69741189f40561b874cb313fec0436a11e55e3 Signed-off-by: Thomas Kosiewski <tk@coder.com>

Also correct the shutdownSnooze and RecordForwardTx docs. Change-Id: Ida509a3c433a7cc4b746c6621ef89e1868414704 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…im held The gate snoozes a push job while its push claim is held, so that the re-run can steal the claim. A run for a repository deleted while queued returns before it claims, so nothing stole the claim, and the job snoozed every 5 seconds forever. On the River lane RunPush now returns ErrRepoGone for a deleted repository, and the push worker completes the job on it. Also correct the pushWorker and RecordForwardTx docs. Change-Id: I7345fd33816f75cf1d1725592507c742f2d77af3 Signed-off-by: Thomas Kosiewski <tk@coder.com>

… log no warning for a deleted repo's run Change-Id: I892bc4f7a4d8cba5b77ed9d4ad4e76543473342d Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: I5b2a152ddfdc740e0956b9e08d1d309eb31c5d31 Signed-off-by: Thomas Kosiewski <tk@coder.com>

… its claim held On PostgreSQL a sync run whose FinishSync fails returns a plain error with its claim still held and the request it consumed, for example the mirror-back pull a forward marked. The sync job's completion gate keeps the job only while SyncPending is set, which the claim cleared, so it completes the job and nothing pulls. TestRiverSyncFailedFinishKeepsJobUntilClaimExpires fails on this commit in all three subtests. TestWorkerDeletedRepoOnRiverLane and TestRiverSyncDeletedRepoCompletesWithClaimHeld pin the deleted-repo exit that the fix needs. Change-Id: Idf901c182691d1d4042cc68ac61a75543687d14d Signed-off-by: Thomas Kosiewski <tk@coder.com>

A River sync run whose FinishSync failed, whose claim committed but reported an error, or that died left its claim holding the request it consumed: a pull-upstream, or the mirror-back pull that a forward marked. The sync job's completion gate completed the job, because the claim had cleared SyncPending, and nothing pulled. The gate now snoozes the job while a sync claim is held, until just past the lease (legacy claims at legacy pace), and the re-run steals the claim. On the River lane Run returns ErrRepoGone for a deleted repository, and the sync worker completes that job. Fixes TestRiverSyncFailedFinishKeepsJobUntilClaimExpires, TestRiverSyncDeletedRepoCompletesWithClaimHeld and TestWorkerDeletedRepoOnRiverLane. Change-Id: I412b17659a9e3d6758992f4854c37a0ccf00822d Signed-off-by: Thomas Kosiewski <tk@coder.com>

… held Change-Id: I2f271296217395ee253f7ceeafc79e7350eca645 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…l lane's deleted-repo return Change-Id: Iaf3ab76ca0c94a5adba58f938e3bb894ea0375fd Signed-off-by: Thomas Kosiewski <tk@coder.com>

…iver job River discards a sync or push job after three crashed attempts, and an operator can delete job rows. The upstream row then keeps push_pending, sync_pending or a held claim, and nothing on PostgreSQL turns it into a run until an unrelated later trigger. The test strands a push request and a pull request with no job and waits for serve to give each a job. Change-Id: I932924f4137bca7f165f269fc1a81969d03c0884 Signed-off-by: Thomas Kosiewski <tk@coder.com>

Each PostgreSQL serve replica now scans the served org's live repos every five minutes (first pass 15 seconds after boot) for push and pull lanes whose claim guard holds: a pending request with no claim, or an expired claim, which holds the request it consumed. A lapsed legacy claim counts only with the steal opt-in. For each lane it inserts the unique-by-repo job that EnqueuePush or EnqueueSync inserts, so River turns the insert into a no-op while a live job covers the lane, and every replica can scan. A new job is kept only if the lane still has work after the insert, in the same transaction, so a run that consumed the work since the list gets no extra run. A pass reads pages of 100 lanes by repo ID and ends within 30 seconds; a pass that runs out of time keeps its cursor. No schema change, no new index and no new job kind: older replicas run the jobs the scan inserts. Fixes TestServeRequeuesLanesWithoutJob. Change-Id: I67b00a4d93ca2119fce37a354cc0e2897dd00124 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ADR-44 said that PostgreSQL recovers a crashed run through River's job rescue; an amendment adds that River discards the job after its third crashed attempt and that the lane scan inserts it again. The production deployment guide names the scan, its cadence and its log line in "Background maintenance in a fleet" and "What to monitor". Two comments that said nothing on PostgreSQL scans for pending markers now name the scan. Change-Id: I88ff2bf2c3d9be5d6e1107fba310e463ad6c2ec2 Signed-off-by: Thomas Kosiewski <tk@coder.com>

Removing either stopLaneScan call leaves the scan goroutine running, and TestRiverLaneScanRunsStrandedLanes now fails then. Change-Id: Ie884ce54674cc77861a5683fe22ae2c721c66420 Signed-off-by: Thomas Kosiewski <tk@coder.com>

… guard with a test A verifier's mutations found 21 scan guards that no test caught. The scan now keeps its cursor on the River, so the pass itself keeps it across a timeout or a failed list, and the timeout is a var for tests. A pass that runs out of time logs at Info: it is expected while many lanes need a job. A page holds up to 1,000 lanes, because each list query costs time in proportion to the linked repositories whatever the page size. New tests: PGLaneWork and PGListLaneWork match TryBeginSync and TryBeginPush for every holder, expiry and pending combination, with the legacy steal opt-in off and on, and leave out deleted, unlinked and other orgs' repos. The scan's jobs equal the enqueued ones. The re-check refuses a lane that has work only on the other lane, a deleted repo and an unlinked repo. A pass that times out ends within the timeout and the next continues after its last lane. A lane whose requeue fails does not stop the pass. Change-Id: I81dfb7f12bc9c999240818c89f6d5bc4ace1a098 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…'s cost A request also loses its job when a process dies or a job insert fails after pull-upstream, push-upstream or the forward poke marked it. The ADR-44 amendment and the deployment guide say so, the guide says that the first scan after an upgrade runs requests stranded before it, and the ADR and the interval's comment say that each list query costs time in proportion to the linked repositories. Change-Id: Ib83ddff7ef39dfde096144b781e33420f23b948f Signed-off-by: Thomas Kosiewski <tk@coder.com>

Without the pass timeout the pass waits on the locked job for good; the test now fails after 15 seconds instead of at the package timeout. Change-Id: Ie4079592f6dd6ca93b0f95d53f523350e4ce1d2f Signed-off-by: Thomas Kosiewski <tk@coder.com>

…n when a scan pass handles no lane A verifier found two gaps at the previous head. A locked job row held every pass at its lane until the pass timeout, so the lanes after it never got a job: each requeue now has its own five-second timeout, and the pass goes on to the next lane. A pass that ran out of time logged at Info even when it had handled no lane, which hid a hung database or a list query slower than the pass: only a pass that handled lanes now ends with errScanOutOfTime (Info); one that handled none returns a deadline error, which the scan logs as a warning. The stop checks in TestRiverLaneScanRunsStrandedLanes now run DrainSyncRuns and Close under their own deadline. Change-Id: I810a37bef0016a1f070108e80f59b144ae8f4405 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…t five seconds Change-Id: I1dccc7a624a1e7a4cb366700e06622299b191637 Signed-off-by: Thomas Kosiewski <tk@coder.com>

A pass that handled lanes logs at Info, and one stuck behind a locked job row logs a warning. Checking for context.DeadlineExceeded in StartLaneScan again, as before the fix, inverts the two, and the test then fails. Change-Id: Ic1020697c927014af8bf2a3056060fcfeb898f1c Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: I2926eb7b9d28092a66d7a95e0160e46eea6433f7 Signed-off-by: Thomas Kosiewski <tk@coder.com>

…SQL lane scan Change-Id: I9fd8f3407927b53d65831063274a41e1df2d800f Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: Ide2a76e72a81a3cea9ecdfa797619c6d8cf3beac Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 · 2026-11-06T09:26:47Z

@codex review

ThomasK33 · 2026-11-06T09:26:53Z

@codex security review

chatgpt-codex-connector · 2026-11-06T09:27:04Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-06T14:54:43.743709Z	`49f5827`	Manual request
🔒 Security Review	✅ Completed 2026-11-06T14:55:18.308873Z	`49f5827`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-11-06T09:30:07Z

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 2dd38975db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-06T09:30:38Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 2dd38975db

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…can's schedule, stop and requeue log - TestRiverLaneScanLogsOutOfTimeAtInfo inserted b's push job as available while workers ran, so a worker could fetch it before the test locked it. The job now waits an hour ahead (schedulePush), and lockJob locks a scheduled row too. - TestRiverLaneScanPassSchedule pins the first delay and the interval between passes. - TestRiverLaneScanStopEndsPassQuietly pins that a stop mid-pass logs no failure and keeps the cursor, and that a later StartLaneScan fails its assertion, also on a River whose scan never started. - TestRiverLaneScanRequeuesLanesWithoutJob pins the requeue warning that operators monitor: its text and its org, repo and lane attributes. - StartLaneScan holds scanMu across scanning.Go, so a stop waits for the pass it starts and a failed assertion does not leave the lock held. Change-Id: If77eea8475dac915992231688a91a92c2279e6d3 Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: Iafc04079d26fd9c3af61d05f0f1fc68db7c7f46e Signed-off-by: Thomas Kosiewski <tk@coder.com>

Change-Id: I00792ede667d226cc721839d17f3e9d9a478c6a8 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 · 2026-11-06T14:51:25Z

@codex review

ThomasK33 · 2026-11-06T14:51:33Z

@codex security review

chatgpt-codex-connector · 2026-11-06T14:54:42Z

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 49f582787e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-11-06T14:55:17Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 49f582787e

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

check.sh --record at the merge: LaneSchedule.cfg PASS (3535660 distinct states) OtherWriters.cfg PASS (56705945 distinct states) Liveness.cfg PASS (348735 distinct states) NoLeaseAssumption.cfg fails as expected RESULT: PASS Change-Id: I0403b6c1d6d32c4351fc6f45eefaf87caa8dd457 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 added 30 commits November 5, 2026 08:33

docs(skills): explain probe UNEXPECTED lines, FAIL over STALE, and th…

90dea59

…e model variables Change-Id: Iafc903e2e9a284d33bd0cb076b93f5c621837eb2 Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs(adr): amend ADR-44 for PostgreSQL forward intent in the CAS tran…

5ef37ea

…saction Change-Id: Ib7dd1c7257f42c1f6a65197cfb066cec9616b9a9 Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs(adr): say that the sync poke skips a forward the CAS recorded

1e74dab

Change-Id: I008f14e5ef848e5505344bebcf6288b8e3bec719 Signed-off-by: Thomas Kosiewski <tk@coder.com>

test(openstorage): pin the sync poke for a win recovered from another…

52c32dc

… caller's write Change-Id: Id162f732f5606bf1f0157e29aa2a6425c5039415 Signed-off-by: Thomas Kosiewski <tk@coder.com>

test(jobs): check that a held push claim's job waits for the lease

20f35b7

Also correct the shutdownSnooze and RecordForwardTx docs. Change-Id: Ida509a3c433a7cc4b746c6621ef89e1868414704 Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs(jobs): name every holder of a claim the push gate waits for, and…

4b4e54c

… log no warning for a deleted repo's run Change-Id: I892bc4f7a4d8cba5b77ed9d4ad4e76543473342d Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs(adr): say that the push gate keeps a job while its claim is held

c0f3093

Change-Id: I5b2a152ddfdc740e0956b9e08d1d309eb31c5d31 Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs(adr): say that the sync gate keeps a job while its sync claim is…

aec17e3

… held Change-Id: I2f271296217395ee253f7ceeafc79e7350eca645 Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs(jobs): say which Run errors steer the sync job, and pin the loca…

1419ef0

…l lane's deleted-repo return Change-Id: Iaf3ab76ca0c94a5adba58f938e3bb894ea0375fd Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 added 8 commits November 5, 2026 17:11

docs(adr): say that the lane scan skips a lane whose insert waits pas…

97cd391

…t five seconds Change-Id: I1dccc7a624a1e7a4cb366700e06622299b191637 Signed-off-by: Thomas Kosiewski <tk@coder.com>

Merge remote-tracking branch 'origin/main' into fix-pg-orphan-push-scan

b2a5882

Change-Id: I2926eb7b9d28092a66d7a95e0160e46eea6433f7 Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs(skills): record the lane-schedule fingerprints after the Postgre…

786a7dd

…SQL lane scan Change-Id: I9fd8f3407927b53d65831063274a41e1df2d800f Signed-off-by: Thomas Kosiewski <tk@coder.com>

Merge remote-tracking branch 'origin/main' into fix-pg-orphan-push-scan

2dd3897

Change-Id: Ide2a76e72a81a3cea9ecdfa797619c6d8cf3beac Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 added 3 commits November 6, 2026 13:16

Merge remote-tracking branch 'origin/main' into fix-pg-orphan-push-scan

4819bfc

Change-Id: Iafc04079d26fd9c3af61d05f0f1fc68db7c7f46e Signed-off-by: Thomas Kosiewski <tk@coder.com>

docs(skills): record the lane-schedule sync commit after merging main

49f5827

Change-Id: I00792ede667d226cc721839d17f3e9d9a478c6a8 Signed-off-by: Thomas Kosiewski <tk@coder.com>

ThomasK33 added this pull request to the merge queue Nov 6, 2026

Merged via the queue into main with commit 2c1c74f Nov 6, 2026
11 checks passed

ThomasK33 deleted the fix-pg-orphan-push-scan branch November 6, 2026 17:15

github-actions Bot mentioned this pull request Nov 6, 2026

chore(main): release 0.20.0 #653

Merged

ThomasK33 mentioned this pull request Nov 6, 2026

fix(sync): hold the ref and record forward intent before a REST write's CAS #737

Merged

Navigation Menu

Repository navigation

fix(jobs): requeue PostgreSQL lanes whose durable work has no River job - #726

fix(jobs): requeue PostgreSQL lanes whose durable work has no River job#726
ThomasK33 merged 41 commits into
mainfrom
fix-pg-orphan-push-scan

ThomasK33 commented Nov 6, 2026 •

edited

Loading

Uh oh!

ThomasK33 commented Nov 6, 2026

Uh oh!

ThomasK33 commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

ThomasK33 commented Nov 6, 2026

Uh oh!

ThomasK33 commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ThomasK33 commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Fix

Tests

Uh oh!

ThomasK33 commented Nov 6, 2026

Uh oh!

ThomasK33 commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

🛡️ Codex Security Review

Uh oh!

ThomasK33 commented Nov 6, 2026

Uh oh!

ThomasK33 commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

Uh oh!

chatgpt-codex-connector Bot commented Nov 6, 2026

🛡️ Codex Security Review

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ThomasK33 commented Nov 6, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Nov 6, 2026 •

edited

Loading