Runtime verification

| Runtime verification helps you check whether a Deep Scan finding is real before you decide how to address it. It has two parts:

• A reproduction guide: step-by-step instructions, written from the source code, that you can follow in your own staging or test environment. Guides are free and don’t run code.
• Verification: CodeRabbit runs the guide in an isolated test environment that it prepares for the scanned commit, then keeps the result and the evidence with the finding. The agent is instructed to use only that test environment, never your deployed services.

By default, Deep Scans create reproduction guides for new or reopened Critical and High findings. You choose which findings also get a verification run, as repository defaults or for a single scan.

​

Before you begin

• Runtime verification is available for GitHub Cloud repositories on CodeRabbit Cloud. If a finding’s Runtime verification tab shows Coming Soon, the feature is not available for that finding.
• Viewing guides, results, and evidence requires Security read access. Generating guides, starting or canceling runs, and saving repository defaults require Security write access.
• Verification runs require an active paid CodeRabbit Agent add-on for your organization. Runs use Verify minutes, which are billed as Agent minutes. See Usage and billing.
• To run a guide in CodeRabbit, the repository needs a prepared environment with the dependencies and setup the application needs. You don’t need one to follow a guide in your own environment.

​

What happens after a Deep Scan

After a Deep Scan of the repository’s code finishes, CodeRabbit looks at the open findings that the scan newly detected or reopened:

1. It generates a reproduction guide for each finding at or above the guide level.
2. When a guide is ready, it runs the guide for each finding at or above the verification level, in the repository’s default environment.

Findings carried over unchanged from earlier scans don’t get new guides or runs. On large scans, guides can arrive well after the scan finishes, so a completed Deep Scan doesn’t mean its guides and runs have finished. Open a finding’s Runtime verification tab to follow its progress. If no guide level is saved for the repository, guides are generated for Critical and High findings, or for every severity selected for verification if that is broader. If the repository has no default environment ready, findings get a guide but no run. Both levels use the same options: Verification can only run on findings that have a guide, so the verification level can’t be broader than the guide level. If verification covers a severity that guides don’t, CodeRabbit shows Generate reproduction guides for every severity selected for verification. and you can’t save the defaults or start the scan. Setting verification to Off still generates guides. To generate neither, set both levels to Off.

​

Set repository defaults

1. Go to Security > Repositories, select the repository, and open the Settings tab.
2. In Reproduction and verification, choose a level for Generate reproduction guides and for Verify in a test environment.
3. Select Save defaults.

The defaults apply to future Deep Scans of the repository, including scheduled and coverage-triggered scans. Saving with verification on makes you the payer for runs from those scans. See Usage and billing.

​

Override for one scan

In the Run deep scan dialog, expand Configuration. Under Runtime Verification, choose levels for Generate reproduction guides and Verify in a test environment. These choices apply to that scan only and don’t change the repository defaults. When the section is collapsed, it summarizes the choices, for example “Guides: High+ · Verification: Critical”.

​

Generate and use a reproduction guide

You can generate a guide for any Deep Scan vulnerability, whatever its severity.

​

Run the guide in CodeRabbit

CodeRabbit can run a published guide in an isolated test environment and keep the evidence from the run. Guide execution stops after 90 minutes. Shutdown and assessment can take additional time. To run the guide again, select Run again, or Retry run after a failed run. If the previous environment is still available, this immediately starts a new billed run in it; otherwise, CodeRabbit asks you to choose an environment. To pick a different one, open Run options and select Change environment….

​

Environment variables and evidence

A run uses the environment you choose. An automatic run uses the repository’s default environment. The agent running the guide can read the environment variables configured for that environment, including values marked as secret, and can run commands in its sandbox. Marking a variable as secret doesn’t hide its value from the agent during the run. CodeRabbit keeps the run’s evidence with the finding: completed steps, command output, observations, and downloadable text evidence. Evidence is text only. Everyone with Security read access who can see the finding can view it.

​

Interpret runtime results

The Runtime analysis section shows one of these results: Evidence verified appears when a finished run that isn’t Inconclusive has readable evidence. Check Application ready and Target exercised alongside the result. They show whether the run confirmed that the application was ready and that the relevant code path was exercised. A setup failure or an untested path is not evidence that a vulnerability is absent. Under Assessment and evidence, review the completed steps, the retained output, Limitations, and Counterevidence. Select View evidence or Download text evidence for closer inspection. The result also shows the Environment, the Tested commit, and the Verify minutes used. Results apply to the published guide and the tested commit. They can change the finding’s state:

• Reproduced can reopen a fixed finding, unless newer activity or a dismissal takes precedence.
• Could not reproduce doesn’t mark the finding fixed. CodeRabbit rechecks the source, and the recheck determines whether the finding still applies.
• Inconclusive doesn’t change the finding’s state.

Use the evidence to decide whether to resolve or dismiss the finding, or start Fix with AI.

​

Filter findings by result

On the Findings page, the Verification column shows each vulnerability’s latest run result or guide status. To filter, use the Runtime verification filter group:

• Run result: Reproduced, Not reproduced, Inconclusive, In progress, Failed, Canceled, or Not run.
• Reproducer guide: Ready, Partial, Unavailable, In progress, Failed, Canceled, or No guide.

​

Usage and billing

Reproduction guides are free. Running a guide in CodeRabbit uses Verify minutes, billed at $0.40 per Agent minute. You pay for the time spent running the guide and assessing the result. Each run shows its Verify minutes when it finishes. You aren’t charged for a run that fails before it publishes a result; it shows Not billed. Published results are charged, including Inconclusive. Each run is charged separately, including reruns. Who pays for a run depends on how it started. The tab shows who started each guide and run, and who is billed for automatic runs.

• Runs you start from a finding: billed to you.
• Deep Scans you start: automatic runs are billed to you, including runs from the levels preselected in the dialog.
• Scheduled and coverage-triggered scans: automatic runs are billed to the person who last saved the repository defaults with verification on. If the defaults were saved with verification Off, or never saved, these scans generate guides only.

If the payer can’t be billed, for example because the organization has no active paid Agent add-on or has reached its Agent spend cap, the guide is still published and the run shows Not run with the reason. See Access and billing for Agent-minute pricing and spending controls.

​

Troubleshoot runtime verification

When an automatic run can’t start, its message begins with “Couldn’t verify automatically.” followed by the reason.

​

What’s next