Skip to content

Model router: name a model by its family, without a version - #348458

Merged
Bhavya U (bhavyaus) merged 4 commits into
mainfrom
bhavyaus/model-router-model-family
Nov 16, 2026
Merged

Bhavya U (bhavyaus) merged 4 commits into
mainfrom
bhavyaus/model-router-model-family

Conversation

@bhavyaus

@bhavyaus Bhavya U (bhavyaus) commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Builds on #347704. Model routers (.model-router.yaml, behind chat.copilot.modelRouter.enabled).

Change

A route's model had to be an exact model id, so a router went stale every time a new version shipped. A model can now also be a family: a model id without its version.

routes:
  - model: opus
    when: Building or changing things over many steps.
  - model: haiku
    when: Quick answers and small chores.

A family resolves to the newest released model of that family the account can use. It is resolved from the account's model list each time a message is routed, with no extra request and no decision-model call.

Rules

  • Exact ids still win. A reference that is a model's id is that model, as before.
  • A digit names a version. claude-opus-5 or gpt-4o asks for that model and never falls back to its family, so an unavailable id still moves on to the route's next model.
  • What a family is. A model belongs when its id has every word of the reference. The models with the fewest other words are the family, so a family is not its variants: gpt is not gpt-5-mini. The family is decided before looking at which of its models can be used, so a family with no usable model is unavailable; it is not replaced by a variant.
  • Newest. Version numbers are compared part by part. A number can follow a word and be followed by one letter, so llama3.2 is version 3.2 of llama and gpt-4o is version 4 of gpt. A snapshot date compares the same written together or apart. For dated snapshot ids on a user's own key, the date is kept apart from the version, and an undated alias beats a dated snapshot of the same version.
  • Models on a user's own key. The family is the last segment: anthropic/haiku, or anthropic/Work/haiku to stay within one group. What comes before the family is written in full or without the group, as for an exact id, and the same family in two groups is reported as two families.
  • Policy. A model turned off by policy is passed over while another of the family can be used.
  • Ambiguity. A name that fits more than one family, such as claude, is reported with the families to write instead.
  • Experimental models are never picked by a family. Otherwise a router would move to an internal or early-access model the day it appears in the account's list. They can still be used by their exact id. A family that has only experimental models picks none and says so.

How experimental models are known

The Copilot API reports the label Experimental in a model's vendor field for these models. The agent host dropped that field; it now keeps it on the model as a _meta flag (vscode.model.experimental). check_model_router lists the flag on each model, and the built-in model-router skill describes families.

Open chats

A chat on a route that names a family stays on that route when a newer model of the family ships, and runs on the newer model from its next message. For an unnamed route this needed a change: its identity was the resolved model id, so a new version made the chat look like it had no route, and it was routed from scratch. A routed turn now also saves the entry of the route's model that matched (routeReference) when that is not the model's id, and a chat's route is found by it first. When the newer model is already the model of an earlier route, the family's route is no route of its own, and the chat is on that earlier route.

How to test

  1. Turn on chat.copilot.modelRouter.enabled and save a router with model: opus and model: haiku routes in ~/.agents/model-routers/.
  2. Pick it in the model picker and send a message. The routing line names the newest released model of the family.
  3. Ask the agent to run check_model_router on the file. Each route shows the model it matched.
  4. Change a route to model: claude. The check reports that it matches more than one family and lists them.

Validation

  • pnpm run typecheck-client: passed.
  • ./scripts/test.sh --runGlob "vs/platform/agentHost/test/**/{modelRouter*,copilotModelRouter*,metadataCompatibility}.test.js": 162 passing.
  • ./scripts/test.sh --runGlob "vs/platform/agentHost/test/**/{agentSideEffects,copilotModelRouterTools}.test.js": 316 passing.
  • ./scripts/test.sh --run src/vs/platform/agentHost/test/node/copilotAgent.test.ts: 1103 passing, 1 pending.
  • pnpm run oxlint on the changed files and pnpm run valid-layers-check: passed.
  • In a local Code OSS build: a router written with families routed messages to the newest released model of each family, and a route written as an experimental-only family followed by a fallback ran on the fallback, with the check reporting why.

A route's `model` can be a family such as `opus`, `sol` or `claude-haiku`:
a model id without its version. It resolves to the newest released model
of that family the account can use, so a router keeps working when a new
version ships.

- An exact id still wins, and a reference with a digit, such as
  `claude-opus-5` or `gpt-4o`, names a version and never falls back to
  its family.
- The family is decided before looking at which of its models can be
  used. A family with no usable model is unavailable, never replaced by
  a variant of it, so the route's next model is tried.
- A name that fits more than one family is reported with the families
  to write instead.
- A family never picks an experimental model, so a router does not move
  to an internal or early-access model the day it appears. The Copilot
  API marks those with the vendor label "Experimental", which the agent
  host now keeps on the model as a flag. They are used by their exact id.
- A chat on an unnamed route that names a family stays on that route
  when a newer model of the family ships.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 05:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Scoped model-family selectors are incorrectly rejected, while unstable route matching can prevent model upgrades or reroute chats despite first-turn-only switching.

2 open issues
What changed in this PR

Extends Agent Host model routers to accept versionless family names, reducing the need to update router files when new model versions become available.

Changes:

  • Resolves families to the newest eligible released model, with policy filtering and ambiguity diagnostics.
  • Preserves experimental-model metadata and exposes it through check_model_router.
  • Adds family-aware follow-up matching, tests, and authoring guidance.
File Description
src/​vs/​platform/​agentHost/​test/​node/​copilotModelRouter.test.ts Tests follow-up routing after a family upgrade.
src/​vs/​platform/​agentHost/​test/​common/​modelRouter.test.ts Tests family selection, versions, policy, and experimental exclusions.
src/​vs/​platform/​agentHost/​node/​copilot/​copilotModelRouterTools.ts Reports experimental models in check results.
src/​vs/​platform/​agentHost/​node/​copilot/​copilotModelRouter.ts Adds family-based matching for unnamed-route follow-ups.
src/​vs/​platform/​agentHost/​node/​copilot/​copilotBuiltinSkills.ts Documents family references and experimental-model behavior.
src/​vs/​platform/​agentHost/​node/​copilot/​copilotAgent.ts Preserves the API’s experimental label as model metadata.
src/​vs/​platform/​agentHost/​common/​modelRouter.ts Implements family resolution, version comparison, and diagnostics.
src/​vs/​platform/​agentHost/​common/​meta/​vscode/​agentModelExperimentalMeta.ts Adds experimental metadata helpers.
src/​vs/​platform/​agentHost/​common/​meta/​agentModelMeta.ts Exports the new metadata helpers.

Review effort: Balanced

…ts model entry

- A family for a model on the user's own key can be scoped to a group or
  an id's own path: `anthropic/Work/haiku`. The same family in two
  groups is reported as two families.
- A routed turn saves the entry of the route's `model` that matched when
  it is not the model's id. A chat on a route without a name is found by
  it, so the route can move to a newer model whatever that model's id
  looks like, and a chat is not taken for one on another route that
  names the earlier model.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Model routing can lose saved family selections during deduplication and skip available fallback models because alphanumeric versions are misclassified.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Recognize alphanumeric version tokens in model family routing

src/​vs/​platform/​agentHost/​common/​modelRouter.ts:724

Alphanumeric versions are treated as family words: gpt-4o becomes words ['gpt', '4o'] with no version, whereas gpt-5 becomes words ['gpt']. If GPT-5 is policy-disabled but GPT-4o is available, model: gpt excludes GPT-4o at the fewest-words filter and returns the disabled GPT-5, so the route is skipped instead of using the available older model. Please recognize alphanumeric version tokens when deriving family membership and ordering, while retaining exact-only lookup for references containing digits, and cover this policy-fallback case.

2 resolved since last review

Review effort: Balanced

… read 4o as a version

- A route without a name that lands on the model of an earlier one is
  not a route of its own. A chat that was on it is now on that earlier
  route, rather than routed from scratch.
- A letter after a number is part of that version, so `gpt-4o` is
  version 4 of `gpt` and can be used while policy has a newer one
  turned off.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bhavyaus

Copy link
Copy Markdown
Collaborator Author

On the previously missed issue (alphanumeric versions): fixed in 6eec59c. A letter after a number is part of that version, so gpt-4o is version 4 of gpt, and a reference with a digit is still exact-only. The family test covers the policy fallback.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Model-family routing can select older snapshots when date formats differ and miss models whose numeric versions are attached to family names.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Medium severity Match attached numeric model versions in family routes

src/​vs/​platform/​agentHost/​common/​modelRouter.ts:721

model: ollama/llama cannot match ollama/llama3, an identifier already used in agentHostUtilityModelService.test.ts:39, because the tokenizer retains llama3 as a word. The same happens for llama3.2 and llama4, so a versionless route skips these models and falls through to its fallback. Recognize numeric versions attached to family names while preserving v-prefixed versions and suffixes such as 4o, and add coverage for this identifier shape.

Medium severity Normalize snapshot date formats before comparing

src/​vs/​platform/​agentHost/​common/​modelRouter.ts:728

Compact and hyphenated snapshot dates are compared in different units. For example, a BYOK family containing gpt-4o-20250805 and gpt-4o-2026-01-01 selects the older snapshot because [20250805] sorts above [2026, 1, 1]; even equivalent dates compare unequal. Normalize compact dates to year/month/day components before comparing them so family routing selects the newest snapshot regardless of its date format.

1 resolved since last review

Review effort: Balanced

…t dates in one unit

- `llama3.2` is version 3.2 of `llama`, so a family finds models whose
  id has the version attached.
- A date written together, such as 20250805, is its year, month and day,
  so it compares with one written apart.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bhavyaus

Copy link
Copy Markdown
Collaborator Author

On the two previously missed issues: both fixed in 59430b7. A number can follow a word, so ollama/llama finds llama3.2, and a snapshot date written together is compared as its year, month and day. The family test covers both.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain to be addressed.

No open issues 🎉

Review effort: Balanced

@bhavyaus
Bhavya U (bhavyaus) added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 8bb5b34 Nov 16, 2026
38 of 39 checks passed
@bhavyaus
Bhavya U (bhavyaus) deleted the bhavyaus/model-router-model-family branch November 16, 2026 06:31
@vs-code-engineering vs-code-engineering Bot added this to the 1.148.0 milestone Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants