Skip to content

[Frontend / Core] Support Running Frontend Only & Validate Stage IDs - #8626

Merged
NickCao merged 7 commits into
vllm-project:mainfrom
alex-jw-brooks:stage_id
Oct 15, 2026
Merged

NickCao merged 7 commits into
vllm-project:mainfrom
alex-jw-brooks:stage_id

Conversation

@alex-jw-brooks

Copy link
Copy Markdown
Collaborator

Purpose

Currently, stage ID can take any numeric value, which isn't properly validated. E.g., users can pass --stage-id -1 as a hack to try run against all stages running as --headless. Similarly, passing an out of range stage id, e .g., 999, isn't validated and behaves the same.

Running just the frontend (i.e., API server & orchestrator) is a useful capability, and will be very important for effectively running all stages independently with multiple replicas & external orchestration, which is being worked on in #6389. As such, we should both support this as a formal feature, and tighten the validation for correctness. This is done as follows:

  • Passing --stage-id none starts just API server and orchestrator and is mutually exclusive with --headless, because --headless doesn't make sense if all stages are remote
  • Passed stage IDs are validated against the stage configs and will now validate if the passed ID is invalid.

Examples:

  1. Starting the server with all remote replicas:
vllm serve --omni \
    Qwen/Qwen3-Omni-30B-A3B-Instruct \
    --stage-id none \
    --omni-master-address 127.0.0.1 \
    --omni-master-port  20000

In the logs, you'll see:

(APIServer pid=2339167) INFO 10-08 06:34:59 [stage_runtime.py:1410] No local stage replicas found in stage plans (server + orchestrator only)
  1. Validation on out of range indices:
vllm serve --omni \
    Qwen/Qwen3-Omni-30B-A3B-Instruct \
    --stage-id 1000 \
    --omni-master-address 127.0.0.1 \
    --omni-master-port  20000 \
    --headless

will raise: ValueError: No stage config found for stage_id=1000. Available stage ids: [0, 1, 2]

Test Plan

Added tests at various levels, for parsing and ensuring that passing the FRONTEND_ONLY_ID_FILTER through engine creation will create a plan for all remote replicas.

vLLM Version:: 0.31.0

vLLM-Omni Commit:: 32ead8e

Test Result

Tests pass.

@vllm-omni-review-bot

Copy link
Copy Markdown

This PR appears to belong to: docs/design/module/observability.md.

Module owners: @lishunyang12 @vraiti @hsliuustc0106

Routing: @lishunyang12 via module of the changed files; @vraiti via module of the changed files; @hsliuustc0106 via CODEOWNERS

@alex-jw-brooks, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

@vllm-omni-review-bot

Copy link
Copy Markdown
Omni ReviewBot routing record

Assigned Strict on cursor (cursor-grok-4.6-high) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

@vllm-omni-review-bot

vllm-omni-review-bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Omni ReviewBot triage note

Automated triage of commit 4fc699873906 produced:

  • Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

@vllm-omni-review-bot

vllm-omni-review-bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Omni ReviewBot: superseded

The CI failure noted on 695224ec4b50 refers to an earlier head; the pull request now points at 8fcb22b0e14a.

@hsliuustc0106 hsliuustc0106 added the high priority high priority issue, needs to be done asap label Oct 8, 2026 — with ChatGPT Codex Connector
Comment thread vllm_omni/entrypoints/cli/serve.py Outdated

def _parse_stage_id(value: str) -> int:
"""Parse `--stage-id` as a non-negative int, or `none` (no local stage) as `FRONTEND_ONLY_ID_FILTER`."""
if value == "none":

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a .lower() since users would likely type None.

@alex-jw-brooks alex-jw-brooks Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, sounds reasonable to me. We may want to take a pass for other flags at some point though and ensure they normalize "none" consistently. I didn't add it initially because other flags don't normalize it. For example

This is ok

$ vllm serve black-forest-labs/FLUX.2-klein-4B --omni --port 8313 --cache-backend none

But this fails:

$ vllm serve black-forest-labs/FLUX.2-klein-4B --omni --port 8313 --cache-backend None

[rank0]: RuntimeError: Orchestrator initialization failed: Unsupported cache backend: None. Supported: 'cache_dit', 'tea_cache', 'mag_cache', 'sea_cache', 'step_cache'

Comment thread vllm_omni/entrypoints/cli/serve.py Outdated
Comment on lines +80 to +82
if not (value.isascii() and value.isdigit()):
raise argparse.ArgumentTypeError(f"--stage-id must be a non-negative integer or 'none', got {value!r}")
return int(value)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't think of an edge case here, but a try catch for ValueError would be more lenient.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be safe since we guard isascii(), because the main edge case would be sub/superscripts 🤔 imo there isn't any intuitive behavior if int coercion fails though, unless you mean still failing loudly here, but reraising with a more direct error string?

Comment thread vllm_omni/engine/stage_runtime.py Outdated
@vllm-omni-review-bot

Copy link
Copy Markdown
Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 120s (try 2 of 3)).

@vllm-omni-review-bot

Copy link
Copy Markdown
Omni ReviewBot attempt record

Review attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 600s (try 3 of 3)).

@vllm-omni-review-bot

Copy link
Copy Markdown
Omni ReviewBot attempt record

Review attempt ended as failed (failed; falling back to direct/cursor/auto).

@vllm-omni-review-bot

vllm-omni-review-bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Omni ReviewBot: superseded

The CI failure noted on 8fcb22b0e14a refers to an earlier head; the pull request now points at 989b3d96ea64.

@alex-jw-brooks alex-jw-brooks added the ready label to trigger buildkite CI label Oct 15, 2026

@NickCao NickCao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested running Qwen/Qwen3-Omni-30B-A3B-Instruct using this.

Signed-off-by: Alex Brooks <albrooks@redhat.com>
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Signed-off-by: Alex Brooks <albrooks@redhat.com>
@NickCao NickCao added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Oct 15, 2026
@NickCao
NickCao merged commit 7812588 into vllm-project:main Oct 15, 2026
9 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

high priority high priority issue, needs to be done asap ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants