[Bug] A Batch cancellation during live input validation is rejected with HTTP 409 #3075

🐛 Describe the bug

On 99325e7, a Batch cancellation sent while a live driver is still validating the input file returns TerminateResult.REJECTED. If validation subsequently succeeds, the job moves to in_progress. The reproduction below exercises the manager in-process. The API handler maps that rejection to HTTP 409 and, while the job remains validating, Cannot cancel a batch with status 'validating'. (python/aibrix/aibrix/metadata/api/v1/batch.py:746). VALID_STATE_TRANSITIONS in python/aibrix/aibrix/batch/batch_manager.py:137 permits validating → cancelling; docs/source/features/batch-inference/batch-api.rst:146 defines the cancelled outcome. These references are at 99325e7. This is the related case mentioned in #3067.

BatchManager.cancel_job (batch_manager.py:402 at 99325e7) builds the update with _build_cancelled_job_update (line 1370), which turns any created or validating job straight into finalized + cancelled. That is right when no driver exists yet. Once admission has attached a live driver, though, the update goes to job_driver.terminate, and BaseJobDriver.terminate rejects every finished or finalizing job, so the cancel is refused.

Steps to Reproduce

From python/aibrix at 99325e7, save this as tests/batch/test_cancel_during_validation.py and run poetry run pytest -p no:cacheprovider -s tests/batch/test_cancel_during_validation.py. It uses the existing _make_job test helper and blocks input validation on an asyncio.Event. This is a standalone reproduction, not an additional committed test:

import asyncio
from unittest.mock import patch

import pytest

from aibrix.batch.batch_manager import BatchManager
from aibrix.batch.job_driver import BaseJobDriver, NoopRuntime, TerminateResult
from aibrix.batch.job_driver import base as base_module
from aibrix.batch.job_entity import BatchJobState
from aibrix.context.infra import InfrastructureContext
from tests.batch.job_driver.test_base_job_driver import _make_job


@pytest.mark.asyncio
async def test_cancel_while_live_driver_validates():
    context = InfrastructureContext()
    manager = BatchManager(context)
    job = _make_job()
    job.status.state = BatchJobState.CREATED
    manager._pending_jobs[job.job_id] = job
    driver = BaseJobDriver(context, manager, NoopRuntime(context), job)
    started, release = asyncio.Event(), asyncio.Event()

    async def blocked_validation(*args):
        started.set()
        await release.wait()
        return 1, None

    with (
        patch("aibrix.batch.batch_manager.create_job_driver", return_value=driver),
        patch.object(base_module.storage, "read_job_input_info", return_value=(1, True)),
        patch.object(base_module.storage, "validate_job_input_file", side_effect=blocked_validation),
    ):
        admission = asyncio.create_task(manager.admit(job.job_id))
        await asyncio.wait_for(started.wait(), timeout=5)
        result = await manager.cancel_job(job.job_id)
        release.set()
        await asyncio.wait_for(admission, timeout=5)
        current = await manager.get_job(job.job_id)
        print(f"cancel={result.name}; state after validation={current.status.state.value}")
        assert result == TerminateResult.ACCEPTED

On 99325e7 it prints cancel=REJECTED; state after validation=in_progress and fails; the log shows Live job rejected cancellation with state=finalized. Cancelling a created or validating job before a driver is attached still finalizes it as cancelled immediately.

Expected behavior

The cancel is accepted and the job moves to cancelling. A successful validation result that arrives afterwards does not move it back to in_progress, and cancellation alone finishes cancelled. A late validation failure retains the existing precedence: failed outranks cancelled; this fix does not change failure or expiry policy.

Environment

• AIBrix version: main at 99325e7
• Deployment environment: Batch API with a live job driver; reproduced in-process on Linux, Python 3.10.12
• LLM(s) being used: none; the reproduction stops before inference

Area

Batch

I'd like to fix this. I have the change and tests ready and will open a PR.