Skip to content

bug(test): Python E2E cleanup deadline is shorter than Kubernetes sandbox termination #4685

Open

Description

@gmenher

Title

bug(test): Python E2E cleanup deadline is shorter than Kubernetes sandbox termination

Body

User Story

As an OpenShell E2E maintainer, I want a successful Python sandbox test to
finish cleanup within the fixture's supported deadline, so that Kubernetes
workload shutdown time does not turn a passed scenario into a false test
failure.

Problem Statement

The Python Sandbox context manager uses its default 30-second RPC timeout for
DeleteSandbox during __exit__. The Kubernetes driver keeps the synchronous
delete RPC open while the workload Pod terminates. In a Kubernetes deployment, supervisor
shutdown and final log upload can take about 45 seconds after the delete is
accepted. The test body succeeds, but context cleanup raises
DEADLINE_EXCEEDED.

This is reproducible with E2E_PARALLEL=1, so it is not caused by xdist
concurrency.

Impact / Why This Matters

The current workaround is to exclude otherwise valid lifecycle scenarios from a
Kubernetes E2E run. That loses regression coverage and makes maintainers
distinguish a fixture deadline from a product failure by manually inspecting
logs. Increasing parallelism does not resolve it, and an asynchronous deletion
protocol would be a much broader API decision than is needed to make the
current synchronous test fixture reliable.

Acceptance Criteria

  • Python E2E cleanup using the Kubernetes driver does not fail merely
    because normal Pod termination exceeds the default 30-second RPC deadline.
  • The test fixture uses an explicit, documented cleanup deadline suitable
    for the driver's bounded termination behavior, without changing ordinary
    SDK caller defaults globally.
  • A focused test proves cleanup remains bounded and reports a useful error
    if the configured cleanup deadline is exceeded.
  • The affected Python lifecycle scenarios pass against a Kubernetes-backed
    gateway without test-local timeout overrides.
  • The existing SDK default remains unchanged for callers that do not opt
    into the E2E fixture behavior.

Reproduction Steps

  1. Deploy an OpenShell gateway with the Kubernetes driver and select it as the
    active gateway.
  2. Run the Python E2E lifecycle scenarios that use
    with sandbox(..., delete_on_exit=True) with E2E_PARALLEL=1.
  3. Let the scenario complete and enter context cleanup.
  4. When workload Pod termination takes longer than 30 seconds, observe
    DEADLINE_EXCEEDED from DeleteSandbox.

For example, run a focused serial scenario against the selected gateway:

E2E_PARALLEL=1 \
uv run pytest -o python_files='test_*.py *_test.py' \
  e2e/python/test_exec_admission.py

Environment

  • OpenShell: main checked at a4c848adb.
  • Deployment: gateway using the Kubernetes driver.
  • Test path: e2e/python, with serial execution to rule out xdist contention.
  • Observed workload termination time: approximately 45 seconds.

Logs

The test body completed, then context cleanup failed:

grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
>

Investigation

  • python/openshell/sandbox.py gives Sandbox a 30-second default timeout.
  • Sandbox.__exit__ calls SandboxSession.delete(), then identity-aware
    wait_deleted() for accepted deletes.
  • e2e/python/conftest.py sets a longer readiness timeout for fixtures that
    pull an explicit workload image, but does not set a cleanup timeout.
  • The Kubernetes driver waits for sandbox runtime and supervisor Pods as part
    of its synchronous deletion path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions