Repository navigation
bug(test): Python E2E cleanup deadline is shorter than Kubernetes sandbox termination #4685
Description
Title
bug(test): Python E2E cleanup deadline is shorter than Kubernetes sandbox termination
Body
User Story
As an OpenShell E2E maintainer, I want a successful Python sandbox test to
finish cleanup within the fixture's supported deadline, so that Kubernetes
workload shutdown time does not turn a passed scenario into a false test
failure.
Problem Statement
The Python Sandbox context manager uses its default 30-second RPC timeout for
DeleteSandbox during __exit__. The Kubernetes driver keeps the synchronous
delete RPC open while the workload Pod terminates. In a Kubernetes deployment, supervisor
shutdown and final log upload can take about 45 seconds after the delete is
accepted. The test body succeeds, but context cleanup raises
DEADLINE_EXCEEDED.
This is reproducible with E2E_PARALLEL=1, so it is not caused by xdist
concurrency.
Impact / Why This Matters
The current workaround is to exclude otherwise valid lifecycle scenarios from a
Kubernetes E2E run. That loses regression coverage and makes maintainers
distinguish a fixture deadline from a product failure by manually inspecting
logs. Increasing parallelism does not resolve it, and an asynchronous deletion
protocol would be a much broader API decision than is needed to make the
current synchronous test fixture reliable.
Acceptance Criteria
- Python E2E cleanup using the Kubernetes driver does not fail merely
because normal Pod termination exceeds the default 30-second RPC deadline. - The test fixture uses an explicit, documented cleanup deadline suitable
for the driver's bounded termination behavior, without changing ordinary
SDK caller defaults globally. - A focused test proves cleanup remains bounded and reports a useful error
if the configured cleanup deadline is exceeded. - The affected Python lifecycle scenarios pass against a Kubernetes-backed
gateway without test-local timeout overrides. - The existing SDK default remains unchanged for callers that do not opt
into the E2E fixture behavior.
Reproduction Steps
- Deploy an OpenShell gateway with the Kubernetes driver and select it as the
active gateway. - Run the Python E2E lifecycle scenarios that use
with sandbox(..., delete_on_exit=True)withE2E_PARALLEL=1. - Let the scenario complete and enter context cleanup.
- When workload Pod termination takes longer than 30 seconds, observe
DEADLINE_EXCEEDEDfromDeleteSandbox.
For example, run a focused serial scenario against the selected gateway:
E2E_PARALLEL=1 \
uv run pytest -o python_files='test_*.py *_test.py' \
e2e/python/test_exec_admission.pyEnvironment
- OpenShell:
mainchecked ata4c848adb. - Deployment: gateway using the Kubernetes driver.
- Test path:
e2e/python, with serial execution to rule out xdist contention. - Observed workload termination time: approximately 45 seconds.
Logs
The test body completed, then context cleanup failed:
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.DEADLINE_EXCEEDED
details = "Deadline Exceeded"
>
Investigation
python/openshell/sandbox.pygivesSandboxa 30-second default timeout.Sandbox.__exit__callsSandboxSession.delete(), then identity-aware
wait_deleted()for accepted deletes.e2e/python/conftest.pysets a longer readiness timeout for fixtures that
pull an explicit workload image, but does not set a cleanup timeout.- The Kubernetes driver waits for sandbox runtime and supervisor Pods as part
of its synchronous deletion path.
Activity