Skip to content

fix(cli): delete dev endpoints when rp flash dev is stopped by a signal - #625

Draft
justinwlin wants to merge 1 commit into
mainfrom
fix/dev-signal-cleanup
Draft

justinwlin wants to merge 1 commit into
mainfrom
fix/dev-signal-cleanup

Conversation

@justinwlin

Copy link
Copy Markdown
Contributor

Stopping rp flash dev with a signal left its endpoints running and billing. An agent's dev loop hits this every time: it starts the session in the background and stops it with kill.

Cause

  • Non-interactive shells start background jobs with SIGINT ignored. Python then never installs its KeyboardInterrupt handler, and asyncio doesn't add its own, so kill -INT does nothing.
  • SIGTERM has no handler, so the process dies without running teardown.
  • A stop during a long entrypoint call waits for that call with no limit.

runpod-flash cleaned up on KeyboardInterrupt in its CLI.

Change

  • Install SIGINT and SIGTERM handlers on the session loop. The first signal cancels the session, and teardown deletes the endpoints. A second signal stops waiting for a running entrypoint.
  • The entrypoint thread is a daemon, and it ignores the loop closing underneath it after a forced stop.
Verification
  • Live: rp flash dev main.py < /dev/null & from a script, then kill -TERM and separately kill -INT after the first run. Both exited with ✓ cleaning up 1 deleted, and a GraphQL endpoint list showed nothing left.
  • New tests: SIGTERM deletes endpoints, SIGINT cleans up even when ignored at start, and a second signal tears down before a blocked entrypoint returns. The last one first failed with Event loop is closed from the abandoned thread, which this change fixes. Each passed three consecutive runs. pytest tests: 1482 passed, coverage 91.64%.

A dev session started in the background from a non-interactive shell
inherits SIGINT as ignored, so Python never raises KeyboardInterrupt,
and SIGTERM has no handler. Either way the process stopped without its
teardown and left the session's endpoints running and billing. A stop
during a long entrypoint call also waited for that call indefinitely.

Install SIGINT and SIGTERM handlers on the session loop. The first
signal cancels the session, which runs teardown and deletes the
endpoints. A second signal stops waiting for a running entrypoint. The
entrypoint thread is a daemon and tolerates the loop closing under it.
Comment thread runpod/rp_cli/main.py
# an abandoned entrypoint's error is not retrieved
done.add_done_callback(lambda f: f.cancelled() or f.exception())
else:
await done
Comment thread runpod/rp_cli/main.py
# an abandoned entrypoint's error is not retrieved
done.add_done_callback(lambda f: f.cancelled() or f.exception())
else:
await done

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants