Skip to content

feat: abort named child runs when the parent is gracefully aborted - #1153

Open
vdusek wants to merge 3 commits into
masterfrom
feat/abort-child-runs-with-parent
Open

vdusek wants to merge 3 commits into
masterfrom
feat/abort-child-runs-with-parent

Conversation

@vdusek

@vdusek vdusek commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Adds abort_with_parent to Actor.start, Actor.call, Actor.start_task and Actor.call_task. It's off by default and needs run_name. The flag is stored as abortWithParent in the name's record under __ACTOR_CHILD_RUNS, so a parent that migrated or was resurrected still knows which children to abort. The latest call under a name sets the value.

The SDK registers its own ABORTING listener on ApifyEventManager when the Actor context starts. It's kept apart from user listeners, so off() without a listener doesn't remove it. On a graceful abort, it sends a graceful abort to every flagged child, one request each, all in parallel. The platform returns a finished run unchanged, so there's no status lookup first. A child the API no longer knows is skipped, and any other failed abort is logged without stopping the rest. Each child is aborted with the client it was started with in this process, which matters when a custom token was passed. A child from before a migration uses the default client.

Once the parent is aborting, a named start or call with abort_with_parent=True raises a RuntimeError and doesn't start, resurrect or reattach a run. A loop that starts one child per batch stops there, so it doesn't leave a new child behind. A start already in flight when ABORTING arrives finishes recording its run, and that run is then aborted. Runs started without abort_with_parent aren't affected.

A hard abort, a timeout or an OOM kill still leaves the children running. Covering that needs a parent/child link on the platform.

Closes: #1128

✍️ Drafted by Claude Code

@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 30, 2026
@vdusek vdusek self-assigned this Sep 30, 2026
@codecov

codecov Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.16%. Comparing base (290af90) to head (ab99472).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1153      +/-   ##
==========================================
+ Coverage   93.00%   93.16%   +0.16%     
==========================================
  Files          54       54              
  Lines        3672     3733      +61     
==========================================
+ Hits         3415     3478      +63     
+ Misses        257      255       -2     
Flag Coverage Δ
e2e 31.60% <25.39%> (-0.12%) ⬇️
integration 51.32% <30.15%> (-0.37%) ⬇️
unit 89.95% <100.00%> (+0.22%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek
vdusek added this pull request to stack #1152 September 30, 2026 08:19
@vdusek vdusek linked an issue Sep 30, 2026 that may be closed by this pull request
@vdusek
vdusek force-pushed the feat/abort-child-runs-with-parent branch from 9554a97 to 1782ca6 Compare October 6, 2026 13:20
Base automatically changed from feat/child-runs-api to master October 9, 2026 14:50
@vdusek
vdusek force-pushed the feat/abort-child-runs-with-parent branch from 2d70b86 to c1f6b59 Compare October 9, 2026 15:11
@vdusek
vdusek requested review from Mantisus and Pijukatel October 9, 2026 15:19
@vdusek
vdusek marked this pull request as ready for review October 9, 2026 15:19
@apify-service-account apify-service-account added the tested Temporary label used only programatically for some analytics. label Oct 9, 2026
Comment thread src/apify/_child_runs.py Outdated
Comment on lines +246 to +249
run = await run_client.get()
if run is None or run.status not in _ABORTABLE_STATUSES:
return
await run_client.abort(gracefully=True)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any benefit in doing GET request for run status instead of sending a blind ABORT anyway? The previous state could probably be guessed from the response to the abort request.

It is important to clean up as quickly as possible to not get aborted midway, so going from 2 requests in series to 1 request is maybe worth it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, done in ba138b3. Each child now gets one abort(gracefully=True) request without a status check first.

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From Claude. Could you check that, pls?

Children started after ABORTING are never aborted. abort_runs_with_parent runs once. A named start or resurrect after that isn't covered. A typical orchestrator loop hits this:

for i, batch in enumerate(batches):
    await Actor.call(actor_id, batch, run_name=f'batch-{i}', abort_with_parent=True)

On a graceful abort, the current child is aborted and call returns ABORTED. The loop then starts the next batch inside the 30 s grace period. Once the parent is force-killed, that child is orphaned. A later call under an existing name can also resurrect the child we just aborted, since the registry resurrects ABORTED runs. I reproduced the first case in a unit test: after emit(ABORTING), a new named start with abort_with_parent=True never gets aborted.

Suggestion: set a "parent aborting" flag in abort_runs_with_parent. After that, find_or_start with abort_with_parent=True shouldn't start or resurrect runs. It could raise, or abort the run right after recording it. A test for this case would help too.

Comment thread src/apify/_actor.py
await self.event_manager.__aenter__()
# Only the platform emits `ABORTING`, and it does so through `ApifyEventManager`.
if isinstance(self.event_manager, ApifyEventManager):
self.event_manager._on_internal(event=Event.ABORTING, listener=self._abort_child_runs) # noqa: SLF001

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method _on_internal was added in that PR. Let's make it public.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, that's the point of them being private. We need to distinguish the SDK's own listeners from the users's.

off() without a listener removes everything a user registered, but the SDK's ABORTING handler that aborts child runs has to survive that.

If {on|off}_internal() were public, users could put their own listeners into that internal group too. Then off() would no longer reliably remove a user's listeners, and the line between SDK listeners and user listeners would be gone.

Comment thread src/apify/_actor.py
@vdusek

vdusek commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

continue here after #1184

@vdusek
vdusek force-pushed the feat/abort-child-runs-with-parent branch from c1f6b59 to ed70f26 Compare October 15, 2026 12:35
@vdusek
vdusek requested review from Mantisus and Pijukatel October 16, 2026 12:34

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Abort child runs when the parent is gracefully aborted

4 participants