gh-74028: `concurrent.futures.Executor.map`: avoid reference cycles when an exception is raised by ebonnal · Pull Request #131701 · python/cpython

ebonnal · 2025-03-24T23:57:32Z

A failed future stores the raised exception in its ._exception attribute. We get a reference cycle if this exception's __traceback__ refers back to the future.

A particular care has been taken to avoid that (e.g. by @graingert in #95169)

This PR tests this behavior.

(follows up discussion in #131467)

Edit:
We extend this PR to ensure that no reference to an Exception is captured in the error's traceback frames.

Issue: Make Executor.map work with infinite/large inputs correctly #74028

ebonnal · 2025-03-25T11:47:21Z

only failing for Ubuntu (free-threading) / build and test (ubuntu-24.04-arm) and I cannot reproduce on my ubuntu arm VM

Edit: actually I did reproduce it after a couple attempts

graingert · 2025-03-30T12:50:12Z

@ebonnal thanks for the test! Your first test was good, and it found a bug https://github.com/python/cpython/actions/runs/14048391667/job/39333996238?pr=131701#step:22:632

I've pushed a commit to revert to the original style of test - which includes the references in the error message, and fixed the bug.

ebonnal · 2025-03-30T20:43:10Z

I've pushed a commit to revert to the original style of test - which includes the references in the error message, and fixed the bug.

Thanks @graingert !

probably we also need to finally: del exc_wrapper here too

Nice, I have cleaned up this exc_wrapper and added an additional test that goes through the error's traceback frames and search for any reference to an Exception.

(I have renamed the PR to show that we have extended its scope)

ebonnal · 2025-03-31T23:22:34Z

@graingert via my recent cosmetic changes I have noticed that the get_referrers test fails from time to time in a non deterministic way (e.g. https://github.com/python/cpython/actions/runs/14169734692/job/39690706469?pr=131701#step:22:777):

build: it only concerns the free-threading builds (configured with --disable-gil)
test: the failing module is either test_interpreter_pool or test_process_pool
platform: only on Ubuntu or Windows, the test never fails on macos. (I confirm that I have successfully reproduced the issue multiple times on an Ubuntu VM, and never on my local macos).

-> as a deterministic workaround in those situations, I have added an explicit gc collection before the test:

if (
    '--disable-gil' in get_config_var("CONFIG_ARGS")
    and sys.platform in ("linux", "win32")
):
    gc.collect()

I wonder if this GC issue with free-threading builds is a known issue. I also wonder if we should investigate this further as part of this PR or if we should open an issue 🤔 Let me know your thoughts!

graingert · 2025-04-01T07:07:39Z

When debugging these sorts of things I usually use a mix of the packages objgraph and refcycle. Can you repeat this locally? I'll see if I can find out what's going on

graingert · 2025-04-03T07:40:48Z

ok I can't repeat it locally, even disabling the GC - sorry for hijacking your PR but I think I need to test it in CI to repeat

ebonnal · 2025-04-05T22:46:10Z

Hey @graingert, update from my investigations with a free-threading build on an Ubuntu ARM machine:
It fails like 50% of the time, looking like some GC race condition, but the test is always successful if we pause for a bit with a time.sleep(1) before checking the referrers.
Maybe for the scope of this PR we should go back to the state of 03f8ab4 and just add this little pause?

graingert · 2025-04-07T07:33:54Z

I don't think it's a gc race condition as the problem still happens with the GC disabled. I think the problem is that the main thread is resumed before the background thread can delete the future.

I'm happy for you to revert back to 03f8ab4

This reverts commit 09b0819.

This reverts commit ef30122.

This reverts commit cae6474.

This reverts commit fc7a569.

…hongh-74028) Hold exc across 'raise exc' keeps the frame in the exception's referrers via its traceback; clear it in a finally like the interpreter.py fix so the strengthened test_map_exception passes for all pool types. Co-authored-by: Muse Code (Meta Muse Spark)

bedevere-app Bot added the awaiting review label Mar 24, 2025

ebonnal mentioned this pull request Mar 24, 2025

gh-131466: concurrent.futures.Executor.map: avoid temporarily exceeding buffersize while collecting the next result #131467

Open

ebonnal changed the title ~~test_concurrent_futures: Executor.map: assert no reference cycle from failed future captured in its exception’s traceback~~ concurrent.futures.Executor.map test: assert no reference cycle from failed future captured in its exception’s traceback Mar 25, 2025

ebonnal changed the title ~~concurrent.futures.Executor.map test: assert no reference cycle from failed future captured in its exception’s traceback~~ concurrent.futures.Executor.map: test no reference cycle from failed future captured in its exception’s traceback Mar 25, 2025

picnixz changed the title ~~concurrent.futures.Executor.map: test no reference cycle from failed future captured in its exception’s traceback~~ gh-74028: concurrent.futures.Executor.map: test no reference cycle from failed future captured in its exception’s traceback Mar 30, 2025

bedevere-app Bot mentioned this pull request Mar 30, 2025

Make Executor.map work with infinite/large inputs correctly #74028

Closed

picnixz added the skip news label Mar 30, 2025

picnixz requested a review from graingert March 30, 2025 11:36

graingert reviewed Mar 30, 2025

View reviewed changes

Comment thread Lib/test/test_concurrent_futures/executor.py Outdated

graingert reviewed Mar 30, 2025

View reviewed changes

Comment thread Lib/concurrent/futures/interpreter.py Outdated

ebonnal changed the title ~~gh-74028: concurrent.futures.Executor.map: test no reference cycle from failed future captured in its exception’s traceback~~ gh-74028: concurrent.futures.Executor.map: test there are no reference cycles when an exception is raised Mar 30, 2025

ebonnal changed the title ~~gh-74028: concurrent.futures.Executor.map: test there are no reference cycles when an exception is raised~~ gh-74028: concurrent.futures.Executor.map: test there are no reference cycles after an exception is raised Mar 30, 2025

ebonnal changed the title ~~gh-74028: concurrent.futures.Executor.map: test there are no reference cycles after an exception is raised~~ gh-74028: concurrent.futures.Executor.map: ensure there are no reference cycles after an exception is raised Mar 30, 2025

ebonnal changed the title ~~gh-74028: concurrent.futures.Executor.map: ensure there are no reference cycles after an exception is raised~~ gh-74028: concurrent.futures.Executor.map: avoid reference cycles after an exception is raised Mar 30, 2025

ebonnal changed the title ~~gh-74028: concurrent.futures.Executor.map: avoid reference cycles after an exception is raised~~ gh-74028: concurrent.futures.Executor.map: avoid reference cycles when an exception is raised Mar 30, 2025

graingert reviewed Mar 31, 2025

View reviewed changes

Comment thread Lib/concurrent/futures/interpreter.py Outdated

ebonnal changed the title ~~gh-74028: concurrent.futures.Executor.map: avoid reference cycles when an exception is raised~~ gh-74028: concurrent.futures.Executor: avoid reference cycles when an exception is raised Mar 31, 2025

ebonnal requested a review from graingert March 31, 2025 20:34

graingert marked this pull request as draft April 3, 2025 07:11

bedevere-app Bot removed the awaiting review label Apr 3, 2025

ebonnal marked this pull request as ready for review April 7, 2025 12:45

ebonnal and others added 25 commits October 20, 2026 23:11

concurrent.futures.process: avoid ref cycle in _sendback_result

f2e4dbe

concurrent.futures.process: avoid ref cycle in _process_worker

1245a9a

format test msg

e984592

refactor ZeroDivisionError assert

d3dee34

Revert "concurrent.futures.process: avoid ref cycle in _process_worker"

fc2a351

This reverts commit 09b0819.

Revert "concurrent.futures.process: avoid ref cycle in _sendback_result"

2e29020

This reverts commit ef30122.

skip referrers test for free-threading build on win/linux

51269d9

call gc.collect for free-threading on linux/windows

ed0e87f

Revert "call gc.collect for free-threading on linux/windows"

b3f3f4a

This reverts commit cae6474.

Revert "skip referrers test for free-threading build on win/linux"

e3e75ce

This reverts commit fc7a569.

more gc referrers

70e22a1

even more gc referrers

12abc0e

delete a ref to work_item

cac4aac

delete reference to result_item before joining process

66b98b4

unpack ResultItem and WorkItem

68fcecb

fix typo

0be8f49

bit of a hack?

6c2406d

Revert investigations

7037c4a

pause before checking referrers

0568262

add comment about the avoided ref cycle with exc_wrapper

cfeaa9b

format test

346b4ea

make the self-captured exceptions check not rely on var name

2a25b9a

use traceback.walk_tb

fd9b9bb

comment the skipping of the current frame

682a719

test_map_exception: improve readability

fb0c522

ambv force-pushed the test/executor-map-ref-cycles branch from 82542f3 to fb0c522 Compare October 20, 2026 21:17

ambv requested a review from ericsnowcurrently as a code owner October 20, 2026 21:17

bedevere-app Bot added the type-feature A feature request or enhancement label Oct 20, 2026

ambv force-pushed the test/executor-map-ref-cycles branch from 05ba98a to e0a031d Compare October 21, 2026 01:52

Navigation Menu

Repository navigation

gh-74028: `concurrent.futures.Executor.map`: avoid reference cycles when an exception is raised - #131701

gh-74028: `concurrent.futures.Executor.map`: avoid reference cycles when an exception is raised#131701
ebonnal wants to merge 40 commits into
python:mainfrom
ebonnal:test/executor-map-ref-cycles

ebonnal commented Mar 24, 2025 •

edited

Loading

Uh oh!

ebonnal commented Mar 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

graingert commented Mar 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

ebonnal commented Mar 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

ebonnal commented Mar 31, 2025 •

edited

Loading

Uh oh!

graingert commented Apr 1, 2025

Uh oh!

graingert commented Apr 3, 2025

Uh oh!

ebonnal commented Apr 5, 2025 •

edited

Loading

Uh oh!

graingert commented Apr 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

ebonnal commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ebonnal commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

graingert commented Mar 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ebonnal commented Mar 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ebonnal commented Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

graingert commented Apr 1, 2025

Uh oh!

graingert commented Apr 3, 2025

Uh oh!

ebonnal commented Apr 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

graingert commented Apr 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ebonnal commented Mar 24, 2025 •

edited

Loading

ebonnal commented Mar 25, 2025 •

edited

Loading

graingert commented Mar 30, 2025 •

edited

Loading

ebonnal commented Mar 30, 2025 •

edited

Loading

ebonnal commented Mar 31, 2025 •

edited

Loading

ebonnal commented Apr 5, 2025 •

edited

Loading