feat(autoscaling)!: Add `ConcurrencySystem` to share one concurrency budget between crawlers by Mantisus · Pull Request #2322 · apify/crawlee-python

Mantisus · 2026-10-14T00:27:25Z

Description

Load monitoring and concurrency scaling move from AutoscaledPool into a new ConcurrencySystem. Pass one system to several crawlers through the new concurrency_system option to cap their combined concurrency. The autoscaling classes are now public in crawlee.autoscaling.

Autoscaling logs move to DEBUG, and max_tasks_per_minute now counts the tasks started over the last 60 seconds. Breaking changes are in the v2 upgrade guide.

Issues

Closes: Share one concurrency budget between multiple crawlers #2133
Closes: Reduce the autoscaled pool logs at INFO level #1591
Closes: Make the AutoscaledPool log understandable #705

✍️ Drafted by Claude Code

Pijukatel · 2026-10-16T08:17:31Z

-The <ApiLink to="class/AutoscaledPool">`AutoscaledPool`</ApiLink> manages a pool of asynchronous, resource-intensive tasks that run in parallel. It keeps `min_concurrency` tasks running even while the system is overloaded, and starts additional tasks only when there is enough free CPU and memory. To monitor system resources, it leverages the <ApiLink to="class/Snapshotter">`Snapshotter`</ApiLink> and <ApiLink to="class/SystemStatus">`SystemStatus`</ApiLink> classes. If any task raises an exception, the error is propagated, and the pool is stopped. Every crawler uses an <ApiLink to="class/AutoscaledPool">`AutoscaledPool`</ApiLink> under the hood.
+Every crawler runs its requests through an <ApiLink to="class/AutoscaledPool">`AutoscaledPool`</ApiLink>, and the pool asks a <ApiLink to="class/ConcurrencySystem">`ConcurrencySystem`</ApiLink> whether there is room for one more request. The system is what keeps `min_concurrency` requests running even while the machine is overloaded, starts further ones only when there is enough free CPU and memory, and scales the budget up and down as the load changes. To monitor system resources, it leverages the <ApiLink to="class/Snapshotter">`Snapshotter`</ApiLink> and <ApiLink to="class/SystemStatus">`SystemStatus`</ApiLink> classes. If any task raises an exception, the error is propagated and the pool is stopped.
+
+The crawler builds its own system from the `ConcurrencySettings` described above. Build one yourself and pass it as `concurrency_system` to go beyond those settings, or to share a single budget between several crawlers that would otherwise each scale against the whole machine and oversubscribe it. The budget goes to whoever asks for it first, so a crawler that joins a saturated system late may have to wait. Browser crawlers default to a lower `desired_concurrency` than the rest, and a supplied system replaces that default too, so set it in the `ConcurrencySettings` you build the system from. Holding the instance also lets you retune `min_concurrency` and `max_concurrency` while the crawl runs. The lifecycle of such a system belongs to you, so keep it open until every crawler using it has finished.


Browser crawlers default to a lower desired_concurrency

Just an idea. Could those default initial desired_concurrencies be guessed by the amount of resources available? For example, browser crawlers would probably look at available memory

This is quite difficult to predict.

The amount of RAM used by a browser-based crawler will depend on the browser used (Firefox, Chrome, Camoufox) as well as on the websites. Some resource-intensive websites consume a huge amount of memory in the browser.

Pijukatel · 2026-10-16T08:33:06Z

+
+@dataclass(frozen=True)
+@docs_group('Autoscaling')
+class ConcurrencyConsumer:


Why have ConcurrencyConsumer and not just a plain string?

This feels like a preparation for something missing in the codebase right now. Is there a specific plan to extend this in the future?

I kept it for parity with TS. It also keeps the protocol stable: a field added to the consumer later doesn't change any method signature, while switching from a plain string to an object afterwards would be a breaking change.

I'm not aware of any concrete plans to extend it yet. That said, I'm thinking about adding a status field that a system could use to allocate slots, so that new consumers don't end up empty-handed and inactive ones give their slots back. That will most likely come after we finish porting the major updates from TS v4.

Pijukatel · 2026-10-16T08:48:28Z

-                        await asyncio.sleep(60 / self._max_tasks_per_minute)
-
-                    continue
+                    task_id = str(self._next_task_id)


Is there any specific benefit to having IDs as increasing numbers or just an implementation detail?

It's just an implementation detail

Pijukatel · 2026-10-16T08:54:00Z

+    """
+
+    _TASKS_PER_MINUTE_WINDOW = 60
+    """Number of one-second buckets the per-minute task cap is counted over."""


Reading this sentence over and over again and still not sure what it means :D

This does not feel like a constant, but an intermediary result that depends on the hardcoded values elsewhere in the code.

I think something like this would be more readable

_BUCKET_DURATION = timedelta(seconds=1)
_WINDOW_DURATION = timedelta(minutes=1)
_BUCKET_COUNT = int(WINDOW_DURATION / BUCKET_DURATION) # 60

vdusek · 2026-10-16T08:02:07Z

+        self._tasks_started = deque([0] * self._TASKS_PER_MINUTE_WINDOW, maxlen=self._TASKS_PER_MINUTE_WINDOW)
+
+        try:
+            await self._measurement.enter_async_context(self._snapshotter)


Important: a system entered once outside the crawlers (the setup the docs recommend) loses its CPU and memory listeners when the first crawler.run() exits the global event manager, because EventManager.__aexit__ clears all listeners on its last exit. Later runs then never see overload and scale to max_concurrency. Outside an Actor, I saw 2 listeners after __aenter__ and 0 after the first run. Holding the event manager open for the life of the system fixes it (needs from crawlee import service_locator):

Suggested change

await self._measurement.enter_async_context(self._snapshotter)

# The CPU and memory snapshots come from the global event manager, which drops every listener on its last

# exit. Holding it open keeps them flowing for as long as the system lives, also across crawler runs.

await self._measurement.enter_async_context(service_locator.get_event_manager())

await self._measurement.enter_async_context(self._snapshotter)

✍️ Drafted by Claude Code

It will be resolved in #2242

vdusek · 2026-10-16T08:02:07Z

+        self._concurrency_system = concurrency_system
+        self._autoscaled_pool = autoscaled_pool = AutoscaledPool(
+            concurrency_system=concurrency_system,
+            consumer=ConcurrencyConsumer(id=str(self._id)),


Note: self._id is shared by crawlers that share state, and a user-set id can collide with an auto-assigned one. Two such crawlers get equal (frozen) consumers, and both pools number tasks from "0", which breaks the "process-unique" contract for a custom per-consumer system. The bundled ConcurrencySystem ignores the consumer, so nothing breaks today. Maybe build one consumer per crawler in __init__ from its own class counter (e.g. f'{self.__class__.__name__}-{n}') and reuse it across runs?

✍️ Drafted by Claude Code

Good point.

I'd prefer that we switch to using strings for IDs and make them more unique. We already have an issue for that: #1699

But this will work as a temporary solution.

vdusek · 2026-10-16T08:02:07Z

+current concurrency 3, desired 5, overloaded: cpu 58% of the time (limit 40%)
+```
+
+To read the concurrency values in code, use `crawler.concurrency_system`.


Maybe we should mention the logging changes as well?

Do you think anything should be added to the “## Autoscaling logs moved to DEBUG” section?

vdusek · 2026-10-16T08:02:07Z

-        # Set default concurrency settings for browser crawlers if not provided
-        if 'concurrency_settings' not in kwargs or kwargs['concurrency_settings'] is None:
+        # Lower the default concurrency for browser crawlers, unless the caller configured it themselves.
+        if kwargs.get('concurrency_settings') is None and kwargs.get('concurrency_system') is None:


Note: nothing tests the concurrency_system half of this guard (same in AdaptivePlaywrightCrawler). Without it, PlaywrightCrawler(concurrency_system=...) raises "cannot be combined", and that is the main way to share a system between browser crawlers. A small test that builds both crawlers with a ConcurrencySystem and checks crawler.concurrency_system is system would cover it.

✍️ Drafted by Claude Code

vdusek · 2026-10-16T08:02:07Z

+        Raises:
+            RuntimeError: If the system is not active, as nothing measures the load or scales the budget then.
+        """
+        _consumer = consumer


This assignment only silences the unused-argument lint, like _task_id = task_id (line 344), _consumer, _task_id = consumer, task_id (line 364) and _task_id = task_id in _basic_crawler.py:1534.

Why are the arguments there?

If it has a reason, I would prefer either naming them _ or putting # noqa: ARG002 on the signature.

I will use # noqa: ARG002 for it.

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 14, 2026

Mantisus self-assigned this Oct 14, 2026

Mantisus force-pushed the split-autoscaled-pool branch from 25a03be to a738bbb Compare October 14, 2026 11:02

add ConcurrencySystem to share one concurrency budget between crawlers

b6e4fad

Mantisus force-pushed the split-autoscaled-pool branch from a738bbb to 97286df Compare October 14, 2026 18:54

Mantisus mentioned this pull request Oct 14, 2026

feat(autoscaling): Add ThroughputConcurrencySystem #2328

Draft

Mantisus marked this pull request as ready for review October 14, 2026 19:48

Mantisus requested review from janbuchar and vdusek October 14, 2026 19:48

Mantisus mentioned this pull request Oct 15, 2026

fix(autoscaling)!: Call is_finished_function only when AutoscaledPool is idle #2332

Draft

Mantisus changed the title ~~feat!: Add ConcurrencySystem to share one concurrency budget between crawlers~~ feat(autoscaling)!: Add ConcurrencySystem to share one concurrency budget between crawlers Oct 15, 2026

vdusek requested a review from Pijukatel October 16, 2026 06:50

vdusek force-pushed the v2 branch from 792e5a6 to 08747f7 Compare October 16, 2026 07:43

vdusek force-pushed the split-autoscaled-pool branch 2 times, most recently from 028fd2c to b6e4fad Compare October 16, 2026 07:59

Pijukatel reviewed Oct 16, 2026

View reviewed changes

vdusek reviewed Oct 16, 2026

View reviewed changes

Mantisus force-pushed the split-autoscaled-pool branch from b9f1b02 to b6e4fad Compare October 16, 2026 13:39

Update src/crawlee/autoscaling/_types.py

39a2094

Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>

-            await self._measurement.enter_async_context(self._snapshotter)
+            # The CPU and memory snapshots come from the global event manager, which drops every listener on its last
+            # exit. Holding it open keeps them flowing for as long as the system lives, also across crawler runs.
+            await self._measurement.enter_async_context(service_locator.get_event_manager())
+            await self._measurement.enter_async_context(self._snapshotter)

Conversation

Mantisus commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Mantisus commented Oct 14, 2026 •

edited

Loading