Celeryd_concurrency, --concurrency And Autoscale

I have a few questions regarding task routing, concurrency and performance. Here is my use case : I've got one dedicated server to run celery tasks, so I can use all the CPUs to ru

Solution 1:

It makes no sense setting both concurrency and autoscale since both are means to control the number of worker subprocesses for a given worker instance, as explained here.

--concurrency N means you will have exactly N worker subprocesses for your worker instance (meaning the worker instance can handle N conccurent tasks).

--autoscale max, min means you will have at least min and at most max concurrent worker subprocesses for a given worker instance.

On which CPU each process (the main worker process or any of it's child subprocesses) will run is not predictable, it's an OS thing, but do not assume subprocesses will all run on the same CPU (chances are they won't - that's part of the point of having concurrent subprocesses actually).