Python: Interdependent Process/thread Queues

I have four queues that each have multiple processes/threads that are interdependent in the following way: Queue 1 is reading a file from disk and copying to RAM Queue 2 takes the

Solution 1:

Is there any particular reason you actually need each of those 4 steps be separate threads/processes? Personally I'd just implement all 4 steps in one function/callable class, and then use multiprocessing.Pool's map to invoke the function in parallel over the filenames of interest.

Simpler example of this sort of pattern (just reading and processing) discussed in this Q&A. As the answer notes, if it appears to bottleneck on I/O rather than processing, just create more processes in the pool.