Python: Threading + Lock Slows My App Down Considerably

Say I have a function that writes to a file. I also have a function that loops repeatedly reading from said file. I have both of these functions running in separate threads. (Actua

Solution 1:

Do you use threading on a multicore machine?

If the answer is yes, then unless your Python version is 3.2+ you are going to suffer reduced performance when running threaded applications.

David Beazly has put considerable effort to find what is going on with GIL on multicores and has made it easy for the rest of us to understand it too. Check his website and the resources there. Also you might want to see his presentation at PyCon 2010. It is rather intresting.

To make a long story short, in Python 3.2, Antoine Pitrou wrote a new GIL that has the same performance on single and multicore machines. In previous versions, the more cores/threads you have, the performance loss increases...

hope it helps :)

Solution 2:

Why aren't you acquiring the lock in the writer for the duration of each write only? You're currently locking for the entire duration of the load function, the reader never gets in until the load function is completely done.

Secondly, you should be using context locks. Your current code is not thread safe:

def load(lock):
    for x in data:
        withlock:
            whatever.write(x)

The same goes for your reader. Use a context to hold the lock.

Thirdly, don't use an RLock. You know you don't need one, at no point does your read/write code need to reacquire, so don't give it that opportunity, you will be masking bugs.

The real answer is in several of the comments to your question: The GIL is causing some contention (assuming it isn't actually your misuse of locking). The Python threading module is fantastic, the GIL sometimes is not, but moreso, the complex behaviours it generates that are misunderstood. It's worth mentioning though that the mainstream belief that throwing threads at problems is not the panacea people believe it to be. It usually isn't the solution.