Comments: Python on Multi-Core

The CPython implementation will continue to use a GIL. Performance would be hit too badly by any attempt to remove it, due to the need to lock data structure access during critical sections.

IronPython, Jython and PyPy are all free of a global interpreter lock, though I don't know how much work has been done on proving them thread-safe under all circumstances.

Posted by Steve Holden at March 7, 2008 08:59 PM

Hi,

See the Wide Finder project

http://www.tbray.org/ongoing/When/200x/2007/09/20/Wide-Finder

by Tim Bray for a useful discussion of concurrency issues.

Comparisons are made of different languages etc

The estimable Mr. Fredik Lundh made a signal contribution to the
discussion

See:http://effbot.org/zone/wide-finder.htm

The techiques presented in this article would seem to be widely applicable.

Tom

Posted by Thomas Crawley at March 7, 2008 11:26 PM

CPython does use native threads for multi-threading - so threaded applications do use more than one OS level thread thread even though your comments about the GIL are true.

Posted by MIchael Foord at March 8, 2008 06:34 AM

As pointed out at the parallel-processing session, multi-core architectures are effectively crippled when synchronization objects are depended on. If you can make your threads spend less time waiting on synchronization objects, your effective processing would parallel much better. Future advancements in processing power is pointing more and more towards multi-core systems rather than increasing amounts of MHZ. Dual core was introduced not too long ago and quad-core is in the pipeline for this year.

CPython depends critically on the GIL. If this implementation is not corrected, it would mean that we are stuck on the current CPython performance 2 years from now, 10 years from now, 20 years from now.

Posted by Hoang at March 8, 2008 10:46 AM

I believe Adam Olsen was working on patches to CPython which permit better usage of system threads. I'm not sure what the status of this work is, however. Apologies to Adam if he isn't actually involved with such work or if it's currently in "stealth mode" and he doesn't want to enter into discussions about it.

Posted by Paul Boddie at March 8, 2008 07:27 PM

Hi Hoang,

Please read the effbot article or the wide finder discussion before you make incorrect statements.

The GIL is not a problem for concurrent use of Python.

The startup and overhead of Python VM is much less than that of the JVM.

Use multiple python processes and communicate between them using shared memory etc..

Tom

Posted by Thomas Crawley at March 10, 2008 10:39 AM

Doug.. on Isotoma" has some deeper thoughts on concurrency, multi-core and Python.

Posted by Hoang at May 19, 2008 04:58 PM
MT::App::Comments=HASH(0x622520) Subroutine MT::Blog::SUPER::site_url redefined at /kunden/homepages/26/d89284986/htdocs/jotsite/cgi-bin/mt/lib/MT/Object.pm line 125.
MT::App::Comments=HASH(0x622520) Subroutine MT::Blog::SUPER::archive_url redefined at /kunden/homepages/26/d89284986/htdocs/jotsite/cgi-bin/mt/lib/MT/Object.pm line 125.
MT::App::Comments=HASH(0x622520) Subroutine MT::Blog::SUPER::archive_url redefined at /kunden/homepages/26/d89284986/htdocs/jotsite/cgi-bin/mt/lib/MT/Object.pm line 125.
MT::App::Comments=HASH(0x622520) Subroutine MT::Blog::SUPER::archive_url redefined at /kunden/homepages/26/d89284986/htdocs/jotsite/cgi-bin/mt/lib/MT/Object.pm line 125.
MT::App::Comments=HASH(0x622520) Subroutine MT::Blog::SUPER::archive_url redefined at /kunden/homepages/26/d89284986/htdocs/jotsite/cgi-bin/mt/lib/MT/Object.pm line 125.
MT::App::Comments=HASH(0x622520) Subroutine MT::Blog::SUPER::archive_url redefined at /kunden/homepages/26/d89284986/htdocs/jotsite/cgi-bin/mt/lib/MT/Object.pm line 125.
MT::App::Comments=HASH(0x622520) Subroutine MT::Blog::SUPER::archive_url redefined at /kunden/homepages/26/d89284986/htdocs/jotsite/cgi-bin/mt/lib/MT/Object.pm line 125.