I thought I’d mention the D programming language here. The 2.0 implementation of the language is currently being equipped with “safe” and a “functional” subsets (more about safeD here: http://www.digitalmars.com/d/2.0/safed.html ).
The functional subset is, as far as I understand, not entirely implemented yet. It is being prepared for, however, and the goal is to be able to create purely functional parts of a program, that can communicate, but not interact, with the imperative pats. Eventually this will mean automatic concurrency for the purely functional parts of the program. There’s a presentation about this here: http://www.digitalmars.com/d/2.0/accu-functional.pdf

Some of the problems with message passing are:

1) Scaling of the number of processes used is not automatic, but is instead a design decision. Changing the number of processes requires a rearchitecturing of the code base.

2) It doesn’t eliminate deadlocks- and worse yet, it makes deadlocks much harder to detect (as they take the form of “thread 1 is waiting for a message that only thread 2 will send, while thread 2 is waiting for a message that only thread 1 will send”, and it’s really hard to prove the “only thread X will send this message”). It does make deadlocks less common, however.

3) It doesn’t eliminate race conditions, at least if race conditions are specified as “event X should not happen between when I send/receive message A and when I send/receive message B”. Again, race conditions are signfigiantly reduced, but not eliminated.

4) It does not solve the composition problem- if I want to take an object out of data structure A “owned” by thread 1, and insert it into data structure B “owned” by thread 2, without any other thread (except threads 1 and 2, obviously) seeing it in either both data structures or neither data structures, without introducing either race conditions or deadlocks.

It does deal with high latency/low bandwidth connections (especially network connections, but signifigantly non-uniform ccNUMA also counts) much better, allow for process failure, and can be retrofitted into existing, single-threaded languages (See JoCaml, MPI, etc.). But the above reasons are why I don’t think it’s sufficient- especially not when you’re looking at designing a program with hundreds or thousands of threads.

What has changed is that the free ride of single-process performance has ended. Numerical algorithms generally parallelize easily- parallelizing everything is work. The logic used to be why bother to spend the time and effort to parallelize any given code base, because by the time you’re done, faster processors will have come out giving you that speed up for free.

But that doesn’t apply any more. And the situation is worse. Multiprocessor used to mean 2-8 cpus, maybe as high as 64 in extreme cases. Now, the number of cores per cpu is doubling every generation. And multi-chip boxes are becoming much more common, as going from 2 (1 dual core chip) to 4 (2 dual core chips) is less weird- especially as those quad cores are comming out anways.

And I think we’ll see an explosion in cores in the near future, as processor makers “back-date” when they gave up on single-threaded performance. If you can make a CPU that is 95% as fast for 1/4th the number of transistors, you can give a 3.8x speed up by giving more of the smaller processors, and in effect “back-dating” when you started adding more processors by two generations. And then they’ll start eyeing giving you 16 processors in the same realestate, each at 80% the speed…

Add all of this up, and I expect that within five or so years, you’ll be seeing work stations with hundreds of cores in them- and home boxes with as many cores as the largest servers of ten years ago had. Single chips will have hundreds, possibly thousands, of processors. And large boxes will harness hundreds of these chips. And the range of multithreaded computers will increase wildly- while it used to be that even multithreaded code only had to deal with one, maybe two, orders of magnitude range in the number of cpus (the top end Sun and SGI servers having ~100 processors), we’ll be seeing differents of three, even four, orders of magnitude.

Sometimes a difference in degree is a difference in kind. Building a bridge over the English Channel is a fundamentally different undertaking than building a bridge over Old Man’s Creek. Different technologies are called for.

This is the point at which functional languages become a necessity- and programmers will have the choice of learn or die. I can not imagine writting a program that effectively uses anywhere from 1 to 1024 processors using old style mutable data structures, threads, and locks.