This is the continuation of a conversation started at Brian’s post on approaches to parallelism infrastructure, and continued at Thinking Parallel.
Although their blog post certainly right in that functional languages are a entirely different paradigm and approach to development, and therefore there is a high barrier to their entry, I’m with Brian’s post here — sooner or later, functional languages are going to make it into the mainstream, because the parallelism advantages are just too great to ignore.
Note that there is a precursor for this kind of change: the coincidental rise of the GUI and object oriented languages. When GUIs simply became necessary for many/most development projects, the business languages shifted to the object-oriented model, which is the most natural for that domain. That said, it took 20-odd years into the wide scale adoption of graphical applications before the change made it.
We will be in a similar case soon regarding parallelism. At this point, single-threaded performance has tapped out: chips aren’t getting the kind of incredible boosts in mHz that they used to. Instead, we are seeing speed growth through multi-core, which provides performance benefits only accessible to multi-threaded/multi-process applications. The problem is that the existing set of business languages and their semaphore-based locking mechanisms are nightmares, and that attempting to retrofit clean performance solutions is an uphill battle (trust me, I know). So, unless the business need for performance stops growing, the push for languages with enhanced parallelism is going to be inevitable.
That said, I think that the idea of a shared virtual machine that can execute both object-oriented code and functional code is a pretty powerful idea, and would get some very major gains. Unfortunately, without some major changes into virtual machine design, it looks like you can either have fast mutable (OO) code or fast immutable (functional) code, not both. And right now, the only popular shared virtual machines (Sun’s JVM for Java/Scala), Microsoft’s CLR for C#/F#) emphasize the fast mutable code, and they don’t offer the “free” performance win that something like JoCaml gets you.
As a final aside, I’d note that I’m with David Heinemeier Hansson on one point: I think the web server/web application is going to be the last place to go, because the standard use case gets you the paralellism for free. If you’ve got a process-per-request model, you can rely on the traffic through your website (and the OS kernel) to take advantage of the multi-core performance for you. So the push to change towards functional models aren’t there. This is somewhat akin to the way the OO push never really made it into sys admin languages like shell scripting or Perl. That said, there’s already a lot of functional leaking into the Ruby way of thinking, and that will continue to increase as functional languages make it into the mainstream.
6 Comments
I agree that we need better languages for cope with parallelism. As I said in my article, I doubt it will be the functional ones. I would be happy if it were, but I doubt it. The problems with parallelism have been there e.g. in high performance computing forever. The functional languages have been available forever. Still, nobody in HPC uses functional languages.
Will it be different, now that parallelism moves out of HPC into the mainstream? I have my doubts, but I am afraid only time will tell :-).
HPC coding is done at or near C, because the performance/productivity value balance is weighed heavily in favor of the former. Functional languages are pretty deep abstractions from machine language, so it’s not a surprise that they have been lacking in traction there: so have the OO languages, excepting a certain subset of C++.
If someone can come up with a good way to code parallelism in the standard business languages (and semaphores don’t count), there’s a Ph.D. and a lot of well-paid contracting gigs waiting for them. The only attempt I know of is Cilk, and that has not fared well. Although I have no proof for this statement, it seems to me that automatic procedural parallelism simply cannot happen: the procedural approach (including procedural OO languages like C#/Java) seems fundamentally broke because of its dependence on mutable data and the second-class quality of functions/code blocks. Immutable data and first-class functions are what empower JoCaml to do its thing, and I think that’s the direction we’re going to have to go.
Multiprocessing has been around forever. Remember that Multics from the 1960’s was designed for asymmetric multiprocessing- basically the equivelent of shoving both a PowerPC and an x86 into the same box.
What has changed is that the free ride of single-process performance has ended. Numerical algorithms generally parallelize easily- parallelizing everything is work. The logic used to be why bother to spend the time and effort to parallelize any given code base, because by the time you’re done, faster processors will have come out giving you that speed up for free.
But that doesn’t apply any more. And the situation is worse. Multiprocessor used to mean 2-8 cpus, maybe as high as 64 in extreme cases. Now, the number of cores per cpu is doubling every generation. And multi-chip boxes are becoming much more common, as going from 2 (1 dual core chip) to 4 (2 dual core chips) is less weird- especially as those quad cores are comming out anways.
And I think we’ll see an explosion in cores in the near future, as processor makers “back-date” when they gave up on single-threaded performance. If you can make a CPU that is 95% as fast for 1/4th the number of transistors, you can give a 3.8x speed up by giving more of the smaller processors, and in effect “back-dating” when you started adding more processors by two generations. And then they’ll start eyeing giving you 16 processors in the same realestate, each at 80% the speed…
Add all of this up, and I expect that within five or so years, you’ll be seeing work stations with hundreds of cores in them- and home boxes with as many cores as the largest servers of ten years ago had. Single chips will have hundreds, possibly thousands, of processors. And large boxes will harness hundreds of these chips. And the range of multithreaded computers will increase wildly- while it used to be that even multithreaded code only had to deal with one, maybe two, orders of magnitude range in the number of cpus (the top end Sun and SGI servers having ~100 processors), we’ll be seeing differents of three, even four, orders of magnitude.
Sometimes a difference in degree is a difference in kind. Building a bridge over the English Channel is a fundamentally different undertaking than building a bridge over Old Man’s Creek. Different technologies are called for.
This is the point at which functional languages become a necessity- and programmers will have the choice of learn or die. I can not imagine writting a program that effectively uses anywhere from 1 to 1024 processors using old style mutable data structures, threads, and locks.
An additional comment- message passing is not a complete solution. I think it’s an important part of the solution, but it isn’t sufficient.
Some of the problems with message passing are:
1) Scaling of the number of processes used is not automatic, but is instead a design decision. Changing the number of processes requires a rearchitecturing of the code base.
2) It doesn’t eliminate deadlocks- and worse yet, it makes deadlocks much harder to detect (as they take the form of “thread 1 is waiting for a message that only thread 2 will send, while thread 2 is waiting for a message that only thread 1 will send”, and it’s really hard to prove the “only thread X will send this message”). It does make deadlocks less common, however.
3) It doesn’t eliminate race conditions, at least if race conditions are specified as “event X should not happen between when I send/receive message A and when I send/receive message B”. Again, race conditions are signfigiantly reduced, but not eliminated.
4) It does not solve the composition problem- if I want to take an object out of data structure A “owned” by thread 1, and insert it into data structure B “owned” by thread 2, without any other thread (except threads 1 and 2, obviously) seeing it in either both data structures or neither data structures, without introducing either race conditions or deadlocks.
It does deal with high latency/low bandwidth connections (especially network connections, but signifigantly non-uniform ccNUMA also counts) much better, allow for process failure, and can be retrofitted into existing, single-threaded languages (See JoCaml, MPI, etc.). But the above reasons are why I don’t think it’s sufficient- especially not when you’re looking at designing a program with hundreds or thousands of threads.
If you’re interested in concurrency then you might like to look at F#’s new asynchronous programming constructs.
I stumbled across this article when hunting for interesting reading regarding parallelization.
I thought I’d mention the D programming language here. The 2.0 implementation of the language is currently being equipped with “safe” and a “functional” subsets (more about safeD here: http://www.digitalmars.com/d/2.0/safed.html ).
The functional subset is, as far as I understand, not entirely implemented yet. It is being prepared for, however, and the goal is to be able to create purely functional parts of a program, that can communicate, but not interact, with the imperative pats. Eventually this will mean automatic concurrency for the purely functional parts of the program. There’s a presentation about this here: http://www.digitalmars.com/d/2.0/accu-functional.pdf
One Trackback
[...] hey hey! It’s like he reads this blog or something. It’s nice to know that this idea doesn’t just exist in the head of Brian [...]