Part II: Tools

The second part of this book comprises five chapters that deal with the implementation of parallel programs. In parallel as in sequential programming, there are many different languages and programming tools, each suitable for different classes of problem. Because it would be   neither feasible nor useful to describe them all, we restrict our   attention to four systems---Compositional C++ (CC++ ), Fortran M   (FM), High Performance Fortran (HPF), and the Message Passing   Interface (MPI)---and explain how each can be used to implement designs developed using the techniques of Part I. We also describe, in Chapter 9, tools that aid in the collection and analysis of performance data.

Except where material is explicitly cross-referenced, each chapter in Part II is self-contained. Hence, it is quite feasible to base a practical study of parallel programming on just one of the four tools described here. However, while each of these tools is of broad utility, each also is most appropriate for different purposes, and we recommend that you become familiar with several systems.

CC++ , described in Chapter 5, is a small set of extensions to C++ . These extensions provide the programmer with explicit control over locality, concurrency, communication, and mapping and can be used to build libraries that implement tasks, channels, and other basic parallel programming abstractions. Designs developed using the techniques of Part I are easily expressed as CC++ programs.

FM, described in Chapter 6, is a small set of extensions to Fortran. These extensions provide explicit support for tasks and channels and hence can implement designs developed using the techniques of Part I directly. A distinguishing feature of FM is that programs can be guaranteed to be deterministic, meaning that two executions with the same input will produce the same output.

HPF, described in Chapter 7, is an example of a data-parallel language and has emerged as a de facto standard for scientific and engineering computation. Parallelism is expressed in terms of array operations---statements that apply to many elements of an array at once. Communication operations are inferred by the compiler, and need not be specified by the programmer.

MPI, described in Chapter 8, is a library of standard subroutines for sending and receiving messages and performing collective operations. Like HPF, MPI has emerged as a standard and   hence supersedes earlier message-passing libraries such as PARMACS,   p4, PVM,   and Express.  

When building parallel programs, our choice of tool will depend on the nature of the problem to be solved. HPF is particularly appropriate for numeric algorithms based on regular domain decompositions (for example, finite difference computations). CC++ and FM are better suited for applications involving dynamic task creation, irregular communication patterns, heterogeneous and irregular computation structures, and concurrent composition. They can also be used to build data-parallel libraries for regular problems. MPI is a lower-level approach to parallel programming than CC++ , FM, or HPF and is particularly appropriate for algorithms with a regular SPMD structure.

CC++ , FM, HPF, and MPI represent very different approaches to parallel programming. Nevertheless, we shall see that good design is independent of our choice of implementation language. The design techniques introduced in Part I apply regardless of language. Issues of concurrency, scalability, locality, and modularity must be addressed in any parallel program.