This is a big undefined area in HPF, and HPFF II was formed in part to discuss issues related to parallel I/O. However, it was decided in the 1995 HPFF kickoff meeting to concentrate on irregular problem support and case parallelism, deferring parallel I/O until a later date. Existing parallel file systems vary in their level of support for data distribution mappings [32]; usually HPF programmers rely on EXTRINSIC routines to do I/O in parallel [21,23].
Parallel I/O can be thought of as a data mapping problem in HPF. As Gilbert pointed out [48], it is common to break the mapping into two stages: first alignment maps the data to an abstract template, and then a distribution maps that template to the processors. But we have found that the two stages are in fact not independent, particularly in xHPF (FULLBLOCK vs. SHRUNKBLOCK is a distribution decision, but can appear in an ALIGN statement).
Nevertheless,
Choudhary et al., used this idea to design
a two-phase run-time library that
could be ported to systems that support a parallel file system (like the SP2
with PIOFS) [30].
The I/O processors (
) are a subset of the compute
processors (P). The I/O processors read in the data and redistribute it
to the compute processors. Note that one compute process is running on the
same node as the I/O process; when the
I/O processor determines that the
compute process is computing, the
I/O process will use the switch to do sending.
They also support a virtual file system where parallel files are distributed to local disks, for fast I/O. The file is updated during quiet times on the switch (see above). You get to specify number of disks involved.
Since Bordawekar did his thesis, the group has gone on to develop PASSION, a runtime library for parallel I/O mainly for out-of-core arrays [176,177,178]. In other words, this would be using the parallel file system as scratch space. The intent is that you would write an ``out-of-core'' HPF program. Don't know what directives you would use, or if it would just be left up to the compiler to decide when to allocate arrays to backing store. In Frontiers'95 the complaint was that existing out-of-core applications are hard to find (perhaps because they are hard to write?).
Gropp et al. [75] describe the Chamelian-based set of parallel I/O primitives in use at Argonne. Their paper lists the actual kinds of I/O needed by grand challenge applications: input, debugging, scratch files, checkpoint/restart, and output, and briefly discusses each but leaves lots out. They have implemented their primitives (which are similar to NPAC's but all different spellings) on top of P4 and Chamelion for the SP-1. They have an PIFArrayPart struct that is similar to Choudhary's ADT (Array Descriptor Table).
Brezany et al. [41] writes distributed data to file in chunks and is supported at the compiler level.
Jacob Gotwals et al. describe their streams-based parallel I/O system designed for use with pC++ [80].
Finally, the Vesta file system is described in [59]. Vesta has evolved into PIOFS at this point, but the statement by the authors that they are designing a loosely synchronous SPMD parallel I/O library, with collective I/O operations, accessible through C, Fortran, and HPF, is probably significant. They also mention that Vesta could be used by HPF programmers for scratch space, so one doesn't care whether scratch space is in memory or on disk. The Vesta primitives are Vesta_Read, Vesta_Write, Vesta_create, etc. The Vesta client runs as part of the application on the compute nodes, and the Vesta server runs on the storage nodes. Overall, it provides a user-definable view of parallel files.
For a classification of parallel file organizations, see the paper by Crockett [60].
Kennedy's suggestion for High Performance Fortran I/O is
!SIO$ IO-DISTRIBUTE A(BLOCK,X)
and let the compiler, HPF runtime, and I/O subsystem handle it.
Coelho [58] suggests further modifications of
this and gives an Irigoin-type underpinning to this argument.