Exploiting Direct Access Shared Memory for MPI On Multi-Core Processors

Abstract

In 2008, the Catamount lightweight kernel was extended to support direct access shared memory between processes running on the same compute node. This extension, called SMARTMAP, allows each process read/write access to another process’ memory by extending the virtual address of each process. Simple virtual address bit manipulation can be used to access the same virtual address in a different process’ address space. In this paper we describe an implementation of the Message Passing Interface (MPI) that uses SMARTMAP for intra-node message passing. SMARTMAP has several advantages over POSIX shared memory techniques for implementing MPI. We present performance results comparing MPI using SMARTMAP to the existing MPI transport layer on a quad-core Cray XT platform.

Keywords

MPI shared memory multi-core lightweight kernel operating system

Get full access to this article

View all access options for this article.

References

Brightwell, R. et al. (2005). Implementation and performance of Portals 3.3 on the Cray XT3. In Proceedings of the 2005 IEEE International Conference on Cluster Computing.

Brightwell, R. et al. (2006). SeaStar interconnect: Balanced bandwidth for scalable performance. IEEE Micro 26(3): 41-57.

Buntinas, D. , Mercier, G. and Gropp, W. ( 2006a). Data transfers between processes in an SMP system: Performance study and application to MPI. In Proceedings of the 2006 International Conference on Parallel Processing.

Buntinas, D. , Mercier, G. and Gropp, W. ( 2006b). Design and evaluation of Nemesis, a scalable, low-latency, message-passing communication subsystem. In Proceedings of the 2006 International Symposium on Cluster Computing and the Grid.

Buntinas, D. , Mercier, G. and Gropp, W. ( 2006c). Implementation and shared-memory evaluation of MPICH2 over the Nemesis communication subsystem. In Proceedings of the 2006 European PVM/MPI Users’ Group Meeting.

Buntinas, D. , Mercier, G. , and Gropp, W. ( 2007). Implementation and evaluation of shared-memory communication and synchronization operations in MPICH2 using the Nemesis communication subsystem . Parallel Computing 33(9): 634-644.

Camp, W.J. and Tomkins, J.L. ( 2002). Thor’s hammer: The first version of the Red Storm MPP architecture. In Proceedings of the SC 2002 Conference on High Performance Networking and Computing, Baltimore, MD.

Gabriel, E. et al. (2004). Open MPI: Goals, concept, and design of a next generation MPI implementation. In Proceedings, 11th European PVM/MPI Users’ Group Meeting, Budapest , Hungary, pp. 97-104.

Graham, R.L. et al. (2007). An evaluation of Open MPI’s matching transport layer on the Cray XT. In Proceedings of the 14th European PVM/MPI Users’ Group Conference.

10.

Graham, R.L. and Shipman, G. ( 2008). MPI support for multi-core architectures: Optimized shared memory collectives. In Proceedings of the 15th European PVM/MPI Users’ Group Conference.

11.

Jin, H.-W. et al. (2005). Limic: Support for high-performance MPI intra-node communication on Linux. In Proceedings of the 2005 Cluster International Conference on Parallel Processing.

12.

Kelly, S.M. and Brightwell, R. ( 2005). Software architecture of the light weight kernel, Catamount . In Proceedings of the 2005 Cray User Group Annual Technical Conference.