Accelerating CMIP data analysis with parallel computing in R

In this Technical Note we examine eight schemes for parallelizing Extreme Value Analysis (EVA) on Coupled Model Intercomparison Project data via R foreach, doParallel, and doMPI packages. We perform strong scaling studies to delineate the performance impacts of factors such as R cluster type (TCP/IP sockets and MPI), communication protocol (Ethernet, IP over InfiniBand, and MPI), loop parallelization (outer or inner loop), and approaches to reading data from the NCAR GLADE parallel filesystem. We elucidate peculiarities of R memory management and overhead associated with interprocess communication and discuss broadcast limitations of Rmpi. The best performing scheme parallelizes the outer EVA loop across latitude and reads only the subset of the data operated on in the inner loop over longitude; the different cluster types and communication protocols all perform about equally for this scheme. This configuration represents a parallel speedup of 50 with 96 R workers, and is scalable for EVA on larger problem sizes than those presented here.

To Access Resource:

Questions? Email Resource Support Contact:

    UCAR/NCAR - Library


Resource Type publication
Temporal Range Begin N/A
Temporal Range End N/A
Temporal Resolution N/A
Bounding Box North Lat N/A
Bounding Box South Lat N/A
Bounding Box West Long N/A
Bounding Box East Long N/A
Spatial Representation N/A
Spatial Resolution N/A
Related Links N/A
Additional Information N/A
Resource Format PDF
Standardized Resource Format PDF
Asset Size N/A
Legal Constraints

Copyright Author(s). This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Access Constraints None
Software Implementation Language N/A

Resource Support Name N/A
Resource Support Email
Resource Support Organization UCAR/NCAR - Library
Distributor N/A
Metadata Contact Name N/A
Metadata Contact Email
Metadata Contact Organization UCAR/NCAR - Library

Author Milroy, Daniel
Chen, Sophia
Vanderwende, Brian
Hammerling, Dorit
Publisher UCAR/NCAR - Library
Publication Date 2017-06-30T00:00:00
Digital Object Identifier (DOI) Not Assigned
Alternate Identifier N/A
Resource Version N/A
Topic Category geoscientificInformation
Progress N/A
Metadata Date 2023-08-18T18:06:48.405376
Metadata Record Identifier edu.ucar.opensky::technotes:549
Metadata Language eng; USA
Suggested Citation Milroy, Daniel, Chen, Sophia, Vanderwende, Brian, Hammerling, Dorit. (2017). Accelerating CMIP data analysis with parallel computing in R. UCAR/NCAR - Library. Accessed 30 September 2023.

Harvest Source