Skip navigation
NASA Logo, National Aeronautics and Space Administration
Currently Being Moderated

My $DIETY, it's full of CODE!

Posted by Shawn Freeman on Jun 24, 2008 3:15:41 PM

Hello everyone and welcome to GEOS 5 Optimization: MERRA Edition! I'm your host, Shawn "Crazy Coder"  Freeman.

 

So what's this all about? Well my specific task is to improve the performance of the history output operation of the GEOS5 model when generating netCDF/HDF files (flat files/GrADS is another story and will be told at another time).

 

The performance issue? The output code is serial. This means the entire model has to wait for output generation to gather, regrid/interpolate (if necessary), and write out the data to a file at the whatever interval the user requested. That adds up to a lot of idle time  that the processors could be using for more useful things (such as painting the computer cabinents or mowing the fiber connections).

 

The obvious solution is to parallelize the output generation and make it asynchronous to the model itself. Unfortunately it appears that the way the current MAPL and ESMF frameworks work makes this difficult without a fair amount of work. At least that's what I've gathered so far (I'm new to the model).The current task really doesn't have the time/resources for that kind of effort.

 

 

Can't do it. Time to go home....yeah right. I will not be so easily defeated (Read: I like my job and want to keep it).

 

 

 

 

So here's my current solution. Have each process write out it's own data to file. This acts to parallelize the output. Then an "aggregator" running asynchronously to the model grabs all the files and consolidates them into a single output file.The individual processor files are uniquely marked so they won't be overwritten, and are deleted after the aggregation. The aggregator would most likely be a python daemon running on a seperate node so as to not impact model operations.

 

 

 

 

This approach will effectively eliminate the serial bottleneck for output generation; i.e, the model can continue on computing without having to wait for the output generation to complete. That keeps those idle processors from getting spare tires.

 

 

 

 

There are drawbacks to this approach. The reliance on an external aggregation program/script to produce the final output file might not be pallatable to some. It presents an added complexity and another possible point of failure. There also may be disk bandwidth contention from writing numerous smaller files in parallel (I'll need to test this to see if it is a concern, but it wouldn't affect the model per se). But given the constraints, this appears to be the best short term approach.

 

 

 

 

 

 

 

 

 

Tune in next $TIME_PERIOD for the next installment of the MERRA Output Optimization blog!

 

 

 

 

 

 

 

 

 

~Shawn

 

 

 

 

 

 

Comments (0)

Bookmarked By (0)

Recent Comments

No recent comments.

USAGov logo NASA Logo - nasa.gov