Skip navigation
NASA Logo, National Aeronautics and Space Administration
2397 Views 19 Replies Latest reply: Oct 8, 2008 1:07 PM by glpotter RSS
Currently Being Moderated

Sep 30, 2008 1:53 PM

workflow testing

Jerry Potter wrote:

 

 

1) In my /archive directory, under my experiment name I don't find any .ctl files or .hdf for that matter. I thought the model produced Grads control files but I don't know where they are located.

2) the Workflow visualization tool still doesn't work so I was going to try to bring what I though was an output file. The file I thought might contain something I want to look at (like "moist_import_restart.e19790103_21z") but I don't see how to read it. I assume this is a flat binary file but I can't find a .ctl file to go with it.

    • Currently Being Moderated
      Oct 2, 2008 2:29 PM (in response to Carlos Cruz)
      Re: workflow testing

       

      Carlos,

       

       

      I don't find anything in the documentation about the directory /gsfsm/dnb20. Where is that set? Also, I see the data now but now I would like to access it  on my workstation. When I sftp in to dirac and look at the /glpotter/WORKFLOW-discover/<experiment name> the directory contains different information than the same directory on discover. Obviously I don't understand where my data is kept.

       

       

      Thanks,

       

       

      Jerry

       

       

        • Currently Being Moderated
          Oct 2, 2008 3:10 PM (in response to Carlos Cruz)
          Re: workflow testing

           

          Carlo,

           

           

          Under the dirac directory, /portal/mapme, none of my experiements show up. The last one I submitted was geosgcs_0771. None of my earlier experiemnts are there either. I must have something set incorrectly in the Workflow tool. By the way, it is quite difficult to use the tool remotely, There is usually a 20 second lag from the time I click on an option and it responds. I'm sure I am the exception but anybody using the tool remotely might have similar delays.

           

           

           

           

           

          Jerry

           

           

          • Currently Being Moderated
            Oct 2, 2008 3:57 PM (in response to glpotter)
            Re: workflow testing

             

            Hi Jerry,

             

             

            There might be a permissions problem since you are not a member of the group "k3002".  We've opened the permissions on that folder.  Can you try touching a file and then deleting it from there?  If that works, re-run your workflow and see if the files appear then.

             

             

            Regards,

            Rob

             

             

            • Currently Being Moderated
              Oct 2, 2008 4:46 PM (in response to Robert Burns, III)
              Re: workflow testing

               

              Rob,

               

               

              No luck. I was able to created a test directory and put a file in it on diract /portal/mapme but, when I just ran a new Workflow job, geosgcs_805 it is not in the directory.

               

               

               

               

               

              Jerry

               

               

              • Currently Being Moderated
                Oct 3, 2008 10:24 AM (in response to glpotter)
                Re: workflow testing

                Hi Jerry,

                 

                I may have made some changes yesterday after your last workflow ran.  To help me diagnose your problem, please run a new workflow at your convenience.  Let me know when you've started the workflow.

                 

                Thanks,

                Megan

                • Currently Being Moderated
                  Oct 3, 2008 11:25 AM (in response to Megan Damon)
                  Re: workflow testing

                   

                  Megan,

                   

                   

                  The workflow started and finished. There is still no entry on dirac. The job name was geosgcs_0811.

                   

                   

                  Jerry

                   

                   

                  • Currently Being Moderated
                    Oct 4, 2008 3:12 PM (in response to glpotter)
                    Re: workflow testing

                    Jerry,

                     

                    If the pull_back_data.j script is still in your geosgcs_0811 directory on discover (/discover/nobackup/glpotter/WORKFLOW-discover), can you run it manually and then send me the output? 

                     

                    Also can you open up the /discover/nobackup/glpotter/WORKFLOW-discover for reading by the world?

                     

                    Thanks for re-running,

                    ~Megan

                    • Currently Being Moderated
                      Oct 4, 2008 9:25 PM (in response to Megan Damon)
                      Re: workflow testing

                       

                      Megan

                       

                       

                      Here is the output:

                       

                       

                      geosgcs_0811/agcm> ./pull_back_data.j

                       

                      ssh glpotter@dirac mkdir -p /portal/map06/mapme/geosgcs_0811

                      ************************************************************

                      This U.S. Government resource is for authorized use only.

                      If not authorized to access this resource, disconnect now.

                      Unauthorized use of, or access to, this resource may

                      subject you to disciplinary action or criminal prosecution.

                      By accessing and using this resource, you are consenting to

                      monitoring, keystroke recording, or auditing.

                      ************************************************************

                       

                      Warning: untrusted X11 forwarding setup failed: xauth key data not generated

                      Warning: No xauth data; using fake authentication data for X11 forwarding.

                      /usr/bin/X11/xauth:  error in locking authority file /home/glpotter/.Xauthority

                      UX:mkdir: ERROR: Cannot create directory "/portal/map06/mapme/geosgcs_0811": Permission denied

                      UX:mkdir: ERROR: Cannot create directory "/portal/map06/mapme/geosgcs_0811": No such file or directory

                       

                      scp -r holding dirac:/portal/map06/mapme/geosgcs_0811

                      ************************************************************

                      This U.S. Government resource is for authorized use only.

                      If not authorized to access this resource, disconnect now.

                      Unauthorized use of, or access to, this resource may

                      subject you to disciplinary action or criminal prosecution.

                      By accessing and using this resource, you are consenting to

                      monitoring, keystroke recording, or auditing.

                      ************************************************************

                       

                      scp: /portal/map06/mapme/geosgcs_0811: No such file or directory

                       

                      ssh glpotter@dirac chmod -R 755 /portal/map06/mapme/geosgcs_0811

                      ************************************************************

                      This U.S. Government resource is for authorized use only.

                      If not authorized to access this resource, disconnect now.

                      Unauthorized use of, or access to, this resource may

                      subject you to disciplinary action or criminal prosecution.

                      By accessing and using this resource, you are consenting to

                      monitoring, keystroke recording, or auditing.

                      ************************************************************

                       

                      Warning: untrusted X11 forwarding setup failed: xauth key data not generated

                      Warning: No xauth data; using fake authentication data for X11 forwarding.

                      /usr/bin/X11/xauth:  error in locking authority file /home/glpotter/.Xauthority

                      UX:chmod: WARNING: Cannot access /portal/map06/mapme/geosgcs_0811: No such file or directory

                       

                      scp -r restarts /archive/u/glpotter/WORKFLOW-discover/geosgcs_0811

                       

                      scp scratch/cap_restart /archive/u/glpotter/WORKFLOW-discover/geosgcs_0811

                       

                      scp scratch/*.rc /archive/u/glpotter/WORKFLOW-discover/geosgcs_0811

                       

                      scp scratch/stdlog.out /archive/u/glpotter/WORKFLOW-discover/geosgcs_0811

                      cp: cannot stat `scratch/stdlog.out': No such file or directory

                       

                      ssh explore mkdir -p /explore/nobackup/glpotter/WORKFLOW-VIZ/geosgcs_0811

                      The authenticity of host 'explore (169.154.162.161)' can't be established.

                      RSA key fingerprint is e4:e5:db:3c:eb:34:c1:4a:dd:cf:e8:ea:0b:c1:5b:c6.

                      Are you sure you want to continue connecting (yes/no)? yes

                      Warning: Permanently added 'explore,169.154.162.161' (RSA) to the list of known hosts.

                       

                      This U.S. Government resource is for authorized use only.

                      If not authorized to access this resource, disconnect now.

                      Unauthorized use of, or access to, this resource may

                      subject you to disciplinary action or criminal prosecution.

                      By accessing and using this resource, you are consenting to

                      monitoring, keystroke recording, or auditing.

                       

                      Access denied for this host

                      Connection closed by 169.154.162.161

                       

                      scp holding/*.hdf explore:/explore/nobackup/glpotter/WORKFLOW-VIZ/geosgcs_0811

                       

                      This U.S. Government resource is for authorized use only.

                      If not authorized to access this resource, disconnect now.

                      Unauthorized use of, or access to, this resource may

                      subject you to disciplinary action or criminal prosecution.

                      By accessing and using this resource, you are consenting to

                      monitoring, keystroke recording, or auditing.

                       

                      Access denied for this host

                      Connection closed by 169.154.162.161

                      lost connection

                      geosgcs_0811/agcm>

                       

                       

                       

                      • Currently Being Moderated
                        Oct 4, 2008 9:51 PM (in response to glpotter)
                        Re: workflow testing

                        Jerry - thanks for doing this.  I see the problem.  The pull_back_data task is trying to create the directory:

                         

                        /portal/map06/mapme/geosgcs_0811

                         

                        But the path: /portal/map06/mapme, no longer exists.  You may be using a configuration that is based on an older .ned file.

                         

                        To fix this, you have a couple options.  You can change the VIS_DIR variable under Workflow setup to:

                         

                        /portal/mapme/$EXP_ID_$NED_UNIQUE_ID

                         

                        or, you can use one of the standard .ned config files for GEOS-5:

                         

                        /home/workflow/sharedExperiments/geos5gcm_merra.ned

                        /home/workflow/sharedExperiments/geos5gcm.ned

                         

                        If you have any problems, feel free to send another message.

                         

                        Cheers.

                        • Currently Being Moderated
                          Oct 5, 2008 11:48 AM (in response to Megan Damon)
                          Re: workflow testing

                           

                          Megan,

                           

                           

                          I tried both suggestions. The first to use one of the standard configurations. The job failed almost immediately. The second (geosgcs_0821) was to change the path to

                           

                           

                          /portal/mapme/$EXP_ID_$NED_UNIQUE_ID. It failed almost immediately too.

                           

                           

                           

                           

                           

                          Jerry

                           

                           

                           

                           

                           

                          • Currently Being Moderated
                            Oct 6, 2008 10:20 AM (in response to glpotter)
                            Re: workflow testing

                            Hi Jerry,

                             

                            I think I see the problem for the workflow where you used the standard config.  The group_list in that workflow is set to: k0620, but according to the groups you are in, that group_list should be set to g0620.

                             

                            I do not know what happened with your 2nd workflow, as there are no log files at all for that workflow.  If you have time, can you also re-run this workflow too?

                             

                            Thanks,

                            Megan

                            • Currently Being Moderated
                              Oct 6, 2008 11:18 AM (in response to Megan Damon)
                              Re: workflow testing

                               

                              Megan,

                               

                               

                              It works now. I am able to see the directory in dirac that contains my output files. My next problem may belong in another discussion. The output files are in HDF4 so I can't read them on my workstation. As I understand it, HDF5 is a new standard.

                               

                               

                              Thanks,

                               

                               

                              Jerry

                               

                               

                              • Currently Being Moderated
                                Oct 6, 2008 11:21 AM (in response to glpotter)
                                Re: workflow testing

                                Jerry,

                                 

                                I'm glad it is working for you now.

                                 

                                I think the question regarding HDF output files would be best in another thread.  I'm checking on that right now.

                                 

                                Thanks,

                                Megan

                                • Currently Being Moderated
                                  Oct 6, 2008 12:32 PM (in response to Megan Damon)
                                  Re: workflow testing

                                   

                                  Megan,

                                   

                                   

                                  I have another question about using the workflow tool. The default format for output is GRADS but I can't find any .ctl files in /portal/mapme. There is a /holding directory and it contains the .hdf files. There must be something else I haven't set in my .ned. Any suggestions?

                                   

                                   

                                   

                                   

                                   

                                  Thanks,

                                   

                                   

                                  Jerry

                                   

                                   

                                   

                                   

                                   

                                   

                                   

                                   

                                   

                                   

                                   

                                  • Currently Being Moderated
                                    Oct 8, 2008 1:03 PM (in response to glpotter)
                                    Re: workflow testing

                                     

                                    Everything is fine now. I have found the output files and moved them to my workstation. I understand better where stuff is kept now.

                                     

                                     

                                    After a Workflow run I can:

                                     

                                     

                                    sftp <username>@dirac.gsfc.nasa.gov

                                     

                                     

                                    then cd to:

                                     

                                     

                                    /portal/mapme/geosgcs_XXXX/holding

                                     

                                     

                                    and get *.ctl

                                     

                                     

                                    This seems to work pretty well and now I can do the analysis I need.

                                     

                                     

                                    Now to do some longer runs...

                                     

                                     

                                     

                                     

                                     

                                    Thanks for your help

                                     

                                     

                                    Jerry

                                     

                                     

                                     

                                     

                                     

                                  • Currently Being Moderated
                                    Oct 8, 2008 1:07 PM (in response to glpotter)
                                    Re: workflow testing

                                    I should mention that I incorrectly changed the file format to "GrADS" the instruction Carlos prepared say to set it the format "flat" to get GrADS formatted .ctl files.

          • Currently Being Moderated
            Oct 2, 2008 4:08 PM (in response to glpotter)
            Re: workflow testing

             

            I forgot to mention that you needn't use the "WMS Viewer" tool from within NED.  It is slow because it is calling up a web browser from the remote "mapme" system.  In the future this will be much faster because it will run locally.  For faster access you can also access the data from any ole browser by going to:

             

             

            http://www.map.nasa.gov/cgi-bin/browse.cgi?type=DF&dir=mapme/wfdemo&action=http: %2F%2Fwww.map.nasa.gov%2Fcgi%2Dbin%2Fmapme.cgi%3Ffile%3D

             

             

            Rob

             

             

More Like This

  • Retrieving data ...

Bookmarked By (0)

USAGov logo NASA Logo - nasa.gov