Scripts for ingesting and processing hmi ground test data

Raw Test configuration files

  Text files made at LMSAL are copied to a temp directory at Stanford.
  The file names will be the prefix from the sun_test files with a
  suffix the same as the image file time format, all .txt.  So example
  file names and keywords they contain will be:

    sun_test_stol_proc_060308_174110.txt
       HSTOLNAM=<string> STOL procedure name
       HSTRTIME=<string> unix time e.g. 09/08/2006 00:51_04.248
       OBSERVER=<string> e.g. Miles, Gradwohl, Bush
       HSTIMIN=<string> Stimulus input source, e.g. Dye Laser

    sun_test_pcu_060308_174110.txt
       HPCUPOLR=<float> Polarizer angle in degrees
       HPCUPOLM=<float> Polarizer position in mm
       HPCURETR=<float> Retarder angle in degrees
       HPCURETM=<float> Retarder position in mm

    sun_test_user_comment_060308_174110.txt
       HUSRCOM0=<string> user entered comments.
       HUSRCOM1=<string> user entered comments.
       HUSRCOM2=<string> user entered comments.
       HUSRCOM3=<string> user entered comments.
       HUSRCOM4=<string> user entered comments.
       HUSRCOM5=<string> user entered comments.
       HUSRCOM6=<string> user entered comments.
       HUSRCOM7=<string> user entered comments.
       HUSRCOM8=<string> user entered comments.
       HUSRCOM9=<string> user entered comments.

    sun_test_camera_config_060308_174110.txt
       HINTGRT=<int> Shutter open time in ms
       HTOREAL=<int> e.g. 1
       HDARKFRM=<int> e.g. 0

  These files will be written before the images are taken, perhaps with
  many images for the same configuration.  The times of the different
  files may be different.

  The file names are like:

      sun_test_stol_proc_060308_174110.txt 

  These files are copied into a local directory
  where they are picked up by:

      ingest_test_config_files <directory>

  and stored indexed by the date and type.

  So, for example, get links in dir XXX for all the config files for PCU settings for a given day:
    retrieve_dir ds=hmi_ground.test_config_files[yyyy.mm.dd/1d][pcu] out=XXX -l 

-------------------------------------------------------------------------------------

  The script:  set_test_config_by_time.csh <interval>
  will get all the image times in the given interval then
  find all the corresponding config files and add the
  config keywords to the series  hmi_ground.test_config_files

  Use show_keys to get the keywords.  For example to get the values for
  the range fsn=ABC to fsn=DEF do:

    show_keys -q -a "ds=hmi_ground.test_config_files[ABC-DEF]"