Scripts to process HMI EGSE generated FITS files into the JSOC.

Ancillary keywords are transferred in text files and placed in ../files.usr
these files get ingested into a series by running the script:
  ingest_test_config_files.csh
with no arguments.

This script copies the files into SUMS and makes the filename time a prime key.
The series is: hmi_ground.test_config_files

Once this is done, one of the scripts "q_import..." is run.
There are presently two scripts:

 1.  q_import_lev0_CIF_from_dir.csh
     This script is given directory and first and last "dir numbers" where the dir number is the
     numerical part of a diectory such as: /tmp20/production/tape_070131/file123 and the directory
     is the base directory.  E.g.:
        q_import_lev0_from_dir.csh /tmp20/production/tape_070131 0 123
     would take 124 files and ingest them.

 2.  q_import_lev0_CIF_from_DSDS.csh
     This script is given first and last FSN where FSN is the filtergram serial number.
     It fetches FITS files from the DSDS datasets: prog:hmi_ground,level:lev0,series:egse_hmifsfm[$FSN]
     and ingests them.

Both of the above scripts call a subscript called q_import_lev0_CIF_from_file.csh
which places commands to do the import in a script and submits the script to the
cluster queue.  This script takes two arguments.  First a pathname of a flag file
to track completion of the cluster job, and the second is the pathname of the
FITS file to be ingested.

The script is placed in ../qscripts/FSN_$FSN

The script establishes its new environment to build paths for JSOC programs
then invokes the import_egse_lev0 program.  When the import program has completed
the script removes the flag file.  the script write a log file named ../logs/log.$FSN

The import_egse_lev0 program source is in ~/cvs/jsoc/src/pipeline/hmi_ground/

This program creates a new record in hmi_ground.lev0 and copies the fits file
into SUMS in that record.  It sets keywords in DRMS and changes some and adds
others to the ingested fits file.  The full list is in ../jsd/hmi_ground.lev0.jsd
The first "word" of the description field is the keyword name in the fits file.

When the import_egse_lev0 program is running it calls (via a "system" call) the
script:  set_config_by_time.csh
This script is passed the FSN and T_OBS values.  It searches the config text files for
PCU and FTS files that are times just before the image T_OBS.  It first finds
the nearest predecessor PCU file and reads its keywords.  Then if there is
a closer predecessor FTS file it reads its keywords.  As an intermediate
step it makes a new record in hmi_ground.lev0_config keyed by FSN with the
found keywords.  [note: this is a place where duplicate FSNs will cause
problems.  It should have also keyed on T_OBS].

The main driving scripts numbers 1 and 2 above can be run stand alone or
inside a single drms session.  It is MUCH faster to run it inside a single
session instead of the default which is to make a new session for each image.

So, to run them in a single session simple add the prefix "drms_run".
e.g.:

  drms_run q_import_lev0_CIF_from_DSDS.csh 51000 51004

to for example import FSN numbers 51000 through 51004

The driving scripts do not terminate until all of the jobs submitted to the cluster
have finished and removed their flag files.
When all the processing is done the driving scripts examine each of the log files made
by the q_import_lev0_CIF_from_file.csh script for successful completion.  It prints
either the result of:  show_keys ds=hmi_ground.lev0[$FSN] key=FSN seg=file -p
or a message of failure and the path of the log file.