Plan for handling HMI ground data for in-air cal activities. ## Linked to README ## Data will come in several parts: 1. Image data in lev0 fits files generated by the EGSE system. These fits files should contain the image status packets as keywords as well as the images and a few image keywords. 2. HK data including the image status packets and other data will be available in APID dayfiles as well as some of it in the HS bus data. 3. Ancillary data will be in text files with event time part of the file names. multiple files per day. 4. Some of the early data will still be in RAL GSE camera fits files with some ancillary data incorportated into the fits files. 5. Mechanism commanded position tables are to be imported and updated when changed. Requirements: 1. The data needs to be accessible from IDL and other tools as fits files with image status and other HK data and ancillary data also available in a convenient form. 2. All the data must be archived. 3. There is some benefit to working toward future lev0 type data, but this should not get in the way. Top level plan: 1. The raw tlm data files will be archived in both DSDS and SUMS. The lev0 fits files will be ingested into DSDS with a 10 day retention time but not archived. During this time the lev0 files will be read and ingested into DRMS with some modifications. 3. The ancillary data will be archived in drms. 4. The HK dayfiles will be archived in drms. 5. Scripts will be made to do the ingests and to make it accessible for analysis in a form convenient for IDL. Ingest details: DSDS Image Data: RAL data: This data has been made since Dec 2006 by the RAL GSE system. It consists of fits files, one file per DSDS dataset and as datasets with day or day-part sets of fits files depending on which processing has been done. The day set data is archived mostly by Jesper as e.g.: prog:hmi_ground,level:raw,series:hmi061201a[0] The series name contains the UT day as yymmdd with a letter suffix a,b,c,... when there is more than one dataset per day. Each dataset contains a collection of fits images obtained on that day or part of day. The series number is always 0. The dataset per file data from the early runs is stored (by Karen) as: prog:hmi_ground,level:lev0,series:ral_hmi1[fsn] where fsn is the filtergram serial number. This data was transported from LMSAL on a USB disk. CIF/DCHRI data: This data is flight like coming as tlm files from the SSIM. The processing in the EGSE is described in: http://hmi.stanford.edu/development/SU_Development_Plan/EGSE_User_Guide.html The raw tlm data is stored in e.g.: prog:hmi_ground,level:raw,series:egse_hmifsfm[tsec] with series number tsec equal to seconds since about 1 Jan 2004. This is 851990432 seconds from the DRMS epoch of 1977. (verify this constant from Jim's code) The level0 fits files generated in the data EGSE are stored in datasets named: prog:hmi_ground,level:lev0,series:egse_hmifsfm[fsn] with fsn the filtergram serial number, which is supposed to be unique. Repeated fsn give increasing level numbers. These are kept on-line in DSDS for 10 days and are not archived onto Ampex tape. The datasets are copied into DRMS and archived into SUMS. ZZZZZZZZZZZZZZZZZ The series name may also be aiafsfm for AIA data. This data is transported from LMSAL on SAIT tapes or USB disk. A sample FITS header from this data - without the image status packet - looks like: SIMPLE = T BITPIX = 16 / Number of bits per data pixel NAXIS = 2 / Number of data axes NAXIS1 = 4096 NAXIS2 = 4096 BLANK = -32768 DATE = '2006-11-10 18:53:18' SHS = 1541875937 SHSS = 2200305664 FSN = 35522 FID = 7169 CROPID = 0 ROMODE = 3 HEADRERR= 1 OVERFLOW= 0 TAPCODE = 0 BITSELID= 0 LUTID = 0 COMPID = 128 OFFSET = 0 END The data with image status packets will have additional keywords. The keywords above come from the image and image unpack routines. DRMS tlm data: The data is stored in the series: hmi_ground.raw_egse_hmifsfm DRMS image data copied from egse lev0 This will be copied as fits segments with fully set keywords in both the fits file in the segment named "file" and in drms. This way the keywords will be available to idl users. The ancillary keywords would still need to be pulled together. The series name is: hmi_ground.lev0[fsn] where fsn is the filtergram serial number. The import program, hmi_import_egse_lev0, is called upon by the script: q_import_lev0_CIF_from_file.csh . Log files of the process can be found at: /home/jsoc/hmi/ground/logs/log.fsn To make this process more automatic the ingest script is called upon by a second script, q_import_lev0_CIF_from_DSDS.csh , that retrieves the requested fsn range from DSDS and runs the previous import script on the path returned by DSDS. To make better use of the clusters a third script, q_multi.csh, was created to split the fsn ranges into groupings of 100 and submit it into the cluster queue in parallel. #?? hmi_ground.lev0_fsfm_0445_0001[FSN][T_OBS] # # where FSN is the filtergram serial number and T_OBS is the shutter #start time. # If the image status packet (and shutter time) are not present use #SHS for T_OBS. # The "_0445_0009" is the version number is the Stanford version of #the image status packet # APID 0445. # Other keywords will be those above from SHS through OFFSET plus #those # in the lev0 data that come from the image status packet. # The .jsd file is located in ~jsoc/hmi/ground/jsd DRMS configuration data: The HMI and test setup configuration data is written to text files by the STOL procedures. These files are copied from hmifm1a:/disk2/files/usr to /home/jsoc/hmi/ground/files.usr and moved in /home/jsoc/hmi/ground/files.usr_done once ingested. They are ingested into the series: hmi_ground.test_config_files[type][date] where type is one of "pcu" or "fts_info" as a generic segment named "file". These files are ingested by the script: ./scripts/ingest_test_config_files.csh Once some image files are in the lev0_fsfm series the matching config data can be extracted from the config files and placed into: hmi_ground.lev0_config[FSN][T_OBS] by the script: ./scripts/set_test_config_by_time.csh This script is called from inside the hmi_import_lev0 program vi a "system" call.