Plan for handling data for early in-air cal activities in Dec 2006. Data will come in several parts: 1. Image data in lev0 fits files generated by the EGSE system. These fits files should contain the image status packets as well as the images and a few image keywords. 2. HK data including the image status packets and other data will be available in APID dayfiles as well as some of it in the HS bus data. 3. Ancillary data will be in text files with event time part of the file names. multiple files per day. 4. Some of the early data will still be in RAL GSE camera fits files with some ancillary data incorportated into the fits files. Requirements: 1. The data needs to be accessible from IDL and other tools as fits files with image status and other HK data and ancillary data also available in a convenient form. 2. All the data must be archived. 3. There is some benefit to working toward future lev0 type data, but this should not get in the way. Top level plan: #? 1. The raw data files will be initially archived in DSDS. at least as tlm files and where available as lev0 fits files as well. (Add: Hao) 1. The raw data files, tlm and qac files, are archived in DSDS, and placed on Ampex tapes, but not DRMS. #? 2. The raw data will be copied to drms. (Add: Hao) 2. The lev0 data generated from the tlm file by the EGSE are copied to drms as ds=hmi_ground.lev0[fsn]. They are not archived onto Ampex tapes and stay on-line in DSDS for 10 days only. 3. The ancillary data will be archived in drms. 4. The HK dayfiles will be archived in drms. 5. Scripts will be made to do the ingests and to make it accessible for analysis in a form convenient for IDL. Ingest details: DSDS Image Data: RAL data: This data has been made since Dec 2006 by the RAL GSE system. It consists of fits files, one file per DSDS dataset and as datasets with day or day-part sets of fits files depending on which processing has been done. The day set data is archived mostly by Jesper as e.g.: prog:hmi_ground,level:raw,series:hmi061201a[0] The series name contains the UT day as yymmdd with a letter suffix a,b,c,... when there is more than one dataset per day. Each dataset contains a collection of fits images obtained on that day or part of day. The series number is always 0. The dataset per file data from the early runs is stored (by Karen) as: prog:hmi_ground,level:lev0,series:ral_hmi1[fsn] where fsn is the filtergram serial number. This data was transported from LMSAL on a USB disk. CIF/DCHRI data: This data is flight like coming as tlm files from the SSIM. The processing in the EGSE is described in: http://hmi.stanford.edu/development/SU_Development_Plan/EGSE_User_Guide.html The raw tlm data is stored in e.g.: prog:hmi_ground,level:raw,series:egse_hmifsfm[tsec] with series number tsec equal to seconds since 1 Jan 2004. This is 851990432 seconds from the DRMS epoch of 1977. The level0 fits files generated in the data EGSE are stored in datasets named: prog:hmi_ground,level:lev0,series:egse_hmifsfm[fsn] with fsn the filtergram serial number, which is supposed to be unique. Repeated fsn give increasing level numbers. (Add: Hao) The are kept on-line in DSDS for 10 days and are not archived onto Ampex tape. The datasets are copied into DRMS and archived into SUMS. The series name may also be aiafsfm for AIA data. This data is transported from LMSAL on SAIT tapes. A sample FITS header from this data - without the image status packet - looks like: SIMPLE = T BITPIX = 16 / Number of bits per data pixel NAXIS = 2 / Number of data axes NAXIS1 = 4096 NAXIS2 = 4096 BLANK = -32768 DATE = '2006-11-10 18:53:18' SHS = 1541875937 SHSS = 2200305664 FSN = 35522 FID = 7169 CROPID = 0 ROMODE = 3 HEADRERR= 1 OVERFLOW= 0 TAPCODE = 0 BITSELID= 0 LUTID = 0 COMPID = 128 OFFSET = 0 END The data with image status packets will have additional keywords. The keywords above come from the image and image unpack routines. DRMS tlm data: No suggestion at this time. DRMS image data copied from egse lev0 This will be copied as fits segments with fully set keywords in both the fits file in the segment named "file" and in drms. This way the keywords will be available to idl users. The ancillary keywords would still need to be pulled together. copy lev0 data as fits segments to a series with all keywords: (Add: Hao) hmi_ground.lev0[fsn] where fsn is the filtergram serial number. The import program, hmi_import_egse_lev0, is called upon by the script q_import_lev0_CIF_from_file.csh . Log files of the process can be found at: /home/jsoc/hmi/ground/log.fsn To make this process more automatic the ingest script is called upon by a second script, q_import_lev0_CIF_from_DSDS.csh , that retrieves the requested fsn range from DSDS and runs the previous import script on the path returned by DSDS. To make better use of the clusters a third script, q_multi.csh, was created to split the fsn ranges into groupings of 100 and submit it into the cluster queue in parallel. #?? hmi_ground.lev0_fsfm_0445_0001[FSN][T_OBS] # # where FSN is the filtergram serial number and T_OBS is the shutter #start time. # If the image status packet (and shutter time) are not present use #SHS for T_OBS. # The "_0445_0009" is the version number is the Stanford version of #the image status packet # APID 0445. # Other keywords will be those above from SHS through OFFSET plus #those # in the lev0 data that come from the image status packet. # The .jsd file is located in ~jsoc/hmi/ground/jsd DRMS configuration data: The HMI and test setup configuration data is written to text files by the STOL procedures. (Add: Hao) These files are copied from hmifm1a:/disk2/files/usr to /home/jsoc/hmi/ground/files.usr and moved in /home/jsoc/hmi/ground/files.usr_done once ingested. # These files are copied from the egse machines to a temp directory where they are ingested into the series: hmi_ground.test_config_files[type][date] where type is one of "pcu" or "fts_info" as a generic segment named "file". These files are ingested by the script: ./scripts/ingest_test_config_files.csh Once some image files are in the lev0_fsfm series the matching config data can be extracted from the config files and placed into: hmi_ground.lev0_config[FSN][T_OBS] by the script: ./scripts/set_test_config_by_time.csh