Son-of-Spectro (SOS) Documentation

This document desribes the real-time reductions of spectroscopic data taken with the SDSS 2.5-m. These reductions are robotically posted to the SOS web page and archived at the Princeton mailing list.

In this document:

Related documents:

Overview of Son-of-Spectro

Son-of-spectro is a set of daemon jobs that runs from the observer account on sos3.apo.nmsu.edu. The actual processes are named sos_runnerd.py and there will be four instances running, one per camera. Although sos_runnerd.py can be started manually, users will seldom if ever start the process using this command. Instead, there are two user commands that know about the setup at APO. They are sos_apocontrol and sos_apostart. sos_apocontrol is the highest level command and should be used most often. sos_apocontrol allows for the starting, stopping and monitoring of the daemon processes using the standard APO options. In order to start the processes, sos_apocontrol calls sos_apostart. sos_apostart can be called manually and allows more flexibility. Each of those commands will start four instances of sos_runnerd, one per camera, with the correct parameters for APO.

At APO the log files are stored in ~observer/sos/sos-logs and there is one log file per camera for all commands and one log file per camera that only stores errors. It is a good idea to periodically check the error log. Each log file is rotated out daily and saved for four days.

The sos daemon processes can not be automatically started at boot because they need a password to be manually entered in order to access svn. Because of this, starting the sos daemons should be part of the SOS3 boot procedures document.

There is also a redo command, sos_aporedo that allows one nights of data to be manually rerun trough aporeduce. Again, sos_aporedo will start four processes, one per camera, to reprocess the images.

For more information be sure and read the sos daemon readme file. Also, each command has usage information via the '-?' option. Finally, for even more information each source file has a document header at the beginning of the file with implementation details.

Procedures for Son-of-Spectro

=== All commands are executed from observer@sos3 ===
=== All commands assume that idlspec2d is setup ===
=== Read the readme file for more information on the commands.

Confirm SOS Processes are Running

$ sos_apocontrol status

There should be four instances of sos_runnerd displayed, one per camera. The first line will be "Running", the last line will be "Done". Between those lines should be four instances of sos_runnerd. Each instance will be a long line separated by a blank line.

If you do not see see four processes running, stop the processes and then start the processes.

Confirm the SOS Process are Not Running

$ sos_apocontrol status

There should be zero instances of sos_runnerd displayed. The first line will be "Running", the last line will be "Done". There should be nothing between those two lines.

Start SOS Processes

$ sos_apocontrol start

When sos_apocontrol is run it will check to make sure that there is an ssh-agent that it can connect to for svn access. If there is not, it will ask for the password of the key it uses. The password is the same as is used for the sdss3 data access web site.

When the command finishes, confirm the sos processes are running.

Start SOS Processes After Observing Has Started For the Night

Use this command if the sos process was not started until after observing has started for the night. This procedure will process all images already present for the current MJD.

$ sos_apocontrol allow
$ sos_apostart -g -e

Then confirm the sos processes are running.

Stop SOS Processes

$ sos_apocontrol stopWait

If sos is processing an image, this command will allow the current idl process to finish before stopping sos. If this command does not return after some time (~5 min), then ^C to kill the process and use the following command:

$ sos_apocontrol stopKill

Then confirm the sos processes are not running.

Reprocess a Nights Worth of Spectro

$ sos_aporedo -w -g -m 00000 (Where 00000 is the MJD to reprocess)

The output files are stored in ~/sos/redo. cat ~/sos/redo/redo-b1.out for the output of the reprocessing of the b1 camera.

Confirm that the ssh-agent Process Used by sos_runnerd is Running

$ ps ax | grep ssh-agent

This line should be present:

"ssh-agent -a /home/observer/sos/control/agent.socket"

If it is not, then stop the sos processes and restart them.

Backing out to an older version

You can run a different version of Son-of-Spectro by setting up the relevant version of idlspec2d before you launch it with "sos_apocontrol". This should be done from the "observer" account on "sos3.apo.nmsu.edu".

For example, to stop the current version, set up v5_4_15, then re-start:

$ sos_apocontrol stopWait
$ setup idlspec2d v5_4_15
$ sos_apocontrol allow
$ sos_apostart -g -e
If "sos_apocontrol stopWait" hangs, then hit control-C and use "sos_apocontrol stopKill".

Installing a new version

SOS is part of the idlspec2d package and uses the normal sdss3 install procedures including being eups compliant and installed via sdss3install in the sdss3tools package. A good place for more information is BOSS Software Install Instructions.

At APO, install both on the SOS machine (currently sos3.apo), and on sdsshost.apo. This is because the documentation on the sdsshost.apo procedures web page is linked to the installed version on that machine.

Now you are ready to launch this new version of the code.

How Son-of-Spectro Works

Son-of-Spectro is a daemon job that runs from the "observer" account on "sos3.apo.nmsu.edu". This daemon job monitors files written to /data/spectro/$MJD and then launches a package of shell scripts and IDL commands to reduce the data one frame at a time. It is part of the idlspec2d data product.

There are actually four daemon processes that run, one for each camera. We did this so that both processors on the computer can be utilized. All 4 cameras should be reduced for an exposure within a few minutes.

The output directors are:

	/data/boss/sos/combined   - Copy of aporeduce output for the web site
	/data/boss/sos/$MJD       - Output of aporeduce for that night
	$SPECLOG_DIR/$MJD         - plugmap files used for running aporeduce
	platedb database          - (s/n)^2 for each science frame reduced

The data reduction is not the same as the full-up spectroscopic pipeline. A number of shortcuts are taken to speed up the reductions and make them very robust. For example, boxcar extraction of the spectra is used instead of optimal extraction.

Files like the following can be found in the /data/boss/sos/$MJD directory:

  splog-b1-00006541.log              - Text history of exp #6541 reduction
  tset-51795-0389-00006541-b1.fits   - Fiber traces (from flat exposure #6541)
  wset-51795-0389-00006542-r1.fits   - Wavelength solution (from arc exp #6542)
  fflat-51795-0389-00006542-b1.fits  - Flat-field vectors (w/ arc exp #6542)
  sci-0389-b2-00006546.fits          - Science spectra for plate 389 (exp #6546)
  logfile-51795.fits                 - Summary info for night in FITS format
  logfile-51795.html                 - Summary info for night in HTML format
  snplot-51795-0389.ps               - Signal-to-noise plot for nplate 389
If a particular frame does not get reduced and appear on the Son-of-Spectro web site, then you can look at its log file (splog*.log). If the file does not end with the line "Finished at", then that exposure must have crashed.

The HTML files and S/N plots for the web page are copied into the directory:

  /data/boss/sos/combined
The most recent HTML file has a second copy as the file "logfile-current.html". This is the file pointed to by the SOS web page.

FITS header cards (and how to change them)

Required FITS header cards

The following FITSheader cards are required to be correct for either Son-of-Spectro or Spectro-2D to reduce the data properly. The AIRMASS is not read from the header, but computed from RADEG,DECDEG and the TAI-BEG,TAI-END keywords.

EXPOSURE= Exposure number.  We always override the value of this keyword
          with the exposure number in the file name.
CAMERAS = Camera name, e.g. "b1".  We always override the value of this keyword
          with the exposure number in the file name.
FLAVOR  = The flavor of the observation, which can be "bias", "dark",
          "arc", "flat", and "science" or "target".  Files with other
          flavors, such as "unknown", are not reduced.
MJD     = MJD of observation, which must agree with the directory in which
          the file resides, "/data/spectro/$MJD".
PLATEID = The plate ID, which must match that in the NAME header card.
NAME    = Plug name, e.g. "0328-52277-01" for plate 328, plugged on 52277
          with a plugging ID of 01.  We assume that there is a plug-map
          file at APO with the name "/astrolog/$MJD/plPlugMapM-$NAME.par".
EXPTIME = The exposure time, which is used along with TAI-BEG to compute
          the airmass.  It is also used to determine whether the sky
          brightness or scattered light is larger than tolerances.
TAI-BEG = The TAI time for the beginning of the exposure, which is used
          to compute the airmass and sky-brightness gradients for sky-
          subtraction.  If this keyword is missing, we guess it from
          EXPTIME and TAI.
TAI-END = Used with TAI-BEG.  If this keyword is missing, we guess it from
          EXPTIME and TAI.
TAI     = This keyword is used if TAI-BEG,TAI-END are missing.
FFS     = Flat-field screen positions, which should be
          "0 0 0 0 0 0 0 0" for science exposures (all petals open)
          "1 1 1 1 1 1 1 1" for flat or arc exposures (all petals closed)
FF      = Flat-field (quartz) lamp status, which should be
          "0 0 0 0" for science or arc exposures (all off)
          "1 1 1 1" for flat exposures (all on)
NE      = Neon lamp status, which should be
          "0 0 0 0" for science or flat exposures (all off)
          "1 1 1 1" for arc exposures (all on)
HGCD    = HgCd lamp status, which should be
          "0 0 0 0" for science or flat exposures (all off)
          "1 1 1 1" for arc exposures (all on)
OBSCOMM = This keyword is used to identify the dithered flat sequence
          taken in the Monthly Checkout with the "specFlats" script in SOP.
          The flats in this sequence must have OBSCOMM="{dithered flats-flat}".
          The arcs in this sequence must have OBSCOMM="{dithered flats-arc}".
          This keyword also identifies the Hartmann focus exposures, with
          the entries OBSCOMM="{focus, hartmann l}" or "{focus, hartmann r}".
          Science exposures all have OBSCOMM="science ".
QUALITY = The observer-input quality for an exposure: "excellent", "test",
          or "bad".  Exposures are declared "test" or "bad" using the
          APOFIX procedure.

Informational FITS header cards

The following FITS header cards are informational. We would very much appreciate the observers to correct errors in these keywords so that our book-keeping of survey progress is done properly:

TILEID  = Tile ID for this plate from the plug-map file
CARTID  = Cartridge ID number (1 through 9)
RA      = Right ascension of telescope boresight (in degrees)
DEC     = Declination of telescope boresight (in degrees)
RADEG   = Right ascension of plate center for plug-map file (in degrees)
DECDEG  = Declination of plate center for plug-map file (in degrees)
AIRTEMP = Temperature (deg C)

How to change FITS header cards

Incorrect FITS header cards for the raw spectro sdR files can be corrected by adding entries in the sdHdrFix file "/astrolog/$MJD/sdHdrFix-$MJD.par". The interesting header cards from these files can be listed with the APOHEADER procedure run from the IDL prompt on sos.apo. The proc APOFIX can be used to add entries to the sdHdrFix file to denote edits.

This structure should have the following definition in the sdHdrFix file:

   typedef struct {
     char fileroot;    # Root of file name, without any ".fit" suffix
     char keyword;     # Keyword name
     char value;       # Keyword value (as a string)
   } ophdrfix;

Following are some examples of what should appear in the sdHdrFix files in order to correct faulty headers. (The typedef struct above must also be in the file.)

If you have edited the sdHdrFix file and need to re-reduce an exposure, you can do so with the sos_redo command. When Son-of-Spectro reads in the raw FITS files, it then patches the headers with the information in the ophdrfix entries of the sdHdrFix file. But note that the one piece of information that cannot be patched this way is MJD, since we need the MJD in order to find the correct sdHdrFix file!!

Son-of-Spectro Outputs

Tabulated values

Son-of-Spectro reduces four flavors of observations: bias/dark, flat, arc, & science/smear. Select information is tabulated for each of these types of observation. These values are tabulated in yellow if they are going out of spec, and in red if they are very much out of spec.

The values reported are:

The exact values of these yellow/red limits and further explanation can be found in the "idlspec2d" product in the file "examples/opLimits.par".

WARNING and ABORT messages

There are a number of WARNING and ABORT messages that can appear if the pipeline runs into trouble when processing a frame. These messages appear at the bottom of each table. Each one-line message begins with the relevant file name, WARNING or ABORT, then a brief plain-text message.

Note that a single problem may cascade into a large number of warning messages. For example, an out-of-focus spectrograph will first produce the "Median spatial widths" message, probably followed by warnings about bad sky-residuals.

The following messages may appear:

The following warning messages are all based upon the quality of the sky-subtraction. Typically, we are able to model the sky spectrum (from the 16 sky fibers on each spectrograph) with a relative chi^2 of around unity. At very bright sky lines, like 5577 Ang, the relative chi^2 may be as large at 25. If the relative chi^2 is large at other wavelengths, this means that there is excess light down the fibers that varies across the plate. This could be due to a light source near the telescope, or possibly a bright, non-uniform sky. Strong auroral activity is something that can do this, since the O I lines at 6300 and 6366 Ang are resolved on the sky.

Log files

If the reduction of an exposure is catastrophically bad, it may not appear at all in the Son-of-Spectro table. However, there should still be a log file for this exposure on the sos.apo computer:

    /data/spectro/spectrologs/$MJD/splog-$CAMERA-$EXPOSURE.log
    
Reading this file should tell you what failed. The first and last lines of these files should contain "Started at" and "Finished at" followed by timestamps.

S/N figures

A median signal-to-noise is computed for each object in the wavelength ranges [4000,5500] Angstroms (synthetic g-band) and [6910,8500] Angstroms (synthetic i-band). We plot these S/N values versus the PHOTO fiber magnitudes, which were measured in approximately a 3-arcsec diameter aperature (the same size as our fibers). If everything is working perfectly, then our S/N values should correlate very well with these PHOTO magnitudes.

We determine whether a plate is "done" based upon the signal-to-noise of the fainter objects on the plate. We do this by fitting a line to the (S/N)-vs.-magnitude plot in a specified wavelength range, then evaluating this fit at g=20.2 mag (blue CCDs), and i=19.9 mag (red CCDs). Approximately 10% of the main galaxy sample, 60% of the BRG's and 15% of the QSO's are fainter than g=20.2. Very few objects (1, 1, and 5%) are fainter than i=19.9.

When the sky level is higher, we gain S/N more slowly at the fainter magnitudes where we are sky-limited rather than photon-limited. Without moon, we have typically found:

  log(S/N_g) = (zeropoint) - 0.31 * g
  log(S/N_i) = (zeropoint) - 0.31 * i
With partial moon, the slope steepens to -0.34 or worse.

The fitting regions are denoted on the plot with vertical dotted lines. Arrows point to the evaluation of the fit on each of the 4 cameras, with the top panels corresponding to the blue CCD's (synthetic g-band) and the bottom panels corresponding to the red (synthetic r-band). The thick blue line on the figure is only a meaningless, fixed reference line.

The right-hand figures plot the residuals of each object from the fit. Symbol sizes on those right-hand plots indicate the magnitude of the deviation from the fit line. Symbol color is the same on the left as on the right, so green objects have more flux and red ones less. If the scale of the telescope is wrong, then you will see a radial drop-off in flux (red points on the edge of the plate). If you are observing too far over in air mass, then typically you correct to first order with a scale change, but a quadropole is left in these residuals. If one spectrograph has problems, then this will show up as red points in half of one of these figures.

Note that the (S/N)^2 totals listed in the table and the figure might not exactly agree. This is because the fitting to (S/N)-vs.-magnitude is done on individual frames for the table, but on the summed S/N over all frames for the figure. The tabulated values are the ones we use to declare a plate done.

Trouble-shooting Son-of-Spectro

The Son-of-Spectro (SOS) reduction robot has proven to be quite robust. However, if it appears to not be working one could check the following:

  1. Has a "loadCartridge" command been issued?
    If this has not been done from SOP, then there is no PLATE entry in the images. This prevents SOS from reducing anything but bias and dark frames.
  2. Is the cron job running?
    Log into "observer@sos.apo.nmsu.edu", and issue a "sos_status" command. This should report that SOS is running. If not, then restart the cron job.
  3. Is the data being copied over?
    The raw data files should be copied from "sdsshost" to "sos.apo" in the /data/spectro/$MJD directory. If it is not, then the disk may be full (see SOS disk is full warning message). Or, the rsync command may not be working. This could be due to a rogue "aporsync" process sitting around. The "sos_status" command will report any such processes. If you suspect this as the problem, just kill that process. It's a harmless thing to do, since SOS will just start it up again within a minute.
  4. Are there old "lock" files sitting around?
    SOS generates "lock" files to prevent multiple processes from changing these files simultaneously. The "sos_status" command will report all "lock" files and their last modification times. If any have been sitting around for several minutes or more, then something is wrong. You should delete the file ending with ".lock". However, this could result in a corrupted "logfile*.fits".
  5. Is the IDL license manager running?
    You can issue the following command to check that the IDL license manager is indeed running:
        % $IDL_DIR/bin/lmstat -a
        
    This should display the number of licenses available, and the current number in use. The server running on galileo.apo has 110 licenses, each IDL session uses 6, so there can be at most 18 IDL sessions running concurrently. Note that Son-of-Spectro, the fiber mapper, and HoggPT all use IDL licenses.
  6. Is anything amiss with running the scripts?
    Perhaps there is a communication problem between sdsshost.apo and sos.apo, or a problem running the code on sos.apo. There is logging information written once per minute into the file
        /data/spectro/spectrologs/sos.log
        
    which may indicate such problems.
  7. Is the logfile corrupted?
    This is probably the worst thing that could happen. If the file
        /data/spectro/spectrologs/$MJD/logfile-$MJD.fits
        
    gets corrupted, then SOS will repeatedly crash. One could just delete this file, which would basically wipe out all the previous reductions for that night from the web page. Alternatively, one could delete all the raw and reduced data for that night from "sos.apo" (but not from sdsshost.apo!). Best to stop the SOS robot before doing this:
        % sos_stop
        % sos_status   <-- Wait for any running processes to complete.
        % rm -rf /data/spectro/spectrologs/$MJD
        % rm -rf /data/spectro/$MJD
        % sos_start
        
    That might sound extreme, but SOS will simply start re-copying everything from "sdsshost" and re-build the reductions for the night. Obviously, this could take a couple of hours if its already late into the night.

Re-Reducing (Failed) Exposures

There is a mechanism to re-reduce exposures with "sos_redo". You may need to do this if a reduction failed, or the pre-calibs failed and SOS could not reduce the science exposures, or if you needed to edit the sdR header information by putting entries in the sdHdrFix file (see APOFIX).

It is fastest to only re-reduce a specific exposure number. To force a re-reduction of exposure number 12345 on all 4 cameras (whether it has already been reduced or not) use the EXPNUM option:

  % ssh observer@sos.apo.nmsu.edu
  % sos_redo expnum=12345
If you want to find and reduce only failed reductions for a plate use the PLATE option. This may take several more minutes because all of the FITS headers for the night are parsed. To re-reduce only missing exposures for plate 666:
  % ssh observer@sos.apo.nmsu.edu
  % sos_redo plate=666
The last exposure taken is not re-reduced unless explicitly told to with the EXPNUM option. Currently, either EXPNUM or PLATE must be specified.

Note that science exposures for a plate cannot be reduced until both flat and arc exposures have been successfully reduced for that same plate.


This is from version $Name$ of procedures.

Maintained by David Schlegel and Scott Burles