Son-of-Spectro (SOS) Documentation
This document desribes the real-time reductions of spectroscopic data
taken with the SDSS 2.5-m. These reductions are robotically posted
to the
SOS web page and archived at the
Princeton mailing list.
In this document:
Related documents:
Son-of-spectro is a set of daemon jobs that runs from the observer account on
sos3.apo.nmsu.edu. The actual processes are named sos_runnerd.py and there will be four
instances running, one per camera. Although sos_runnerd.py can be started manually, users
will seldom if ever start the process using this command. Instead, there are two user
commands that know about the setup at APO. They are sos_apocontrol and sos_apostart.
sos_apocontrol is the highest level command and should be used most often. sos_apocontrol
allows for the starting, stopping and monitoring of the daemon processes using the
standard APO options. In order to start the processes, sos_apocontrol calls sos_apostart.
sos_apostart can be called manually and allows more flexibility. Each of those commands
will start four instances of sos_runnerd, one per camera, with the correct parameters for
APO.
At APO the log files are stored in ~observer/sos/sos-logs and there is one log file per
camera for all commands and one log file per camera that only stores errors. It is a
good idea to periodically check the error log. Each log file is rotated out daily and
saved for four days.
The sos daemon processes can not be automatically started at boot because they need a
password to be manually entered in order to access svn. Because of this, starting the
sos daemons should be part of the SOS3 boot procedures document.
There is also a redo command, sos_aporedo that allows one nights of data to be manually
rerun trough aporeduce. Again, sos_aporedo will start four processes, one per camera,
to reprocess the images.
For more information be sure and read the sos daemon
readme
file. Also, each command has usage information via the '-?' option. Finally, for even
more information each source file has a document header at the beginning of the file
with implementation details.
=== All commands are executed from observer@sos3 ===
=== All commands assume that idlspec2d is setup ===
=== Read the readme file for more information on the commands.
Confirm SOS Processes are Running
$ sos_apocontrol status
There should be four instances of sos_runnerd
displayed, one per camera. The first line will be "Running", the last line will be "Done".
Between those lines should be four instances of sos_runnerd. Each instance will be a long
line separated by a blank line.
If you do not see see four processes running, stop the processes and then start the
processes.
Confirm the SOS Process are Not Running
$ sos_apocontrol status
There should be zero instances of sos_runnerd displayed. The first line will be "Running",
the last line will be "Done". There should be nothing between those two lines.
Start SOS Processes
$ sos_apocontrol start
When sos_apocontrol is run it will check to make sure that there is an ssh-agent that
it can connect to for svn access. If there is not, it will ask for the password of
the key it uses. The password is the same as is used for the sdss3 data access web site.
When the command finishes, confirm the sos processes are running.
Start SOS Processes After Observing Has Started For the Night
Use this command if the sos process was not started until after observing has started
for the night. This procedure will process all images already present for the current
MJD.
$ sos_apocontrol allow
$ sos_apostart -g -e
Then confirm the sos processes are running.
Stop SOS Processes
$ sos_apocontrol stopWait
If sos is processing an image, this command will allow the current idl process to finish
before stopping sos. If this command does not return after some time (~5 min), then
^C to kill the process and use the following command:
$ sos_apocontrol stopKill
Then confirm the sos processes are not running.
Reprocess a Nights Worth of Spectro
$ sos_aporedo -w -g -m 00000 (Where 00000 is the MJD to reprocess)
The output files are stored in ~/sos/redo. cat ~/sos/redo/redo-b1.out for the output of
the reprocessing of the b1 camera.
Confirm that the ssh-agent Process Used by sos_runnerd is Running
$ ps ax | grep ssh-agent
This line should be present:
"ssh-agent -a /home/observer/sos/control/agent.socket"
If it is not, then stop the sos processes and restart them.
You can run a different version of Son-of-Spectro by setting up
the relevant version of idlspec2d before you launch it with "sos_apocontrol".
This should be done from the "observer" account on "sos3.apo.nmsu.edu".
For example, to stop the current version, set up v5_4_15, then re-start:
$ sos_apocontrol stopWait
$ setup idlspec2d v5_4_15
$ sos_apocontrol allow
$ sos_apostart -g -e
If "sos_apocontrol stopWait" hangs, then hit control-C and
use "sos_apocontrol stopKill".
SOS is part of the idlspec2d package and uses the normal sdss3 install procedures
including being eups compliant and installed via sdss3install in the sdss3tools package. A
good place for more information is
BOSS Software Install
Instructions.
At APO, install both on the SOS machine (currently sos3.apo), and
on sdsshost.apo. This is because the documentation on the sdsshost.apo
procedures web page is linked to the installed version on that machine.
Now you are ready to launch this new version of the code.
Son-of-Spectro is a daemon job that runs from the "observer" account
on "sos3.apo.nmsu.edu". This daemon job monitors files written to /data/spectro/$MJD and
then launches a package of shell scripts and IDL commands to reduce
the data one frame at a time. It is part of the idlspec2d data product.
There are actually four daemon processes that run, one for each camera. We did
this so that both processors on the computer can be utilized. All 4 cameras should be
reduced for an exposure within a few minutes.
The output directors are:
/data/boss/sos/combined - Copy of aporeduce output for the web site
/data/boss/sos/$MJD - Output of aporeduce for that night
$SPECLOG_DIR/$MJD - plugmap files used for running aporeduce
platedb database - (s/n)^2 for each science frame reduced
The data reduction is not the same as the full-up spectroscopic pipeline.
A number of shortcuts are taken to speed up the reductions and make them
very robust. For example, boxcar extraction of the spectra is used instead
of optimal extraction.
Files like the following can be found in the /data/boss/sos/$MJD directory:
splog-b1-00006541.log - Text history of exp #6541 reduction
tset-51795-0389-00006541-b1.fits - Fiber traces (from flat exposure #6541)
wset-51795-0389-00006542-r1.fits - Wavelength solution (from arc exp #6542)
fflat-51795-0389-00006542-b1.fits - Flat-field vectors (w/ arc exp #6542)
sci-0389-b2-00006546.fits - Science spectra for plate 389 (exp #6546)
logfile-51795.fits - Summary info for night in FITS format
logfile-51795.html - Summary info for night in HTML format
snplot-51795-0389.ps - Signal-to-noise plot for nplate 389
If a particular frame does not get reduced and appear on the Son-of-Spectro
web site, then you can look at its log file (splog*.log). If the file
does not end with the line "Finished at", then that exposure must have
crashed.
The HTML files and S/N plots for the web page are copied into the directory:
/data/boss/sos/combined
The most recent HTML file has a second copy as the file "logfile-current.html".
This is the file pointed to by the
SOS web page.
The following FITSheader cards are required to be correct for either
Son-of-Spectro or Spectro-2D to reduce the data properly.
The AIRMASS is not read from the header, but computed from RADEG,DECDEG
and the TAI-BEG,TAI-END keywords.
EXPOSURE= Exposure number. We always override the value of this keyword
with the exposure number in the file name.
CAMERAS = Camera name, e.g. "b1". We always override the value of this keyword
with the exposure number in the file name.
FLAVOR = The flavor of the observation, which can be "bias", "dark",
"arc", "flat", and "science" or "target". Files with other
flavors, such as "unknown", are not reduced.
MJD = MJD of observation, which must agree with the directory in which
the file resides, "/data/spectro/$MJD".
PLATEID = The plate ID, which must match that in the NAME header card.
NAME = Plug name, e.g. "0328-52277-01" for plate 328, plugged on 52277
with a plugging ID of 01. We assume that there is a plug-map
file at APO with the name "/astrolog/$MJD/plPlugMapM-$NAME.par".
EXPTIME = The exposure time, which is used along with TAI-BEG to compute
the airmass. It is also used to determine whether the sky
brightness or scattered light is larger than tolerances.
TAI-BEG = The TAI time for the beginning of the exposure, which is used
to compute the airmass and sky-brightness gradients for sky-
subtraction. If this keyword is missing, we guess it from
EXPTIME and TAI.
TAI-END = Used with TAI-BEG. If this keyword is missing, we guess it from
EXPTIME and TAI.
TAI = This keyword is used if TAI-BEG,TAI-END are missing.
FFS = Flat-field screen positions, which should be
"0 0 0 0 0 0 0 0" for science exposures (all petals open)
"1 1 1 1 1 1 1 1" for flat or arc exposures (all petals closed)
FF = Flat-field (quartz) lamp status, which should be
"0 0 0 0" for science or arc exposures (all off)
"1 1 1 1" for flat exposures (all on)
NE = Neon lamp status, which should be
"0 0 0 0" for science or flat exposures (all off)
"1 1 1 1" for arc exposures (all on)
HGCD = HgCd lamp status, which should be
"0 0 0 0" for science or flat exposures (all off)
"1 1 1 1" for arc exposures (all on)
OBSCOMM = This keyword is used to identify the dithered flat sequence
taken in the Monthly Checkout with the "specFlats" script in SOP.
The flats in this sequence must have OBSCOMM="{dithered flats-flat}".
The arcs in this sequence must have OBSCOMM="{dithered flats-arc}".
This keyword also identifies the Hartmann focus exposures, with
the entries OBSCOMM="{focus, hartmann l}" or "{focus, hartmann r}".
Science exposures all have OBSCOMM="science ".
QUALITY = The observer-input quality for an exposure: "excellent", "test",
or "bad". Exposures are declared "test" or "bad" using the
APOFIX procedure.
The following FITS header cards are informational. We would very much
appreciate the observers to correct errors in these keywords so that
our book-keeping of survey progress is done properly:
TILEID = Tile ID for this plate from the plug-map file
CARTID = Cartridge ID number (1 through 9)
RA = Right ascension of telescope boresight (in degrees)
DEC = Declination of telescope boresight (in degrees)
RADEG = Right ascension of plate center for plug-map file (in degrees)
DECDEG = Declination of plate center for plug-map file (in degrees)
AIRTEMP = Temperature (deg C)
Incorrect FITS header cards for the raw spectro sdR files can be
corrected by adding entries in the sdHdrFix file
"/astrolog/$MJD/sdHdrFix-$MJD.par". The interesting header cards
from these files can be listed with the
APOHEADER
procedure run from the IDL prompt on sos.apo. The proc
APOFIX
can be used to add entries to the sdHdrFix file to denote edits.
This structure should have the following definition in the sdHdrFix file:
typedef struct {
char fileroot; # Root of file name, without any ".fit" suffix
char keyword; # Keyword name
char value; # Keyword value (as a string)
} ophdrfix;
Following are some examples of what should appear in the sdHdrFix files
in order to correct faulty headers. (The typedef struct above must
also be in the file.)
-
Example 1: The exposure time is in the file sdR-b1-00001142.fit should be
60 seconds:
ophdrfix sdR-b1-00001142 EXPTIME 60
-
Example 2: Exposure number 1234 was bad, and should be given an 'unkown'
flavor for all cameras:
ophdrfix sdR-??-00001234 FLAVOR "unknown"
-
Example 3: The mapping name for all cameras of exposures 1137-1139
should be '0198-51433-01':
ophdrfix sdR-??-0000113[7-9] NAME "0198-51433-01"
-
Example 4: Telescope status information is missing for the pre-flat
exposure 10850:
ophdrfix sdR-??-00010850 FFS "1 1 1 1 1 1 1 1"
ophdrfix sdR-??-00010850 FF "1 1 1 1 "
ophdrfix sdR-??-00010850 NE "0 0 0 0 "
ophdrfix sdR-??-00010850 HGCD "0 0 0 0 "
If you have edited the sdHdrFix file and need to re-reduce an exposure,
you can do so with the sos_redo command.
When Son-of-Spectro reads in the raw FITS files, it then patches the
headers with the information in the ophdrfix entries of the sdHdrFix file.
But note that the one piece of information that cannot be patched this
way is MJD, since we need the MJD in order to find the correct sdHdrFix file!!
Son-of-Spectro reduces four flavors of observations:
bias/dark, flat, arc, & science/smear. Select information is tabulated
for each of these types of observation. These values are tabulated in
yellow
if they are going out of spec, and in
red
if they are very much out of spec.
The values reported are:
- BIAS/DARK PERCENTILE98
The value in electrons of the 98-th percentile on the (overscan-corrected)
image. For example, if this is 8, then 98% of the pixels are below 8
electrons. Bad regions on the CCD and saturated pixels are excluded
from this evaluation.
The CCDs can also accumulate charge during the day that may need several
bias exposures to completely flush. If this does not work, then the
CCD is probably warm.
- FLAT NGOODFIBER
The number of illuminated fibers. This should be 320 if all the
fibers are plugged and unbroken. Note that fibers that fall on particularly
bad parts of the CCD can also be excluded from the good fiber list
(the red CCDs have some of these cases). If this number is less than
about 315, then you should be suspicious that some fibers have dropped out.
- FLAT XMIN (XMAX)
The minimum (maximum) X position of the spectra on the CCD. If this is
less than 0 (greater than 2047), then some of the spectra fall off the
left (right) side of the CCD. This probably means that no one ran
the Spectro Monthly Checkout. If it is off by more than 5 pix or so,
you should probably run the Monthly Checkout.
- FLAT XSIGMA
The profiles in the spatial (X) dimension have a gaussian fit with
a width of this sigma. The median of this width is taken independently in
each of 4 quadrants on a CCD, and the maximum of those 4 values reported.
If the spectrographs are in focus, then this value should be about 1.0 pix.
If it is larger, then the spectrographs may be out-of-focus, or the
slit-heads may not be properly latched.
- ARC WAVEMIN (WAVEMAX)
The minimum (maximum) wavelength (in Angstroms) of any spectra on this CCD.
Because of the optical distortions, this is always for the central
fiber. The edge fibers have less wavelength coverage.
This probably means that no one ran
the Spectro Monthly Checkout. If it is off by more than 5 pix or so,
you should probably run the Monthly Checkout.
- ARC BESTCORR
The linear correlation coefficient between the arc spectra and a template
arc spectrum. If they agree perfectly, then this is 1. The value of
this correlation is typically 0.80. If it drops too much lower, then
the arc spectra do not look as they should. This could happen if
the arcs did not turn on, the flat field screens did not close, or some
of the arc lines are missing -- if, for example, the Hg lamps all failed.
If the correlation is less than 0.5 or so, then it's even possible that
the code has found the incorrect wavelength solution.
- ARC NLAMPS
The number of arc lines used to generate the wavelength solution.
There are many more on the red CCD's because neon has many more lines
there. If this drops to too few lines, then some of the lamps have not
turned on properly, have not warmed up, or have warmed up too much.
- ARC WSIGMA
The arc-line profiles in the wavelength (Y) dimension have a gaussian
fit with a width of this sigma. The median of this width is taken
independently in each of 4 quadrants on a CCD, and the maximum of
those 4 values reported.
If the spectrographs are in focus, then this value should be about 1.0 pix.
If it is larger, then the spectrographs may be out-of-focus, or the
slit-heads may not be properly latched.
(One would expect that both XSIGMA and WSIGMA would go out-of-focus
at the same time.)
- SCIENCE/SMEAR SKY/SEC
The median sky counts in electrons per pixel.
If this is too large, then there must be a light source near the
telescope, or the night sky (the moon) is bright, or the CCD's are
warming up and generating dark current.
- SCIENCE/SMEAR (S/N)^2
The signal-to-noise squared for objects at the fiducial magnitude limit.
We choose (S/N)^2 because it is an additive quantity with additional
integration time. When all the cameras reach a total (S/N)^2 of 15,
then all these entries turn from red to black, and
we can declare a plate done. Exactly how this quantity is
measured is described below.
- EXPTIME
The exposure time (EXPTIME) header keyword from the first camera
of this exposure to be reduced. The assumption is that all 4 cameras
have the same EXPTIME.
- AIRTEMP
The air temperature (AIRTEMP) header keyword from the first camera
of this exposure to be reduced. The assumption is that all 4 cameras
have the same AIRTEMP. We think this is the temperature from
the galileo weather station.
- UT
The UT time computed from TAI in the header from the first camera
of this exposure to be reduced.
- QUALITY
This is the observer-input quality for this exposure.
It can be set independently for each of the 4 cameras, but only one
camera value (the first reduced frame) is reported in this table.
The default value is "excellent" for everything except for dithered
flats or spectro focus frames which are "test". The observers have
the option of declaring exposures "excellent", "test", or "bad"
using the APOFIX command.
The exact values of these yellow/red limits and further explanation can
be found in the "idlspec2d" product in the file "examples/opLimits.par".
There are a number of
WARNING
and
ABORT
messages that can appear if the pipeline runs into trouble when processing
a frame. These messages appear at the bottom of each table.
Each one-line message begins with the relevant file name, WARNING or ABORT,
then a brief plain-text message.
Note that a single problem may cascade into a large number of warning
messages. For example, an out-of-focus spectrograph will first produce
the "Median spatial widths" message, probably followed by warnings about
bad sky-residuals.
The following messages may appear:
- Error reading sdHdrFix file
The sdHdrFix file (sdHdrFix-$MJD.par) in sos.apo:/data/spectro/astrolog/$MJD
exists but is not a valid Yanny parameter file. This file is written
by the IDL procedure
APOFIX
if the observers' have run that proc on sos.apo, but can also be edited
by hand. If the file is invalid, you should edit it to be valid,
or delete it. If the
APOFIX
command somehow generated an invalid file, then file a PR against sospectro.
- Exposure number in header disagrees w/filename
This means that IOP/SOP is in a very confused state, and it
is putting a different exposure number in the header (EXPOSURE
keyword) from what is being used to generate file names.
You should probably exit SOP, re-start SOP, then issue a "goStare -init".
- Sun above the horizon by ... deg for non-test exposure
This warning message is to trap flat, arc, or science exposures taken
during the day that have not been marked as either "test" or "bad" data.
If these are test data, be sure to mark them as such. This is to prevent
such data from being used later in the full reductions.
- Amp #... expected read noise = ..., measured = ... DN
The number reported is the standard deviation in the bias region
for either amplifier #2 (left side) or amplifier #3 (right side).
This calculation is done clipping the half-percent of lowest and
highest values. We trigger this warning if the value is ever 1.0 DN
above the expected value. This can happen
if there are a huge number of cosmic rays (if you've been integrating
for hours), or if there is an electronics problem. This would pick
up the problem seen on 22/23 Aug 2000 (MJD 51779) when the left amplifier
on r2 was randomly flipping bits (c.f., PR #2335). Or, the problems
seen in Aug 2004 (PR #6143), when r2 was digitally mixing bits between the
left and right amplifiers.
- Amp #... bias region difference at xxx-th-percentile =... DN
This measures another statistic of the bias region for either
amplifier #2 (left side) or amplifier #3 (right side).
This measures the difference between the 16th-percentile and
84th-percentile, which should be equal to twice the read noise
(e.g., 1-sigma). This test is done at the 68.2-percentile (1 sigma),
95.4-percentile (2 sigma), and 99.7-percentile (3 sigma).
A warning is reported if this difference is either significantly too
small or too large, as compared to what gaussian statistics dictate.
This should catch the same sorts of electronics problems as listed
for the above warning message.
- Amp #... way too many pixels (xxx%) below bias-5*sigma=... DN
This test looks for anomolously low pixel values in the data region
of the CCD. There should essentially never be any pixel values below
5-sigma less than the bias level, unless there is something wrong with
the electronics. This did happen in Aug 2004 (see PR #6143).
- Fixing shifted rows (from electronics)
This is very, very bad. This is indicative of an electronics problem
that we first saw on MJD=51578 to 51580 (3 to 5 Feb 2000).
The raw images have rows shifted to the right, sometimes by
many pixels. Call Connie immediately.
- Fixing dropped-pixel rows (from electronics)
This is very bad. This is indicative of an electronics problem
that we first saw on MJD=51688 (23 May 2000).
The raw images actually have some rows shifted by 1 or 2 pixels,
usually more so near the bottom of the CCD (the first rows to be read).
Call Connie immediately.
- Electronics shifted xxx rows by 2 pix
This is indicative of an electronics problem
with the BOSS electronics that shifts all or some of the rows in the
raw images in all 4 amplifiers (ticket 423).
- More than 10% of the image is rejected
This is very bad. This can probably only happen if most of the CCD
has saturated pixels, which probably means you're observing during
twilight, the CCDs are warm, or the dome lights are on.
- Flat-field screens not closed
The "FFS" keyword in the header indicates that at least one of the
flat-field petals was not closed. When this happens, the flat or arc
is not reduced.
- Flat-field lamps not turned on
The "FF" keyword in the header indicates that at least one of the four
flat-field lamps was not turned on.
When this happens, the flat is not reduced.
- Reject flat (or arc) ... % bad pixels
This condition is triggered when more than 2% of the non-masked pixels
on the image are bad (saturated).
When this happens, the flat (or arc) is not reduced.
This probably happens if the CCDs are warm, the dome lights are on,
or if for some reason the shutters were open too long.
- Reject flat (or arc) ... saturated rows
This condition is triggered when there are more than 100 saturated rows
on the image.
When this happens, the flat (or arc) is not reduced.
This probably happens if the CCDs are warm, the dome lights are on,
or if for some reason the shutters were open too long.
- Reject flat as too faint
This condition is triggered when the 80-th percentile of the image
is less than 1000 electrons.
When this happens, the flat is not reduced.
Either the flat field screens were not closed, the lamps were not
turned on, or the shutter didn't open.
- Reject science ...
A science exposure can be rejected if the header keywords indicate
that the flat-field petals are closed, any flat-field or arc lamps
are turned on, too many pixels are bad or saturated, or if the
25-th percentile of the image too large.
- Possible Argon lines in superflat
Emission lines are present in the quartz-halogen flat-field images,
which are supposed to be featureless. When a number follows this message,
that is a measure of the line strength -- the trigger is set to 0.01,
but we usually see it as 0.1 to 0.5 when present.
We have identified these rogue lines as argon.
JEG's best guess is that these
contaminating lines come from trace amounts of argon in the HgCd lamps,
which must still have current running through them when they are supposed
to be off. We can still reduce the data with these lines, but you should
re-open PR 1859 if this problem is seen again.
- Arc lamps not turned on
The "NE" and "HGCD" keywords in the header indicates that either the
neon or HgCd lamps are not fully turned on.
When this happens, the arc is not reduced.
- Best arc correlation = ...
The cross-correlation of the arc spectrum with the template arc spectrum
was bad. This problem also triggers BESTCORR as bad in the table (see
above). This may be due to too little signal in the lamps, e.g. if
the flat field petals did not close or the lamps did not turn on.
If those are not the problems, then look at the raw image.
- Big wavelength gap
Some arc lines were not found in the arc spectra, and there is a large
gap in wavelength space without any lines. This will produce a poor
wavelength solution. More arcs should be taken until this message
does not appear.
- Cd I 3610 line missing
This particular arc line is the tie-down of the UV wavelength calibration
and is either missing or has a poor fit. This is likely due to the HgCd
lamps not being sufficiently warm before the exposure start.
- Scattered light
There was a high baseline count rate on the CCD, and appears even
between fibers. This can most obviously occur if there are light
sources in the CCD (such as the LED's we had for some time), or
if the CCD's are warming up and generating dark current.
There can also be a scattered light contribution just from a very
bright sky, or if there are super-bright objects on some fibers that
are scattering or bleeding light across the CCD.
It's best to carefully inspect the raw images for problems
(see Displaying Spectra at APO).
- Median spatial widths = ...
The spatial (X) widths of the fibers typically are gaussians with a
sigma of 0.85 to 1.05 pixels. If the sigma is larger than 1.10 pixels
in any of the 4 quadrants of a CCD (lower-left, lower-right, upper-left,
upper-right), then this warning is triggered. It means
that either the spectrographs are out of focus, or the slit-heads are
not properly latched. Note that these widths are computed for both
the flat and science exposures (but not for arcs -- we compute the
widths in the dispersion dimension for arcs, and not for smears since
they have low S/N).
- Median wavelength widths = ...
The widths of the arc lines in the dispersion (Y) dimension typically
are gaussians with a sigma of 0.90 to 1.10 pixels.
If the sigma is larger than 1.10 pixels
in any of the 4 quadrants of a CCD (lower-left, lower-right, upper-left,
upper-right), then this warning is triggered. It means
that either the spectrographs are out of focus, or the slit-heads are
not properly latched.
- Too few sky fibers to model sky-sub variance
There are not enough good sky fibers on the CCD. This will only happen
if for some reason there were far fewer than the mandated 16 sky fibers
on a CCD, or most of those fibers just happened to be dead fibers.
You should never see this message. If you do, the data is un-reducable.
You could try one more exposure, but something is probably horribly
wrong with this plate.
- Airmass range = ...
This warning is triggered if the airmass exceeds 2.5. At such large
airmasses, the atmospheric refraction terms are getting large and the
sky background is bright. The data is still perfectly usable, it's
just not as good as taking data at lower airmass. Keep observing if
you must, but the airmass will rapidly approach infinity!
- Large flexure flat<->science
There is a large shift (more than 1.00 pix) between the flat-field and
the science exposure, presumably from flexure in the spectrographs.
When this happens, another set of flat-fields (post-calibs) is
recommended. (However, don't bother to take a set of calibrations
on a different night. The Spectro-2D reductions never use calibrations
from one night for science exposures on another.)
- Whopping fiber ....
The fibers listed have very bright objects that affect their neighbors
on the CCD. If the objects are bright enough (12th mag?), then this can
trigger other warnings such as "scattered light". I think it's safe
to say that whopping fibers only show up when there has been a mistake
made in the plate designs -- this is already recorded in PR 2471.
You should check that this object does not saturate (> 30,000 ADU)
the raw sdR image
(see Displaying Spectra at APO).
Should it be saturating, reduce the exposure time to prevent saturation
or move on to the next plate.
- Wrong number of elements for REDDEN_MED
This is a warning that the reddening vector for this plate (reddeningMed
in the plPlugMapM file) is not a 5-element vector, as expected).
In this case, reddening values of zero are assumed. This means that
the plate will not be observed as deep as it would have if non-zero
reddening values were provided.
- SOS disk is ...% full
The specified disk on sos.apo is more than 95% full. This could be
either the disk used to copy the raw sdR images from sdsshost.apo,
or the disk used for SOS outputs (/data/spectro/spectrologs/$MJD).
If you must clean out old data, do not delete files from sos.apo
whose original copies still exist on sdsshost.apo or they will be
re-rsynced and re-reduced
(see How Son-of-Spectro Works).
You could delete files from a previous night first on sdsshost.apo,
then from sos.apo.
The following warning messages are all based upon the quality of
the sky-subtraction. Typically, we are able to model the sky spectrum
(from the 16 sky fibers on each spectrograph) with a relative chi^2
of around unity. At very bright sky lines, like 5577 Ang, the relative
chi^2 may be as large at 25. If the relative chi^2 is large at
other wavelengths, this means that there is excess light down the fibers
that varies across the plate. This could be due to a light source near the
telescope, or possibly a bright, non-uniform sky. Strong auroral activity
is something that can do this, since the O I lines at 6300 and 6366 Ang
are resolved on the sky.
- Median sky-residual chi2 = ... at ... Ang
The median reduced chi^2 for sky-subtraction is always around unity.
If it is greater than 2, this message is triggered. If this message
appears and you are not observing during twilight or with warm CCDs,
then there must be a serious problem. Seriously out-of-focus
spectrographs might trigger this, or lights on near the telescope.
- Max sky-residual chi2 = ... at ... Ang (ignoring 5577)
This is an informational message triggered if the reduced chi^2 is
greater than 100 anywhere other than at the 5577 Ang sky line.
If there is auroral activity, then this could be triggered at a few
specific lines like O I at 6300 and 6366 Ang. These O I lines has
reduced chi^2 values of about 100 on MJD 51999 during a solar storm
(i.e. plate 336/51999).
The Spectro-2D pipeline will automatically mask these wavelengths
for any down-stream analyses.
- Bad sky residuals at ....
This warning is triggered if there is a range of at least 25 Angstroms
with a reduced chi in the sky-subtraction worse than 2.
This could be due to a light source near the telescope, a bright,
non-uniform sky, a warm CCD, scattered light, or out-of-focus spectrographs.
For most of those cases, there should be other relevant warning messages
preceding this one (like a warm CCD message).
This message should be ignored at the edges of the CCDs
wavelength coverage -- near 3800 or 6200 Ang for b1,b2 or near
5800 or 9200 Ang for r1,r2 (c.f. PR 3656).
If none of the above explanations apply, then there is some real problem.
I would look at the images and/or spectra to look for any obvious problems,
and file a PR against sospectro
(see Displaying Spectra at APO).
Might as well keep observing?
- Red Monster at ....
This warning is triggered if we see bad sky residuals (above) that
is specifically in the wavelength range of about 6400-6600 Ang.
We think this happens when the handpaddle is still plugged in,
in which case the observers should unplug it immediately.
The treshhold is set at reduced chi=2, and the worst that we have ever seen
is at about the level chi=6. The last time we definitively saw this was on
plate 426 on 3/4 Dec 2000 (MJD 51882).
You should be able to see a bump in the extracted spectra, especially
for the sky fibers
(see Displaying Spectra at APO).
If you cannot find the source
of the excess light, keep observing but file a PR against sospectro.
If the reduction of an exposure is catastrophically bad, it may not
appear at all in the Son-of-Spectro table. However, there should still
be a log file for this exposure on the sos.apo computer:
/data/spectro/spectrologs/$MJD/splog-$CAMERA-$EXPOSURE.log
Reading this file should tell you what failed. The first and last lines
of these files should contain "Started at" and "Finished at"
followed by timestamps.
A median signal-to-noise is computed for each object in the
wavelength ranges [4000,5500] Angstroms (synthetic g-band)
and [6910,8500] Angstroms (synthetic i-band).
We plot these S/N values versus the PHOTO fiber magnitudes, which were
measured in approximately a 3-arcsec diameter aperature (the same size
as our fibers). If everything is working perfectly, then our S/N values
should correlate very well with these PHOTO magnitudes.
We determine whether a plate is "done" based upon the signal-to-noise
of the fainter objects on the plate. We do this by fitting a line to
the (S/N)-vs.-magnitude plot in a specified wavelength range, then evaluating
this fit at g=20.2 mag (blue CCDs), and i=19.9 mag (red CCDs).
Approximately 10% of the main galaxy sample, 60% of the BRG's and 15% of
the QSO's are fainter than g=20.2. Very few objects (1, 1, and 5%) are
fainter than i=19.9.
When the sky level is higher, we gain S/N more slowly at
the fainter magnitudes where we are sky-limited rather than photon-limited.
Without moon, we have typically found:
log(S/N_g) = (zeropoint) - 0.31 * g
log(S/N_i) = (zeropoint) - 0.31 * i
With partial moon, the slope steepens to -0.34 or worse.
The fitting regions are denoted on the plot with vertical dotted lines.
Arrows point to the evaluation of the fit on each of the 4 cameras, with
the top panels corresponding to the blue CCD's (synthetic g-band) and
the bottom panels corresponding to the red (synthetic r-band).
The thick blue line on the figure is only a meaningless, fixed reference line.
The right-hand figures plot the residuals of each object from the fit.
Symbol sizes on those right-hand plots indicate the magnitude of the
deviation from the fit line. Symbol color is the same on the left
as on the right, so
green
objects have more flux and
red
ones less.
If the scale of the telescope is wrong, then you will see a radial drop-off
in flux (red points on the edge of the plate).
If you are observing too far over in air mass, then typically you correct
to first order with a scale change, but a quadropole is left in these
residuals.
If one spectrograph has problems, then this will show up as red points
in half of one of these figures.
Note that the (S/N)^2 totals listed in the table and the figure might
not exactly agree. This is because the fitting to (S/N)-vs.-magnitude
is done on individual frames for the table, but on the summed S/N over
all frames for the figure. The tabulated values are the ones we use
to declare a plate done.
The Son-of-Spectro (SOS) reduction robot has proven to be quite robust.
However, if it appears to not be working one could check the following:
- Has a "loadCartridge" command been issued?
If this has not been done from SOP, then there is no PLATE entry
in the images. This prevents SOS from reducing anything but bias
and dark frames.
- Is the cron job running?
Log into "observer@sos.apo.nmsu.edu", and issue a "sos_status"
command. This should report that SOS is running.
If not, then restart the cron job.
- Is the data being copied over?
The raw data files should be copied from "sdsshost" to "sos.apo"
in the /data/spectro/$MJD directory. If it is not, then the disk
may be full (see SOS disk is full warning message).
Or, the rsync command may not be working.
This could be due to a rogue
"aporsync" process sitting around. The "sos_status" command will
report any such processes. If you suspect this as the
problem, just kill that process. It's a harmless thing to do, since
SOS will just start it up again within a minute.
- Are there old "lock" files sitting around?
SOS generates "lock" files to prevent multiple processes from changing
these files simultaneously. The "sos_status" command will report
all "lock" files and their last modification times.
If any have been sitting around for several minutes or more,
then something is wrong. You should delete the file ending with ".lock".
However, this could result in a corrupted "logfile*.fits".
- Is the IDL license manager running?
You can issue the following command to check that the IDL license
manager is indeed running:
% $IDL_DIR/bin/lmstat -a
This should display the number of licenses available, and the current
number in use. The server running on galileo.apo has 110 licenses,
each IDL session uses 6, so there can be at most 18 IDL sessions
running concurrently. Note that Son-of-Spectro, the fiber mapper,
and HoggPT all use IDL licenses.
- Is anything amiss with running the scripts?
Perhaps there is a communication problem between sdsshost.apo and sos.apo,
or a problem running the code on sos.apo. There is logging information
written once per minute into the file
/data/spectro/spectrologs/sos.log
which may indicate such problems.
- Is the logfile corrupted?
This is probably the worst thing that could happen. If the file
/data/spectro/spectrologs/$MJD/logfile-$MJD.fits
gets corrupted, then SOS will repeatedly crash. One could just
delete this file, which would basically wipe out all the previous
reductions for that night from the web page. Alternatively, one
could delete all the raw and reduced data for that night from
"sos.apo" (but not from sdsshost.apo!). Best to stop the SOS robot
before doing this:
% sos_stop
% sos_status <-- Wait for any running processes to complete.
% rm -rf /data/spectro/spectrologs/$MJD
% rm -rf /data/spectro/$MJD
% sos_start
That might sound extreme, but SOS will simply start re-copying
everything from "sdsshost" and re-build the reductions for the night.
Obviously, this could take a couple of hours if its already late
into the night.
There is a mechanism to re-reduce exposures with "sos_redo".
You may need to do this if a reduction failed, or the pre-calibs failed
and SOS could not reduce the science exposures, or if you needed
to edit the sdR header information by putting entries in the sdHdrFix
file (see APOFIX).
It is fastest to only re-reduce a specific exposure number.
To force a re-reduction of exposure number 12345 on all 4 cameras
(whether it has already been reduced or not) use the EXPNUM option:
% ssh observer@sos.apo.nmsu.edu
% sos_redo expnum=12345
If you want to find and reduce only failed reductions for a plate
use the PLATE option. This may take several more minutes because
all of the FITS headers for the night are parsed.
To re-reduce only missing exposures for plate 666:
% ssh observer@sos.apo.nmsu.edu
% sos_redo plate=666
The last exposure taken is not re-reduced unless explicitly told to
with the EXPNUM option. Currently, either EXPNUM or PLATE must be
specified.
Note that science exposures for a plate cannot be reduced until both
flat and arc exposures have been successfully reduced for that same plate.
This is from version $Name$ of procedures.
Maintained by
David Schlegel
and Scott Burles