Michelle Troubleshooting Guide
Tips, Tricks & Troubleshooting
Mechanical
|
Movie
|
Electronic (including Edict)
|
CCS
|
Sequence Console
|
Telescope-related issues
|
Mechanical
Mechanism fails to find datum: After a reboot
of the cryostat control system (CCS), all the mechanisms will first have
to find their reference point, or datum, before they can move to a
specified position. Although rare, a mechanism does occasionally fail to
find datum and will usually time out, an error status message appearing in
the status DM screen (see Running
Michelle). The first thing to try is to move the mechanism again from
the CCS DM screen while watching the datum switch fields in the relevant
DM screen. If the datum switches move (they switch between closed and open)
and the mechanism still fails to find datum, there could be a software, firmware
or communication problem. You could try rebooting the CCS via the CCS serial
console. If this fails to cure the problem, you will need to call for help.
If the datum switches fail to operate, the microswitch could be faulty and
the backup switch will need to be used. This requires a change to the CCS
software. Call for assistance.
Illuminated area on the array has shifted (spectroscopy
path): This is rare but has occurred in the past when doing echelle
spectroscopy. If the instrument has been moved or worked upon during the
day and received a minor shock, the grating drum can shift slightly causing
the illuminated area on the area to move. If this occurs, simply go into
the grating DM screen via the CCS DM window, select "force" and hit the
start button. This will force the mechanism to find datum and fix the problem.
Do not forget to switch back to "discretionary" after the move, otherwise
the grating will re-find datum every time a move is requested. See 20020813.012.
Large "blobs" seen in raw chopped frames: If the
Michelle sacrificial or main window is exposed to water, it will become
damaged. Over time this damage is probably unavoidable on the sacrificial
window, although the main cryostat window should remain protected or dry.
If either window becomes water-damaged, blobs of high emissivity (or high
counts) become obvious in the raw chopped frames in imaging mode. If the
damage is slight, these emissive regions subtract out when chopping and are
not a big issue. However, if the damage is extensive, the regions cause the
array to saturate and the images become badly affected. The only recourse
is to change the window which requires daytime engineering effort. See www.jach.hawaii.edu/~tkerr/window_damage.html
for an example of a badly damaged window.
Unstable array temperature: Small variations in
the array temperature are normal when switching between imaging filters,
the thermal load on the array changes as the flux incident on the array
changes. However, if you see changes in temperature of 1-2K or greater
over a time period of an hour or two, this could indicate a blocked JT
valve. Call for assistance. If the JT valve is becoming blocked, a temporary
fix to to open the JT valve to 5 turns and then quickly return the valve
to a setting of 0.7 turns. This will move the obstruction away from the
valve and the JT temperature should return to normal after twenty minutes
or so. However, this does not clear the contaminants from the JT system and
the blockage will return in a few hours (less if the contamination is severe).
To clear the blockage a full JT purge is required which requires daytime
engineering effort. See 20020211.003.
Electronic (including Edict)
Array counts are 0 or > 65000: This probably
means the array has not been enabled. Try enabling the array from the mich_oper
DM screen (see Running Michelle).
If enabling the array does not work (i.e., the indicator light on the
DM screen remains yellow or the temperatures fail to rise after a few seconds)
then the array safety software may be preventing the array from being enabled.
This occurs when the array temperature is too warm for the array to be
used safely (the current cut-off temperature is 12K). It can also occur
if the safety software has failed to load. If the temperatures look normal
you can attempt to load the arraySafe software from the WFG serial console.
Open up the console and type isArraySafe. The response to this command
might give some clues as to what the problem is. If things look normal, try
starting the software with the command arraySafeStart, waiting a few
seconds (try 30 sec) and then enabling the array again. If the array still
fails to enable, you may have to reload a valid waveform. To do this you
need to open or reopen the WFG serial console and type the following: wfgClear
<return> followed by setIdle <return>. The setIdle
command will prompt you for four responses, which are: 1, 0.03, 10,
0. After this, type wfgLoad "mch_str_bw" <return>. Now
try enabling the array again. If this fails, call for help.
The array is saturated: Saturation is rarely
an issue in spectroscopy mode, but can be when imaging, especially through
the broadband filters. The data reduction automatically detects possible
contamination in the individual chop frames and should warn you if saturation
is approached. If this occurs, check the raw chop frames using gaia to see
the counts, which should be lower than 50,000. Typical Si filter counts
are ~35,000 but the broadband filters can result in counts of around 48,000
to 50,000 depending on atmospheric conditions. If the CSO tau is high, it's
likely that the conditions are not good enough for imaging and you should
switch to another project. If you are desperate, you can change the exposure
times in the OT programme but this is not recommended. A list of working exposure times is available.
If conditions are good, also check that you haven't
left the Michelle window shutter closed!
Noisy array: Noise on the Michelle array is
generally caused by electronic pickup. This could in turn be caused by poor
grounding, loose cables or boards, dirty connections etc. If the array looks
noisy, you can run a test to measure the read noise. From the sequence console,
go into the File menu and then choose load. You should then select the sequence
called Mich_read_noise.exec. Run this sequence. The DR will report
and averaged standard deviation for each of the 10 pairs of data taken.
Note that the array is capable of a read noise of 3500 electrons in deep
well STARE mode. Note the reported standard deviation and call for help
if the noise is significantly higher than around 4000 electrons.
One quadrant "missing" on the array: A channel
or "quadrant" on the Michelle array consists of four contiguous "subchannels"
or outputs, each of these consisting of 20 pixel columns. Occasionally after
the Edict crate has been powered up one of the four quadrants is missing
from the data files. Although a soft rest of Edict may fix this, usually
the Edict crate requires power-cycling using the green rocker switch at the
bottom of the crate. If this does not fix the problem you will need to call
for assistance as one of the Edict slaves may have a problem. Look to see
if any of the Edict slaves are showing red error lights in the crate.
Noisy or intermittent subchannel: Loose cabling
or faulty connections can cause one subchannel or output to behave erratically,
showing few counts when the array is enable, yet appearing quite active
even when the array is disabled. The problem can come and go with telescope
slews as well. See 20040208.003. Please call for assistance first,
but the place to start checking is the front-end Edict electronics underneath
the main cryostat. Please ensure that you wear a grounding strap when
checking cables in this area, otherwise there is potential for array damage
to occur.
Edict reports a damaged waveform: This problem
will result in a MOCS error which is reported via the sequence console
in normal observing procedures. If you see a MOCS error, check both the
WFG and CCS serial consoles for error messages. If the WFG serial console
is reporting a series of error messages, and amongst them is "damaged waveform",
then a waveform has failed to load correctly. Although this problem should
not occur anymore since the location of the waveforms has changed, if you
do see this problem then you should try an do a soft reset of Edict. If this
doesn't fix the problem, power cycle the Edict crate. Call for help if
neither of these attempts cure the problem.
You will also very likely have to run the OCS software
down before you can take data again, or at least do a kill_inst Michelle
from the sequence console, and then restart the connection to the instrument.
Edict slaves report "read 0" error: The symptoms
of this may include a sequence console MOCS error, a simple failure to take
a data frame or movie failing to produce a quick look image. If the array
appears to be enabled and the reason for a failure is not apparent, open
all the Edict serial consoles. If one or all the slaves are reporting something
along the lines of "0x80f662a0 (tAccum): daqAccumThread: Failed waiting for
read 0" then Edict is in a bad state and needs rebooting. To save time, you
should probably just go ahead and power cycle the crate, as a soft reset
rarely fixes this problem. See 20020901.005.
Array doesn't enable or temperature
fails to rise on enable command: See Array counts
are 0 or > 65000.
Unable to run the WFG Serial console or open the slaves:
Only one occurrence of the Edict or CCs serial consoles can run at any given
time. If someone else is running them you will not be able to successfully
telnet to the console. Unless you happen to be running the consoles yourself
elsewhere, then you should log into Ohi as observer and type ps -ef and look
for observer process similar to "telnet ukirt9 3008". The last four digits
may be different, but it's ukirt9 that you need to connect to. You can simply
kill these process with the "kill -9 processid" command and then try to
reopen the consoles. Alternatively, after killing the process, you can log
directly into the console with "telnet ukirt09 ****" where **** is the 4
digit number of a process you just killed. Remember to disconnect before
trying to run the consoles again.
Sequence Console
The dreaded MOCS error message: See 20040204.009 for an extensive discussion. Although
a variety of problems may cause the sequence console to produce a MOCS error,
it is usually caused by either a problem with Edict or the CCS, and is probably
a communications issue. Firstly, it is rare that you can simply acknowledge
the error message the sequence console produces and continue, although on
the odd occasion this does work and probably depends on the specific underlying
problem. Usually, you must run down the OCS software and then either reset
Edict or the CCS. If you look at the Edict serial consoles and see an error
message (usually a damaged waveform reported in the WFG serial console, or
read 0 errors in the slaves), then reset Edict from the WFG serial console,
restart the OCS and retry the current sequence. If no error is reported by
Edict, then the chances are the CCS needs rebooting from the CCS serial console.
You may look for an error message in this console, but more often than not,
one isn't obvious. After the CCS is rebooted, run the OCS and retry the
sequence.
Fortunately, for a reason no one yet understands, these
problems seem to have stopped occurring. Therefore, if you do see one, it's
important to report the problem with as much relevant information as possible,
including what's reported in the Michelle serial consoles, in case the problem
indicates that another batch of MOCS errors are about to occur.
Movie
Movie fails to take data: Probably due to a race
condition, when doing target acquisition with Michelle, or peakup up through
a slit using movie, the first movie attempt fails. The sequence console will
display "Failure taking movie image" in the status field and the quick look
display will no longer update. Simply click on the stop button in the movie
window and then hit start. Unless Edict has been given an inappropriate configuration
(unlikely, unless the observer has specified their own exposure times),
movie should work the second time around. There is no need to close the
movie window and open it again, hitting stop and start should fix the problem.
Remember to stop movie before continuing with the sequence!
The movie quick look display is garbage: This
has only happened once in my experience, but is something to watch for if
the observer account is being used on both Ohi and Kauwa. I believe that
problem is fixed, but....
If gaiadisp is used to display an image on Kauwa, it
can attempt to display the image on the quick look display on Ohi instead
and get you into quite a mess. Because of what's known as a byte-to-byte
error, the quick look display on Ohi will display garbage (pixels with counts
of 1e-38 for instance). If you do see this problem, the observer must kill
whatever process is being run on Kauwa to display the image, and the OCS
software will almost certainly have to be restarted.
If you are stuck with running the observer account
on both Kauwa and Ohi, and an image needs to be displayed on a gaia display
on Kauwa, simply start a gaia session and open the relevant file from within
the gaia menu.
The movie quick look display shows data but is very
noisy: This can occur for numerous reasons, but the most likely is that
too few coadds are being used for successful chop subtraction. If only one
coadd is used, the resulting image can be extremely messy due to the fact
that the array often needs a few null reads or exposures at the beginning
of an observation. Check to see how many coadds are being used in the sequence
console and alter the OT programme if possible.
If the array is saturated (see "The array
is saturated"), then this can also result in an extremely noisy looking
difference frame. Call for assistance if the reason for noisy movie
frames is unclear.
CCS
The mechanisms are taking a long time to move:
After the CCS is rebooted, all the mechanism must find their datum position
the first time a request is sent for them to move to a position. Although
all the mechanisms move simultaneously (unlike CGS4 for instance), two or
three mechanisms take a minute or two to find their reference position. This
is necessary as these mechanisms either need to be positioned extremely accurately
so find datum in a rather tedious but necessary manner, or require other
mechanisms to move before they are able to find datum successfully.
If you feel the mechanisms are slow to move in normal
operations, then check the status DM screen to see which mechanism is moving
("configuring" and "busy" will be displayed in the screen against the relevant
mechanism). If you are switching gratings, then it can take a minute or two
for the mechanism to find its new position - this is normal. If a mechanism
takes a long time and eventually times out, then this indicates a more serious
problem. See "Mechanism fails to find datum".
Checking the current status: See "Running Michelle" to find out how
to open the Status DM window. In usual operations, the mich_oper DM window
reports where the mechanisms are, e.g., whether you are in spectroscopy or
imaging mode, which grating is being used, the filter name etc. When mechanisms
are moving, this DM window will not display the next setting until the relevant
mechanism is in place, but will report that the CCS state is "BUSY" if mechanisms
are moving, or "IDLE" if they are in position. Likewise, the mich_oper window
also reports if Michelle is taking data, with "Busy" appearing in the Edict
state field when taking a frame, and "Idle" when no exposure is taking place.
When mechanisms are moving, you can launch the status
DM window via the STATUS button on the mich_oper screen. This screen will
indicate which mechanism is moving (it will say "Configuring" and "busy")
and which are idle ("Running" and "Idle). Note that when "Running" is displayed,
this actually means the mechanism is healthy rather than moving. If any mechanism
reports an error state, it will be reported in this screen. If you see a red
line next to a mechanism, this means that the mechanism has yet to find datum.
Telescope-related issues
Top-end doesn't chop: Firstly, check the the topend.dl
window. If a Michelle sequence has been executed from the sequence console
(excepting echelle sequences) then the chop should automatically be turned
on and the chop set to external so Michelle can drive the secondary. Bear
in mind that these will not be set until the ChopOn command has been
reached in the sequence (the Chopon command can only be seen when the engineering
display is turned on). If the sequence is running but one of these fields
is incorrect, check the sequence itself to see if chopping has been set up
(look for commands such as SET_CHOPTHROW, SET_CHOPPA). If the sequence doesn't
contain these commands, it's likely that chopping hasn't been selected in
the OT programme target component (see "Observing Preparation").
If all looks well with the sequence then check the front
of the CCS cabinet towards the bottom right. There should be a cable leading
from the JK2 output on the Xycom card to the UKIRT chop connection a few inches
to the left and below. If the cable is there, check the connections. If the
cable is missing, call for help.
The chop throw looks wrong on the guider draw window:
If a new sequence is run from the sequence console which uses a different
chop throw, the guider draw display does not automatically update.
You will need to click on the fast guider button in order for the window to
update with the new chop throw. (NB. It is important to have the correctly
sized draw window - guiding will fail if the size is incorrect). If this
does not fix the problem, check the topend.dl DM window and make sure the
chop and throw settings match those of the actual sequence. If they don't
then for some reason the settings have not been passed onto the TCS. You
will have to stop the sequence and restart. If this does not fix the problem,
run down the OCS and start again. If the problem persists, call for help.
Telescope doesn't slew when the slew command is reached
in the sequence console: Remember that the sequence pauses on the first
slew command in a sequence. If you expected the telescope to slew, check to
see if the observer has hit the continue button in the sequence console. If
the telescope still fails to slew, then it's possible the sequence has been
"sent to engineering" rather than to the queue, and in this case telescope
slew commands are disabled. Stop the sequence, requeue it and send for execution
from the queue monitor and restart it from the top.
"I put the star in the guide-box but it moved away a
few seconds later": This always catches people out when Michelle hasn't
been used for a while. When executing a Michelle sequence, the chop throw,
chop angle and the chop beam need to be set up by the TCS after the
instructions are sent by the sequence console. The important thing to remember
is not to turn guiding on until the telescope is in CHOPBEAM A and the sequence
has gone through this point. If you turn guiding on before this command is
reached, then the star will move out of the guide box when the telescope offsets
to chopbeam A. Also remember that it takes a little while for the guider
camera to rotate if the chop angle changes, so you need to wait for this
to finish as well.
Inability to focus - only 2 images of the star appear:
Currently, we are only able to focus on a CMC or guidestar when the guider
camera angle is 0. It may well be possible to focus when the guider is at
an angle of 45 degrees or more, but so far we have not been able to focus
if the guider is at 90 or -90 degrees. Although the reason for this is not
certain, it looks very much as if the CCD is vignetted slightly at these
angles, so all four images of the star do not appear. In order to focus,
you will have to manually set the camera angle to 0 in the topend.dl screen.
Engineering mode - why isn't the telescope chopping?:
This will probably only occur when Tom Kerr is observing and is being a pain
by running Michelle via the engineering interface. Michelle itself only sends
out a chop signal, it cannot send the chop throw or chop angle (during normal
queue observing, it is the OCS software that sends the throw and angle to
the TCS). In engineering mode, the chop throw and position angle will have
to be set manually in the topend.dl screen. You will also probably have to
set up the same parameters using the TCS NODSET command. I have no
idea why the same numbers have to be set in two different places. You will
also have to manually turn on chopping in the topend .dl screen and set the
mode to external.
|
