UIST: Troubleshooting

CAR or Edict Error

• Mechanism errors (on uist_oper screen): datum the mechanism that has the error, set the highlight on the sequence console to the previous "set OBJECT|DARK|FLAT|ARC|whatever" and run from highlight. You DON'T need to run from load config.
• Array disabled: see below
• Edict Error: the command and/or observe status on the uist_oper screen, and/or the Edict box on the sequence console, are in "error" - because (1) array problems or (2) mechanism error. If (2) datum the mechanism (just that one) and run sequence from the "set OBJECT|DARK|FLAT|whatever", otherwise see "array disabled" error below.
• Problem loading (thermal?) waveform: UOCS reports a CAR error followed by ``OOS UIST : OOS command set completed with error 226525274''. Recover from this by:
  (1) acknowledging the error
  (2) stop ASAP
  (3) go back to "set OBJECT|DARK|FLAT|whatever" before break
  (4) re-enable array
  (5) run from highlight
  

Array disabled

There are no error messages but data frames show characteristic top left quadrant bad image:

Check the oper_screen where most likely you will see the array has disabled and the button is yellow. Try the following:

  - enable the array from the uist_oper screen
    - works: continue
    - doesn't work:
                                                                                
    - gives error (on WFG console) containing
      "output status is disabled after enable command"
      - set sequence console highlight back to previous
        "set OBJECT|DARK|FLAT|ARC|" and run from highlight. It should
        disable and enable the array by itself when it loads the waveform.
        If the array is not enabled when it stops at the break, enable the
        array.
        - works: continue
        - same error:
          - check for slave errors (need to open the slaves' serial consoles
            and look at the buffer for that)
            - errors (usually at read 0) on all slaves:
              - repeat running from "set ".
                (maybe once or twice - not endlessly)
                - if that doesn't work power cycle Edict
            - error on one slave only, usually not at read 0 but some
              random number: power cycle Edict (? I'm not sure about this one,
              but power cycling Edict should fix this.)
            (power cycle, not soft reboot. This is almost certainly a hardware
            problem and needs the power to go down to get reset. Power it
            off, wait 3-5 seconds and then power it on again.)
                                                                                
                                                                                
    - check array temperature
      - high: check for cryogenics related faults
      - normal: check for error messages on the WFG console.
        If there are error messages about the temperature not being read
        type arraySafeStop and arraySafeStart into the WFG serial console
        - works: continue
        - doesn't work: (there are two failure modes here):
          (there are two failure modes here):
          - arraySafeStart starts, but it doesn't let you enable the array:
            try this a second (and possibly a few more) time, and if that 
            doesn't work reboot the CCS and type arraySafeStop and 
            arraySafeStart into the WFG console again after the CCS is 
            done rebooting unless it recovers by itself 
            - works: continue
            - doesn't work: reboot Edict (soft reboot should do it, this is a
              communications problem)
          - arraySafeStart fails to start and exits: type k into
            the WFG console (this is vxWorks' way of doing a command line
            recall) and hit return. Repeat this a few times if necessary.
            - works: continue
            - doesn't work: reboot Edict (soft reboot should do it, this is a
              communications problem)
 

Array wont enable at start of night

You run up UIST, datum everything by going left to right through the startup seqeunce (on the uist_oper screen) with no errors. But when you click "array enable" nothing happens. The FIX: open the wfg console (found under "serial consoles") and type arraySafeStop then arraySafeStart. The array should then enable.

NewGroup Error

newGroup errors never occur by themselves and are (almost?) always caused by running from too high up in a sequence after an error has occurred.
- run the OCS down; fix underlying errors (see above: CCS-Edict communication problems, mechanism errors, disabled array); run the OCS back up.

WFG console hangs

Do one of :
- type reboot or reset in the DAQMaster console (unless that's hung too)
- type cntrlx in the DAQMaster console or into each serial console (all 10 of them)
- reset or power cycle the EDICT crate in the computer room. To reset use the reset toggle switch on the Master board which is the leftmost board at the top of the rack. To power cycle use the rocker switch below the top rack.

Aborting an unwanted/long integration

Here is how to get to the abort button and how it works:
From the uist_oper screen go to UIST edict control from the drop down menu next to the attenuation plate label. From the edict control button select the sequence button on the right edge of the screen. This brings up a menu with several buttons, one of which is abort. Hit abort.
This will abort the current observation - stop it in its tracks. Edict will be in an error state, but when a valid config is loaded it should clear the error. The observer must therefore start the next observation from "load config" in the sequence console.

Slanted spectra/Grism wheel 1 failure

From the bottom of the TSS's oper screen, under "other displays" select "CCS Engineering DM screen", which will open a second, small epics window. Here select Grism wheel 1, which will open another screen. Here set Datum Force to true and then click the green start button. This will datum the wheel first, and then set it to the position it should have been at (for your chosen grism). There is no need to repeat your calibrators.

Rebooting CCS

To pick up changes in the data files the CCS has to be rebooted. To do this bring up the CCS serial console either by typing uistMenu& or via the operators screen and the "other displays" buttons. Type reboot in the CCS serial console xterm.

To check array voltages

Bring up the WFG serial console via uistMenu& or the operators other displays buttons and type readBias. For standard (non-thermal) observing they should be:

Bias                LOW     HIGH     GAIN 
Chan         V       mA       uA       mA 
  0     -3.408               1.0            VdetCom 
  1     -4.009               1.0            VddUC
  2     -6.011               2.0            VnRow
  3     -6.006               1.0            VnCol
  4      0.000              -1.0            VssCm
  5     -1.504      0.0                     VddOut
  6     -2.510               1.0            Vref
  7     -0.005      0.0                     Vload
  8      0.000      0.0                     bias8
  9      0.000      0.0                     bias9
 10      0.000      0.0                     bias10
 11      0.000      0.0                     bias11
 12     -0.005      0.0                     bias12
 13     -0.005      0.0                     bias13
 14     -3.809               1.0            VddCL
 15     -0.005             706.1            VrowOff
 16      0.000      0.0                     VggCL_Low
 17     -6.001      0.0                     VggCL_Hi
 18     -0.005      0.0                     VrowON_Low
 19     -0.005      0.0                     VrowON_Hi
 20     -1.699     12.1                     VrstON_Low
 21    -10.000     22.0                     VrstON_Hi
 22    -10.000     22.0                     VrstOFF_Low
 23    -10.000     22.0                     VrstOFF_Hi

Wheels aren't moving

The motor drive cards are in the motor control crate in the computer room. Check power is going to the crate as indicated by the illumination of the three rocker switches. To check individual drive cards, remove the front panel and check the LEDs - green is OK, red is faulted.

Calibration unit faults

Check power is on to the UIST electronics rack on the mirror cell. In the lower part of the rack there are labelled power supplies for each lamp. The Krypton lamp is known to be intermittent.

Recovering from a power failure