Filter drum report

Summary
The SCUBA Filter Drum is currently stuck in one position only - that holding the new wideband 450w:850w filters. The next attempt to repair the drum will be made during August/September 2000.

Details

Pre-July 1999
Prior to the July Upgrades programme the SCUBA filter drum had been experiencing two kinds of problems. The first was a non-functioning microswitch which prevented moving the drum under computer control to position #1 (350/750 filter pair). The second was an occasional missed move - which was somewhat random, but in general happened when moving from the photometric pixel position (#7) to the 450/850 pair (#2).

9th - 14th July 1999
As part pf the SCUBA Phase I Upgrades work we decided to completely overhaul the filter drum. Mark Cliffe came out from the ATC to lead this work. The filter drum was removed from the instrument and brought down from the summit to JAC for inspection. In Hilo, operation by hand revealed that the outer drum did have a certain amount of rotational resistance and also sounded a little rough. It was decided to disassemble the drum in order to identify any mechanical faults that were present, or in the process of getting worse.

A complete strip-down revealed that there was a considerable amount of contamination within the drum. This contamination was probably due to particulates from the copper thermal cooling wicks which were rubbing on their top and bottom edges within the channel in which they are housed, a gasket film is in place top and bottom to keep this to an absolute minimum. A certain amount of rubbing is however unavoidable due to the nature of the design of the filter drum.

Contamination was also present due to particles of molybdenum-type dry lubrication migrating out of the bearings that the outer filter ring rotates on. All the components of the filter drum were carefully cleaned with alcohol to remove all traces of contamination with particular attention being paid to the thermal cooling wicks. New cleaned bearings with vespel cages and no dry lubrication were fitted into new bearing rollers and reassembled into their roller units. A set of cleaned micro-switches were put into place as a precaution in case they were at all contaminated. The filter drum was then carefully and cleanly reassembled in preparation for testing. The re-assembly of the filter drum was watched by Chris Yamasaki and partly by Vernon DeMattos. Once the filter drum was assembled, checks were carried out to ensure that micro-switches actuated in their correct positions. The reassembled drum certainly felt smoother after the maintenance was carried out.

15th July 1999
The filter drum was taken to the summit to be attached to the instrument for testing. The drive shaft was also examined and found to be okay. There is a certain amount of resistance but that is to be expected due to the vacuum feed through and heat sinking assemblies. The filter drum was fitted to the instrument ensuring the correct alignment of the drive coupling. The drum was then turned manually a few times to check movement using the handwheel on the top of the cryostat. Testing was carried out using drive software (SCUMOMO) to rotate the drum to it's end-stops and then random filter positions, during this time no mechanical problems arose. Comprehensive testing with software will be carried out after some cabling maintenance has been done. Filter drum was then removed to await the fitting of filters and subsequent re-assembly of the cryostat.

29th July 1999
Following the completion of the Upgrades programme, the drum worked flawlessly.

8th November 1999
The first reported problems occurred during the week of the 8th November. Most of the faults were in moving to (or from) the photometric pixel position (#7), although eventually almost any motion of the drum was generating errors. Reducing the acceleration and velocity helped to select the 350/750 pair, but by the 16th Nov it was clear that the resistance to motion was becoming worse. By the 16th no motion was possible under computer control.

15th November
A slight red-herring occurred on 15th when instead of giving position failures the error, "Scuhp:error opening MUX tree switch" was seen. IAS swapped out various boards in the HP, cpu, mux, digital I/O, but the symptoms persisted. In the end it was realised that the filter drum was stuck hard, at least as far as the stepper motor was concerned. This meant that each attempt to move the filter drum timed out. It looked as though a bell character, ^G, is sent to the HP when a time out occurs. This causes the HP to complain back to the vax, and the vax thinks that the HP has been power cycled, and then reloads the code(!). This circle of events was happening for each requested move of the filter drum. The drum was manually moved to position 2 for observing on the 16th. IAS disconnected the cable to the filter drum stepper motor, and modified the SCUHP code so that it pretends to have moved the filter drum without actually trying. This allows the observing queue to proceed without problems, though with the restriction that only filter drum #2 is used.

At this stage it was believed that the filter drum could be moved to other positions by hand. One person turning the wheel, and another watching the status display on the front of the HP, even though the filter drum was presenting a large load which is fairly uniform from end to end.

24th November 1999
The SCUBA filter drum suffered a major failure. We were trying to establish whether moving the drum by hand (say, to the photometric pixel position) would be a relatively straightforward process. Initial moves of the drum were successful, and although there was resistance to movement, every filter could be selected. However, after some other interim tests, we found that the drum was binding so much that the clutch mechanism was slipping and failing to rotate the drum. Having allowed a 20 min break we tried again but to no avail. The best diagnosis is that the copper wicks have become tangled (maybe some strands have even broken), causing the drum to bind in certain positions. Most worrying is that the situation had obviously become worse after the initial moves. At that time SCUBA was declared non-operational. The drum was at the CCW endstop and despite considerable effort we could not move to any useful filter combination. Applying more torque to the drive shaft might be possible, but it is contained within a vacuum seal. Warming the drum has also been suggested, but if the diagnosis is correct this is unlikely to help.... We may be forced with another warmup situation and have to carry out an overhaul of the drum (we may not even have sufficient spares for this e.g. copper wicks).

25th November 1999
A telecon was held on Thanksgiving day between the JAC and ATC. It was decided that there was no other course of action but to warm up the system. The following options emerged from the meeting and subsequent discussions (in order of complexity/minimising lost time - easiest first):

1. The simplest option is that the drum will move when the instrument is warm. If it does move we have agreed to move the drum to position #5 (wide band 450/850 filter pair) and pump-out and cool back down again.

2. If the drum is still stuck, one suggestion to avoid the need to open up is to disassemble the stepper motor and clutch mechanism on top of the cryostat. We would then try and move the drum with a torque wrench as far as position 5. Obviously, this is risky and may not even be practical - the drive shaft is fibreglass and any excess force could cause it to break. However, it is worth considering, because it is a relatively quick way of getting the instrument back on-line, without the necessity to open up the cryostat. Of course, as in (1) we don't solve the problem, but at least (for the time being) we minimise down-time, and can schedule a repair slot some time in the future.

3. If the above options don't work or are infeasible, then we have no option but to open up. Another suggestion is that we only disassemble the cryostat as far as the drum, and then try to move it by hand to position 5. This should be possible, but we would have to be careful not to put too much stress on the drum, and be wary of the delicate filters. As in (1) and (2) we don't solve the problem, but we avoid the need to take out the ribbon cables and arrays (and the associated problems that this work may cause).

4. We fix the problem by removing the drum completely. This would require a complete disassembly (more time and effort) and a close inspection and repair of the drum. It should be noted that we don't have much experience in this area, and would have to ask for considerable assistance (via videocom?) from the ATC. If the copper strap is the likely problem we have a spare that could be introduced. This is obviously risky, as we have to remove the arrays and cables - bearing in mind we have lost pixels the two previous occasions we have done this. However, this will have to be done at some time and it should cure the problem - if we are confident about doing the work - and will give us back the multi-wavelength capability. If this is the chosen option then we will also be faced with the questions of whether we put a spare LW array cable in (to fix the 2mm pixel), and whether we replace the indium O-ring seals.

29th November 1999
A meeting was held in Hilo to discuss the course of action for the day. The 4-option list given above was discussed and agreed. Upon arrival at the telescope it was found that the drum moved freely by hand. It was rotated to position #5, and the pump-out procedure started. Having obtained position #5, suitable for the majority of SCUBA programmes, we opted not to move the drum for further tests (despite great temptation otherwise from an engineering perspective). The plan now will be to pump down and then not move again until a suitable engineering period in the (distant) future. This is a very cautious approach, but warranted given the priority of SCUBA operations and shortage of effort at this time of year. The motor is still unplugged and computer control is deactivated.

Given that the shaft moved freely (about as much torque by hand as normal) when warm points either to a mechanical clearance having opened up, or that the problem may have been ice related. A mechanical fault could include tight bearings, wick failure, debris in the wick rollers or a buckled microswitch. Ice may have been present in any of the above, or in the drive shaft bearing or seal interfaces.

It was recalled yesterday that there was a near loss of vacuum on 22nd November (due to vacuum pump power failure because of a circuit breaker problem). There was speculation that since the main cryostat vacuum had risen to 10e-3 mbar ice may have formed (or migrated) at this time. (However, note we did have problems with the drum before the loss of vacuum). This may have produced an increase in torque to just below the slipping clutch threshold level, and if there is any variability of the torque to drive the drum (e.g. slight eccentricities in rolling diameters, wind up of the wick etc.) then that may explain the phenomena of failure in one or two places only. The phenomena on 25th Nov where it was apparently sticking at a microswitch bump may have been related to just the extra torque to push over this. Adjusting or bypassing the slipping clutch may have overcome this, but on inspection we decided that access to this unit is just too hard on the crowded top plate anyway.

The only two small concerns remaining for this cooldown are that, (1) if some wick damage has occurred we may have problems getting down to temperature, or that (2) the filters may shift slightly on cooldown, and then not be moveable. Neither of these is felt very likely at the moment.

As to the real problem underlying all this - investigative tests would be performed prior to a scheduled engineering (sometime next year): if the drum works perfectly them we might conclude ice was our problem and not go for a full filter drum strip down at that point. Other possibilities for future engineeing work have been suggested as the installation of a polypropylene cryostat window, repair for the 2mm pixel, and the infamous indium seal.

4th December 1999
SCUBA has reached base temperature and the system is currently operating in a wide-band 450/850 micron filter pairing. No attempt will be made to move the drum for the forseeable future...

07 Mar 2000
During a warmup, the latest attempt to free or operate the drum meets with no success.

Effects on the observing programme
Until the planned repairs to the filter drum in Aug/Sep 2000, only wideband 450 & 850 micron work is possible, although, in anticipation of success, proposals to use other filters are being accepted for the latter part of Semester 00b.