On-Sky Again - AuxTel Observations Resume

February 8, 2021 - So far 2021 has been exciting for the Rubin Construction project; after the rollout of new safety procedures, more personnel are returning to the summit and restarting activities that have been paused since the COVID-19 construction shutdown in March 2020. 

One noteworthy success in January was the restart of on-sky observing with the Rubin Observatory Auxiliary Telescope on January 19th. This telescope, which will measure atmospheric transmission above the Rubin Observatory Simonyi Survey Telescope, is housed in a separate enclosure near the main summit facility. On-sky observations, including the collection of spectral data, originally began early in 2020 as part of the Commissioning effort for the telescope but these had to be suspended when the summit closed in March. 

During the almost ten months that the AuxTel couldn't be operated, the team's attention turned to tasks that could be done remotely. Many of these tasks involved updating the communications middleware (which benefits the entire Rubin Observatory system, including AuxTel). Other updates included bringing the Camera and Archiver systems back to using the main codebase used for the LSST Camera and the Commissioning Camera, because they had diverged during the process of getting the AuxTel system functioning on-sky. 

So what was it like to restart the system on the summit after a ten-month nap? From a hardware perspective the AuxTel was in good shape, especially considering that routine maintenance tasks had to be deferred during the shutdown. The software challenges were more significant. During the shutdown it wasn't possible to follow the usual workflow of development and deployment, in which software changes are made, tested on a test stand at the National Center for Supercomputing Applications (NCSA), and then pushed out on the summit where they ultimately go into production. But it still made sense to continue developing the software, since that was an activity that could continue while the team was working remotely. As a consequence, by the time the AuxTel started operating again, the software running on the test stand at NCSA was much more developed than what was last used on the summit. That means that instead of working with incremental upgrades, the team was integrating the AuxTel hardware with software that had changed substantially, so there were a number of bugs to work out. But since the team had anticipated this extra work, they were able to accomplish their goals in the allotted amount of time. 

The onsite observing team for the first run of 2021 included Craig Lage, Kevin Reil, and Alysha Shugart. Each night of the three-night run, observations began at about 9:30 p.m., which is when it gets dark this time of year in Chile. But there were a couple of hours of work to do before that, including opening the dome shutters to equilibrate the dome temperature with the air outside, and taking dome flat images for calibration. The observers would then go to the control room in the main Rubin facility building, and observe through the night until about 5:30 a.m. While driving the telescope, the observers communicated with remote support staff via Zoom and Slack. At the end of each night's observing shift, after returning to the AuxTel to close the dome shutters and make sure the telescope was stowed safely, the observers would head to the summit hotel to sleep—or at least try to.

The team is excited to be doing observing runs again, although circumstances are different because of increased safety protocols and travel restrictions. Team members who would usually be present in person to troubleshoot engineering issues were not able to travel to the summit site or gather in the remote observing room in Tucson, AZ. Instead, the entire remote support team worked from their homes, managing the challenges posed by having access to fewer display screens, less reliable internet connections, and, in some cases, children who weren't entirely on board with their parents evening work obligations.

Overall, the restart has been successful. The AuxTel has now achieved the same level of observatory functionality (including the telescope, dome, and camera) as it had before the shutdown, while the software is at a higher level of readiness, and the networking/middleware is far more robust. The team has been able to demonstrate scheduler-driven observing of a list of specific targets, and has taken a reasonable dataset to help with data pipeline development. 

Upcoming observing runs, which will be scheduled for about three nights every few weeks, will continue to shift from engineering to more science-driven operation. One near-term goal, planned for the next observing run, is to improve the telescope image quality and pointing at low elevations (~30 degrees above the horizon). This will allow the team to get better data from a target star near the South Celestial pole, which doesn't move across the sky; data from a star that is fixed is particularly useful for constraining the atmospheric transmission measurements. 

Congratulations to everyone who contributed to the success of the first AuxTel on-sky observing run in 2021.