School's In For Summer

Project Members Learn How to Use Rubin's Engineering and Facility Database

July 14, 2020 - Even though Rubin Observatory's physical construction site is closed, there's no shortage of work going on across the Project. And, with the focus on activities that can be done while staff are working from home, it's a great time to implement training that will benefit Project members in the long term. One of these training opportunities, which took place in late May and early June, educated staff from across the Project about a resource many of them will need to access and interact with during Commissioning and Operations: the Engineering and Facility Database (EFD). The course, titled EFD-101, was open to Project members who anticipate using the EFD for their work, and about 30 people participated virtually.

The EFD is the central repository for telemetry data from different subsystems relating to the Rubin summit facility. Telemetry streams, which combine physical or electrical data with timing data, are collected from many sources, both physically on the summit and elsewhere, and streamed to the EFD. Although it's not technologically feasible to include data from every Rubin component in the EFD, the goal is to capture as much useful data as possible and make it available to the scientists, engineers, and technicians who need it to make sure everything is working properly.

The EFD, powered by InfluxDB, a time-series database, will store data collected by Service Abstraction Layer (SAL), an API on top of the Project’s middleware/Data Distribution Service (DDS). The SAL, in concert with Commandable SAL Components (CSCs) pull data from various sources, including the weather station outside the summit facility, environmental sensors inside the facility, the Telescope Mount Assembly and the Dome of the main observatory, the Auxiliary Telescope, the DIMM tower, and more. With input from so many different sources the EFD must be extremely robust, as well as flexible enough to accommodate data streams that vary widely in terms of context and throughput.

The EFD is commonly referred to as a single system, but in fact there are currently two EFDs. The one on the summit continuously collects data and can operate independently without a link to the base for up to 30 days. Another EFD, located at the US Data Facility at the National Center for Supercomputing Applications (NCSA), replicates new data from the summit EFD, and then archives the data permanently.

The EFD system was a requirement of the Rubin Observatory Construction Project, and was built by the Data Management Science Quality and Reliability Engineering (SQuaRE) team led by Frossie Economou. SQuaRE's Angelo Fausti and Jonathan Sick collaborated with Data Management and Telescope and Site engineer Russell Owen to connect the system to SAL. SQuaRE also prepared training material to help users access and use the database, but the EFD-101 class was requested specifically as a way to efficiently train a lot of Project members at the same time.

Currently, the primary way Project members access and use the EFD is through Chronograf, an open-source, web-based interface. With Chronograf, users employ queries and construct dashboards from which they can view multiple contextually-related data outputs on the same screen. The images accompanying this article illustrate two sample dashboards. The image above and to the right is the state monitor for the system on March 20th, 2020; it shows which components are disabled (1), enabled (2), offline (4), and on standby (5). The other sample dashboard, at the top of the page, displays external weather station data from January 1-2, 2020.

In addition to monitoring systems using Chronograf, Project members will also be able to use EFD data with Python and Jupyter Notebooks for more in-depth analysis and troubleshooting activities. If the Auxiliary Telescope isn't tracking precisely, for example, or if an image looks unusual, Project team members can analyze detailed time-series data from the EFD to pinpoint the issue.

The EFD-101 course was already in the works when Rubin construction shut down in March; systems like the Auxiliary Telescope and the Commissioning Camera had recently come online and started feeding data into the EFD for the first time. Simon Krughoff, the SQuaRE team member who facilitated the EFD-101 course, originally planned to use current data for training activities and was excited about the amount of data beginning to populate the system. But the shutdown led to a change of plans—fortunately the EFD-101 course could still proceed using archived data from the Rubin Observatory components that were online and functioning before the summit site was closed.

Participant feedback has been very positive, both for the course and the EFD itself, which is proving to be a user-friendly and very valuable resource as Commissioning and verification activities increase. The Data Management team is now working with the Commissioning team to determine next steps, which will likely include more in-depth workshops to practice EFD data analysis.