Moor - European Southern Observatory

Please verify that (1) all pages are present, (2) all figures are acceptable, (3) all fonts and special characters are correct, and (4) all text and figures fit within the margin lines shown on this review document. Return to your MySPIE ToDo list and approve or disapprove this submission.

Moor: Web access to end-to-end data flow information at ESO A. M. Chavan∗ , M. Peron, J. Anwunah, T. Canavan, D. Dorigo, N. Kornweibel and F. Sogni European Southern Observatory, Karl-Schwarzschild-Str. 2, D-85748 Garching, Germany ABSTRACT All ESO Science Operations teams operate on Observing Runs, loosely defined as blocks of observing time on a specific instrument. Observing Runs are submitted as part of an Observing Proposal and executed in Service or Visitor Mode. As an Observing Run progresses through its life-cycle, more and more information gets associated to it: Referee reports, feasibility and technical evaluations, constraints, pre-observation data, science and calibration frames, etc. The Manager of Observing Runs project (Moor) will develop a system to collect operational information in a database, offer integrated access to information stored in several independent databases, and allow HTML-based navigation over the whole information set. Some Moor services are also offered as extensions to, or complemented by, existing desktop applications. Keywords: Science Operations, databases, Web

1. INTRODUCTION

1.1. Supporting operations of a world-class observatory The European Southern Observatory’s “24/7” operations model spans two continents, two hemispheres, two time zones and two observatories. Several operations groups are involved in the submission, selection and execution of hundreds of observing programmes per semester, producing terabytes of archived data that must be distributed to the community. To keep such a system running, much diversified information must be kept flowing at all times. The Observatory’s operations model evolved over time, and the information it needs found different ways into and out of the system. Some of it was bound to database storage by design, but the databases were not meant to be used together. Some information, like finding charts, was uploaded via FTP; other documents are still stored as text files, email threads, or into Problem Reporting Systems. As a result, it is not easy to follow the life-cycle of an observing programme (or, better said, its instrument-specific components called Observing Runs) from the moment it was submitted as a proposal to the moment its data is returned to the Primary Investigator (PI). Over the five years of VLT operations, day-to-day needs have been identified and addressed by the different groups involved, generally resulting in the development of databases, procedures, and Web-based applications serving rather specific purposes. This has gradually moved away from an original scheme in which information could be often found only in local disks or private directories. However, the complexity of operations and the multiple levels of interaction among the activities of the different groups managing them have made increasingly obvious the need for a more integrated approach. The first goal of the Moor is therefore to provide for consolidated storage of all operations information that is not yet centrally available, making extensive use of databases; functionalities for storing missing information will be implemented either as Web forms or as desktop applications (some of which already exist)... Providing integrated access to operations information is the Moor’s second goal. Once all operations data can be accessed in a unified and secure way, software tools can be developed to allow authorized support staff to review and alter that data any time and anywhere (that is, regardless of the physical location of the staff and the data).

∗

[email protected]; phone +49-89-32006-536

SPIE USE, V. 2 5493-8 (p.1 of 9) / Color: No / Format: A4/ AF: A4 / Date: 2004-05-25 11:48:21


Data will be presented via Web browsers as HTML documents; specialized applications will be implemented as desktop tools. Finally, we want to allow hyperlink-style navigation over the set of operations data. Where a logical connection exists, a corresponding navigation link should be available. Given an observation specification (called Observation Block [OB] in the Moor’s dictionary), for instance, one should be able see the Observing Run it is associated with. From that run, one could then navigate to the initial technical and scientific evaluations of the run; see a trace of interactions with the PI; assess its current status; list data produced by it that’s available in the Archive; etc.

1.2. Periods, Phases and Documents ESO operations are structured in Periods. A Period is nominally a semester, beginning on April 1st or October 1st: in fact, though, operations for a Period begin with the submission of proposals for that Period and end when the last data packages is delivered to its PI, spanning over one year. Within a Period one can identify specific phases, each producing or updating some document types. Phase I Observing Time Proposals are submitted, reviewed, (possibly) approved and scheduled during Phase I. Approved proposals are called Observing Programmes; Observing Runs (or simply runs) originate during this phase, as instrument-specific components of a Programme. If a Programme’s observations are executed by someone from the Principal Investigator’s team (PI), that programme is said to be Visitor Mode; observations of Service Mode programmes, on the other hand, are executed by ESO staff observers. The Moor project’s primary focus is on Service Mode. Phase II During Phase II the actual observations of Service Mode Observing Runs are prepared: Observation Blocks (often called OBs) are prepared, submitted, reviewed, and queued for execution at one of the ESO observatories in Chile (the VLT, Very Large Telescope, or La Silla). A standard-format description of the Run (called README File) and all relevant Finding Charts must be provided by the PI together with the OBs. User Support astronomers write a Run Abstract, and all interactions with the PI are recorded in the Support History. Execution Execution of Observation Blocks produces science and calibration Frames. A detailed Night Log is kept, and a first quality assessment of the resulting Frames is performed right after the observations. Note that the execution phase for a given OB may be repeated, e.g. if the quality control process reveals that the OB was not executed under the constraints specified by the PI. Post-Execution Data-Flow astronomers review the raw data Frames originating from the instrument, combine them with the most appropriate calibrations, and produce Science- and Calibration Products for the PI and the Archive; all Data-Flow operations are logged. Finally, all user-relevant data associated with an Observing Run is written by the Operations Technical Support group to removable media in the form of a Data Package, and shipped to the PI. Note that phases are not rigidly sequential, as they may overlap. Some OBs of a run could be still queued for execution while science products are being prepared for other OBs of the same run. Also, the execution phase for a given OB can be repeated several times, in which case several sets of raw data frames will be created.



1.3. Moor users Phases and documents are handled by different operations groups, each having a well defined mission within the overall operations scheme. • • • • •

The Visiting Astronomers Section (Visas), in charge of handling the proposal submission and evaluation process as well as the time allocation. The User Support Group (USG), providing assistance to the community in the preparation and definition of their programs, and also maintaining a number of reporting services both to the mountain operation groups and to the general users. The Paranal and La Silla Science Operations groups (PSO and LSO respectively), in charge of the execution of observations at the telescope The Data Flow Operations group (DFO), which carries out quality control, instrument trending, and preparation of packages containing both raw and processed data for distribution to the users. The Archive group, in charge of storing and distributing data to the users, in addition to maintaining and providing accessibility to the archive.

PIs will be able to access some of the Moor’s features as well. 1.4. Documents, presentation, navigation and views In the context of the Moor project, a Document is any data item the Moor itself can display, create or modify. An Observing Run is obviously a document, like Finding Charts, Observation Blocks, Frames, and so on. Documents are presented in different ways to different users. For instance, the Moor presents only a minimal view of an Observing Run to the PI, while the User Support Group has access to a much richer set on OR information, including its history. In general, only a subset of a document's contents is presented: in the case of a data Frame, for instance, only selected FITS headers are displayed. Navigation is an important concept that is already in extensive use in current ESO operations: for example, users can follow the progress of their runs by means of dedicated Web pages that give access to items such as the files stored in the archive or the ambient conditions database at the time that a given observation was performed. The Moor greatly extends these capabilities by allowing users to navigate from one Moor document to the next; for instance, from a run to the resulting Data Package, from there to the data Frames, and from those to the OBs. All navigation links are available from the Web, and some of them are available also from within desktop applications. A Moor View is defined as a set of tools, documents, presentations, navigation links and access rights. Different Moor user types enjoy different views. For instance, the PI view of an Observation Block differs from the User Support view in that the PI no access to the Support History. The following views will be offered by the Moor: • PI view: shows Proposals and Phase II information submitted by external PIs, who can also request support from USG. • USG view: the Moor gives the User Support Group access to a wide range of documents, to support the Phase II and Execution phases. Access to Observing Programmes is also provided, to review the scientific background of operations. • PSO view: Paranal Science Operations with the same documents as USG. A PSO-specific view displays Observing Programmes, to asses their feasibility. • DFO view: the Web pages dedicated to the Data Flow Operations group present mostly science and calibration data, for quality control purposes • OTS view: offers the Operations Technical Support group support for the preparation and shipping of Data Packages.



Fig. 1: Moor view of a set of Observing Runs (fragment of an actual page; names and ID were altered) The write icons allow operations staff to enter/edit run comments 1.4.1. What does not belong to the Moor? The Moor offers access to operational data originating from the entire Observatory, but it is not some global Enterprise Information System. The Moor does not replace existing ESO applications such as the ESO Archive; instead, it will integrate with those tools when possible, in order to duplicate interfaces and functionalities. Existing data entry tools such as P2PP (ESO’s Phase II Proposal Preparation system) are not part as such of the project either, even though the Moor may generate new requirements for them.

1.5. Document history Changes to some Moor documents, like the Observing Runs, Observation Blocks, Abstracts, etc. are logged. Logging is expressed in terms of events: • • • •

Events include a timestamp and, where available, the identity of the user performing the change. Logs are distributed to all operation sites. The collection of log entries about a specific document constitutes that document's history. A document's history is accessible from the document itself.

Note that some document types, like the Support History, are intrinsically "histories".

2. OBSERVING RUNS An Observing Run is an amount of observing time requested by, or granted to, a PI. It applies to one instrument, and it is identified by a string like 074.A-1234(B). A run is part of an Observing Programme, and is characterized by information including: • whether its observations should be performed in Visitor or Service mode;



• • • • • • •

the instrument to be used and how that should be configured; a set of general timing, like specific observing time windows, and observing constraints, like required seeing, sky transparency and lunar illumination; a list of targets; instrument-specific information, like required VLTI baselines; a set of yes/no detailed conditions, like “Chilean Programme?” or “Time critical?”; completion level; status, such as “Open”, “Completed”, or “OnHold”; and much more.

Fig. 2: Moor presentation of an Observing Run (fragment of a mockup page, USG View) Some of that information can be inferred from the original Proposal, or computed on the basis of the Observation Blocks associated to the run. Some other information, however, must be entered manually by support astronomers.

2.1. Life-cycle of the Observing Runs Service Mode runs follow a well-defined life-cycle, from the time they are created - at proposal submission time - to the time their data is shipped to the PI. The life-cycle itself can be described as a sequence of states; an Observing Run advances from one state to the next as the result of an event. Events are logical entities corresponding to actions performed on, or about, Observing Runs. The full life-cycle of an Observing Run is quite complex, but it can be neatly segmented along operations phases: Phase I, Phase II, Execution and Post-execution. The following figure shows, for instance, the state transitions for Phase II.



From Phase I

Review

Issue

RunReady Period Changed

OnHold

Issue

Open OBExecuted

Issues Clarified

Terminated RunTerminated PeriodChanged To Post-Execution

AllOBsExecuted Completed RunExecuted

To Post-Execution

Fig. 3: Observing Run state transition diagram for Phase II (draft) After Phase I, Observing Runs undergo a revision stage (state Review), during which some issues my arise; in that case, the run is put OnHold and support astronomers work with the PI to clarify them. Once all issues are cleared and Observation Blocks are prepared, the run is Open and remains in that state until all OBs are executed. After that, the Observing Run is marked Completed, a RunExecuted signal is sent to the relevant group (DFO), and the Post-Execution phase begins. Issues may arise also as a result of OB execution, in which case the run is put again OnHold until problems are solved. Incomplete Observing Runs are generally Terminated at the end of the observing Period.

Fig. 4: Web page for changing an Observing Run’s status (fragment of a mockup page)



Moor pages are available to advance an Observing Run along its life-cycle. Fig. 4 shows how a run can be put OnHold; note that every state transition must be justified (in plain text). In general, only “sane” state transitions can be applied – that is, predefined transitions like Open–Completed on an AllOBsExecuted event. However, an operator with super-user role can override the system-imposed constraints and terminate, for instance, an Observing Run while it’s still in the Review state. 2.2. Observing Run History All status changes of an Observing Run are logged; the collection of all status changes, from creation to end state, constitutes the history of a run. History entries are time-stamped; they include the transition’s justification and details about the operator advancing the run along its life-cycle.

3. OTHER DOCUMENT TYPES As explained in sec. 1.4, the Moor can display, create or modify several document types, including: • • • • • • • • •

Finding Charts: images of the field to observe, to facilitate the target acquisition phase; prepared by the PI. README Files: a semi-structured set of notes, summarizing the scientific features and constraints of an Observing Run; prepared by the PI. Observation Blocks: the individual observations to be executed; prepared by the PI. Abstracts: they summarizing operational aspects of an Observing Run; prepared by support astronomers for Science Operations. Support History: a record of all reviews of the Abstract, interactions with the PI, etc., performed by the support astronomer who’s responsible for an Observing Run. Technical Feasibility Review: a technical evaluation of a run (see an example in Fig. 5). It is provided by Science Operations. Night Logs: all night-time activities of Science Operations are logged. Details include quality control information, problem descriptions, data frames lists, etc. Frames: the FITS files created by ESO telescopes. Information about frames is stored in the observations database; the system provides navigation facilities to access that information from Moor documents. Science products and Calibration products: are generated from science and calibration frames by the Data Flow Operations group, for archiving and distribution – together with the raw Frames – to the original PI (in the form of a Data Package).

071.A-3456(B)

Some user ISAAC SM If L=16, 15 min of integration in LWS-LR will give S/N=0.04. To achieve S/N=40, as intended, in 15 min of integration, one needs a source of L=9. If V=16, this would correspond to V-L=7, which seems very unlikely. The proposal does not specify to which band the magnitudes appearing in the target list pertain, but in any case, it seems that the S/N aimed at cannot be achieved, by a wide margin.

RECOMMENDATION: Based on the information given in the proposal, this programme is not feasible and it should not be allocated time on this basis. However, if it appears strongly desirable to give time to this programme, the PI should first be contacted to clarify with him if its apparent non-feasibility results from ambiguity in the proposal or from an error in his exposure time calculations.

Fig. 5: An Observing Run’s technical evaluation



4. VIEWS AND NAVIGATION The following figure shows an overview of the end-to-end Moor document navigation scheme, from Observing Programmes to Data Packages. Arrows represent navigation links. Note that not all links are available to all Moor users, nor are all documents presented in the same way. Also, some of the logical links depicted in Fig. 6 can be followed using specialized desktop applications: for instance, the Observing Tool allows operations teams to review Observation Blocks, their History, and the associated Finding Charts.

README History

Observing Programmes

OB History

README files Observing Runs Observation Blocks

OR History

Night Logs

Frames Finding Charts Abstracts

FC History

Support History Technical Review

Abstract History

Data Packages Science Products Calibration Products

Fig. 6: The Moor’s overall navigation scheme (simplified view)

5. ARCHITECTURE AND TECHNOLOGIES The Web-specific components of the Moor are implemented an object-oriented, three-tier system.



The system’s back end is built on top of a commercial RDBM server, the same server supporting all ESO science operation. The Moor’s persistence layer – the object/relational mapping – is being developed in-house, capitalizing on the development team’s multi-year experience with related projects. (ESO’s first Java-based proposal preparation tool was released in 2000). The middle-tier is implemented in Java using Java-enabled technologies, including JDOM, Servlets, Java Server Pages, and Struts. We are using Tomcat as Servlet container; we are currently developing no J2EE components, although in the future we could consider using JBoss as application server. For compatibility with the hardware installed at the Observatory, we need to support relatively old browsers; our usage of Javascript is therefore very limited.

6. PROJECT STATUS The Moor project is in its inception phase. In addition to Observing Programmes, Observing Runs and Observation Blocks, as of observing Period 73 (April to October 2004) we are able to store Finding Charts into the database, and retrieve them using desktop tools. (Previously, Finding Charts were FTP’d to ESO; the whole batch was then printed and kept in folders close to the telescope consoles.) We plan to accommodate README Files for the next period: the files will be prepared using the same Phase II tools PIs are accustomed to use, but the review will be Web-based: effectively, the first Web-based Moor components will be put in place to support reviewing README Files. Other Moor views will be added in the following months, according to operational priorities.

7. CONCLUSIONS Operating a large astronomical observatory requires a timely flow of information among all participating groups and individuals. Centralizing information storage and offering integrated access to operations data will allow ESO staff to follow an Observing Run and all associated data in real-time along its life-cycle, from proposal submission to data distribution. The Moor will offer ESO staff end-to-end, Web-based access to several document types that were previously only accessible using ad-hoc programs on specific servers, if at all. It will integrate with, complement, and in some cases replace the software tools supporting the VLT operations model.

8. ACKNOWLEDGEMENTS The authors would like to thank two ESO colleagues: David Silva, for being the initial driving force for the Moor project and developing many of the concepts presented here; and Fernando Comerón, who carefully reviewed a draft of this paper and provided corrections and constructive criticism.
