Introduction
Purpose
The purpose of the present document is to provide an installation guide and to describe the mode of operation of the Auxiliary Data Conversion System Next Generation (ADCSng) and a summary of its main functionalities. After reading this document the user should be able to install, work with and fully understand the software package.
Running ADCSng in a nutshell
-
Obtain ADCSng from BitBucket: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng/
-
Install from top level directory with "pip3 install -e ."
-
Generate an ADCSng configuration file as from adcsng/adcsng/tests/bc/bepicolombo.json
-
Generate the appropriate directory structure as derived from the configuration file
-
Obtain and install your SPICE Kernel Dataset: https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS
-
Run with "adcsng yourconfig.json"
Overview
Auxiliary data are those that help scientists and engineers to determine the location and orientation of the spacecraft, when and how an instrument was acquiring scientific data. These data also help to determine what other relevant events were occurring on the spacecraft or ground that might affect the interpretation of the scientific observation or the S/C systems performance. Software applications are required to know what were the location, size, shape and orientation of the observed target in addition to these auxiliary data. Almost all NASA and ESA planetary missions have embraced the use of the SPICE system for ancillary data archiving and for science data analysis. Although the Flight Dynamics Division provides software to read the position and orientation files of the orbiters, most of the Principal Investigators (PI) have pointed out their interest in having all the auxiliary data distributed in SPICE format.
The ESA SPICE Service (ESS) of the Data Science and Archive Division (SCI-SA) is responsible for supporting the ESOC auxiliary data transformation into SPICE kernels and the distribution of these files to the instrument teams using the public network. NASA’s Planetary Data System discipline node NAIF (Navigational Ancillary Information Facility) is responsible for designing a transformation software tool to generate spacecraft orbit and attitude SPICE kernels. The ESS, helped by the individual instrument and FDy teams, the SGSs, and, in some missions, supported by NAIF, design and support all the necessary instrument, spacecraft and timing kernels.
Applicable and Reference Documents
ADCSng is based on different technologies, in particular most of the code is written in Python. FORTRAN is also used along with other technologies such as JSON, being familiar with those technology will certainly facilitate the usage of ADCSng.
Applicable Documents
ID |
Title |
Reference |
AD-E40 |
ECSS-E40 Space Software Engineering |
ECSS-E-ST-40C |
AD-PYP |
Packaging and Distributing Projects |
https://packaging.python.org/tutorials/distributing-packages/ |
AD-JSN |
Introducing JSON |
|
AD-BBT |
Bitbucket |
|
AD-SPI |
The SPICE Toolkit |
Reference Documents
ID |
Title |
Reference |
RD-ADC |
Auxiliary Data Conversion into SPICE Kernel And Distribution |
MEX-EST-PL-10210 |
RD-ANG |
Auxiliary Data Conversion into SPICE Kernels for ESA Solar System Exploration Missions |
SPICE-CROSS-TN-001 |
RD-ECM |
SPICE-CROSS-TN-002 |
|
RD-PIP |
Configuring a .pypirc File for Easier Python Packaging |
Terms, Definitions and Abbreviated Terms
Acronym |
Name |
OEM |
Orbit Ephemeris Message |
AEM |
Attitude Ephemeris Message |
ADCSng |
Auxiliary Data Conversion System next generation |
TCP |
Time Correlation Packets |
JSON |
JavaScript Object Notation |
EM16 |
ExoMars2016 |
BC |
BepiColombo |
ESS |
ESA SPICE Service |
PI |
Principal Investigator Team |
SOLO |
Solar Orbiter |
ESOC |
European Space Operations Center |
NAIF |
Navigational Ancillary Information Facility |
JUICE |
Jupiter Icy Moons Explorer |
MEX |
Mars Express |
FDy |
Flight Dynamics |
S/C |
Spacercaft |
SGS |
Science Ground Segment |
SOC |
Science Operations Center |
OEM |
Orbit Ephemeris Message |
AEM |
Attitude Ephemeris Message |
SKD |
SPICE Kernel Data Set |
MK |
Meta-kernel |
SCLK |
Spacecraft Clock Kernel |
CK |
Camera-Matrix Kernel |
SPICE |
Spacecraft Planets Instruments C-Camera and Events |
Software Overview
ADCSng together with some other scripts and pieces of software cover from the automated download of orbital, attitude and spacecraft (S/C) and Payload elements position and orientation data from the given mission Science Ground Segment (SGS) uplink and downlink systems as provided by FDy in the shape of Orbit Ephemeris Messages (OEM), Attitude Ephemeris Messages (AEM) -or another Orbit and Attitude information container files- and Raw Housekeeping Telemetry data processed following the NASA Planetary Data System Standard (PDS4). More information on its concept and design is available here: [RD-ANG]
Installation
Hardware and Software requirements
ADCSng is a Python 3.6.x package that uses FORTRAN 77 and FORTRAN 90 applications and a set of standard Python libraries. ADCSng is meant to interact with different nodes: SGS Uplink and Downlink sub-systems, the ESA SPICE FTP Server and the NAIF FTP Server.
ADCsng is designed to work with UNIX systems; namely with Linux and with MacOS. The hardware requirements are minimal and any modern computer/server will suffice. The software is able to run for example in a minimal CentOS 7 Virtual Machine:
-
Architecture: x86_64 (64 bits)
-
Memory: 1GB/logical CPU
-
Disk space: 10GB
Nevertheless it is recommended to allocate more resources for ADCSng to run appropriately, the "Regression Tests" section includes further information on resource allocation and expected performances.
The software requirements are the following:
-
Python 3.6 or higher
-
Python packages:
-
numpy >= 1.8.0
-
pytest >= 2.9.0
-
pandas > 0.17.1
-
html-testRunner
-
-
gcc
-
libgfortran
-
Git (possibly, for the obtention of SPICE Kernel Datasets)
Please note that the FORTRAN applications are already compiled and therefore the user should not compile and link them. Nevertheless this is possible since the source code is distributed with ADCSng, the only remaining part would be to link them with the SPICELIB library of the SPICE Toolkit. The compilation of these applications is out of the scope of this SUM and any interested user is encourage to try by himself or to contact the author.
Obtaining the Software
Installing from the ESS Nexus repository with PiPy
ADCSng can be installed directly from PiPy without the need to download the source code, in order to do so, you need to configure the given PIP installation by editing your local Python Package Index configuration file. More information in RD-PIP.
The file is accessible with: more ~/.pypirc what follows is the current configuration for ADCSng:
index-servers = nexus-master nexus-develop
[nexus-master]
repository = https:
//repos.esac.esa.int/repository/pypi-spice-releases/
username = spicedep
password = Esacvilspa1!
[nexus-develop]
repository = https:
//repos.esac.esa.int/repository/pypi-spice-snapshots/
username = spicedep
password = Esacvilspa1!
Installing the source code
The source code of ADCSng is under configuration control with Git the BitBucket repository that hosts it is the following: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng/browse, since this is a private repository access might not be granted, if so please contact the author of this document. Alternatively the source code can be obtained under request from esa_spice@sciops.esa.int.
If the user is to install ADCSng from its source code it is recommended to use PiPy to do so (taking advantage of the Python setup file), in order to do so using the Terminal go to the top level adcsng directory, adcsng and type:
pip3
install
[--user] -e .
Please note that the --user parameter is optional; if you are using your user Python installation it needs to be indicated.
Once installed, you will be able to run ADCSng anywhere in your computer.
Source Code structure
A brief overview of the code structure for ADCSng is useful for its installation, configuration and usage:
ADCSng Package top level structure |
||||
LICENSE |
|
License file (as per any other Python package). |
||
MANIFEST.in |
Manifest file (as per any other Python package). |
|||
README.md |
Readme file (as per any other Python package). |
|||
setup.cfg |
Configuration for Python Setup. |
|||
setup.py |
Setup Python routine. |
|||
adcsng |
Package directory. |
|||
|
command_line.py |
|
Main function called when invoking ADCSng. |
|
|
classes |
Directory that contains Python source code classes. |
||
|
config |
Directory that contains the mission-specific developer configuration items for ADCSng. |
||
|
core |
Directory that contains Python source code core functions. |
||
|
etc |
Directory that contains different templates used by ADCSng (mostly the ones used by the FORTRAN utilities). |
||
|
exe |
Directory that contains Linux and MacOS FORTRAN utilities executables. |
||
|
src |
Directory that contains the FORTRAN utilities source code. |
||
|
utils |
Directory that contains Python source code utility functions. |
||
|
tests |
Directory that contains the tests performed for ADCSng. |
||
|
|
bc |
|
Directory that contains the test files for BepiColombo (shown as example). |
|
|
|
bepicolombo.json |
User configuration file for BepiColombo testes (shown as example). |
Configuration
The next step to use ADCSng is to configure it. ADCSng has two levels of configuration:
-
User configuration
-
Developer configuration (placed under the adcsng/adcsng/config directory)
User Configuration
As a user you only need to take care of the user configuration. The user configuration has to be edited and prepared by the user, it consists of a JSON file and an example from the test directory of ADCSng can be used as baseline (adcsng/adcsng/tests/bc/bepicolombo.json). Any name can be provided to the configuration file, the file extension has to be *.json. The configuration file will determine the directory structure that you need to setup for ADCSng. The recommendation is to create a directory out of the ADCSng directory structure and to create one per intended ADCSng execution/project.
The following table describes all the available configuration items (please note that all the items are strings):
Key |
Value(s) |
Description |
Required? |
name |
ADCSng Configuration File |
Description of the JSON file |
NO |
mission |
|
Name of the mission, has to be one of the provided in the list. (corresponds to the name of the SKDs replacing the dashes). |
YES |
operational_host |
It is recommended to obtain the operational host by running the hostname command in your terminal. |
Name of the operational Host. If you use multiple environment and wish to distinguish in between test/development environments from the operational environment, this will allow you to receive notifications with an indication of whether if the run has been done with the operational pipeline or not. |
YES |
source_dir |
Directory path for the input files. |
Directory path for the input files. This directory is the one that ADCSng will look for input. |
YES |
temp_dir |
Directory path for ADCSng temporary directory. |
Directory path for the where ADCSng will place temporary files. WARNING: DO NOT PUT ANY FILE IN THIS DIRECTORY: It will be deleted by ADCsng. |
YES |
log_dir |
Directory path for ADSCng log files. |
More information on log files below. |
YES |
kernels_dir |
Directory path for the SPICE Kernel Dataset of the mission to be executed. |
This has to point to the kernels directory of the SDK. More information below. |
YES |
output_dir |
Directory path for the output of ADCSng. |
Directory where the output will be placed. WARNING: It is HIGHLY RECOMMENDED that this path is the same as kernels_dir. More information below. |
YES |
processed_dir |
Directory path to place the used and processed input. |
More information below. |
OPTIONAL |
log_rotation |
|
This key indicates the log rotation. The log will be split on a daily, weekly or monthly basis. More information below. If not indicated the default will be Monthly. |
OPTIONAL |
daily_log |
Directory path to place an extra log generated per pipeline execution. |
More information below. |
OPTIONAL |
lock_file |
Path and name of the lock file. |
More information below. |
OPTIONAL |
mkgen |
|
Parameter that indicates whether if ADCSng generates a meta-kernel for the run or not or if it is only executed to generate a meta-kernel (and ignore the input). |
YES |
email/send_email |
|
Parameter that indicates whether if e-mails will be sent for clients after an execution and for the developer for logs and for failed executions. More information below. |
YES |
email/clients |
|
E-mail list for clients to receive notifications of the pipeline execution. More info below. |
YES |
email/developer |
|
E-mail list for developers to receive notifications of the pipeline execution. More info below. |
YES |
email/git_repository |
HTTP(S) of the Git repository that contains the kernels |
Free text to introduce the HTTP(S) address of the Git repository for the SKD. |
YES |
email/ftp_address |
HTTP(S) of the FTP repository that contains the kernels |
Free text to introduce the HTTP(S) address of the FTP repository for the SKD. |
YES |
email/skd_name |
Name of the Git repository for the given mission. |
Free text to introduce the name of the SPICE Kernel Dataset as in: https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS |
YES |
email/mission |
Name of the FTP directory for the given mission. |
Free text to introduce the name of the SPICE Kernel Dataset as in: ftp://spiftp.esac.esa.int/data/SPICE/ |
YES |
email/contact |
Contact name and/or email |
Free text to introduce the Contact information for clients. |
YES |
Once the configuration file has been generated it can be placed anywhere in your computer, it is recommended to put it in the same directory level as where you will define the directory structure of your ADCSng configuration. What follows is an example of an ADCSng configuration file:
{
"name"
:
"ADCSng Configuration File"
,
"mission"
:
"BEPICOLOMBO"
,
"operational_host"
:
"ESAC1102875"
,
"source_dir"
:
"/Users/esaspice/bc/incoming/"
,
"temp_dir"
:
"/Users/esaspice/bc/temp"
,
"log_dir"
:
"/Users/esaspice/bc/log"
,
"lock_file"
:
"/Users/esaspice/bc/adcsng.lock"
,
"kernels_dir"
:
"/Users/esaspice/SPICE/BEPICOLOMBO/kernels"
,
"output_dir"
:
"/Users/esaspice/SPICE/BEPICOLOMBO/kernels"
,
"processed_dir"
:
"/Users/esaspice/ANCDR/BEPICOLOMBO"
,
"log_rotation"
:
"daily"
,
"daily_log"
:
"True"
,
"mkgen"
:
"True"
,
"email"
: [{
"send_email"
:
"True"
,
"clients"
:
"mcosta@sciops.esa.int"
,
"developer"
:
"mcosta@sciops.esa.int"
,
"mission"
:
"BepiColombo"
,
"skd_name"
:
"BEPICOLOMBO"
,
"ftp_address"
:
"ftp://spiftp.esac.esa.int/data/SPICE/"
,
"git_repository"
:
"https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS/repos/"
,
"contact"
:
" Marc Costa\n mcosta@sciops.esa.int"
}]
}
If we consider the previous configuration file we could have a directory structure as follows:
bc
|-- adcsng.json
|-- incoming
|-- kernels -> ../ftp/data/SPICE/BEPICOLOMBO/kernels/
|-- log
| |-- adcsng_weekly_2019Oct04.log
| '-- adcsng_weekly_2019Oct09.log
|-- misc -> ../ftp/data/SPICE/BEPICOLOMBO/misc/
|-- processed -> ../ftp/data/ANCDR/BEPICOLOMBO/
'-- temp
Remember that you need to generate this directory structure for ADCSng will not generate it for you.
Developer configuration
The developer configuration should not be modified by the user. Further details on the developer configuration can be found on RD-ANG.
Kernels and Output directories requirements
Regardless of the kernels/output directory structure you setup the following subdirectories for each path are mandatory in order to execute ADCSng:
-
kernels: 'ck', 'spk', 'lsk', 'sclk', 'pck', 'ik', 'fk' and 'mk'
-
output: 'ck', 'spk', 'sclk', 'mk'
If any of those kernels sub-directories is not present ADCSng will trigger an error and the execution will fail.
You need a working SPICE Kernel Dataset to run ADCSng
It is important to note that you need a local copy of a working SKD in order to execute ADCSng. This document does not intend to provide you a detailed explanation on how to obtain the latest version of a given SKD, instead we recommend you to read the information in the following page: https://www.cosmos.esa.int/web/spice/data
What is important to note though is that you will need to have a working SKD. In order to check that you do we recommend you to use a SPICE utility such as BRIEF on the MK in order to ensure that the SKD is properly installed.
ADCSng needs the SKD as an input and preferably also output directory for the execution: the idea is that ADCSng updates the SKD itself and therefore it should be considered as both an input and an output. That is why in general is recommended to set the kernel and the output directories with the same value.
If you still consider to have different directories please note that the SCLK that you generate with an ADCSng run will not be taken into consideration to generate the rest of the CK kernels; instead the SCLK in the kernel directory will be used. The same issue will happen with the versioning of the kernels you generate: if the same kernel is generated the version will not be derived from the original SKD.
Processed directory requirements
The processed directory should have a given sub-directory structure. This structure is determined by the 'originators' of each input file. ADCSng will generate these sub-directories if they do not exist. The 'originator' acronyms can be found in the developer configuration. In general they are the following:
-
man: Files coming from Mission Analysis (typically test trajectories)
-
fdy: Files coming from Flight Dynamics (typically OEMs and AEMs)
-
soc: Files coming from the Science Operations Center (typically planning AEMs)
-
hkt: Files from the Housekeeping Telemetry typically processed by the SOC Downlink group
-
tcp: Time Correlation Packets coming from the S/C
-
pit: Files from the PI teams
Please note that the files under these directories are usually published in the ESA SPICE FTP. An example can be seen in the following link: ftp://spiftp.esac.esa.int/data/ANCDR/BEPICOLOMBO/
If the 'processed_dir' is not specified in the condiguration file (is left empty) then the processed files will simply not be copied (this is the case for ExoMarsRSP for instance).
Executing ADCSng
In order to execute ADCSng you only need to type the command: adcsng, the command usage is as follows:
adcsng [-h] [-v] [-m] [-l] [-d] [-e] CONFIG [CONFIG ...]
Indicating the configuration file path is mandatory, the options are displayed by the -h or --help options:
___ ___ _________ Auxiliary Data Conversion System next generation
/ _ | / _ \/ ___/ __/__ ___ _
/ __ |/
// / /___\ \/ _ \/ _ `/ a command-line utility program that
/_/ |_/____/\___/___/_
//_/\_, / converts auxiliary data into SPICE
/___/ Kernels
for
the European Space Agency
v2.
5.0
positional arguments:
CONFIG Mission specific JSON configuration file
optional arguments:
-h, --help show
this
help message and exit
-v, --version Display the version of ADCSng
-m, --metakernel Execute ADCSng to generate meta-kernel only
-l, --log Prompt log during execution
-d, --debug Run in debug mode; run files not cleaned-up, etc.
-e, --noemail Do not send execution notification e-mail to users.
Source and documentation is available here:
https:
//repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng
developed and maintained by the
__ __ __ __ __ __ ___ __ ___ __ __ ___
/__\ /__` '__\ /__` |__) | / ` |__ /__` |__ |__) \ / | / ` |__
\__, .__/ \__/ .__/ | | \__, |___ .__/ |___ | \ \/ | \__, |___
esa_spice
@sciops
.esa.
int
for
Mars-Express
// ExoMars2016 // ExoMarsRSP
http:
//spice.esac.esa.int BepiColombo // Solar Orbiter
Please note that if you do not use the optional argument -l or --log you will not be prompted the log of the execution. Error messages prior to the log creation might be shown though.
Once ADCSng is executed, in a nutshell it checks the incoming directory for input data. If present it processes the input data and generates the corresponding kernels (if no input data is present but ADCSng is set to generate a meta-kernel it generates it) and then moves the generated kernels to the output directory and the input and processed data to the processed directory. Finally if specified ADCSng notifies the users and developers with an e-mail.
Please note that the synchronisation with BitBucket and other services such as the SKD validation are performed by other packages of the ESA SPICE Service (such as spigit and spival).
ADCSng internal executables
ADCSng has a number of internal executables that may be used during the execution depending on the ADCSng module that is used. These executables are compiled and built FORTRAN applications and are provided as part of the ADCSng package for the working environments: PC Linux 64 bits and Mac Intel OSX 64 bits. The ADCSng package also includes the source files to compile these executables in any working environment (The SPICE Toolkit library for that given environment is required). More information on how to install SPICE for different environments is available in AD-SPI.
Please note that these executables can be used independently of ADCSng if desired. The list of executables is provided by the following table:
Executable |
Description |
Originator |
User Guide |
aem2ck |
AEM2CK is a command-line utility program that converts AEM files to CK files. |
NAIF |
Not available. |
brief |
BRIEF is a command-line utility program that displays a contents and time coverage summary for one or more binary SPK or binary PCK files. |
NAIF |
|
ck5to6 |
CK5TO6 is a command-line utility program that converts Type 5 CKs to Type 6 Cks. |
NAIF |
Not avaiable. |
ckbrief |
CKBRIEF is a command-line utility program that displays a contents and time coverage summary for one or more binary CK files. |
NAIF |
|
commnt |
COMMNT is a command-line program that reads, adds, extracts, or deletes comments from SPICE binary kernel files. |
NAIF |
|
dafcat |
DAFCAT is a command-line utility program that concatenates together SPICE DAF files of the same type. |
NAIF |
|
dafmod |
DAFMOD is an interactive program that modifies various segment attributes of a DAF based kernel -- SPK, CK, or binary PCK. |
NAIF |
|
delutc |
DELUTC is a command-line utility that provides the number of leapseconds for given UTC date. |
ESS |
Not available |
diffsclk |
DIFFSCLK is a command-line utility that provides the difference between to SCLKs. |
NAIF |
Not available |
frmdiff |
FRMDIFF is a program that samples orientation of a reference frame known to SPICE or computes differences between orientations of two reference frames known to SPICE, and either displays this orientation or these differences, or shows statistics about it or them. |
NAIF |
|
makclk |
MAKCLK is a program that converts a SCLKSCET (also known as SCLKvSCET) file to a SPICE spacecraft clock (SCLK) kernel file. |
NAIF |
|
mex2ker |
MEX2KER is a command-line utility program that converts OASW attitude and trajectory files (ESA Flight Dynamics Legacy format, smilar to OEM and AEM v1) to CK or SPK files. |
NAIF |
Not available |
mkspk |
MKSPK is a program that creates an SPK file from a text file containing trajectory information. |
NAIF |
|
msopck |
MSOPCK is a command-line program that converts attitude data provided in a text file as UTC, SCLK, or ET-tagged quaternions, Euler angles, or matrices, optionally accompanied by angular velocities, into a type 1, 2, or 3 SPICE C-kernel. |
NAIF |
|
oem2spk |
OEM2SPK is a utility program that converts a CCSDS ``Orbit Ephemeris Message'' text file (also referred to as an ``OEM'' or ``OEM file'') to a binary SPICE SPK file containing type 9 or type 13 segments. |
NAIF |
|
orbnum |
The ORBNUM program creates an ASCII orbit number table file for an object, keyed on the time of a geometric event. |
NAIF |
|
prediCkt |
prediCkt is a program that creates C-kernels (usually predicted C-kernels) for orientations that can be described by a set of simple geometric descriptions. |
NAIF |
|
spk13to19 |
SPK13TO19 is a command-line utility program that converts Type 13 SPKs to Type 19 SPKs. |
NAIF |
Not available. |
spk18to19 |
SPK18TO19 is a command-line utility program that converts Type 18 SPKs to Type 19 SPKs. |
NAIF |
Not available. |
spkdiff |
SPKDIFF provides means for comparing the trajectories of two bodies or sampling the trajectory of a single body using data from SPICE kernels. |
NAIF |
|
tcp2scet |
TCP2SCET is a program converts ESOC Time Correlation Packet (TCP) file |
NAIF |
tcp2scet.ug
|
tcpescan |
TCPESCAN is a command-line utility that converts Time Correlation Packets in binary format to text format. Used by ExoMars2016. |
EM16 SOC |
Not available. |
tcpescan_bc |
TCPESCAN_BC is a command-line utility that converts Time Correlation Packets in binary format to text format for BepiColombo MPO. |
BC SOC |
Not available. |
tcpescan_emrsp |
TCPESCAN_BC is a command-line utility that converts Time Correlation Packets in binary format to text format for the ExoMars RSP Rover |
ESS |
Not available. |
tdb2utc |
TDB2UTC is a command-line utility that converts a data in ephemeris time to a UTC date. |
ESS |
Not available. |
utc2tdb |
UTC2TDB is a command-line utility that converts a data in UTC calendar format into an ephemeris time. |
ESS |
Not available. |
Please note that all the available User Guides can be found here: http://naif.jpl.nasa.gov/pub/naif/utilities/PC_Linux_64bit/
Logging
ADCSng provides multiple options for the logging process, the logging is provided via:
-
Log file with a specified rotation written in a specified directory. Filename is adcsng_rotation_YYYYMonDD.log where:
-
rotation is 'monthly', 'weekly' or daily,
-
YYYY is the year, Mon is the three letter capitalised month acronym and DD is the day with two digits.
-
-
Log file per execution written in a specified directory. Filename is adcsng_YYYYMMDDTHHMMSS.eop where:
-
YYYY is year, MM month, DD day, HH hour, MM minute, SS second
-
eop stands for End of Process.
-
-
E-mail containing the log file text per execution
-
Prompted Log in the screen if -l or --log optional argument is provided
Log format
The log format per line is as follows (Python syntax):
'%(asctime)s.%(msecs)03d %(levelname)-8s %(module)-16s %(message)s'
This translates into the following columns: Time | Information Level | Module logging | Log message (please note that prompted log only includes the Log message).
There are four information levels:
-
INFO : Log information of all kinds, nothing to worry about.
-
WARNING : A foreseen/predictable action by the pipeline (input processing, present files) nothing to worry about but sometimes worth checking.
-
ERROR : Something which should not have happened and that the developer might want to look into but that did not interrupt the execution.
-
CRITICAL : Something bad which interrupted the execution and that will most probably translate into human intervention either from the user or a bug-fix from the developer.
What follows is a log example:
2019
-
10
-09T03:
09
:
52.876
INFO director ================================================================================
2019
-
10
-09T03:
09
:
52.877
INFO director ADCSng v0.
13.2
for
Solar_Orbiter run on ESAC01102875 at
2019
-
10
-
09
03
:
09
:
52.8
2019
-
10
-09T03:
09
:
52.877
INFO director ================================================================================
2019
-
10
-09T03:
09
:
52.877
INFO director
2019
-
10
-09T03:
09
:
52.917
WARNING director MK solo_ANC_soc-pred-mk_V101_20191009_001.tm has not been updated
2019
-
10
-09T03:
09
:
52.955
WARNING director MK solo_ANC_soc-flown-mk_V101_20191009_001.tm has not been updated
2019
-
10
-09T03:
09
:
52.957
INFO director - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2019
-
10
-09T03:
09
:
52.957
INFO director No kernels have been generated.
Sending emails
If properly configured ADCSng will send e-mail notifications to users and developers. In order to be able to use it Python needs to connect to the following SMTP server: smtp.sciops.esa.int, the user name and the password are hard-coded in ADCSng and the sender will be esa_spice@sciops.esa.int, so far allowing for an alternative server is not considered. ADCSng will then send notifications in the following cases:
-
To the specified users after a successful (or partially successful) run, the e-mail template is available here: adcsng/adcsng/etc/email.commnt
-
To the developer after a successful run with the log of the run.
-
To the developer in case of a failed run with the log (if applicable) and the Python standard error output (this one is important to be able to detect glitches in the pipeline).
-
To the developer in case of a lock file present in the directory (more information below).
Exception and Error handling
ADCSng has a limited but concise amount of exception and error handling taking advantages of the possibilities of Python and limited by the error handling of CSPICE (C version of SPICE). Wherever possible ADCSng will provide enough information for the developer to fix the issue.
The ADCSng lock file
When ADCSng is executed, ADCSng works/modifies the source and output directories. Due to this parallel runs of ADCSng on the same directory structure could lead into disaster. ADCSng is triggered by a daemon (crontab) in the SPICE operational server and although in principle executions are planned in such a way this event does not happen, if multiple long orbit files were to be processed, executions might overlap.
The solution has been to implement a locking mechanism. When ADCSng is executed, it will generate an empty lock file as per configuration. If no lock file is provided in the configuration, then the default will be to write it in the directory where ADCSng is being executed with the following name: adcsng_mission.lock
The lock file will be automatically removed at the end of an execution even if it is a failed execution. If a lock file is present when a new execution is requested, ADCSng will not run and if emails are enabled, it will send an e-mail to the developer indicating so.
ADCSng exit codes
When a command finishes execution, it returns an exit code. The exit code is not displayed on the command line screen by default. To examine the exit code, you need to examine a special variable, "$?", that can be displayed with the command "echo $?". In bash in general the exit status is 0 if the command succeeded, and 1 if failed. The exit codes for ADCSng are as follows:
-
Run successfully: 0
-
Dry-run (without input): 0
-
Display usage 'adcsng -h': 0
-
Run with errors: 1
-
Executed with wrong arguments: 2
Regression Tests
ADCSng incorporates a number of regression tests that are run in all the operational, development, test and integration environments managed by ESS. It is recommended that the ADCSng user also runs these tests before integration of the pipeline in her/his environment.
The tests can can be run in any Python test environment and after ADCSng is installed they can also be run with:
python3 adcsng/tests/test_regression.py
The regression tests are performed for every project that ADCSng is configured for. The directory structure is the same for each project and contains a number of files and directories to be able to carry out the test. For each project there is one directory for the kernels used for the execution (ker_dir), a directory for the generated kernels (ker_gen), a directory for the kernels to be validated against the generated ones (ker_val) and then the required directories to run ADCSng (described above). The incoming directory is compressed to minimize the size of the ADCSng package whilst containing all the necessary data to run the tests. The resulting directory structure is as follows:
.
|-- ancdr
| |-- fdy
| |-- hkt
| |-- man
| |-- pit
| |-- soc
| `-- tcp
|-- exomars2016.json
|-- exomars2016.py
|-- exomars2016mk.json
|-- incoming.tar.gz
|-- ker_dir
|-- ker_gen
|-- ker_val
|-- log
|-- misc
| |-- deprecated_kernels
| |-- release_notes
| `-- reports
`-- temp
Regression tests content
The following table describes the implemented test. Please note that not all the tests are defined for all projects for not all projects generate the same types of kernels.
Function |
Name |
Description |
runADCSng |
ADCSng execution tests |
Test passes if execution is correct. |
SPKcompare |
SPK Comparison |
Compare resulting SPK with validated ones. Fails for a given threshold. |
CKcompare |
CK comparison |
Compare resulting CKs with validated ones for selected periods of time. Fails for a given threshold. |
ORBNUMcompare |
ORBUN file comparison, |
Compare resulting ORBNUM with validated ones. Fails for a given threshold. |
SCLKcompare |
SCLK comparison |
Compare resulting SCLK with validated one. Fails if not equal. |
MKcompare |
MK comparison |
Compare data of resulting MK with validated one. Fails it not equal. |
LOGcompare |
ADCSng log comparison |
Compare if not-date specific log content is the same as for the validated one. Fails if not equal. |
GAPcompare |
Comparison of GAP files |
Compare if not-date specific Gap files content is the same as for the validated one. Fails if not equal. |
runMkADCSng |
ADCSng for MK execution test. |
Test passes if execution is correct (ADCSng is executed only to generate MK) |
An HTML test report is generated after every test in the 'reports' directory. The regression tests of the ADCSng releases are available here: http://spice.esac.esa.int/status/index_adcsng_former.html the test report of the latest release is available here: http://spice.esac.esa.int/status/adcsng_regression_test.html
Please note that the HTML report provides the start time, duration, test summary and the list of the executed tests providing names, status and the option to view the log of each individual test. The following Figure provides an example of a test report.
Tests performances
A good indicator of whether if your environment is adept for executing ADCSng is by using the metrics provided by the regression tests. Tests are continuously evolving but you can always use the reference version of ADCSng used here (v2.5.0) to compare performances. ESS develops ADCSng in Mac Intel machines, performs integration with PC Linux machines and deploys ADCSng on a PC Linux machine. Tests are performed in all environments. The following table provides the figures of these environments and tests:
Environment |
Operating System |
Specifications |
Test Duration (v2.5.0) |
|
Mac Intel OSX |
macOS Mojave v10.14.5 |
Processor |
2.3GHz Intel Core i9 |
44 seconds |
Memory |
16 GB |
|||
PC Linux Docker |
Centos 7 |
Processor |
8 CPUs Docker engine v19.03.8 |
68 seconds |
Memory |
2 GB |
|||
PC Linux Docker |
RedHat Rhel7 |
Processor |
8 CPUs Docker engine v19.03.8 |
50 seconds |
Memory |
2 GB |
|||
PC Linux (Virtual) |
RedHat Rhel7 |
Memory |
7.6 GB |
89 seconds |
Processor |
2 Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz |