Introduction

Purpose

The purpose of the present document is to provide an installation guide and to describe the mode of operation of the Auxiliary Data Conversion System Next Generation (ADCSng) and a summary of its main functionalities. After reading this document the user should be able to install, work with and fully understand the software package.

Running ADCSng in a nutshell

  1. Obtain ADCSng from BitBucket: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng/

  2. Install from top level directory with "pip3 install -e ."

  3. Generate an ADCSng configuration file as from adcsng/adcsng/tests/bc/bepicolombo.json

  4. Generate the appropriate directory structure as derived from the configuration file

  5. Obtain and install your SPICE Kernel Dataset: https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS

  6. Run with "adcsng yourconfig.json"

Overview

Auxiliary data are those that help scientists and engineers to determine the location and orientation of the spacecraft, when and how an instrument was acquiring scientific data. These data also help to determine what other relevant events were occurring on the spacecraft or ground that might affect the interpretation of the scientific observation or the S/C systems performance. Software applications are required to know what were the location, size, shape and orientation of the observed target in addition to these auxiliary data. Almost all NASA and ESA planetary missions have embraced the use of the SPICE system for ancillary data archiving and for science data analysis. Although the Flight Dynamics Division provides software to read the position and orientation files of the orbiters, most of the Principal Investigators (PI) have pointed out their interest in having all the auxiliary data distributed in SPICE format.

The ESA SPICE Service (ESS) of the Data Science and Archive Division (SCI-SA) is responsible for supporting the ESOC auxiliary data transformation into SPICE kernels and the distribution of these files to the instrument teams using the public network. NASA’s Planetary Data System discipline node NAIF (Navigational Ancillary Information Facility) is responsible for designing a transformation software tool to generate spacecraft orbit and attitude SPICE kernels. The ESS, helped by the individual instrument and FDy teams, the SGSs, and, in some missions, supported by NAIF, design and support all the necessary instrument, spacecraft and timing kernels.

Applicable and Reference Documents

ADCSng is based on different technologies, in particular most of the code is written in Python. FORTRAN is also used along with other technologies such as JSON, being familiar with those technology will certainly facilitate the usage of ADCSng.

Applicable Documents

ID

Title

Reference

AD-E40

ECSS-E40 Space Software Engineering

ECSS-E-ST-40C

AD-PYP

Packaging and Distributing Projects

https://packaging.python.org/tutorials/distributing-packages/

AD-JSN

Introducing JSON

http://www.json.org/

AD-BBT

Bitbucket

https://bitbucket.org/product

AD-SPI

The SPICE Toolkit

https://naif.jpl.nasa.gov/naif/toolkit.html

Reference Documents

ID

Title

Reference

RD-ADC

Auxiliary Data Conversion into SPICE Kernel And Distribution

MEX-EST-PL-10210

RD-ANG

Auxiliary Data Conversion into SPICE Kernels for ESA Solar System Exploration Missions

SPICE-CROSS-TN-001

RD-ECM

ESA SPICE Service Configuration Management Plan

SPICE-CROSS-TN-002

RD-PIP

Configuring a .pypirc File for Easier Python Packaging

https://truveris.github.io/articles/configuring-pypirc/

Terms, Definitions and Abbreviated Terms

Acronym

Name

OEM

Orbit Ephemeris Message

AEM

Attitude Ephemeris Message

ADCSng

Auxiliary Data Conversion System next generation

TCP

Time Correlation Packets

JSON

JavaScript Object Notation

EM16

ExoMars2016

BC

BepiColombo

ESS

ESA SPICE Service

PI

Principal Investigator Team

SOLO

Solar Orbiter

ESOC

European Space Operations Center

NAIF

Navigational Ancillary Information Facility

JUICE

Jupiter Icy Moons Explorer

MEX

Mars Express

FDy

Flight Dynamics

S/C

Spacercaft

SGS

Science Ground Segment

SOC

Science Operations Center

OEM

Orbit Ephemeris Message

AEM

Attitude Ephemeris Message

SKD

SPICE Kernel Data Set

MK

Meta-kernel

SCLK

Spacecraft Clock Kernel

CK

Camera-Matrix Kernel

SPICE

Spacecraft Planets Instruments C-Camera and Events

Software Overview

ADCSng together with some other scripts and pieces of software cover from the automated download of orbital, attitude and spacecraft (S/C) and Payload elements position and orientation data from the given mission Science Ground Segment (SGS) uplink and downlink systems as provided by FDy in the shape of Orbit Ephemeris Messages (OEM), Attitude Ephemeris Messages (AEM) -or another Orbit and Attitude information container files- and Raw Housekeeping Telemetry data processed following the NASA Planetary Data System Standard (PDS4). More information on its concept and design is available here: [RD-ANG]

Installation

Hardware and Software requirements

ADCSng is a Python 3.6.x package that uses FORTRAN 77 and FORTRAN 90 applications and a set of standard Python libraries. ADCSng is meant to interact with different nodes: SGS Uplink and Downlink sub-systems, the ESA SPICE FTP Server and the NAIF FTP Server.

ADCsng is designed to work with UNIX systems; namely with Linux and with MacOS. The hardware requirements are minimal and any modern computer/server will suffice. The software is able to run for example in a minimal CentOS 7 Virtual Machine:

  • Architecture: x86_64 (64 bits)

  • Memory: 1GB/logical CPU

  • Disk space: 10GB

Nevertheless it is recommended to allocate more resources for ADCSng to run appropriately, the "Regression Tests" section includes further information on resource allocation and expected performances.

The software requirements are the following:

  • Python 3.6 or higher

  • Python packages:

    • numpy >= 1.8.0

    • pytest >= 2.9.0

    • pandas > 0.17.1

    • html-testRunner

  • gcc

  • libgfortran

  • Git (possibly, for the obtention of SPICE Kernel Datasets)

Please note that the FORTRAN applications are already compiled and therefore the user should not compile and link them. Nevertheless this is possible since the source code is distributed with ADCSng, the only remaining part would be to link them with the SPICELIB library of the SPICE Toolkit. The compilation of these applications is out of the scope of this SUM and any interested user is encourage to try by himself or to contact the author.

Obtaining the Software

Installing from the ESS Nexus repository with PiPy

ADCSng can be installed directly from PiPy without the need to download the source code, in order to do so, you need to configure the given PIP installation by editing your local Python Package Index configuration file. More information in RD-PIP.

The file is accessible with: more ~/.pypirc what follows is the current configuration for ADCSng:

index-servers = nexus-master nexus-develop
 
[nexus-master]
repository = https://repos.esac.esa.int/repository/pypi-spice-releases/
username = spicedep
password = Esacvilspa1!
 
[nexus-develop]
repository = https://repos.esac.esa.int/repository/pypi-spice-snapshots/
username = spicedep
password = Esacvilspa1!

Installing the source code

The source code of ADCSng is under configuration control with Git the BitBucket repository that hosts it is the following: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng/browse, since this is a private repository access might not be granted, if so please contact the author of this document. Alternatively the source code can be obtained under request from esa_spice@sciops.esa.int.

If the user is to install ADCSng from its source code it is recommended to use PiPy to do so (taking advantage of the Python setup file), in order to do so using the Terminal go to the top level adcsng directory, adcsng and type:

pip3 install [--user] -e .

Please note that the --user parameter is optional; if you are using your user Python installation it needs to be indicated.

Once installed, you will be able to run ADCSng anywhere in your computer.

Source Code structure

A brief overview of the code structure for ADCSng is useful for its installation, configuration and usage:

ADCSng Package top level structure

LICENSE

 

License file (as per any other Python package).

MANIFEST.in

Manifest file (as per any other Python package).

README.md

Readme file (as per any other Python package).

setup.cfg

Configuration for Python Setup.

setup.py

Setup Python routine.

adcsng

Package directory.

 

command_line.py

 

Main function called when invoking ADCSng.

 

classes

Directory that contains Python source code classes.

 

config

Directory that contains the mission-specific developer configuration items for ADCSng.

 

core

Directory that contains Python source code core functions.

 

etc

Directory that contains different templates used by ADCSng (mostly the ones used by the FORTRAN utilities).

 

exe

Directory that contains Linux and MacOS FORTRAN utilities executables.

 

src

Directory that contains the FORTRAN utilities source code.

 

utils

Directory that contains Python source code utility functions.

 

tests

Directory that contains the tests performed for ADCSng.

 

 

bc

 

Directory that contains the test files for BepiColombo (shown as example).

 

 

 

bepicolombo.json

User configuration file for BepiColombo testes (shown as example).

Configuration

The next step to use ADCSng is to configure it. ADCSng has two levels of configuration:

  • User configuration

  • Developer configuration (placed under the adcsng/adcsng/config directory)

User Configuration

As a user you only need to take care of the user configuration. The user configuration has to be edited and prepared by the user, it consists of a JSON file and an example from the test directory of ADCSng can be used as baseline (adcsng/adcsng/tests/bc/bepicolombo.json). Any name can be provided to the configuration file, the file extension has to be *.json. The configuration file will determine the directory structure that you need to setup for ADCSng. The recommendation is to create a directory out of the ADCSng directory structure and to create one per intended ADCSng execution/project.

The following table describes all the available configuration items (please note that all the items are strings):

Key

Value(s)

Description

Required?

name

ADCSng Configuration File

Description of the JSON file

NO

mission

  • BEPICOLOMBO

  • ExoMars2016

  • ExoMarsRSP

  • JUICE

  • MARS_EXPRESS

  • VENUS_EXPRESS

  • Solar_Orbiter

Name of the mission, has to be one of the provided in the list. (corresponds to the name of the SKDs replacing the dashes).

YES

operational_host

It is recommended to obtain the operational host by running the hostname command in your terminal.

Name of the operational Host.

If you use multiple environment and wish to distinguish in between test/development environments from the operational environment, this will allow you to receive notifications with an indication of whether if the run has been done with the operational pipeline or not.

YES

source_dir

Directory path for the input files.

Directory path for the input files. This directory is the one that ADCSng will look for input.

YES

temp_dir

Directory path for ADCSng temporary directory.

Directory path for the where ADCSng will place temporary files.

WARNING: DO NOT PUT ANY FILE IN THIS DIRECTORY: It will be deleted by ADCsng.

YES

log_dir

Directory path for ADSCng log files.

More information on log files below.

YES

kernels_dir

Directory path for the SPICE Kernel Dataset of the mission to be executed.

This has to point to the kernels directory of the SDK. More information below.

YES

output_dir

Directory path for the output of ADCSng.

Directory where the output will be placed.

WARNING: It is HIGHLY RECOMMENDED that this path is the same as kernels_dir.

More information below.

YES

processed_dir

Directory path to place the used and processed input.

More information below.

OPTIONAL

log_rotation

  • Daily

  • Weekly

  • Monthly

This key indicates the log rotation. The log will be split on a daily, weekly or monthly basis. More information below. If not indicated the default will be Monthly.

OPTIONAL

daily_log

Directory path to place an extra log generated per pipeline execution.

More information below.

OPTIONAL

lock_file

Path and name of the lock file.

More information below.

OPTIONAL

mkgen

  • True

  • False

  • Only

Parameter that indicates whether if ADCSng generates a meta-kernel for the run or not or if it is only executed to generate a meta-kernel (and ignore the input).

YES

email/send_email

  • True

  • False

Parameter that indicates whether if e-mails will be sent for clients after an execution and for the developer for logs and for failed executions. More information below.

YES

email/clients

  • email address for clients

  • Blank (if send_email false)

E-mail list for clients to receive notifications of the pipeline execution. More info below.

YES

email/developer

  • email address for developers

  • Blank (if send_email false)

E-mail list for developers to receive notifications of the pipeline execution. More info below.

YES

email/git_repository

HTTP(S) of the Git repository that contains the kernels

Free text to introduce the HTTP(S) address of the Git repository for the SKD.

YES

email/ftp_address

HTTP(S) of the FTP repository that contains the kernels

Free text to introduce the HTTP(S) address of the FTP repository for the SKD.

YES

email/skd_name

Name of the Git repository for the given mission.

Free text to introduce the name of the SPICE Kernel Dataset as in: https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS

YES

email/mission

Name of the FTP directory for the given mission.

Free text to introduce the name of the SPICE Kernel Dataset as in: ftp://spiftp.esac.esa.int/data/SPICE/

YES

email/contact

Contact name and/or email

Free text to introduce the Contact information for clients.

YES

Once the configuration file has been generated it can be placed anywhere in your computer, it is recommended to put it in the same directory level as where you will define the directory structure of your ADCSng configuration. What follows is an example of an ADCSng configuration file:

{
"name": "ADCSng Configuration File",
"mission": "BEPICOLOMBO",
"operational_host": "ESAC1102875",
"source_dir": "/Users/esaspice/bc/incoming/",
"temp_dir": "/Users/esaspice/bc/temp",
"log_dir": "/Users/esaspice/bc/log",
"lock_file": "/Users/esaspice/bc/adcsng.lock",
"kernels_dir": "/Users/esaspice/SPICE/BEPICOLOMBO/kernels",
"output_dir": "/Users/esaspice/SPICE/BEPICOLOMBO/kernels",
"processed_dir": "/Users/esaspice/ANCDR/BEPICOLOMBO",
"log_rotation": "daily",
"daily_log": "True",
"mkgen": "True",
"email": [{
"send_email": "True",
"clients": "mcosta@sciops.esa.int",
"developer": "mcosta@sciops.esa.int",
"mission" : "BepiColombo",
"skd_name" : "BEPICOLOMBO",
"ftp_address": "ftp://spiftp.esac.esa.int/data/SPICE/",
"git_repository": "https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS/repos/",
"contact": " Marc Costa\n mcosta@sciops.esa.int"
}]
}

If we consider the previous configuration file we could have a directory structure as follows:

bc
|-- adcsng.json
|-- incoming
|-- kernels -> ../ftp/data/SPICE/BEPICOLOMBO/kernels/
|-- log
|   |-- adcsng_weekly_2019Oct04.log
|   '-- adcsng_weekly_2019Oct09.log
|-- misc -> ../ftp/data/SPICE/BEPICOLOMBO/misc/
|-- processed -> ../ftp/data/ANCDR/BEPICOLOMBO/
'-- temp

Remember that you need to generate this directory structure for ADCSng will not generate it for you.

Developer configuration

The developer configuration should not be modified by the user. Further details on the developer configuration can be found on RD-ANG.

Kernels and Output directories requirements

Regardless of the kernels/output directory structure you setup the following subdirectories for each path are mandatory in order to execute ADCSng:

  • kernels: 'ck', 'spk', 'lsk', 'sclk', 'pck', 'ik', 'fk' and 'mk'

  • output: 'ck', 'spk', 'sclk', 'mk'

If any of those kernels sub-directories is not present ADCSng will trigger an error and the execution will fail.

You need a working SPICE Kernel Dataset to run ADCSng

It is important to note that you need a local copy of a working SKD in order to execute ADCSng. This document does not intend to provide you a detailed explanation on how to obtain the latest version of a given SKD, instead we recommend you to read the information in the following page: https://www.cosmos.esa.int/web/spice/data

What is important to note though is that you will need to have a working SKD. In order to check that you do we recommend you to use a SPICE utility such as BRIEF on the MK in order to ensure that the SKD is properly installed.

ADCSng needs the SKD as an input and preferably also output directory for the execution: the idea is that ADCSng updates the SKD itself and therefore it should be considered as both an input and an output. That is why in general is recommended to set the kernel and the output directories with the same value.

If you still consider to have different directories please note that the SCLK that you generate with an ADCSng run will not be taken into consideration to generate the rest of the CK kernels; instead the SCLK in the kernel directory will be used. The same issue will happen with the versioning of the kernels you generate: if the same kernel is generated the version will not be derived from the original SKD.

Processed directory requirements

The processed directory should have a given sub-directory structure. This structure is determined by the 'originators' of each input file. ADCSng will generate these sub-directories if they do not exist. The 'originator' acronyms can be found in the developer configuration. In general they are the following:

  • man: Files coming from Mission Analysis (typically test trajectories)

  • fdy: Files coming from Flight Dynamics (typically OEMs and AEMs)

  • soc: Files coming from the Science Operations Center (typically planning AEMs)

  • hkt: Files from the Housekeeping Telemetry typically processed by the SOC Downlink group

  • tcp: Time Correlation Packets coming from the S/C

  • pit: Files from the PI teams

Please note that the files under these directories are usually published in the ESA SPICE FTP. An example can be seen in the following link: ftp://spiftp.esac.esa.int/data/ANCDR/BEPICOLOMBO/

If the 'processed_dir' is not specified in the condiguration file (is left empty) then the processed files will simply not be copied (this is the case for ExoMarsRSP for instance).

Executing ADCSng

In order to execute ADCSng you only need to type the command: adcsng, the command usage is as follows:

adcsng [-h] [-v] [-m] [-l] [-d] [-e] CONFIG [CONFIG ...]

Indicating the configuration file path is mandatory, the options are displayed by the -h or --help options:

___ ___ _________ Auxiliary Data Conversion System next generation
/ _ | / _ \/ ___/ __/__ ___ _
/ __ |/ // / /___\ \/ _ \/ _ `/ a command-line utility program that
/_/ |_/____/\___/___/_//_/\_, / converts auxiliary data into SPICE
/___/ Kernels for the European Space Agency
v2.5.0
 
positional arguments:
CONFIG Mission specific JSON configuration file
 
optional arguments:
-h, --help show this help message and exit
-v, --version Display the version of ADCSng
-m, --metakernel Execute ADCSng to generate meta-kernel only
-l, --log Prompt log during execution
-d, --debug Run in debug mode; run files not cleaned-up, etc.
-e, --noemail Do not send execution notification e-mail to users.
 
Source and documentation is available here:
https://repos.cosmos.esa.int/socci/projects/SPICE/repos/adcsng
 
developed and maintained by the
__ __ __ __ __ __ ___ __ ___ __ __ ___
/__\ /__` '__\ /__` |__) | / ` |__ /__` |__ |__) \ / | / ` |__
\__, .__/ \__/ .__/ | | \__, |___ .__/ |___ | \ \/ | \__, |___
 
esa_spice@sciops.esa.int for Mars-Express // ExoMars2016 // ExoMarsRSP
http://spice.esac.esa.int BepiColombo // Solar Orbiter

Please note that if you do not use the optional argument -l or --log you will not be prompted the log of the execution. Error messages prior to the log creation might be shown though.

Once ADCSng is executed, in a nutshell it checks the incoming directory for input data. If present it processes the input data and generates the corresponding kernels (if no input data is present but ADCSng is set to generate a meta-kernel it generates it) and then moves the generated kernels to the output directory and the input and processed data to the processed directory. Finally if specified ADCSng notifies the users and developers with an e-mail.

Please note that the synchronisation with BitBucket and other services such as the SKD validation are performed by other packages of the ESA SPICE Service (such as spigit and spival).

ADCSng internal executables

ADCSng has a number of internal executables that may be used during the execution depending on the ADCSng module that is used. These executables are compiled and built FORTRAN applications and are provided as part of the ADCSng package for the working environments: PC Linux 64 bits and Mac Intel OSX 64 bits. The ADCSng package also includes the source files to compile these executables in any working environment (The SPICE Toolkit library for that given environment is required). More information on how to install SPICE for different environments is available in AD-SPI.

Please note that these executables can be used independently of ADCSng if desired. The list of executables is provided by the following table:

Executable

Description

Originator

User Guide

aem2ck

AEM2CK is a command-line utility program that converts AEM files to CK files.

NAIF

Not available.

brief

BRIEF is a command-line utility program that displays a contents and time coverage summary for one or more binary SPK or binary PCK files.

NAIF

brief.ug

ck5to6

CK5TO6 is a command-line utility program that converts Type 5 CKs to Type 6 Cks.

NAIF

Not avaiable.

ckbrief

CKBRIEF is a command-line utility program that displays a contents and time coverage summary for one or more binary CK files.

NAIF

ckbrief.ug

commnt

COMMNT is a command-line program that reads, adds, extracts, or deletes comments from SPICE binary kernel files.

NAIF

commnt.ug

dafcat

DAFCAT is a command-line utility program that concatenates together SPICE DAF files of the same type.

NAIF

dafcat.ug

dafmod

DAFMOD is an interactive program that modifies various segment attributes of a DAF based kernel -- SPK, CK, or binary PCK.

NAIF

dafmod.ug

delutc

DELUTC is a command-line utility that provides the number of leapseconds for given UTC date.

ESS

Not available

diffsclk

DIFFSCLK is a command-line utility that provides the difference between to SCLKs.

NAIF

Not available

frmdiff

FRMDIFF is a program that samples orientation of a reference frame known to SPICE or computes differences between orientations of two reference frames known to SPICE, and either displays this orientation or these differences, or shows statistics about it or them.

NAIF

frmdiff.ug

makclk

MAKCLK is a program that converts a SCLKSCET (also known as SCLKvSCET) file to a SPICE spacecraft clock (SCLK) kernel file.

NAIF

makclk.ug

mex2ker

MEX2KER is a command-line utility program that converts OASW attitude and trajectory files (ESA Flight Dynamics Legacy format, smilar to OEM and AEM v1) to CK or SPK files.

NAIF

Not available

mkspk

MKSPK is a program that creates an SPK file from a text file containing trajectory information.

NAIF

mkspk.ug

msopck

MSOPCK is a command-line program that converts attitude data provided in a text file as UTC, SCLK, or ET-tagged quaternions, Euler angles, or matrices, optionally accompanied by angular velocities, into a type 1, 2, or 3 SPICE C-kernel.

NAIF

msopck.ug

oem2spk

OEM2SPK is a utility program that converts a CCSDS ``Orbit Ephemeris Message'' text file (also referred to as an ``OEM'' or ``OEM file'') to a binary SPICE SPK file containing type 9 or type 13 segments.

NAIF

oem2spk.ug

orbnum

The ORBNUM program creates an ASCII orbit number table file for an object, keyed on the time of a geometric event.

NAIF

orbnum.ug

prediCkt

prediCkt is a program that creates C-kernels (usually predicted C-kernels) for orientations that can be described by a set of simple geometric descriptions.

NAIF

prediCkt.ug

spk13to19

SPK13TO19 is a command-line utility program that converts Type 13 SPKs to Type 19 SPKs.

NAIF

Not available.

spk18to19

SPK18TO19 is a command-line utility program that converts Type 18 SPKs to Type 19 SPKs.

NAIF

Not available.

spkdiff

SPKDIFF provides means for comparing the trajectories of two bodies or sampling the trajectory of a single body using data from SPICE kernels.

NAIF

spkdiff.ug

tcp2scet

TCP2SCET is a program converts ESOC Time Correlation Packet (TCP) file
to JPL SCLK-vs-SCET (SCLKvSCET) file, which can then be converted to
SPICE SCLK file using the SPICE MAKCLK program.

NAIF

tcp2scet.ug

tcpescan

TCPESCAN is a command-line utility that converts Time Correlation Packets in binary format to text format. Used by ExoMars2016.

EM16 SOC

Not available.

tcpescan_bc

TCPESCAN_BC is a command-line utility that converts Time Correlation Packets in binary format to text format for BepiColombo MPO.

BC SOC

Not available.

tcpescan_emrsp

TCPESCAN_BC is a command-line utility that converts Time Correlation Packets in binary format to text format for the ExoMars RSP Rover

ESS

Not available.

tdb2utc

TDB2UTC is a command-line utility that converts a data in ephemeris time to a UTC date.

ESS

Not available.

utc2tdb

UTC2TDB is a command-line utility that converts a data in UTC calendar format into an ephemeris time.

ESS

Not available.

Please note that all the available User Guides can be found here: http://naif.jpl.nasa.gov/pub/naif/utilities/PC_Linux_64bit/

Logging

ADCSng provides multiple options for the logging process, the logging is provided via:

  1. Log file with a specified rotation written in a specified directory. Filename is adcsng_rotation_YYYYMonDD.log where:

    1. rotation is 'monthly', 'weekly' or daily,

    2. YYYY is the year, Mon is the three letter capitalised month acronym and DD is the day with two digits.

  2. Log file per execution written in a specified directory. Filename is adcsng_YYYYMMDDTHHMMSS.eop where:

    1. YYYY is year, MM month, DD day, HH hour, MM minute, SS second

    2. eop stands for End of Process.

  3. E-mail containing the log file text per execution

  4. Prompted Log in the screen if -l or --log optional argument is provided

Log format

The log format per line is as follows (Python syntax):

'%(asctime)s.%(msecs)03d %(levelname)-8s %(module)-16s %(message)s'

This translates into the following columns: Time | Information Level | Module logging | Log message (please note that prompted log only includes the Log message).

There are four information levels:

  1. INFO : Log information of all kinds, nothing to worry about.

  2. WARNING : A foreseen/predictable action by the pipeline (input processing, present files) nothing to worry about but sometimes worth checking.

  3. ERROR : Something which should not have happened and that the developer might want to look into but that did not interrupt the execution.

  4. CRITICAL : Something bad which interrupted the execution and that will most probably translate into human intervention either from the user or a bug-fix from the developer.

What follows is a log example:

2019-10-09T03:09:52.876 INFO director ================================================================================
2019-10-09T03:09:52.877 INFO director ADCSng v0.13.2 for Solar_Orbiter run on ESAC01102875 at 2019-10-09 03:09:52.8
2019-10-09T03:09:52.877 INFO director ================================================================================
2019-10-09T03:09:52.877 INFO director
2019-10-09T03:09:52.917 WARNING director MK solo_ANC_soc-pred-mk_V101_20191009_001.tm has not been updated
2019-10-09T03:09:52.955 WARNING director MK solo_ANC_soc-flown-mk_V101_20191009_001.tm has not been updated
2019-10-09T03:09:52.957 INFO director - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2019-10-09T03:09:52.957 INFO director No kernels have been generated.

Sending emails

If properly configured ADCSng will send e-mail notifications to users and developers. In order to be able to use it Python needs to connect to the following SMTP server: smtp.sciops.esa.int, the user name and the password are hard-coded in ADCSng and the sender will be esa_spice@sciops.esa.int, so far allowing for an alternative server is not considered. ADCSng will then send notifications in the following cases:

  1. To the specified users after a successful (or partially successful) run, the e-mail template is available here: adcsng/adcsng/etc/email.commnt

  2. To the developer after a successful run with the log of the run.

  3. To the developer in case of a failed run with the log (if applicable) and the Python standard error output (this one is important to be able to detect glitches in the pipeline).

  4. To the developer in case of a lock file present in the directory (more information below).

Exception and Error handling

ADCSng has a limited but concise amount of exception and error handling taking advantages of the possibilities of Python and limited by the error handling of CSPICE (C version of SPICE). Wherever possible ADCSng will provide enough information for the developer to fix the issue.

The ADCSng lock file

When ADCSng is executed, ADCSng works/modifies the source and output directories. Due to this parallel runs of ADCSng on the same directory structure could lead into disaster. ADCSng is triggered by a daemon (crontab) in the SPICE operational server and although in principle executions are planned in such a way this event does not happen, if multiple long orbit files were to be processed, executions might overlap.

The solution has been to implement a locking mechanism. When ADCSng is executed, it will generate an empty lock file as per configuration. If no lock file is provided in the configuration, then the default will be to write it in the directory where ADCSng is being executed with the following name: adcsng_mission.lock

The lock file will be automatically removed at the end of an execution even if it is a failed execution. If a lock file is present when a new execution is requested, ADCSng will not run and if emails are enabled, it will send an e-mail to the developer indicating so.

ADCSng exit codes

When a command finishes execution, it returns an exit code. The exit code is not displayed on the command line screen by default. To examine the exit code, you need to examine a special variable, "$?", that can be displayed with the command "echo $?". In bash in general the exit status is 0 if the command succeeded, and 1 if failed. The exit codes for ADCSng are as follows:

  • Run successfully: 0

  • Dry-run (without input): 0

  • Display usage 'adcsng -h': 0

  • Run with errors: 1

  • Executed with wrong arguments: 2

Regression Tests

ADCSng incorporates a number of regression tests that are run in all the operational, development, test and integration environments managed by ESS. It is recommended that the ADCSng user also runs these tests before integration of the pipeline in her/his environment.

The tests can can be run in any Python test environment and after ADCSng is installed they can also be run with:

python3 adcsng/tests/test_regression.py

The regression tests are performed for every project that ADCSng is configured for. The directory structure is the same for each project and contains a number of files and directories to be able to carry out the test. For each project there is one directory for the kernels used for the execution (ker_dir), a directory for the generated kernels (ker_gen), a directory for the kernels to be validated against the generated ones (ker_val) and then the required directories to run ADCSng (described above). The incoming directory is compressed to minimize the size of the ADCSng package whilst containing all the necessary data to run the tests. The resulting directory structure is as follows:

.
|-- ancdr
| |-- fdy
| |-- hkt
| |-- man
| |-- pit
| |-- soc
| `-- tcp
|-- exomars2016.json
|-- exomars2016.py
|-- exomars2016mk.json
|-- incoming.tar.gz
|-- ker_dir
|-- ker_gen
|-- ker_val
|-- log
|-- misc
| |-- deprecated_kernels
| |-- release_notes
| `-- reports
`-- temp

Regression tests content

The following table describes the implemented test. Please note that not all the tests are defined for all projects for not all projects generate the same types of kernels.

Function

Name

Description

runADCSng

ADCSng execution tests

Test passes if execution is correct.

SPKcompare

SPK Comparison

Compare resulting SPK with validated ones. Fails for a given threshold.

CKcompare

CK comparison

Compare resulting CKs with validated ones for selected periods of time. Fails for a given threshold.

ORBNUMcompare

ORBUN file comparison,

Compare resulting ORBNUM with validated ones. Fails for a given threshold.

SCLKcompare

SCLK comparison

Compare resulting SCLK with validated one. Fails if not equal.

MKcompare

MK comparison

Compare data of resulting MK with validated one. Fails it not equal.

LOGcompare

ADCSng log comparison

Compare if not-date specific log content is the same as for the validated one. Fails if not equal.

GAPcompare

Comparison of GAP files

Compare if not-date specific Gap files content is the same as for the validated one. Fails if not equal.

runMkADCSng

ADCSng for MK execution test.

Test passes if execution is correct (ADCSng is executed only to generate MK)

An HTML test report is generated after every test in the 'reports' directory. The regression tests of the ADCSng releases are available here: http://spice.esac.esa.int/status/index_adcsng_former.html the test report of the latest release is available here: http://spice.esac.esa.int/status/adcsng_regression_test.html

Please note that the HTML report provides the start time, duration, test summary and the list of the executed tests providing names, status and the option to view the log of each individual test. The following Figure provides an example of a test report.

images/socciwiki/download/attachments/42245115/Screenshot_2020-10-13_at_22.56.59.png

Tests performances

A good indicator of whether if your environment is adept for executing ADCSng is by using the metrics provided by the regression tests. Tests are continuously evolving but you can always use the reference version of ADCSng used here (v2.5.0) to compare performances. ESS develops ADCSng in Mac Intel machines, performs integration with PC Linux machines and deploys ADCSng on a PC Linux machine. Tests are performed in all environments. The following table provides the figures of these environments and tests:

Environment

Operating System

Specifications

Test Duration (v2.5.0)

Mac Intel OSX

macOS Mojave v10.14.5

Processor

2.3GHz Intel Core i9

44 seconds

Memory

16 GB

PC Linux Docker

Centos 7

Processor

8 CPUs Docker engine v19.03.8

68 seconds

Memory

2 GB

PC Linux Docker

RedHat Rhel7

Processor

8 CPUs Docker engine v19.03.8

50 seconds

Memory

2 GB

PC Linux (Virtual)

RedHat Rhel7

Memory

7.6 GB

89 seconds

Processor

2 Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz