Introduction
Purpose
The purpose of the present document is to provide an installation guide and to describe the mode of operation of the Archive Generation pipeline (ARCGEN) and a summary of its main functionalities. After reading this document the user should be able to install, work with and fully understand the software package.
Running ARCGEN in a nutshell
-
Obtain ARCGEN from BitBucket: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/arcgen/
-
Install from top level directory with "pip3 install -e ."
-
Generate an ARCGEN configuration file as from arcgen/arcgen/test/vex.json
-
Generate the appropriate directory structure as derived from the configuration file
-
Obtain and install your SPICE Kernel Dataset: https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS
-
Run with "arcgen yourconfig.json"
Overview
Auxiliary data are those that help scientists and engineers to determine the location and orientation of the spacecraft, when and how an instrument was acquiring scientific data. These data also help to determine what other relevant events were occurring on the spacecraft or ground that might affect the interpretation of the scientific observation or the S/C systems performance. Software applications are required to know what were the location, size, shape and orientation of the observed target in addition to these auxiliary data. Almost all NASA and ESA planetary missions have embraced the use of the SPICE system for ancillary data archiving and for science data analysis. Although the Flight Dynamics Division provides software to read the position and orientation files of the orbiters, most of the Principal Investigators (PI) have pointed out their interest in having all the auxiliary data distributed in SPICE format.
The ESA SPICE Service (ESS) of the Cross Mission Support Office is responsible for supporting the ESOC auxiliary data transformation into SPICE kernels and the distribution of these files to the instrument teams using the public network. NASA’s Planetary Data System discipline node NAIF (Navigational Ancillary Information Facility) is responsible for designing a transformation software tool to generate spacecraft orbit and attitude SPICE kernels. The ESS, helped by the individual instrument and FDy teams, the SGSs, and, in some missions, supported by NAIF, design and support all the necessary instrument, spacecraft and timing kernels.
Applicable and Reference Documents
ARCGEN is based on different technologies, in particular most of the code is written in Python. FORTRAN is also used along with other technologies such as JSON, being familiar with those technology will certainly facilitate the usage of ADCSng.
Applicable Documents
ID |
Title |
Reference |
AD-E40 |
ECSS-E40 Space Software Engineering |
ECSS-E-ST-40C |
AD-PYP |
Packaging and Distributing Projects |
https://packaging.python.org/tutorials/distributing-packages/ |
AD-JSN |
Introducing JSON |
|
AD-BBT |
Bitbucket |
Reference Documents
ID |
Title |
Reference |
|
|
|
Terms, Definitions and Abbreviated Terms
Acronym |
Name |
OEM |
Orbit Ephemeris Message |
AEM |
Attitude Ephemeris Message |
TCP |
Time Correlation Packets |
JSON |
JavaScript Object Notation |
EM16 |
ExoMars2016 |
BC |
BepiColombo |
SOLO |
Solar Orbiter |
JUICE |
Jupiter Icy Moons Explorer |
MEX |
Mars Express |
FDy |
Flight Dynamics |
SGS |
Science Ground Segment |
Software Overview
ARCGEN consists on a set of Python based scripts devoted to the Archiving of SPICE Kernel Data Sets following the NASA Planetary Data System Standards (PDS3 and PDS4). It is a semi-automated package used to generate most of the files which have to be present inside a Data Set (PDS3) or Bundle (PDS4) based on the files included in the SKD. It includes validating functionalities which should be used for consistency checks after the execution of the pipeline.
Installation
Hardware and Software requirements
ARCGEN is a Python 3.6.x package that uses
FORTRAN 77 and FORTRAN 90
applications and a set of standard Python libraries.
ARCGEN is designed to work with UNIX systems namely with Linux and with MacOS. The hardware requirements are minimal and any modern computer/server will suffice. The software is able to run for example in a minimal CentOS 7 Virtual Machine:
-
Architecture: x86_64 (64 bits)
-
Memory: 1GB/logical CPU
-
Disk space: 10GB
The software requirements are the following:
-
Python 3.6 or higher
-
Python packages:
-
numpy >= 1.8.0
-
spiceypy
-
-
gcc
-
libgfortran
-
Git (possibly, for the obtention of SPICE Kernel Datasets)
Obtaining the Software
Installing the source code
The source code of ARCGEN is under configuration control with Git. The BitBucket repository that hosts it is the following: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/arcgen/browse, since this is a private repository access might not be granted, if so please contact the author of this document. Alternatively the source code can be obtained under request from esa_spice@sciops.esa.int.
If the user is to install ARCGEN from its source code it is recommended to use PiPy to do so (taking advantage of the Python setup file), in order to do so using the Terminal go to the top level ARCGEN directory, arcgen and type:
pip
install
[--user] -e .
Please note that the --user parameter is optional; if you are using your user Python installation it needs to be indicated.
Once installed, you will be able to run ARCGEN anywhere in your computer.
Source Code structure
A brief overview of the code structure for ARCGEN is useful for its installation, configuration and usage:
ADCSng Package top level structure |
||||
aescalante |
|
VEX and MEX example config files. |
||
bgrieger |
ROS example config file. |
|||
sandbox |
Some routines for various checks. |
|||
README.md |
Readme file (as per any other Python package). |
|||
setup.py |
Setup Python routine. |
|||
arcgen |
Package directory. |
|||
|
command_line.py |
|
Main function called when invoking ARCGEN. |
|
|
classes |
Directory that contains Python source code classes. |
||
|
config |
Directory that contains the mission-specific developer configuration items for ARCGEN. |
||
|
core |
Directory that contains Python source code core functions. |
||
|
etc |
Directory that contains different templates used by ARCGEN (mostly the ones used by the FORTRAN utilities). |
||
|
utils |
Directory that contains Python source code utility functions. |
||
|
tests |
Directory that contains the tests performed for ARCGEN. |
Configuration
The next step to use ARCGEN is to configure it.
The configuration has to be edited and prepared by the user. It consists of a JSON file and an example from the test directory of ARCGEN can be used as baseline (arcgen/arcgen/tests/vex.json). Any name can be provided to the configuration file, the file extension has to be *.json. The configuration file will determine the directory structure that you need to setup for ARCGEN. The recommendation is to create a directory out of the ARCGEN directory structure and to create one per intended ARCGEN execution/project.
The following table describes all the available configuration items (please note that all the items are strings):
Key |
Value(s) |
Description |
Required? |
version |
Data set Version number |
Data set Version number |
YES |
name |
ARCGEN Configuration File |
Description of the JSON file |
NO |
bundle_directory |
Bundle/Data Set output directory |
Directory where ARCGEN puts the generated archive |
YES |
kernels_directory |
SKD directory |
Directory where ARCGEN looks for Kernels as input |
YES |
increment |
Archive last increment |
Existing version of the archive to increment |
YES |
increment_stop |
Increment Stop date |
Stop date for the new increment |
YES |
mission_name |
|
Name of the mission, has to be one of the provided in the list. (corresponds to the name of the SKDs replacing the dashes). |
YES |
mission_accronym |
|
Mission accronym, has to be one of the provided in the list. |
YES |
dataset |
Data Set name |
Name of the Data Set |
YES |
dataset_name |
Data Set descriptive name |
Data Set descriptive name |
NO |
volume_id |
Volume Identifier |
Identifier of the Archive Volume |
YES |
spacecraft |
Spacecraft Name |
Name of the Spacecraft |
YES |
target |
Target Name |
Name of the Target (Mars, Venus, Mercury,...) |
YES |
mission_start |
Start UTC date |
Start date of the archive (YYYY-MM-DD) |
YES |
mission_stop |
Stop UTC date |
Stop date of the archive (YYYY-MM-DD) |
YES |
secondary_spacecrafts |
Secondary spacecrafts names |
Name of any secondary spacecraft (e.g. PHILAE) |
YES |
secondary_targets |
Secondary target names |
Name of any secondary target (e.g. DEIMOS, PHOBOS) |
YES |
metakernel |
Metakernel Path |
SKD Metakernel used for archive generation |
YES |
mk_grammar |
MK Grammar config file |
Configuration file with MK Grammar. Used to include/exclude files from the SKD into the generated archive. |
YES |
pds |
PDS Version
|
Version of PDS format |
YES |
author |
Author name |
Name of the Generated archive author |
NO |
Once the configuration file has been generated it can be placed anywhere in your computer, it is recommended to put it in the same directory level as where you will define the directory structure of your ARCGEN configuration. What follows is an example of an ARCGEN configuration file:
{
"version"
:
"2.0"
,
"name"
:
"ARCGEN Configuration File"
,
"bundle_directory"
:
"/Users/aescalante/spice/missions/mex/MEX-E-M-SPICE-6-V2.0"
,
"kernels_directory"
:
"/Users/aescalante/spice/missions/mex/kernels"
,
"increment"
:
""
,
"increment_stop"
:
"2020-12-31T23:59:59"
,
"mission_name"
:
"MARS EXPRESS"
,
"dataset"
:
"MEX-E/M-SPICE-6-V2.0"
,
"dataset_name"
:
"MARS EXPRESS SPICE KERNELS V2.0"
,
"volume_id"
:
"MEXSP_2000"
,
"mission_accronym"
:
"MEX"
,
"spacecraft"
:
"MARS EXPRESS"
,
"target"
:
"MARS"
,
"mission_start"
:
"1997-10-31"
,
"mission_stop"
:
"2020-05-31"
,
"secondary_spacecrafts"
:
""
,
"secondary_targets"
:
""
,
"xml_model"
:
"xml_model"
,
"schema_location"
:
"schema_location"
,
"information_model_version"
:
"information_model_version"
,
"metakernel"
:
"/Users/aescalante/spice/missions/mex/kernels/mk/MEX_OPS.TM"
,
"mk_grammar"
:
"/Users/aescalante/spice/pipelines/arcgen/arcgen/config/mex.grammar"
,
"pds"
:
"3"
,
"author"
:
"Alfredo Escalante Lopez"
}
If we consider the previous configuration file we could have a directory structure as follows:
mex.json
MEX-E-M-SPICE-
6
-V2.
0
|-- AAREADME.TXT
|-- CATALOG
|-- DATA (ARCGEN will put kernels with labels here)
|-- DOCUMENT
|-- ERRATA.TXT
|-- EXTRAS
| |-- MK
| '-- ORBNUM
|-- INDEX
'-- VOLDESC.CAT
kernels
|-- spk
|-- ck
:
'-- mk
Remember that you need to generate this directory structure for ARCGEN will not generate it for you.
Kernels and Output directories requirements
You need a working SPICE Kernel Dataset to run ARCGEN
It is important to note that you need a local copy of a working SKD in order to execute ARCGEN. This document does not intend to provide you a detailed explanation on how to obtain the latest version of a given SKD, instead we recommend you to read the information in the following page: https://www.cosmos.esa.int/web/spice/data
What is important to note though is that you will need to have a working SKD. In order to check that you do, we recommend you to use a SPICE utility such as BRIEF on the MK in order to ensure that the SKD is properly set up.
Executing ARCGEN
In order to execute ARCGEN you only need to type the command: arcgen, the command usage is as follows:
arcgen [-h] [-e] [-l] [-t] CONFIG [CONFIG ...]
Indicating the configuration file path is mandatory, the options are displayed by the -h or --help options:
ARCGEN -- Version
1.0
.
0
, SPICE PDS3/PDS4 Archive Generation Pipeline
ARCGEN is the command-line utility program that generates PDS4
Bunldes and PDS3 Datasets
for
SPICE Kernel Datasets. Source and
documentation are available here:
https:
//repos.cosmos.esa.int/socci/projects/SPICE/repos/arcgen
positional arguments:
CONFIG Mission specific JSON configuration file
optional arguments:
-h, --help show
this
help message and exit
-e, --labels Execute ARCGEN to generate labels only
-l, --log Prompt log during execution
-t, --transfer Execute ARCGEN to generate PSA transfer manifest
__ __ __ __ __ __ ___ __ ___ __ __ ___
/__\ /__` '__\ /__` |__) | / ` |__ /__` |__ |__) \ / | / ` |__
\__, .__/ \__/ .__/ | | \__, |___ .__/ |___ | \ \/ | \__, |___
esa_spice
@sciops
.esa.
int
http:
//spice.esac.esa.int
Please note that if you do not use the optional argument -l or --log you will not be prompted the log of the execution. Error messages prior to the log creation might be shown though.
Once ARCGEN is executed, in a nutshell it checks the kernels directory for input SKD. The kernel grammar is used to include/exclude kernels from the SKD into the generated data set/bundle and generate the metakernel of the archive. If a previous archive is provided with "increment" keyword, ARCGEN searches for previous files to be included in the current increment. Finally, the index file is generated listing the files in the archive.
In the case of generating a PDS3 Data Set, after running the pipeline the user has to manually update several files of the archive:
-
AAREADME.TXT
-
CATALOG
-
DOCUMENT
-
ERRATA.TXT
-
VOLDESC.CAT
It is also a good practice to run additional checks on the generated archive to make sure the files have been generated correctly, the format is adequate and the PDS standards have been followed properly. In the directory arcgen/aescalante, the python script checkText.py can be used to check some PDS3 format consistencies, end of lines, line lengths and label keywords.
DVal tool may be used as well to perform some checks on the Data Set. Download the Data Validator command line tool from ftp://ssols01.esac.esa.int/pub/software/DValNG/index.html#advanced. Unpack the gzipped tar file. Executables are in DatasetValidator/bin. Before running any checks, run ConfigSynchronizer. Calling GrammarChecker without arguments lists possible options and parameters. In order to create a log file listing the issues found by the tool, GrammarChecker as:
DatasetValidator/bin/GrammarChecker -f grammarcheckerlog.txt -i ARCGEN_OUTPUT |
It will find issues related to incorrect line endings, non ascii characters or incorrect label structure.