Introduction

Purpose

The purpose of the present document is to provide an installation guide and to describe the mode of operation of the Archive Generation pipeline (ARCGEN) and a summary of its main functionalities. After reading this document the user should be able to install, work with and fully understand the software package.

Running ARCGEN in a nutshell

  1. Obtain ARCGEN from BitBucket: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/arcgen/

  2. Install from top level directory with "pip3 install -e ."

  3. Generate an ARCGEN configuration file as from arcgen/arcgen/test/vex.json

  4. Generate the appropriate directory structure as derived from the configuration file

  5. Obtain and install your SPICE Kernel Dataset: https://repos.cosmos.esa.int/socci/projects/SPICE_KERNELS

  6. Run with "arcgen yourconfig.json"

Overview

Auxiliary data are those that help scientists and engineers to determine the location and orientation of the spacecraft, when and how an instrument was acquiring scientific data. These data also help to determine what other relevant events were occurring on the spacecraft or ground that might affect the interpretation of the scientific observation or the S/C systems performance. Software applications are required to know what were the location, size, shape and orientation of the observed target in addition to these auxiliary data. Almost all NASA and ESA planetary missions have embraced the use of the SPICE system for ancillary data archiving and for science data analysis. Although the Flight Dynamics Division provides software to read the position and orientation files of the orbiters, most of the Principal Investigators (PI) have pointed out their interest in having all the auxiliary data distributed in SPICE format.

The ESA SPICE Service (ESS) of the Cross Mission Support Office is responsible for supporting the ESOC auxiliary data transformation into SPICE kernels and the distribution of these files to the instrument teams using the public network. NASA’s Planetary Data System discipline node NAIF (Navigational Ancillary Information Facility) is responsible for designing a transformation software tool to generate spacecraft orbit and attitude SPICE kernels. The ESS, helped by the individual instrument and FDy teams, the SGSs, and, in some missions, supported by NAIF, design and support all the necessary instrument, spacecraft and timing kernels.

Applicable and Reference Documents

ARCGEN is based on different technologies, in particular most of the code is written in Python. FORTRAN is also used along with other technologies such as JSON, being familiar with those technology will certainly facilitate the usage of ADCSng.

Applicable Documents

ID

Title

Reference

AD-E40

ECSS-E40 Space Software Engineering

ECSS-E-ST-40C

AD-PYP

Packaging and Distributing Projects

https://packaging.python.org/tutorials/distributing-packages/

AD-JSN

Introducing JSON

http://www.json.org/

AD-BBT

Bitbucket

https://bitbucket.org/product

Reference Documents

ID

Title

Reference

 

 

 

Terms, Definitions and Abbreviated Terms

Acronym

Name

OEM

Orbit Ephemeris Message

AEM

Attitude Ephemeris Message

TCP

Time Correlation Packets

JSON

JavaScript Object Notation

EM16

ExoMars2016

BC

BepiColombo

SOLO

Solar Orbiter

JUICE

Jupiter Icy Moons Explorer

MEX

Mars Express

FDy

Flight Dynamics

SGS

Science Ground Segment

Software Overview

ARCGEN consists on a set of Python based scripts devoted to the Archiving of SPICE Kernel Data Sets following the NASA Planetary Data System Standards (PDS3 and PDS4). It is a semi-automated package used to generate most of the files which have to be present inside a Data Set (PDS3) or Bundle (PDS4) based on the files included in the SKD. It includes validating functionalities which should be used for consistency checks after the execution of the pipeline.

Installation

Hardware and Software requirements

ARCGEN is a Python 3.6.x package that uses FORTRAN 77 and FORTRAN 90 applications and a set of standard Python libraries.

ARCGEN is designed to work with UNIX systems namely with Linux and with MacOS. The hardware requirements are minimal and any modern computer/server will suffice. The software is able to run for example in a minimal CentOS 7 Virtual Machine:

  • Architecture: x86_64 (64 bits)

  • Memory: 1GB/logical CPU

  • Disk space: 10GB

The software requirements are the following:

  • Python 3.6 or higher

  • Python packages:

    • numpy >= 1.8.0

    • spiceypy

  • gcc

  • libgfortran

  • Git (possibly, for the obtention of SPICE Kernel Datasets)

Obtaining the Software

Installing the source code

The source code of ARCGEN is under configuration control with Git. The BitBucket repository that hosts it is the following: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/arcgen/browse, since this is a private repository access might not be granted, if so please contact the author of this document. Alternatively the source code can be obtained under request from esa_spice@sciops.esa.int.

If the user is to install ARCGEN from its source code it is recommended to use PiPy to do so (taking advantage of the Python setup file), in order to do so using the Terminal go to the top level ARCGEN directory, arcgen and type:

pip install [--user] -e .

Please note that the --user parameter is optional; if you are using your user Python installation it needs to be indicated.

Once installed, you will be able to run ARCGEN anywhere in your computer.

Source Code structure

A brief overview of the code structure for ARCGEN is useful for its installation, configuration and usage:

ADCSng Package top level structure

aescalante

 

VEX and MEX example config files.

bgrieger

ROS example config file.

sandbox

Some routines for various checks.

README.md

Readme file (as per any other Python package).

setup.py

Setup Python routine.

arcgen

Package directory.

 

command_line.py

 

Main function called when invoking ARCGEN.

 

classes

Directory that contains Python source code classes.

 

config

Directory that contains the mission-specific developer configuration items for ARCGEN.

 

core

Directory that contains Python source code core functions.

 

etc

Directory that contains different templates used by ARCGEN (mostly the ones used by the FORTRAN utilities).

 

utils

Directory that contains Python source code utility functions.

 

tests

Directory that contains the tests performed for ARCGEN.

Configuration

The next step to use ARCGEN is to configure it.

The configuration has to be edited and prepared by the user. It consists of a JSON file and an example from the test directory of ARCGEN can be used as baseline (arcgen/arcgen/tests/vex.json). Any name can be provided to the configuration file, the file extension has to be *.json. The configuration file will determine the directory structure that you need to setup for ARCGEN. The recommendation is to create a directory out of the ARCGEN directory structure and to create one per intended ARCGEN execution/project.

The following table describes all the available configuration items (please note that all the items are strings):

Key

Value(s)

Description

Required?

version

Data set Version number

Data set Version number

YES

name

ARCGEN Configuration File

Description of the JSON file

NO

bundle_directory

Bundle/Data Set output directory

Directory where ARCGEN puts the generated archive

YES

kernels_directory

SKD directory

Directory where ARCGEN looks for Kernels as input

YES

increment

Archive last increment

Existing version of the archive to increment

YES

increment_stop

Increment Stop date

Stop date for the new increment

YES

mission_name

  • BEPICOLOMBO

  • ExoMars2016

  • ExoMarsRSP

  • JUICE

  • MARS EXPRESS

  • VENUS EXPRESS

  • Solar_Orbiter

Name of the mission, has to be one of the provided in the list. (corresponds to the name of the SKDs replacing the dashes).

YES

mission_accronym

  • BC

  • EM16

  • EMRSP

  • JUICE

  • MEX

  • VEX

  • SOLO

Mission accronym, has to be one of the provided in the list.

YES

dataset

Data Set name

Name of the Data Set

YES

dataset_name

Data Set descriptive name

Data Set descriptive name

NO

volume_id

Volume Identifier

Identifier of the Archive Volume

YES

spacecraft

Spacecraft Name

Name of the Spacecraft

YES

target

Target Name

Name of the Target (Mars, Venus, Mercury,...)

YES

mission_start

Start UTC date

Start date of the archive (YYYY-MM-DD)

YES

mission_stop

Stop UTC date

Stop date of the archive (YYYY-MM-DD)

YES

secondary_spacecrafts

Secondary spacecrafts names

Name of any secondary spacecraft (e.g. PHILAE)

YES

secondary_targets

Secondary target names

Name of any secondary target (e.g. DEIMOS, PHOBOS)

YES

metakernel

Metakernel Path

SKD Metakernel used for archive generation

YES

mk_grammar

MK Grammar config file

Configuration file with MK Grammar. Used to include/exclude files from the SKD into the generated archive.

YES

pds

PDS Version

  • 3

  • 4

Version of PDS format

YES

author

Author name

Name of the Generated archive author

NO

Once the configuration file has been generated it can be placed anywhere in your computer, it is recommended to put it in the same directory level as where you will define the directory structure of your ARCGEN configuration. What follows is an example of an ARCGEN configuration file:

{
"version": "2.0",
"name": "ARCGEN Configuration File",
"bundle_directory": "/Users/aescalante/spice/missions/mex/MEX-E-M-SPICE-6-V2.0",
"kernels_directory": "/Users/aescalante/spice/missions/mex/kernels",
"increment": "",
"increment_stop": "2020-12-31T23:59:59",
"mission_name": "MARS EXPRESS",
"dataset": "MEX-E/M-SPICE-6-V2.0",
"dataset_name": "MARS EXPRESS SPICE KERNELS V2.0",
"volume_id": "MEXSP_2000",
"mission_accronym": "MEX",
"spacecraft": "MARS EXPRESS",
"target": "MARS",
"mission_start": "1997-10-31",
"mission_stop": "2020-05-31",
"secondary_spacecrafts": "",
"secondary_targets": "",
"xml_model": "xml_model",
"schema_location": "schema_location",
"information_model_version": "information_model_version",
"metakernel": "/Users/aescalante/spice/missions/mex/kernels/mk/MEX_OPS.TM",
"mk_grammar": "/Users/aescalante/spice/pipelines/arcgen/arcgen/config/mex.grammar",
"pds": "3",
"author": "Alfredo Escalante Lopez"
}

If we consider the previous configuration file we could have a directory structure as follows:

mex.json
MEX-E-M-SPICE-6-V2.0
|-- AAREADME.TXT
|-- CATALOG
|-- DATA (ARCGEN will put kernels with labels here)
|-- DOCUMENT
|-- ERRATA.TXT
|-- EXTRAS
| |-- MK
| '-- ORBNUM
|-- INDEX
'-- VOLDESC.CAT
kernels
|-- spk
|-- ck
:
'-- mk

Remember that you need to generate this directory structure for ARCGEN will not generate it for you.

Kernels and Output directories requirements

You need a working SPICE Kernel Dataset to run ARCGEN

It is important to note that you need a local copy of a working SKD in order to execute ARCGEN. This document does not intend to provide you a detailed explanation on how to obtain the latest version of a given SKD, instead we recommend you to read the information in the following page: https://www.cosmos.esa.int/web/spice/data

What is important to note though is that you will need to have a working SKD. In order to check that you do, we recommend you to use a SPICE utility such as BRIEF on the MK in order to ensure that the SKD is properly set up.

Executing ARCGEN

In order to execute ARCGEN you only need to type the command: arcgen, the command usage is as follows:

arcgen [-h] [-e] [-l] [-t] CONFIG [CONFIG ...]

Indicating the configuration file path is mandatory, the options are displayed by the -h or --help options:

ARCGEN -- Version 1.0.0, SPICE PDS3/PDS4 Archive Generation Pipeline
 
ARCGEN is the command-line utility program that generates PDS4
Bunldes and PDS3 Datasets for SPICE Kernel Datasets. Source and
documentation are available here:
 
https://repos.cosmos.esa.int/socci/projects/SPICE/repos/arcgen
 
positional arguments:
CONFIG Mission specific JSON configuration file
 
optional arguments:
-h, --help show this help message and exit
-e, --labels Execute ARCGEN to generate labels only
-l, --log Prompt log during execution
-t, --transfer Execute ARCGEN to generate PSA transfer manifest
 
__ __ __ __ __ __ ___ __ ___ __ __ ___
/__\ /__` '__\ /__` |__) | / ` |__ /__` |__ |__) \ / | / ` |__
\__, .__/ \__/ .__/ | | \__, |___ .__/ |___ | \ \/ | \__, |___
 
esa_spice@sciops.esa.int
http://spice.esac.esa.int

Please note that if you do not use the optional argument -l or --log you will not be prompted the log of the execution. Error messages prior to the log creation might be shown though.

Once ARCGEN is executed, in a nutshell it checks the kernels directory for input SKD. The kernel grammar is used to include/exclude kernels from the SKD into the generated data set/bundle and generate the metakernel of the archive. If a previous archive is provided with "increment" keyword, ARCGEN searches for previous files to be included in the current increment. Finally, the index file is generated listing the files in the archive.

In the case of generating a PDS3 Data Set, after running the pipeline the user has to manually update several files of the archive:

  • AAREADME.TXT

  • CATALOG

  • DOCUMENT

  • ERRATA.TXT

  • VOLDESC.CAT

It is also a good practice to run additional checks on the generated archive to make sure the files have been generated correctly, the format is adequate and the PDS standards have been followed properly. In the directory arcgen/aescalante, the python script checkText.py can be used to check some PDS3 format consistencies, end of lines, line lengths and label keywords.

DVal tool may be used as well to perform some checks on the Data Set. Download the Data Validator command line tool from ftp://ssols01.esac.esa.int/pub/software/DValNG/index.html#advanced. Unpack the gzipped tar file. Executables are in DatasetValidator/bin. Before running any checks, run ConfigSynchronizer. Calling GrammarChecker without arguments lists possible options and parameters. In order to create a log file listing the issues found by the tool, GrammarChecker as:

DatasetValidator/bin/GrammarChecker -f grammarcheckerlog.txt -i ARCGEN_OUTPUT

It will find issues related to incorrect line endings, non ascii characters or incorrect label structure.