ID

Title

Version

Author

Operator

Category

Status

SPICE-PR-EMRSP-001

spice-pipeline-datapackage-delivery

2

Marc Costa

ESS Engineer

ROCC Engineer

ADCSNG

CONSOLIDATED

Objective

This procedure describes how to generate a SPICE Pipelien Datapackage delivery for the ROCC and how to test it on the ROCC side.

Start

Release of an ADCSng, ARCGEN or ExoMarsRSP SKD version for the ROCC.

Frequency:

Every 4-6 months

Duration:

Half a day

In case of problem(s)

Contact Marc Costa

Detailed Procedure

Steps highlighted in green only need to be followed by the ESA SPICE Service Engineer.

Step

Description

Action

1

Obtain repository

Obtain the last version of the spice-pipeline-datapackage repository from: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/spice-pipeline-datapackage/

2

ADCSng update

Obtain and test the latest version of ADCSng. Either locally or on a ESS Server, install the latest version of ADCSng and run the regression tests. Do any required modifications.

3

ARCGEN update

Obtain and test the latest version of ARCGEN. Either locally or on a ESS Server, install the latest version of ARCGEN and run the functional test for label generation. Do any required modifications.

4

SKD update

Obtain and test the latest version of the ExoMarsRSP kernels from the BitBucket repository.

5

Documentation update

If required update the following documentation:

These are the files required as per ICD. Please note that the files are provided as HTML documents. These HTML documents are exported automatically by the pipeline.

6

Update the repository

Copy the updated documentation and any other issue found in the datapackage of the repository (excluding adcsng and arcgen but including their configurations).

Commit and push the changes (later on you will repeat this process).

7

Copy repository to deliver.

Copy the repository to a directory named after the ADCSng version. For example:

cp -r spice-pipeline-datapackage spice-pipeline-datapackage-v250

This directory will be the basis with which you will prepare the delvery.

8

Copy adcsng, arcgen and the ExoMarsRSP repositories in the delivery

rsync -av --progress /Users/mcosta/adcsng ./spice-pipeline-datapackage-v250/spice-pipeline/ --exclude '.*'
rsync -av --progress /Users/mcosta/arcgen ./spice-pipeline-datapackage-v250/spice-pipeline/ --exclude '.*'
rsync -av --progress /Users/mcosta/SPICE/ExoMarsRSP ./spice-pipeline-datapackage-v250/spice-pipeline/ --exclude '.*

9

Generate the Docker image

Generate the Docker image using the script in the spice-pipeline-datapackage: /build_docker.sh

The script shall look something like this:

#!/usr/bin/env bash
TAG_NAME='spice-pipeline:latest'
cd spice-pipeline
git clone ssh://git@repos.cosmos.esa.int:7998/spice/adcsng.git
git clone ssh://git@repos.cosmos.esa.int:7998/spice/arcgen.git
git clone https://repos.cosmos.esa.int/socci/scm/spice_kernels/exomarsrsp.git
sudo docker build . -t $TAG_NAME

10

Check that the Docker image has been created.

->> docker $ docker ps -all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0bbfd859f983 spice-pipeline:latest "/bin/bash" 2 minutes ago Exited (0) 2 minutes ago great_visvesvaraya
->> docker $ docker cp incoming/. 0bbfd859f983:/root/incoming
->> docker $ docker cp kernels/. 0bbfd859f983:/root/kernels

11

Run the Docker container and test the pipeline

Run the SPICE pipeline in the docker container and check that the pipeline is executed with the test files.

docker run spice-pipeline:latest ./run_spice-pipeline.sh

Start the container and perform the tests inside the container.

docker run -it spice-pipeline:latest

If any issues are found perform any required changes to the scripts and data to make it work

12

Remove the PaxHeader files

Remove ant PaxHeader files present in the datapack directory, those include .idea or .DS_Store directories.

13

Commit & Push the repository

After implementing all the changes and verifying that the new versions of the SKD, ARCGEN and ADCSng are working, commit and push the relevant changes in the spice-pipeline-datapackage repository (as done before).

Tag the repository with the corresponding ADCSng version with the x.y.z format only.

14

Datapack publication

Compress the repository as a *.tar.gz file

Publish the compressed datapackage in ftp://spiftp.esac.esa.int/temp/rocc/datapackages/

15

Delivery notification

Send an e-mail notification to the relevant people.

1

Download datapackage

Download the datapackage from: ftp://spiftp.esac.esa.int/temp/rocc/datapackages/

The datapackage follow this convention: spice-pipeline-datapackage-vXYZ.tar.gz

where XYZ is the ADCSng release version. The datapackage has the following structure:

.
|-- doc
|   |-- ADCSng Software User Manual
|   |-- ARCGEN Software User Manual
|   '-- ADCSng Release Notes
|   '-- ARCGEN Release Notes
|-- build_docker.sh
|-- run_docker.sh
|-- setup_centos.sh
'-- spice-pipeline
|-- Dockerfile
|-- ExoMarsRSP
|   |-- MANIFEST.in
|   |-- README.md
|   |-- kernels
|   |-- misc
|   '-- version
|-- adcsng
|-- adcsng.json
|-- arcgen
|-- arcgen.json
|-- incoming
|   |-- emrsp_rm_mes_gnc_0001_v001.aem
|   |-- emrsp_rm_mes_gnc_0001_v001.oem
|   |-- emrsp_rm_mes_mech_0001_v001.aem
|   |-- emrsp_rm_mes_mech_0001_v001.oem
|   |-- emrsp_rm_rec_gnc_0001_v001.oem
|   '-- emrsp_rm_rec_gnc_0002_v001.aem
'-- run_spice-pipeline.sh

The relevant documentation is available in spice-pipeline-datapackage-vXYZ/spice-pipeline/doc. The documents are:

Please note that the OEM/AEM/TCPs provided in the incoming directory are for testing purposes only.

2

Build the CentOS VM

In order to build the CentOS VM to integrate in ROCS the following files are relevant:

  • setup_centos.sh

    • This script contains what is required to be executed in the VM in order to set it up.

    • DISCLAIMER: Please note that the 'commands' to copy the files/directories required for the VM are not specified.

  • spice-pipeline

    • adcsng: This directory contains ADCSng and must be copied in the CentOS VM root dir

    • adcsng.json: This is the ADCSng configuration file and must be copied in the CentOS VM root dir

    • arcgen: This directory contains ADCSng and must be copied in the CentOS VM root dir

    • arcgen.json: This is the ADCSng configuration file and must be copied in the CentOS VM root dir

    • run_spice-pipeline.sh: This is the script that needs to be run to execute the pipeline. Must be copied in the CentOS VM root dir

3

Docker testing

It is recommended that the ROCC operator tests the pipeline in Docker as specified earlier in the procedure.

4

Follow ROCS integration procedures

Needless to say the last step is to follow the procedures to integrate, test and deploy the SPICE Pipeline in the ROCS.

5

Run the pipeline

Please note that in order to execute the pipeline the following needs to be provided to the VM:

  1. Source files (OEM, AEM, TCP) must be copied to the root/incoming/ directory

  2. Latest version of the SPICE Kernel Dataset (Complete!) must be copied to the root/incoming/kernels/ directory (relevant ck, spk, lsk, etc directory structure, including aareadme.txt files)

After providing those files, run the 'run_spice-pipeline.sh' script in the VM. This script does the following:

  1. Ensures that the aareadme.txt files are present in the output directory.

  2. Executes ADCSng

  3. Executes ARCGEN to generate the PDS4 Label files for the generated kernels

After the execution has run successfully the generated files have to be extracted from root/output (with the relevant directory structure).

6

Follow-up of a failed run

If you execute a run and it fails, due to the special nature of some of the EMRSP developments you will need to cleanup the 'incoming' directory and the 'temp' directory; please delete the intermediate files left there.

Future versions of the datapack will address this issue.

7

Feedback the ESA SPICE Service

Any feedback should be provided to the ESA SPICE Service: esa_spice@sciops.esa.int

We expect feedback such as how to specify in the setup_centos.sh script how to copy the files in the VM, bug reporting, integration issues, etc.

History

Revision

Date

Author

Approved by

Comments

0

14 Apr 2020

Marc Costa

Marc Costa

First release.

1

13 Oct 2020

Marc Costa

Marc Costa

Updates for datapack v2.5.0 release