ID |
Title |
Version |
Author |
Operator |
Category |
Status |
SPICE-PR-EMRSP-001 |
spice-pipeline-datapackage-delivery |
2 |
ESS Engineer ROCC Engineer |
ADCSNG |
CONSOLIDATED |
|
Objective |
This procedure describes how to generate a SPICE Pipelien Datapackage delivery for the ROCC and how to test it on the ROCC side. |
|||||
Start |
Release of an ADCSng, ARCGEN or ExoMarsRSP SKD version for the ROCC. |
Frequency: |
Every 4-6 months |
Duration: |
Half a day |
|
In case of problem(s) |
Contact Marc Costa |
Detailed Procedure
Steps highlighted in green only need to be followed by the ESA SPICE Service Engineer.
Step |
Description |
Action |
1 |
Obtain repository |
Obtain the last version of the spice-pipeline-datapackage repository from: https://repos.cosmos.esa.int/socci/projects/SPICE/repos/spice-pipeline-datapackage/
|
2 |
ADCSng update |
Obtain and test the latest version of ADCSng. Either locally or on a ESS Server, install the latest version of ADCSng and run the regression tests. Do any required modifications. |
3 |
ARCGEN update |
Obtain and test the latest version of ARCGEN. Either locally or on a ESS Server, install the latest version of ARCGEN and run the functional test for label generation. Do any required modifications. |
4 |
SKD update |
Obtain and test the latest version of the ExoMarsRSP kernels from the BitBucket repository. |
5 |
Documentation update |
If required update the following documentation:
These are the files required as per ICD. Please note that the files are provided as HTML documents. These HTML documents are exported automatically by the pipeline. |
6 |
Update the repository |
Copy the updated documentation and any other issue found in the datapackage of the repository (excluding adcsng and arcgen but including their configurations). Commit and push the changes (later on you will repeat this process). |
7 |
Copy repository to deliver. |
Copy the repository to a directory named after the ADCSng version. For example: cp -r spice-pipeline-datapackage spice-pipeline-datapackage-v250 This directory will be the basis with which you will prepare the delvery. |
8 |
Copy adcsng, arcgen and the ExoMarsRSP repositories in the delivery |
rsync -av --progress /Users/mcosta/adcsng ./spice-pipeline-datapackage-v250/spice-pipeline/ --exclude '.*' rsync -av --progress /Users/mcosta/arcgen ./spice-pipeline-datapackage-v250/spice-pipeline/ --exclude '.*' rsync -av --progress /Users/mcosta/SPICE/ExoMarsRSP ./spice-pipeline-datapackage-v250/spice-pipeline/ --exclude '.* |
9 |
Generate the Docker image |
Generate the Docker image using the script in the spice-pipeline-datapackage: /build_docker.sh The script shall look something like this: #!/usr/bin/env bash TAG_NAME= 'spice-pipeline:latest' cd spice-pipeline git clone ssh : //git @repos.cosmos.esa.int:7998 /spice/adcsng .git git clone ssh : //git @repos.cosmos.esa.int:7998 /spice/arcgen .git git clone https: //repos .cosmos.esa.int /socci/scm/spice_kernels/exomarsrsp .git sudo docker build . -t $TAG_NAME |
10 |
Check that the Docker image has been created. |
->> docker $ docker ps -all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0bbfd859f983 spice-pipeline:latest "/bin/bash" 2 minutes ago Exited (0) 2 minutes ago great_visvesvaraya ->> docker $ docker cp incoming/. 0bbfd859f983: /root/incoming ->> docker $ docker cp kernels/. 0bbfd859f983: /root/kernels |
11 |
Run the Docker container and test the pipeline |
Run the SPICE pipeline in the docker container and check that the pipeline is executed with the test files. docker run spice-pipeline:latest . /run_spice-pipeline .sh Start the container and perform the tests inside the container. docker run -it spice-pipeline:latest If any issues are found perform any required changes to the scripts and data to make it work |
12 |
Remove the PaxHeader files |
Remove ant PaxHeader files present in the datapack directory, those include .idea or .DS_Store directories. |
13 |
Commit & Push the repository |
After implementing all the changes and verifying that the new versions of the SKD, ARCGEN and ADCSng are working, commit and push the relevant changes in the spice-pipeline-datapackage repository (as done before). Tag the repository with the corresponding ADCSng version with the x.y.z format only. |
14 |
Datapack publication |
Compress the repository as a *.tar.gz file Publish the compressed datapackage in ftp://spiftp.esac.esa.int/temp/rocc/datapackages/ |
15 |
Delivery notification |
Send an e-mail notification to the relevant people. |
1 |
Download datapackage |
Download the datapackage from: ftp://spiftp.esac.esa.int/temp/rocc/datapackages/ The datapackage follow this convention: spice-pipeline-datapackage-vXYZ.tar.gz where XYZ is the ADCSng release version. The datapackage has the following structure: . |-- doc | |-- ADCSng Software User Manual | |-- ARCGEN Software User Manual | '-- ADCSng Release Notes | '-- ARCGEN Release Notes |-- build_docker.sh |-- run_docker.sh |-- setup_centos.sh '-- spice-pipeline |-- Dockerfile |-- ExoMarsRSP | |-- MANIFEST. in | |-- README.md | |-- kernels | |-- misc | '-- version |-- adcsng |-- adcsng.json |-- arcgen |-- arcgen.json |-- incoming | |-- emrsp_rm_mes_gnc_0001_v001.aem | |-- emrsp_rm_mes_gnc_0001_v001.oem | |-- emrsp_rm_mes_mech_0001_v001.aem | |-- emrsp_rm_mes_mech_0001_v001.oem | |-- emrsp_rm_rec_gnc_0001_v001.oem | '-- emrsp_rm_rec_gnc_0002_v001.aem '-- run_spice-pipeline.sh The relevant documentation is available in spice-pipeline-datapackage-vXYZ/spice-pipeline/doc. The documents are:
Please note that the OEM/AEM/TCPs provided in the incoming directory are for testing purposes only. |
2 |
Build the CentOS VM |
In order to build the CentOS VM to integrate in ROCS the following files are relevant:
|
3 |
Docker testing |
It is recommended that the ROCC operator tests the pipeline in Docker as specified earlier in the procedure. |
4 |
Follow ROCS integration procedures |
Needless to say the last step is to follow the procedures to integrate, test and deploy the SPICE Pipeline in the ROCS. |
5 |
Run the pipeline |
Please note that in order to execute the pipeline the following needs to be provided to the VM:
After providing those files, run the 'run_spice-pipeline.sh' script in the VM. This script does the following:
After the execution has run successfully the generated files have to be extracted from root/output (with the relevant directory structure). |
6 |
Follow-up of a failed run |
If you execute a run and it fails, due to the special nature of some of the EMRSP developments you will need to cleanup the 'incoming' directory and the 'temp' directory; please delete the intermediate files left there. Future versions of the datapack will address this issue. |
7 |
Feedback the ESA SPICE Service |
Any feedback should be provided to the ESA SPICE Service: esa_spice@sciops.esa.int We expect feedback such as how to specify in the setup_centos.sh script how to copy the files in the VM, bug reporting, integration issues, etc. |
History
Revision |
Date |
Author |
Approved by |
Comments |
0 |
14 Apr 2020 |
First release. |
||
1 |
13 Oct 2020 |
Updates for datapack v2.5.0 release |