Disclaimer

These instructions are heavily based on similar instructions prepared for the "eumed" VO. Indeed, you will find a lot of references to EUMED which you should substitute with VO_EU_DECIDE_EU.

Note for the impatient

If you are only interested in getting things installed, please read the "Installation Instructions" section and carefully follow the outlined steps.

*NOTE:* the software tags requested by applications in the Science Gateway are managed manually by SGM users, once all pre-requisite software has been installed using the grid-swinstall package.

Otherwise, you are welcome to read the following paragraphs and get some deeper insight in how the machinery works.

USING THE grid-swinstall PACKAGE

All packages are installed using a simplified procedure offered by the grid-swinstall package, see at the bottom of the page for the latest available version of grid-swinstall_<date>.tgz tarfile.

The grid-swinstall package is maintained by Fulvio Galeazzi any software manager are free (and welcome) to provide changes to the current scripts and/or add new installation/validation procedures; in such a case please keep updated Fulvio Galeazzi about the changes in order to incorporate changes in forthcoming versions of the package.

Overview of the grid-swinstall package

The distributed tarfile contains a script GridManageSW which is a generic procedure performing a lot of useful actions. Generally speaking, to install a software package you will, in general, need to:

  • download a tarfile from the GRID and untar it: this step is called installation
  • compile the package (if need be): this step is called validation

For each package, you will thus need to launch the script twice, once for "installation" and, as soon as installation has successfully completed, a second time for validation. Keep reading, for a detailed example.

The script takes care of publishing intermediate tags (read next chapter) to signal pending installation activity, and the successful or problematic completion of each step. If a software package is in status "processing" and you submit again another installation/validation job, this will exit immediately: this ensures that steps are performed one at a time. The script will also recursively fix ownership and privileges for each file/directory such that all eumed users may access those files, and all different sgmeumed users have write access (this is needed because over time the same person may be mapped to two different sgmeumed accounts). It also produces a quite verbose log of what is being done, so that in case of troubles it should be rather easy to understand what has gone wrong.

Tags format being published by grid-swinstall package

The tags which are published when using grid-swinstall have a format, explained in detail below, which has a direct correspondance to the disk path under which the software will be found. For example, a package tagged VO-eumed-prod-pkg1-verX-gccany will be found at the path $VO_EUMED_SW_DIR/prod/pkg1/verX/gccany/: as you see, we have stripped the VO-eumed part, and replaced each further - with /, it's so simple.

Package VO-eumed-prod-local provides the script getrelpath.sh which does the work for you: you supply a tag name and it returns the relevant subpath under $VO_EUMED_SW_DIR:

[galeazzi@ui2-4 Grid]$ $VO_EUMED_SW_DIR/prod/local/bin/getrelpath.sh VO-eumed-prod-pkg1-verX-gccany
/prod/pkg1/verX/gccany

The general format for tags is the following VO-<VO_name>-<status>-<pkgName>-<pkgVersion>-<SOname>-<SOarch>-<compilerName>, like for example VO-eumed-prod-gromacs-v040503-SL5-x86_64-gccany, where:

  • <VO_name>: the name of the VO, eumed most probably since you are reading this
  • <status>: this can be prod or test, the former for official production packages, the latter for packages to be used for test purposes only
  • <pkgName>: the name of the package
  • <pkgVersion>: the version of the package
  • <SOname>: the name of the operating system the package is supposed to run on, typically it will be SL5
  • <SOarch>: the architecture for which the package was compiled, typically it will be i386 or x86_64
  • <compilerName>: the compiler used. If the default compiler found was used, the string will be gccany

Also note that the tag name may contain the string ANY: any string after ANY is intended as a comment, and will not be interpreted by the string-to-pathname converter getrelpath.sh.

While software installation is ongoing, you will see some other tags being published. Assuming you are installing VO-eumed-pkg1-verX, you will see:

VO-eumed-pkg1-verX-processing
whenever a script is running (installation, validation, uninstallation)
VO-eumed-pkg1-verX-aborted-<action>
if the <action> script failed (installation, validation)
VO-eumed-pkg1-verX-to-be-validated
after successful installation of the package. As the tag suggests, you are now ready to run the validation script.
VO-eumed-pkg1-verX
after successful running of the validation script

Last important note: after successful installation, GridManageSW will also touch a hidden file .<software-tag>, for example:

[galeazzi@ui2-4 ~]$ $VO_EUMED_SW_DIR/prod/local/bin/getrelpath.sh VO-eumed-prod-gamess-v101001-SL5-x86_64-gccany
/prod/gamess/v101001/SL5/x86_64/gccany
[galeazzi@ui2-4 ~]$ ls -al $VO_EUMED_SW_DIR/prod/gamess/v101001/SL5/x86_64/gccany | tail -1
-rw-rw-r--  1 sgmeumed001 sgmeumed      0 Apr  5 17:37 .VO-eumed-prod-gamess-v101001-SL5-x86_64-gccany

Installation instructions

If you want to install package pkg1, make sure:

  • a tarfile is available and registered in the catalogue under /grid/vo.eu-decide.eu/SoftRepo/ (I use LFC catalogue lfc-01.ct.trigrid.it): the tarfile name is referenced in the .jdl scripts, see option --tar
  • you have files needed for installation and validation: install_decide_pkg1.jdl, validate_decide_pkg1.jdl, and possibly also validate_decide_pkg1.sh: check in the .jdl scripts the tarfile name, and the tag name which the scripts are supposed to publish
  • run glite-wms-job-submit -a -o install_pkg1.id -r <theCEwhereYouWantToInstall> install_decide_pkg1.jdl this will download the tarfile and explode it in the proper place
  • after successful job completion, run: glite-wms-job-submit -a -o install_pkg1.id -r <theCEwhereYouWantToInstall> validate_decide_pkg1.jdl this will (possibly) run compilation of the package (also using validate_decide_pkg1.sh script) and publish the tag if everything went fine

If the validation job ran correctly, the software tag should have been published by the site.

*NOTE:* the software tags requested by applications in the Science Gateway are managed manually by SGM users, once all pre-requisite software has been installed using the grid-swinstall package.

ADVANCED USE: preparing your own installation scripts

The philosophy of grid-swinstall is rather simple. You can read the GridManageSW script as well as the various install_*.jdl and validate_*.jdl for further information, but I will present here a full example.

Let's assume you have:

  • a tarfile myPkg-v000001.tar.gz
  • a software tag you'd like to publish: VO-eumed-test-myPkg-v000001
  • corresponding to the tag name above, our software will be installed under $VO_EUMED_SW_DIR/test/myPkg/v000001/. This imposes some constraint on the content of the tarfile:
    • if you are installing pre-compiled binaries, you need to make sure that the tarfile does not contain a common first-level subdirectory. I mean, people will expect to find the application under path $VO_EUMED_SW_DIR/test/myPkg/v000001/ so typically the binaries will be under $VO_EUMED_SW_DIR/test/myPkg/v000001/bin/: hence, if the tarfile has a structure similar to:
      [galeazzi@ui2-4 Packages]$ tar tzvf scilab-5.3.1.bin.linux-x86_64.tar.gz | head
      drwxr-xr-x scilab/scilab     0 2011-03-03 17:19:22 scilab-5.3.1/
      -rw-r--r-- scilab/scilab  8099 2011-03-03 17:19:22 scilab-5.3.1/CHANGES_4.X
      -rw-r--r-- scilab/scilab  1633 2011-03-03 17:19:22 scilab-5.3.1/README_Unix
      -rw-r--r-- scilab/scilab 30661 2011-03-03 17:19:22 scilab-5.3.1/CHANGES_5.1.X
      -rw-r--r-- scilab/scilab  1907 2011-03-03 17:19:22 scilab-5.3.1/RELEASE_NOTES_5.1.X
      -rw-r--r-- scilab/scilab  4854 2011-03-03 17:19:22 scilab-5.3.1/RELEASE_NOTES_5.0.X
      drwxr-xr-x scilab/scilab     0 2011-03-03 17:19:22 scilab-5.3.1/bin/
      -rwxr-xr-x scilab/scilab 413344 2011-03-03 17:18:42 scilab-5.3.1/bin/XML2Modelica
      -rwxr-xr-x scilab/scilab  30706 2011-03-03 17:19:11 scilab-5.3.1/bin/scilab
      lrwxrwxrwx scilab/scilab      0 2011-03-03 17:19:22 scilab-5.3.1/bin/scilab-adv-cli -> scilab
      [galeazzi@ui2-4 Packages]$ tar tzvf scilab-5.3.1 | head
      scilab-5.3.1.bin.linux-x86_64.tar.gz  scilab-5.3.1-x86_64.tar.gz
      you need to re-package it such that it looks like:
      [galeazzi@ui2-4 Packages]$ tar tzvf scilab-5.3.1-x86_64.tar.gz | head
      drwxr-xr-x galeazzi/grid     0 2011-03-03 17:19:22 ./
      -rw-r--r-- galeazzi/grid 28189 2011-03-03 17:19:22 ./CHANGES_5.0.X
      -rw-r--r-- galeazzi/grid 21097 2011-03-03 17:19:22 ./COPYING
      -rw-r--r-- galeazzi/grid  7372 2011-03-03 17:19:22 ./ACKNOWLEDGEMENTS
      -rw-r--r-- galeazzi/grid  2337 2011-03-03 17:19:22 ./RELEASE_NOTES_5.2.X
      drwxr-xr-x galeazzi/grid     0 2011-03-03 17:19:11 ./share/
      drwxr-xr-x galeazzi/grid     0 2011-03-03 17:19:12 ./share/locale/
      drwxr-xr-x galeazzi/grid     0 2011-03-03 17:19:11 ./share/locale/ca_ES/
      drwxr-xr-x galeazzi/grid     0 2011-03-03 17:19:11 ./share/locale/ca_ES/LC_MESSAGES/
      -rw-r--r-- galeazzi/grid 461101 2011-03-03 17:19:11 ./share/locale/ca_ES/LC_MESSAGES/scilab.mo
    • if you are compiling from source code, you can adjust paths in the validation stage (read below). Note however a very important thing: for obvious reasons, the validation stage needs to be run unattended. Hence, you may have to apply some tricks and patch the original configuration scripts (see for example the GAMESS tarfile, containing a patch for the configure script).

Once you have your tarfile ready, you need to copy to the GRID, bearing in mind:

  • the catalogue to use is LFC_HOST = lfc.ulakbim.gov.tr
  • the GridManageSW scripts assumes that tarfiles will be found under /grid/eumed/SoftRepo/
  • for the sake of redundancy, it is advisable to copy the tarfile to at least two different SE

Find below the list of commands I am typically using: in this example, first copy is registered at GARR, second copy at DZ-01-ARN.

[galeazzi@ui2-4 Packages]$ lcg-cr -d gridsrv3-4.dir.garr.it -l lfn:/grid/eumed/SoftRepo/myPkg-v000001.tar.gz file:myPkg-v000001.tar.gz
guid:53f6a7f1-e08c-418e-a907-bdb3e6199078
[galeazzi@ui2-4 Packages]$ lcg-lr lfn:/grid/eumed/SoftRepo/myPkg-v000001.tar.gz
srm://gridsrv3-4.dir.garr.it/dpm/dir.garr.it/home/eumed/generated/2011-04-20/file01e269b4-af1c-4886-a0de-29d51fc68454
[galeazzi@ui2-4 Packages]$ lcg-rep --vo eumed -n 4 -d se01.grid.arn.dz -v -P /grid/eumed/SoftRepo/myPkg-v000001.tar.gz srm://gridsrv3-4.dir.garr.it/dpm/dir.garr.it/home/eumed/generated/2011-04-20/file01e269b4-af1c-4886-a0de-29d51fc68454
Using grid catalog type: UNKNOWN
Using grid catalog : lfc.ulakbim.gov.tr
VO name: eumed
Checksum type: None
Trying SURL srm://gridsrv3-4.dir.garr.it/dpm/dir.garr.it/home/eumed/generated/2011-04-20/file01e269b4-af1c-4886-a0de-29d51fc68454 ...
Source SE type: SRMv2
Source SRM Request Token: d97c8978-3dbf-480e-b0dc-327faf522eb2
Destination SE type: SRMv2
Destination SRM Request Token: 9c03aa1b-7d33-485e-a666-69a808556421
Source URL: srm://gridsrv3-4.dir.garr.it/dpm/dir.garr.it/home/eumed/generated/2011-04-20/file01e269b4-af1c-4886-a0de-29d51fc68454
File size: 1902051
Destination specified: se01.grid.arn.dz
Source URL for copy: gsiftp://gridsrv3-4.dir.garr.it/gridsrv3-4.dir.garr.it:/data/eumed/2011-04-20/file01e269b4-af1c-4886-a0de-29d51fc68454.58471.0
Destination URL: gsiftp://se01.grid.arn.dz/se01.grid.arn.dz:/data/eumed/2011-04-20/myPkg-v000001.tar.gz.2008.0
# streams: 1
      1902051 bytes     72.84 KB/sec avg    130.94 KB/sec inst
Transfer took 27070 ms
Using LFN: lfn:(null)
Using GUID: guid:53f6a7f1-e08c-418e-a907-bdb3e6199078
Registering SURL: srm://se01.grid.arn.dz/dpm/grid.arn.dz/home/eumed//grid/eumed/SoftRepo/myPkg-v000001.tar.gz (53f6a7f1-e08c-418e-a907-bdb3e6199078)
Destination URL registered in file catalog: srm://se01.grid.arn.dz/dpm/grid.arn.dz/home/eumed//grid/eumed/SoftRepo/myPkg-v000001.tar.gz
[galeazzi@ui2-4 Packages]$ lcg-lr lfn:/grid/eumed/SoftRepo/myPkg-v000001.tar.gzsrm://gridsrv3-4.dir.garr.it/dpm/dir.garr.it/home/eumed/generated/2011-04-20/file01e269b4-af1c-4886-a0de-29d51fc68454
srm://se01.grid.arn.dz/dpm/grid.arn.dz/home/eumed//grid/eumed/SoftRepo/myPkg-v000001.tar.gz

You are now ready for the installation phase. By installation I mean the following actions:

  • downloading of the package from the GRID
  • creation of the destination directory
  • un-tar of the archive
  • recursively change ownership and access rights (add SGMgroup write-ability)
  • publication of the software tag -to-be-validated

All the above tasks are accomplished by the GridManageSW script alone, you just need to submit one job like:

glite-wms-job-submit -a -e https://wms-4.dir.garr.it:7443/glite_wms_wmproxy_server -r ce-cr-02.ts.infn.it:8443/cream-lsf-grid -o install_myPkg.id install_myPkg.jdl

where install_myPkg.jdl looks like:

Executable = "GridManageSW";
InputSandbox = {"GridManageSW"};
OutputSandbox = {"stdout"};
stdoutput = "stdout";
stderror = "stdout";
Environment = { "theVO=eumed", "LCG_GFAL_INFOSYS=bdii.eumedgrid.eu:2170" , "LFC_HOST=lfc.ulakbim.gov.tr" };
Arguments = "--install --tar myPkg-v000001.tar.gz  --tag VO-eumed-test-myPkg-v000001";

namely, you are submitting GridManageSW passing as arguments:

  • --install: this means, obviously, you are going to install something
  • --tar : the name of the tarfile to be downloaded (from /grid/eumed/SoftRepo/, remember?)
  • --tag : the tag you will eventually publish</verbatim>

After the installation has started at the remote site, the CE will publish tag myPkg-v000001-processing. After some time, you may check the output of the installation job and the tags being published by the CE: if installation went OK the site should publish: myPkg-v000001-to-be-validated otherwise it will publish myPkg-v000001-aborted-installation (or in some exceptional cases, if things were really bad, the job will have completed and the site will still publish the processing tag).

Let's assume we can proceed with the validation phase. For this step you will also need a validation script which should do whatever is needed to get a fully working software: hence the validation script should perform both configuration and compilation, or just nothing (in the case of pre-compiled packages). However, it is mandatory to prepare a script which, in case of success, prints a line containing SUCCESS: the presence of such line is used by GridManageSW to understand which tag should be published: VO-eumed-test-myPkg-v000001 in case of success, or VO-eumed-test-myPkg-v000001-aborted-validation otherwise.

For the validation you would submit a validate_myPkg.jdl which may look like:

Executable = "GridManageSW";
InputSandbox = {"GridManageSW", "validate_dummy.sh"};
OutputSandbox = {"stdout"};
stdoutput = "stdout";
stderror = "stdout";
Environment = { "theVO=eumed", "LCG_GFAL_INFOSYS=bdii.eumedgrid.eu:2170" };
Arguments = "--validate --tag VO-eumed-test-myPkg-v000001 --validate_script validate_dummy.sh";

Of course we are using as arguments:

  • --validate: do I really need to say more about this?
  • --validate_script : which is the script you want to execute. The validate_dummy.sh script is copied below, and as you can verify it's really dummy and does nothing, apart from printing the SUCCESS line:
    #!/bin/bash
    #
    # $Id: InstallSwAsSGM.txt,v 1.1 2013/01/30 00:44:40 galeazzi Exp $
    #
    echo "This is a dummy script, which only returns success"
    echo "=== SUCCESS "
    

For more complicated setup, please take a look at the validate_root.jdl and accompanying validate_root.sh scripts. I paste here the .jdl

:

Executable = "GridManageSW";
InputSandbox = {"GridManageSW", "validate_root.sh"};
OutputSandbox = {"stdout"};
stdoutput = "stdout";
stderror = "stdout";
Environment = { "theVO=eumed", "LCG_GFAL_INFOSYS=bdii.eumedgrid.eu:2170", "LFC_HOST=lfc.ulakbim.gov.tr" };
#  validate_script_arg is name of link which will be placed under ./prod/root/ hence it MUST be different from 5th field in tag name VO-argo-prod-root-<something_else>-<rest_of_tag_name>
Arguments = "--validate --tag VO-eumed-prod-root-v052800b-SL5-x86_64-gccany --validate_script validate_root.sh --validate_script_arg 5.28.00b";

Note in particular the argument --validate_script_arg which is an argument (for a list, enclose it in "ticks") to be passed to the --validate_script.

Note that in the validation script you may use some variables which are defined by the calling script GridManageSW, namely:

  • ${theSWdir}: is set to the base software directory for the VO, $VO_EUMED_SW_DIR/ in our example
  • ${sw_dir}: is set to the destination directory for the software package, $VO_EUMED_SW_DIR/test/myPkg/v000001/ in our example

You are welcome to play with grid-swinstall. Please don't forget to contact Fulvio so that your updates may be inserted in future versions of the package. Enjoy!

-- Main.galeazzi - 2013-01-29

%TWISTY{id="topicattachmentslist" mode="div" remember="on" link="Attachments" showimgleft="/pub/TWiki/TWikiDocGraphics/toggleopen.gif" hideimgleft="/pub/TWiki/TWikiDocGraphics/toggleclose.gif"}%

I Attachment Action Size Date Who Comment
ziptgz grid-swinstall_20130129.tgz manage 17.8 K 2013-01-30 - 00:37 UnknownUser Package grid-swinstall, as of 2013-01-29, complete with DECIDE specific files
%ENDTWISTY%
Topic revision: r1 - 2013-01-30 - 00:44:40 - galeazzi
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback