Data Preservation

Aus HERMESwiki
Zur Navigation springen Zur Suche springen

Page maintainer: Dich

Workingpage.jpg This is a working page which will continuously change. The content is not reviewed and is intended for people working on this specific topic.

Intro

HERMES Efforts on Data Preservation

In the data preservation and final analysis period it is planned to phase out the HERMES local computing farm completely, discard unneeded data and services, and redirect the remaining services to DESY IT supported centralized alternatives.

Working group with following members:

Gunar Schnell, Caroline Riedl, Eduard Avetisyan, Larry Felawka, Zaven Akopov, Alexander Kisselev.


Working on hermes-wgs02:

See below for a short summary.

Update of web page to come soon!

Current Status (as of August 15, 2012)

Computing resources

With the phaseout of the current local computing farm several alternatives have been considered from the computing facilities DESY provides. The least-effort alternative was found to be the BIRD cluster with total job slots of ~600, AFS support, interactive work option and several queues for convenience.

Interactive login and work

In the past, HERMES had two interactive nodes (worf and geordi), used for logins (also from outside DESY) and interactive work.

In the near future, when the PCfarm is phased out completely, one needs to use the pal cluster of DESY for that purposes. Please note that pal is not accessible from outside DESY via ssh, so one has to use the bastion.desy.de node as intermediate login point, and only then ssh pal. (HINT: if you type faq while on bastion, you may learn some useful tricks, like how to login to pal semi-directly tunneling through bastion).

The pal cluster has several OSes available:

Name OS CPU cores
palx SLD3 2
pal-sld4 SLD4 12
pal, pal-sld5 SLD5 60
pal-sld6 SLD6 32

Among these, SLD3 is the OS which is currently used on worf/geordi, it is badly outdated and unsupported. The corresponding palx host may disappear in near future. SLD4 is a system that HERMES never actually migrated to. SLD5 is the current default platform, and will be supported for a few more years, however it is being slowly superseded by the newer SLD6 platform, which will be soon available on larger amount of nodes, so if your analysis software works well on SLD6 is it recommended to migrate to it already now.

Please note, that (like worf/geordi) pal computers are shared resources for all DESY users, and hence are intended for low-load tasks or short-term tests only. For long/multiple/heavy jobs, please use the BIRD batch computing cluster.

Access to root for hermes-wgs02

- Produce a list of available modules: 

module avail

Attention, unlike the old 'ini' command, where modules were listed in a single column, 'module avail' uses multi-column output

- Add the necessary environment for running root v. 5.34: 

module add root/5.34

Currently, versions 5.32, 5.34 and 6.02 are available under 'module' utility. In addition, under /opt/root there is the old version 5.28 available (has to be set up manually - discouraged). Latex, python, gcc and other utilities can be configured in a similar manner.

- Show the currently active list of modules in your environment: 

module list

- Remove that particular module and allow adding another one instead: 

module remove root/5.34

Batch computing

To submit a batch job to BIRD one only needs to login to pal ('ssh pal'), then do 

ini bird

Afterwards the environment will be set up such, that the qsub, qstat and similar commands can be executed. The man command will give the descriptions.

The batch system on BIRD is SUN SGE, while the HERMES pcfarm used OpenPBS. Though very similar in many senses, there are a few differences in the command line and batch script which are listed in the table below:

Analogs of qsub options for PBS and SGE
PBS(worf/geordi) SGE(pal) Description
-j -j y Join the standard error and output
-q M -q default.q Run the job in the shortest queue (max. 8 h HERMES, 3 hours SGE)
-q L -q short.q Run the job in the medium queue (max. 24h)
-q XXL -q long.q Run the job in the shortest queue (max. 10 days HERMES, 7 days SGE)
-I qlogin -q login.q Run an interactive job
-l nodes=sld5 -l os=sld5 Select a specific OS for the batch environment.
-v var1[=value1],var2[=value2]... -v var1[=value1],var2[=value2]... Export (or assign) some of the environment variables from the current shell to the batch job
-V -V Export all the environment variables of the current shell to the batch job
$PBS_JOBID $JOB_ID Access the job ID from within the batch script
$PBS_JOBNAME $JOB_NAME Access the job name (e.g. set via -N option)
$PBS_O_WORKDIR $SGE_O_WORKDIR The current directory where qsub was executed
#PBS some-option #$ some-option One may specify the default options for the job inside the batch script via this syntax

Few hints:

  • -cwd option in SGE takes you do $SGE_O_WORKDIR automatically
  • add a line like "#$ -S /bin/bash" to your batch script if you want it to be executed by bash, or another shell if you like.
  • you should most likely specify the -l os=sld5 option, otherwise you may end up on an SLD6 host which should be treated differently in some cases. If your job needs SLD6, then specify -l os=sld6.
  • note that the HERMES dCache is not (yet) mounted on the BIRD (or pal) nodes, so you have to use the dCache preload library in the URL format
  • if you don't specify which queue you need, the default.q will be used with CPU time limit of 3 hours. Make sure to select an appropriate queue for your tasks, e.g. short.q for jobs running 1 day or less, and long.q for jobs running 1 week or less. Pay attention that requesting a longer queue may result in longer waiting period before the execution.
  • Each job creates a temporary location where you can store intermediate data. This location may vary from host to host, so the best way to access it is through the environment variable $TMP or $TMPDIR (both point to the same location), instead of directly specifying something like /scratch/...

An example command to submit a job will be then: 

ini bird
qsub -q short.q -cwd -j y -l os=sld5 mybatchscript.sh

and a subsequent qstat will let you see if your jobs is running (r state) or still waiting (qw state).

Interactive work

BIRD system allows also (limited) interactive access to its nodes. For that the login.q queue is deployed, e.g. 

qlogin -q login.q

will initiate an interactive session on one of the bird nodes (upon availability). One may also specify a particular node via 

qlogin -q login.q@bird007.desy.de

this can be useful when your job is running but you don't have access to the intermediate outputs or log files, because in particular AFS only shares the contents of a newly created file when it's closed. It is however available on the execution node, hence the use of the interactive usage.

Please note that the number of interactive slotes is limited and don't abuse it without need.

Storage

During HERMES data taking and active analysis period our computing farm included in total 21 NFS-based fileservers providing about 100TB of disc space for:

  • HRC and uDST data productions (mounted under /data01-98)
  • MC productions (/mcdata01-17)
  • user data (/user01-07)
  • group data (/group01-03)

Additionally, about 350TB of data (including RAW data) is stored on tape robot, accessible under generic mount point /pnfs/desy.de/hermes (conveniently mounted under /acs on HERMES pcfarm), where full backups of uDST and HRC productions, as well as some MC productions are stored.

As alternative solutions for these data in the preservation period the following central services have been chosen/evaluated:

dCache

  • HERMES acquired 5 dCache disc-only pools (e.g. without tape backend), with total space ~100TB, for large data storage
  • access to these data is possible via methods described here
    • mounted under /pnfs/desy.de/hermes/scratch on pcfarm, currently stores the following data:
      • all recent GRID-generated MC (not available elsewhere, no backup)
      • all last versions of uDST (one version per year)
      • full copy of /mcdata01-16 (/mcdata17 only holds a backup of /data09), currently verifying the consistency.
      • PLANNED: full copy of the group space (/group01-03)

AFS

  • AFS XXL pool directories, created upon request for active users, meant to hold the data previously stored in /user01-07:
    • currently such pool directories are created for 34 users with individual quota settings between 20 and 200GB upon needs
    • few more directories for active users are planned to be created
    • the rest of /user/* directories (in total 270) of former (or inactive) HERMES users will be archived and stored on tapes (directly accessible) till 2014, when the tapes will be removed from the robot and stored in a safe location for 10 more years, in case the data will be needed.
    • Please note that the usual unix file permissions are not effective under AFS, rather the special AFS ACL directory-based mechanism is to be used.

It is planned that as soon as all crucial data is copied over to the dCache and/or tapes, the oldest 6 fileservers (data,borg,nerys,reed,crusher,paris) will be switched off. Approximate deadline is by the end of this month (August 2012). Upon completion of the important data transfers and corresponding backups the rest of fileservers will be phased out as well.

Software

Locations

The migration to new central computing resources was only possible under the condition of having the HERMES software available under modern OS like SLD6 or at least SLD5, preferable in 64bit as all future computing nodes will run 64bit OSes. Such an effort has been successful, with the full HERMES software made available under AFS tree: 

/afs/desy.de/group/hermes

(please note that on the central computing nodes like pal or bird there will be no /hermes shortcut and one has to use the full path)

A mandatory prerequisity for that was the availability of CERNLIB 2005 copy for each system, with a corresponding (old) version of ypatchy needed to compile ADAMO. This version has been added to the HERMES software tree to avoid confusion with the centrally provided version of CERNLIB which isn't always compatible, and made available through a generic AFS symlink 

/afs/desy.de/group/hermes/cern/2005

that would point to the correct version of CERNLIB on all systems (currently SLD3(32bit), SLD5(32 and 64bit) and SLD6(64bit)).

Environment

It is highly recommended that your analysis code uses the configure and Makefile.in scripts analogous to the ones that can be found in the example codes of HERMES: 

/afs/desy.de/group/hermes/example

if you have a hand-made Makefile, you should convert it to the above setup as it makes the code porting much easier.

As soon as you have the corresponding Makefile.in, there are a few definitions that will help you compile your software on the new systems (need to define these before running the configure script, recommended to put into your $HOME/.profile): 

export HERMES_TOP=/afs/desy.de/group/hermes
export HERMES_ROOT=$HERMES_TOP/r25
export CERN_ROOT=$HERMES_TOP/cern/2005
export PATH=$HERMES_TOP/bin:$HERMES_ROOT/bin:$CERN_ROOT/bin:$PATH:.

If you plan to use ROOT for your analysis, you should initialize your binary and library paths accordingly. DESY provides a convenient way of doing so via the ini tool: 

ini ROOT528_64

or, from within a batch script 

eval "`/afs/desy.de/common/etc/local/ini/ini.pl -b ROOT528_64`"

Alternatively, you can setup ROOT defining manually the paths: 

export ROOTSYS=/opt/products/root64/5.28.00
export PATH=$ROOTSYS/bin:$PATH
export LD_LIBRARY_PATH=$ROOTSYS/lib

It is recommended to stick to ROOT version 5.28 for the time being as Hanna++ has been compiled using that version only. If you do not depend on Hanna++ you are free to choose any other ROOT version - you can find the list of supported version by typing ini without arguments. For compiling your code against Hanna++ libraries, the following additional variables are needed (this works on SLD5@64bit only!): 

export HANNAPLUSPLUS=$HERMES_ROOT/Hanna++
export PATH=$HANNAPLUSPLUS/bin:$PATH
export LD_LIBRARY_PATH=$HANNAPLUSPLUS/lib:$LD_LIBRARY_PATH
export PYTHONPATH=$HANNAPLUSPLUS/lib:$PYTHONPATH
export RDPIDLIBBASE=$HERMES_TOP/rdPIDLIB
export RDPIDLIB=$RDPIDLIBBASE/PDs
export RDANALYSISTOOLSBASE=$HERMES_TOP/rdAnalysisTools
export RDANALYSISTOOLS=$RDANALYSISTOOLSBASE/data

Note: In case you need to compile your own version of Hanna++, we recommend you to follow the instructions in the Hanna++ installation page.

If you want to use the pink browser, set up following environment variables (temporary solution - might not be needed in the future) 

export TCL_LIBRARY=/afs/desy.de/group/hermes/lib/tcl
export TK_LIBRARY=/afs/desy.de/group/hermes/lib/tk
Environment setup for worf/geordi and pal
Common settings 
export HERMES_TOP=/afs/desy.de/group/hermes
export HERMES_ROOT=$HERMES_TOP/r25
export CERN_ROOT=$HERMES_TOP/cern/2005
export PATH=$HERMES_TOP/bin:$HERMES_ROOT/bin:$CERN_ROOT/bin:$PATH:.
Environment setup for worf/geordi and pal
PCfarm(worf/geordi) DESY(pal)
ini ROOT518 ini ROOT528_64
export HANNAPLUSPLUS=/group01/rcoilgrp/Software/Hanna++ export HANNAPLUSPLUS=$HERMES_ROOT/Hanna++
export LD_PRELOAD=$HERMES_TOP/dcap/lib/libpdcap.so export LD_PRELOAD=$HERMES_TOP/dcap/lib/libpdcap.so
./configure ./configure --cc-compiler gcc --f77-compiler gfortran --use-environment
/pnfs/desy.de/hermes/scratch/udst/06f1/smlinks/ dcap://dcache-door-hera03.desy.de:22125//pnfs/desy.de/usr/hermes/scratch/udst/06f1/smlinks/
/production/udst/06f1/pol_burstlist dcap://dcache-door-hera03.desy.de:22125//pnfs/desy.de/usr/hermes/scratch/udst/06f1/pol_burstlist

This is an example script to setup the basic environment variables for the HERMES analysis framework on pal cluster: 

#!/bin/bash

echo 'Loading new HERMES environment...'

# Setting up HERMES environment:
export HERMES_TOP=/afs/desy.de/group/hermes
export HERMES_ROOT=$HERMES_TOP/r25
export CERN_ROOT=$HERMES_TOP/cern/2005

# Setting up Root environment:
export ROOTSYS=/opt/products/root64/5.28.00

# Hanna++
export HANNAPLUSPLUS=$HERMES_ROOT/Hanna++
export RDPIDLIBBASE=$HERMES_TOP/rdPIDLIB
export RDPIDLIB=$RDPIDLIBBASE/PDs
export RDANALYSISTOOLSBASE=$HERMES_TOP/rdAnalysisTools
export RDANALYSISTOOLS=$RDANALYSISTOOLSBASE/data

# Setting up the PATHS:
export PATH=./:$HANNAPLUSPLUS/bin:$ROOTSYS/bin:$HERMES_TOP/bin:$HERMES_ROOT/bin:$CERN_ROOT/bin:$PATH
export LD_LIBRARY_PATH=$ROOTSYS/lib64:$HANNAPLUSPLUS/lib:$HERMES_TOP/lib:$HERMES_ROOT/lib:$CERN_ROOT/lib:$LD_LIBRARY_PATH
export PYTHONPATH=$HANNAPLUSPLUS/lib:$ROOTSYS/lib64

# HERMES CVS repository
export CVSROOT=:pserver:anoncvs@kirk.desy.de:/hermescvsroot

# Give access to data tapes
# See: http://hermes-wiki.desy.de/index.php/PCFarm_Info#Direct_access_to_tapes
# WARNING: This currently break job submission to BIRD cluster.
# The user should better define the LD_PRELOAD in the script being submitted.
# export LD_PRELOAD=/afs/desy.de/group/hermes/dcap/lib/libpdcap.so

#Use the pink browser - might not be needed in the future
export TCL_LIBRARY=/afs/desy.de/group/hermes/lib/tcl
export TK_LIBRARY=/afs/desy.de/group/hermes/lib/tk

Note: You can add a script like the one above to your login scripts to load the HERMES environment variables automatically each time you login pal.

Compilation

Having set all the necessary variables, one may proceed to the compilation phase of the analysis code. Using the configure script, one should generate the corresponding Makefile for the given system (SLD5 or SLD6) by running: 

./configure --cc-compiler gcc --f77-compiler gfortran --use-environment

and if everything went fine, build the code by running make. Pay attention that many pathnames have changed, e.g. /hermes->/afs/desy.de/group/hermes, /cern->/afs/desy.de/group/hermes/cern,/production/udst->dcap://dcache-door-hera03.desy.de:22125//pnfs/desy.de/usr/hermes/scratch etc. In particular, the uDST DDL files for each production are now located in 

/afs/desy.de/group/hermes/r25/ddl/udst/...

instead of /production/udst/.../ddl.

If your build is successful, you may try to run your code for testing on a small amount of runs, and for full-scale analysis submit it to the BIRD batch system.

Remember that to access the uDST or MC files you will need to modify your runlists as described here.

Documentation

All HERMES non-digital documentation is being collected, sorted and archived in a dedicated key-access area under the DESY Library. If you find some important papers in your office that you think should be saved for the future, please let the management know.

Web service

The current wiki pages are residing on DESY IT supported central server. The 'plain' http server located on kirk has been duplicated on a virtual machine, currently named http://hermesdp.desy.de that will be resynchronised with kirk, and eventually replace the actual http://www-hermes.desy.de host.

eLogBook

The PC which had the eLogBook running has been switched off and preserved, presumably in a running condition. The data has been backed up in several locations, and recently made available online under:

http://ttfinfo.desy.de/HERMESelog/index.jsp

Mailing lists

Most of the HERMES mailing lists will be disabled by the end of the year or earlier. The web-archives will be available though. Instead a few active lists will be redirected to DESY IT supported mailing list engine or the management list.

Productions/MC

The following uDST productions are made available online on /pnfs/desy.de/dphep/online/hermes/udst 

95e5  96d0  97d1  98e1  99d1 00d0 00e1  02c1 02d1  03c1 03d1  04c1 04d1 04d2 05c2 05d1  05d2  06f1  07d1

For the availability of all officially generated MC productions throughout the years, "Check the MC productions page for the locations of the MC output"

Other non-official MC productions are available under mc_unofficial from tapes on request.

RAW data and HRC productions are all available from the tape robot, as well as the backups of all old uDST productions and some old MC, at least till 2014. If necessary, these can be made available also later, however the possibility of doing a full data production starting from RAW data is infinitesimal.

Meetings

Meeting 11.06.2010, Discussion of plans.:

In accordance with the Desy DPHEP plans to proceed in the direction of a so-called "rolling" data preservation model, where the current software shall be recompiled onto new OS/hardware as long as it is feasible and produces trustworthy results, and otherwise frozen and virtualized, Hermes needs to accomplish the following tasks:

  1. Make sure the HERMES software compiles and runs on (at least) SL5.
    1. Update: Compilation successful, production chain runs upto slowcontrol part, where it crashes. Reasons being investigated.
    2. Second problem: HMC compiled on SLD5 produces results that differ from the (old) one produced on SLD3. Being investigated, the usual suspects are Geant, cernlib and libc.
  2. Prepare an efficient verification procedure that will use min. 2 well-established and reproducable HERMES results - moreso, a subset thereof - for subsequent software versions/compilations. The verification package should be a all-in-one box that includes the neccessary microDST's, code and scripts that produce an easily recognizable pattern which hints on the check to be successful or not based on direct comparison.
    1. Update: since April 2011 in contact with IT for a centrally supported verification and testing platform. Minimal tests run successfully.
  3. Another direction of data preservation efforts is assessment of the documentation resources that HERMES posesses in form of older web pages, wiki pages and internal/technical notes, that need to be eventually preserved by a trustworthy third-party service like INSPIRE.
    1. Update: As the current webserver (kirk) will eventually phase out due to lack of support from HERMES side, it is suggested and planned that all HERMES static web-paged (e.g. non-wiki) will be transferred to the IT CMS (Content Management System) based server, with support from DESY. Drawbacks: our drafting author confirmation scripts will stop working. Solution: move everything else, keep the kirk up till the end of active drafting, aka end of HERMES.

Meeting 24.05.2011

Agenda/Topics:

  • Summary of the 5th DPHEP workshop (Eduard)
  • Brief update on InSpire: notes, papers, plots, e-logbooks? (Zaven)
  • Software status: SLD5 problems, 1-click compilation and install, productions (Larry)
  • Hardware status: warranty periods, conversions after 2012, new purchases (Alexander)
  • Validation efforts: what analysis can be selected for the validation process? (AC/DC)
  • MC validation: do we still need all the ancient MC we have on disk/tapes? (AC/DC)
  • Virtual Management, Librarian, Open Access: is there any progress? (SP)

Virtual Images

ATTENTION! The Virtual Images mentioned below are not fully functional and outdated. Please either update them or better use existing installations with HERMES software support!

In the PCfarm directory /user/dich/HERMES_SLD5.4 an exported Virtual Machine with SL5.4 and some additional software packages can be found. The "ovf" file has to be imported to VirtualBox (can be downloaded for free from www.virtualbox.org). Importing to VMWare should work, too, yet not tested.

What's in?

The VM has several snapshots, made in various steps of the installation. It can be started from any of the snapshots, as well as new ones saved any time. It contains:

  • "Vanille" SL5.4 installation with some additional software
  • VMBox gues additions for more comfortable guest/host interaction
  • CERNlib
  • HERMES software, compiled for SLD3 and copied over
  • ROOT
  • HANNAPLUSPLUS (not working)

The CERN and HERMES software can be run "as is" without recompiling, as well as ROOT macros and applications not needing HannaPlusPlus.

What's missing?

  • For a reliable use we need to have the whole HERMES software chain recompiled, preferably through a single Makefile (comments welcome). It's expected that the problems with HannaPlusPlus compilation will be gone then.
  • Upon need the DAD and web servers can be transferred to the VM, so that the whole thing works in a nutshell, mimicking a mini-version of the PCfarm.

What's it good for?

Current plan is to use this installation as a platform for future roll-on of the software updates as long as it works, with an option to freeze it at the end. Anyone interested may download the disc images and give it a try, e.g. run his analysis code in a box and compare the results with the ones obtained on pcfarm. Additional well documented snapshots are highly welcome.

Links

Abgerufen von „https://hermes-wiki.desy.de/index.php?title=Data_Preservation&oldid=13860“