Productions

Aus HERMESwiki
Zur Navigation springen Zur Suche springen

Page maintainer: Caro

Checked.png This page is considered done. It been reviewed by an expert. There may be missing elements, but they are all flagged and the text has no errors.

This page contains both general information on the production structure and specific information on produtions from various years. Note that the page is not static but keeps on being updated.

Useful Links

Traditional Data Quality webpage

Data Structure Logruns ddls uDST contents MC uDST contents
Data Quality Burstlists Bad Bits Data Quality Masks Data Quality Plots (uDST) Data Quality Plots (hrc) Data Quality in the Bootcamp
Results Statistics and integrated lumi Lumi constants

New Analysis Platform

NEW

Introduction

Production structures and names

  • The main (hrc) productions are labelled YYL, where YY is the data taking year and L is a letter describing the level of the main production.
  • The first production for each year is the a- (online) production, all later productions (b, c, d, ...) are called higher productions.
  • The b-, c- etc. productions contain better calibrations, better knowledge on tracking or any better knowledge of algorithms which are handled in hrc (xtc), the Spectrometer (Recoil) tracking codes.
  • The main productions contain everything which was collected during data taking, down to the hit level. Usually only detector experts analyze files of the main production.
  • A main production takes typically several weeks on the batch nodes (but even for productions including Recoil data less than a month).
  • Location of hrc files:
     hrc files: worf:/production/hrc/YYL/root/run*****/hrc*****.devents.gz
  • For b- and higher productions, the hrc files contain all information down to the spacepoint/cluster level (but no hits).
  • The uDST production YYLN bases on the main production YYL, where N is the version number of the uDST production. This means that everything about tracking and calibration which applies to a certain main production applies to all daughter uDST productions.
  • The YYL0 productions are called PID productions, as they are produced only for the purpose of having (Spectrometer) parent distributions for the YYL1 production.
  • A new uDST production is produced if there happens to be any better knowledge of algorithms handled in the uDST writer (the code which merges the hrc files with the slow production files).
  • The uDSTs contain all collected information down to the track level and (unlike the main productions) slow control information. For the Spectrometer, also some energy depositions are available to allow the user to feed the PIDLIB.
  • To produce a new uDST production takes typically several days.
  • Location of uDST files:
     uDST files: worf:/production/udst/YYLN/smlinks/run*****.smdst.gz

The Logrun

  • The Hermes logruns are essential for all analyses because they contain fundamental information about the data. There is one logrun for each year of data taking.
  • It contains one line per run and holds the following information (example for Recoil period):
# Run
# _    analyze (1) / don't analyze(0)/alignment+calibration+
# _    detector+checkout+Empty_Target(2) LPOL (3)
# _    _
# _    _ Target mode (0=normal density, recoil magnet ON 
# _    _ _            1=normal density, recoil magnet OFF
# _    _ _            16=high density, recoil magnet ON)
# _    _ _ Target gas (0=empty target, 1=H, 2=D, 3=3He, 4=4He, 5=N, 6=Kr,
# _    _ _ _                   7=Neon, 8=Mix 9=Xenon)
# _    _ _ _ Comments (do not write beyond column 80! Don't use Tabs!)     -->_
# _    _ _ _ _
#15098 1 3 1 TRD@3000V
 00001 3 0 0 Start the LPOL run, set run number as 1, 48196 in old scheme.   
 00002 1 0 1 first normal density H2 run, 42%
 00003 1 0 1 
 00004 1 0 1 
 00005 1 0 1 
 00006 1 0 1 
 00007 1 0 1 
 00008 1 0 1
  • Only analyze==1 runs may be chosen for physics analysis
    • Target mode: gives density and polarization state of target
      • High density is always unpolarized
      • Recoil period is always unpolarized
      • Nuclei heavier than in deuterium are always high density
    • Target gas: gives gas (element) filled into target
  • An example linux script how to create a runlist containing only runs with certain conditions can be found below in this Code Repository section. There are also some examples on the data taking summary* pages. Read also this mail about "logrun vs. badburstlist".
  • The logrun was filled out by the shiftcrew during data taking. It was checked by the period coordinator and later by a DataCops member (for example, for consistency with the electronic logbook). It might still contain errors, though. For example, if the entries in g1Unpol.iGasType and the logrun are inconsistent with each other concerning the unpolarized state, then g1Unpol.iGasType is correct.

Info about productions

General

News

Last update: April 19, 2012 by CKR

Data Productions
  • 06f1: released (16.4.2012)
  • 07d2: released (25.5.2011)
  • 98e1 and 99d1: released (27.1.2011) (HTC)
  • 00e1: released (15.6.2010) (HTC)
  • 04d2 and 05d2: released (5.2.2010) (new RICH background files)
  • 06e1: released (11.8.2009)
  • Combined release of all transverse data: 13.7.2009. 02, 03, 04, 05 d1 productions, HTC + new alignment
  • 07c2: re-released (11.6.2009) including caveats
  • OLD 04d1, 05d1 UNreleased: Mail by Larry, Reported Problems (29.5.2009)
Memory Board
Software Updates

Official Production Releases

Chamber Thresholds

  • Chamber Thresholds in the FCs and the BCs have been changed several times between runs. In contrast to HV trips, changes of the chamber thresholds from the default values are not considered by a badbit in the burstlists. They are important for any analysis of yield-sensitive quantities. Links are given below.

Calorimeter Thresholds

New Calorimeter Calibration (2009 style)

  • For all productions later than and including 07c*, the "new calorimeter calibration 2009" is applied to tracked clusters (but not untracked ones). The calibration is based on the momentum dependent E/p ratio for leptons, where E is the energy measured by the calorimeter and p the momentum of the track. Taking into account the momentum dependence automatically also corrects for the incident angle. See also Photon Reconstruction page.

PID Parent Distributions and Fluxes

  • Parent Distributions (PDs): the Spectrometer PDs entering each uDST production have been obtained from an earlier uDST production. From those parent distributions, the PID values in g1Track were calculated for each track. This information can be obtained by:
zless /production/opa/udstprod/udst_****/operation/udst/fill.*.sm/output.file.01.gz _ grep PDs

RICH EVT

  • RICH EVT and background files see the RICH PID page, specifically the Productions section
    • RICH background file for 2007: has in the old productions a mix of low and high density. File with consistent low-density sent by Rebecca July 14, 2010:
/group01/richgrp/BackgroundFiles/background07.txt_July14_2010

Tracking

  • TMC, RMC, HTC
  • TMC down shifting scheme common in the Transversity group
    • if available, always use TMC2, otherwise TMC1
    • if none of the two is available, discard the track
  • First productions with HTC: 07c; 06e; 02d; 03d; 04d; 05d; 00e
    • All later productions are also HTC productions!

Target Cells

Year Dependent Info 1995-2007

1995 1996 1997 1998 1999 2000 2002 2003 2004 2005 2006 2007

More year dependent information related to the calorimeter

2007

  • Running conditions 2007
  • The lumi constant for 2007 was updated in October 2011
  • 07d and all uDST productions based on it
    • HTC production with latest HTC version. Alexander: HTC version is now (virtually) the same for all years; 07c was using one of the first production versions with an older vertexing code; relative changes between two versions should be small though.
    • Beam line is variable and taken from BPM database in HTC
    • Improved Recoil tracking (compared to 07c).
    • SFT-only tracking added in XTC.
    • New Recoil photon detector calibration.
    • First 2007 production with Recoil data quality (e.g. 07d1) available in bad burst list
    • Noisy calorimeter period fixed (which caused a drop in the DVCS yield if requiring exactly one trackless cluster) by corrected pedestal value for a certain run range where a connector was misplaced during data taking. The calibrations haven't changed.
    • In 2007 productions prior to 07d, there was a flag in ACE (which runs after HRC and XTC in the production chain) that was cutting away photons with E_gamma<1GeV in events with certain configuration, namely if the event had only 1 charged track. In events with more tracks, photons with E_gamma>0.5GeV were kept. This created a step in the photon distribution. For the 07d and 06f productions this threshold cut was set to 0.5GeV for all photons, independent on the track count.
  • 07c and all uDST productions based on it.
    • First production with HTC (refitting of tracks). The track parameters in g1Track come from HTC, the familiar HRC tracking results are stored in different tables.
      • Both 07c1 and 07c2 are HTC productions!
    • Calorimeter calibration 2009
    • PID is based on HTC track parameter result
    • Delia's mail about fluxes for 07c
    • Attention: there is no 1-to-1 relation anymore from g1Track to smTrack. Use the ADAMO relation (as adviced ever since)
    • Correct TRD calibration and first production with new calorimeter calibration and full calibration for whole Recoil
    • Calo energy correction: is performed by the uDSTwriter to the tracked clusters (smTrack.rECaloBackPT), but not to the untracked ones in smCluster.
    • TRDxy correction
    • Mail from GMS expert about run range for which the Calorimeter GMS cannot be trusted due to misplaced connector. (is taken into account in badburstlists of 2007, bit 17 is NOT set in this run range)
    • 07c2: short tracks ok
    • 07c1: short tracks are included but not refit by htc
  • ADAMO input buffer size increased from 5,000,000 -> 30,000,000 words; all analyzers should do the same in their "makeddl" scripts:
for/set initap INITAP:30000000

Smaller numbers will give you a ZEBRA fatal error for some runs.

  • 07b1: has wrong TRD calibration. Do not use for analysis.
    • 07b1 PDs: obtained from 07b0
  • All 2007 uDST productions until including 07c2: do not use Photon Detector information (use hrc-format data)
  • Run ranges 4299-4321 and 22676-22710 should be discarded if normal-density data is analyzed (Trigger21 threshold 1.4 GeV dataset). These runs are marked in logrun as unpol (target mode 0), while in the udst files one sees that Trigger threshold is 3.5 GeV (= characteristic for HD runs, target mode 16 in the year 2007).

2006

  • Running conditions 2006
  • Front Chamber Thresholds in 2006
  • Back Chamber Thresholds in 2006
  • H0 hardware information in 2006
  • 2006 positron beam data: use 06e or later productions only! All earlier productions have wrong TRD calibration.
  • The lumi constants for 2006 were updated in October 2011
  • 06f and all uDST productions based on it
    • General: same settings (software versions) as for 07d.
    • HTC production
    • Beam line is variable and taken from BPM database in HTC
    • New Recoil photon detector calibration.
    • Calorimeter calibration 2009
    • First 2006 production with Recoil data quality (e.g. 06f1) available in bad burst list.
    • In 2006 productions prior to 06f, there was a flag in ACE (which runs after HRC and XTC in the production chain) that was cutting away photons with E_gamma<1GeV in events with certain configuration, namely if the event had only 1 charged track. In events with more tracks, photons with E_gamma>0.5GeV were kept. This created a step in the photon distribution. For the 07d and 06f productions this threshold cut was set to 0.5GeV for all photons, independent on the track count.
  • 07c and all uDST productions based on it.
    • Noisy calorimeter period fixed by corrected pedestal value for a certain run range where a connector was misplaced during data taking. The calibrations haven't changed.
  • 06e and all uDST productions based on it
    • First 2006 production with a good TRD calibration and data quality for both electron and positron data
    • HTC production
    • Calorimeter calibration 2009
    • Delia's mail about fluxes for 06e
  • 06d and all uDSTs productions based on it
    • 06d0 PDs: obtained from 06b2
  • 06c: special production with only certain events processed. No uDST
  • 06b and all uDST productions based on it
    • 06b1: contains only electron-beam data.
    • 06b2: contains both electron- and positron-beam data.
    • 06b1 and 06b2: E/p drops with time, caused by gradual magnetization of the calorimeter holding frames. Was fixed in 06d0 and all productions afterwards by splitting the calibration periods into smaller pieces
    • 06b1, 06b2 PDs: obtained from 06b0.
  • All 2006 uDST productions until including 06e1: do not use Photon Detector information (use hrc-format data)
  • All 2006 data: ADAMO input buffer size increased from 5,000,000 -> 30,000,000 words; all analyzers should do the same in their "makeddl" scripts:
for/set initap INITAP:30000000

Smaller numbers will give you a ZEBRA fatal error for some runs.

  • Recoil magnet ON/OFF periods
# OF 00001-00755 till 05.03.2006 06:14
# ON 00756-00761 05.03.2006 06:14 till 05.03.2006 07:29
# OF 00762-00795 05.03.2006 13:21 till 05.03.2006 23:44 (alignment, unusable)
# ON 00798-00811 06.03.2006 22:10 till 07.03.2006 07:23 (unusable - switching magnet on-off)
# ON 00813-00822 07.03.2006 11:57 till 07.03.2006 16:17
# OF 00824-01030 07.03.2006 16:17 till 09.03.2006 09:41
# ON 01032-01741 09.03.2006 09:41 till 13.03.2006 11:00
# ON 01742-01920 13.03.2006 16:00 till 14.03.2006 05:40 (unusable, calo low open)
# ON 01921-02154 14.03.2006 05:43 till 16.03.2006 01:20
# OF 02155-04610 16.03.2006 01:20 till 28.03.2006 16:38 <- the biggest chunk with OF
# ON 04612-04794 28.03.2006 16:38 till 29.03.2006 01:39
# OF 04795-04870 29.03.2006 09:58 till 29.03.2006 18:10 (unusable? just one fill)
# ON 04871-13846 29.03.2006 18:10 till 18.06.2006 13:48 <- the biggest and smoothest chunk !
# OF 13847-13862 18.06.2006 13:48 till 18.06.2006 18:08 (unusable, empty target)
# ON 13863-14241 18.06.2006 18:08 till 24.06.2006 07:49 join with previous big chunk
# ON 14242-17000 24.06.2006 07:49 till 07.08.2006 00:00
# here an ADC board was changed, which resulted in pedestal file change
# ON 17000-48194 07.08.2006 00:01 till 31.12.2006 23:59 split into 6 periods

2005

2004

2003

2002

2000

1999

  • Running conditions 1999
  • H1 and H2 gains were set erratically during the first ~2000 runs. Results in:
    • Premature saturation in some paddles, lowered resolution in other paddles
    • Calibration suffers from lower stability.
  • 99d and all uDST productions based on it
    • HTC production with new alignment
    • Calorimeter calibration 2009
    • 99d1 vs. 99c1: Larry's release mail. In particular, due to the usage of a new version of the GMS-code, a stricter data quality for the calorimeter is active in this latest production, resulting in more runs marked as bad.

1998

  • Running conditions 1998
  • 98e and all uDST productions based on it
  • 98d1: TOF failed
  • H1 and H2 gains were set erratically during the entire 98 production. Results in:
    • Premature saturation in some paddles, lowered resolution in other paddles
    • Calibration suffers from lower stability.

1997

  • Running conditions 1997
  • Read about accidental coincidences in the lumi detector for 1996/97 polarized running on the Luminosity Page
  • Nitrogen data (unpolarized, high density)
    • From run 27991 on: varying calorimeter thresholds (like 5 GeV...)
    • runs 18158-19650: problems with BCs

1996

1995

Related Pages

Code Repository

HTC

Runlists

Online Data Quality

Offline Data Quality

udST offline code #1

About
~oma/udst_DQ_fortran
  • Maintained by Caroline Riedl since November 2005 (taken over from Ulrike Elschenbroich)
In case of new uDST production

What to do if there is a new uDST production, on the example of 00e1. CR 6.4.2010. Log in as user oma.

  • Produce runlist: cd RUNLISTS; ../bin/x_runlist 00e1 (will assign non-existing runs a -1). Check result of 00e1runlist.all.
  • Add 00e1 to Makefile in main directory
  • mkdir src00e1
  • copy /production/udst/00e1/ddl/g1_00E1.ddl into src00e1
  • link.sh 00e1
  • Add 00e1 to source/Makefile.in
  • cd source; configure
  • in main directory: make 00e1
    • The message >>> udst00e1 has been built successfully! <<< should appear (otherwise: debug code :-)
  • 'mkdir plots/00e1; mkdir results/00e1
  • Start job: cd scripts; sendDQplots.sh --queue XL 00e1 (choose a different length M, L, XXL for very small or big productions)
  • Checklist for webpage updating:
    • Edit the main webpage /www-hermes/html/groups/daqlgrp/OFFLINE_DQ/uDST/index.html
    • Add the link to the 00e1 DQ plots (copy and paste).
      • As long as the production is not officially released, put link in gray, later blue for an official production).
      • While the job is still running, the webpage with the 00e1 DQ plots does not yet exist. In the very end, a script is run that produces this webpage and everything should be generated automatically. If an intermediate result is desired, cd /www-hermes/html/groups/daqlgrp/OFFLINE_DQ/uDST and run this script by hand: update_web.sh 00e1
    • Put ddls on the web: cd /group01/daqlgrp/dataquality/webpages/00e1 and mkdir docu and cd docu and ln -s /production/udst/00e1/ddl/g1_00E1.ddl. Add link on main web page (copy and paste)
    • Add a link to both definition and explanation of the bad bits. Usually at this late stage, where for all years higher productions are available already, one does not need new pages for this. For example, for 05d2 just a reference to 05c0 is given. If there are changes in the badbit pattern, it would be nice to have pages on its own for this production. cd /group01/daqlgrp/dataquality/webpages/00e1/docu and copy the existing BB* files from another production in this directory. ls -ltr ../../*/docu/
    • Add link to the e-mail with the announcement of the production
    • Finally, add location of badburstlist for this production, including the release date.
    • Is there anything to be added on the Productions page about this production?
Change log
  • 7.5.2010: Some runs for badbits 17, 18, 10, and 21 are hardcoded for the 00d productions (mostly bad GMS runs). Need them also for 00e (and if there will be ever, 00f, etc...). Extended badbit.f accordingly. CR

uDST offline code #2

  • Used to produce the official badburstlists and the number of DIS events.
  • Main directory:
~oma/udst_DQ_c
  • Maintained by Dominik Gabbert until 2009. No successor.

Recoil Detector Data Quality

  • Recoil data quality for the physics user will be part of the bad burst lists of the final 2006 and 2007 uDST productions, providing one bit for each sub-detector (SSD, SFT, PD).
  • The basic performance of the detector, i.e. average energy deposition, tracking and also some physics yield is checked by the private code of C. Riedl for every new (main) production.
    • Here is the corresponding weblink.
    • Short docu (probably only useful for Caro): MASTER_DIR=/afs/desy.de/group/hermes/pool/criedl/
      • Add new main prod in source/Makefile.in
      • Compile code from the main directory with the "create ${prod}" command, where prod=07d for example. cd exe and copy successfully created executable recoil${prod} to recoil${prod}_i, where "i" is a running exe-version number (typically 1 for a new production).
      • Find a version number >100 for the output - check directory "output" for non-existing numbers.
      • cd runlists. Find the makelist_rd_* scripts, create a copy (e.g. makelist_rd_07d) for the new production, edit it and fill in the correct year, path to the run files (check correct root*-path) and the logrun.
      • Master script for main productions (not suitable for uDST productions): doscripts/dohrc_rddaql. This script should need no modification, as it is written in a very general form.
      • Edit kumacs/rddaql.kumac and add in the begining the new production together with its directory version number. The current "try" version as of Feb.2011 is "F".
      • Find the RDdaql* scripts in the doscripts directory. Create a copy for the new production, e.g. RDdaql07d. Edit it and fill in the correct version numbers, the year, the production. The default for bunch is 500 (runs), the default for periodfrequency is 1 (i.e. per 1 run) for the basic quantities, 10 (i.e. per 1o runs) for the physics quantities. Finally, give the start and stop run. The package is optimized for start=N+1 and stop=M, where both N and M are dividible by 500. E.g. start=1, stop=500; start=501, stop=1000, etc. In principle, the job needs to be sent only once quoting the first and last run of the production. If there is time pressure, individual packages in units of n*500 runs can be sent. Note that the job needs to be sent separetely for periodfrequency 1 and 10.
      • Start the job in the "doscripts" directory by qsub -V -q XXL RDdaql07d. (outdated - for pcfarm)
      • Plots will be stored in the directory MASTER_DIR/recoil/kumacs/epsfiles/recoil-daql/F-07d/ (example).
      • In the directory MASTER_DIR/rddaql, link MASTER_DIR/recoil/kumacs/epsfiles/recoil-daql/F-07d/ to 07d (example).
      • The webpages *.html are automatically created by the script MASTER_DIR/rddaql/dowebpage_rddaql. The only things that remains to be done is to log in as user oma, cd /www-hermes/html/groups/daqlgrp/OFFLINE_DQ/uDST/recoil, check with ls -ltr and link the new production directory here.
      • Finally, edit (as user oma) http://www-hermes.desy.de/groups/daqlgrp/OFFLINE_DQ/uDST/RDdaql.html and add the links and some description about the new production.
    • Change.log
      • 16.10.2012: rddaql/dowebpage_rddaql updated with new "user disk"

uDST Production HowTo

Overview of the uDST documentation

The uDST production aims to produce a single data file for each run which contains the tracking data from the main (hrc) production for selected events, the relevent slow control data from the slow control production, and the data quality information. These files, called uDSTs, can be reliably used for physics analysis without concern for the details of the data collection. The purpose of this document is to describe, in detail, the operation of the uDST production and, thereby, enable anyone not only to understand the contents of the produced uDSTs but also run the production.

Like the slow control production, the uDST production is organized by fills instead of by runs. This organization is reasonable because, until the slow control production has processed a fill, there is no slow control data available for the uDSTs. To manage the uDST production, there are four principle software components:

  • the daemon which manages the production of fills,
  • a user interface program for monitoring and, to some extent, directing the daemon,
  • a series of preprocessors which shape data for entry into the uDST production program, and
  • the uDST production program itself.

The general documentation for these components is organized as:

  1. the uDST production daemon and the flow of a typical uDST production
  2. the use of the monitor program to communicate and, to some extent, control the uDST production daemon,
  3. the additional needs of an online production,
  4. the organization of the uDST production root directory,
  5. the steps for setting up a new uDST production,
  6. the description of the uDST production program, and
  7. the description of the contents of the uDSTs
Abgerufen von „https://hermes-wiki.desy.de/index.php?title=Productions&oldid=13852“