Udstd files

The Control Files for the uDST Production Daemon

The uDST production is handled by a daemon which receives its initial instructions from a set of three files and all subsequent commands via the monitor program. The three files and their contents are described in this document.

---

the udst.conf file

This file must be stored in the same directory as the uDST daemon program, i.e. the bin subdirectory of a production. From this file, the uDST daemon reads in the configuration setting for the uDST production. These settings include:

(a)	testbase	-	the base directory for the fills made in test mode using this production. For example, via the line line testbase udsttest/, the uDST daemon is configured to produce the test fills in directories with the following prefix: /data??/udsttest/???? where: ?? = the selected data disk for the production ???? = the version number for the uDST production
(b)	prodbase	-	the base directory for uDSTs for fills made in production mode using this production. For example, via the line line prodbase udst/, the uDST daemon is configured to produce the uDSTs for fills in directories with the following prefix: /data??/udst/???? where: ?? = the selected data disk for the production ???? = the version number for the uDST production
(c)	nanobase	-	the base directory for nanoDSTs for fills made in production mode using this production. For example, via the line nanobase nano/, the uDST daemon is configured to produce the nanoDSTs for fills in directories with the following prefix: /data??/nano/????_?? where: ?? = the selected data disk for the production ???? = the version number for the uDST production ?? = the nano DST type (01 thru 05)
(d)	compath	-	the path containing the scripts executed by the uDST daemon in order to produce, redo, and remove fills. Typically, this path is set to the bin subdirectory of the uDST production. For example, the line compath /production/opa/udstprod/udst_07d1/bin was used for the 07d1 production.
(e)	command	-	the command which is executed by uDST daemon in order to do the uDST production for a fill. Typically, this command is a shell script located in the defined compath (see previous item). For example, the line command handle_uDSTproduction instructs the uDST daemon to execute the handle_uDSTproduction script located in the "compath" directory.
(f)	nanocom	-	the command which is executed by the uDST daemon in order to the nanoDST production for a fill. Typically, this command is a shell script located in the defined compath (see previous item). For example, the line nanocom handle_nanoDSTproduction instructs the uDST daemon to execute the handle_nanoDSTproduction script located in the "compath" directory.
(g)	wipe-command	-	the command which is executed by the uDST daemon in order to wipe, inspect, or redo a fill. As with the command, this wipe-command is typically a shell script located in the defined compath. For example, in a typical production, the following line is used: wipe-command wipe_production This line instructs the uDST daemon to execute the wipe_production script located in the "compath" directory.
(h)	dqroot	-	the path for locating the data quality bit information generated by Kevin McIlhany. (only applicable to 96 data, currently obsolete)
(i)	udstroot	-	the path for the uDST base directory for the production. This field should not have the version put onto it because the daemon tacks it on automatically. For example, in in the 07d1 production, the following line was used: udstroot /production/opa/udstprod/udst_ to set /production/opa/udstprod/udst_07d1 as the base directory.
(j)	processes	-	the maximum number of processes which the daemon can spawn.
(k)	reserve	-	the minimum amount of disk space (in kilobytes) which a disk must have free before the disk will be used to produce uDSTs for a fill.
(l)	diskin	-	the file (with a path relative to the directory containing the uDST daemon) from which the disk permission file will be read by the uDST daemon when it is started. Typically, this file will be called disk.conf and be located in the config subdirectory, so the following line would be found in the udst.conf file: diskin ../config/disk.conf
(m)	diskout	-	the file (with a path relative to the directory containing the uDST daemon) into which the disk status file will be written by the uDST daemon as it handles the production. Typically, this file will be called disk.usage and be located in the config subdirectory, so the following line would be found in the udst.conf file: diskout ../config/disk.usage
(n)	fills	-	the file (with a path relative to the directory containing the uDST daemon) containing the fill list for a production. The format of this file is explained below. Typically, this file will be called fill.conf and be located in the config subdirectory, so the following line would be found in the udst.conf file: fills ../config/fill.conf
(o)	sleep	-	the number of seconds the uDST daemon will wait before it will re-read the "fills" file and, if appropriate, start another production.
(p)	HERMES_ROOT	-	the definition of HERMES_ROOT passed by the daemon via an environment variable to the subprocesses which make the uDSTs.

NOTE:	unlike the fills.conf file, this file is only read when the uDST daemon is executed. So, to change any of these parameters, the daemon must be killed and then restarted after the file has been updated.

---

the fill.conf file

Located in the config subdirectory of a production, this file serves many purposes. First, for the production of uDSTs for each fill, this file contains the parameter to be used by the daemon and the uDST production scripts to decide which class of uDSTs to create. Second, this file contains the status of the production for each fill. And, finally, by modifying the status field, the production manager can instruct the daemon to process the uDSTs.

For each fill, this file contains one line. The format of this line is:

column 1	:	full name for the fill, e.g. fill.10.07.96-13:48:52
column 2	:	fill number assigned to this fill; the fill numbers are assigned in chronological order and, for a single year's data, remain fixed for all uDST productions.
column 3	:	udst version, e.g. 96b4
column 4	:	the type of uDSTs to be produced for this fill; the supported choices are "g1" for inclusive (g1) uDSTs, "sm" for semi-exclusive uDSTs, and "pd" for PID uDSTs.
column 5	:	type mode employed to make the uDSTs; the supported options are "prod" for the production uDSTs and "test" for test uDSTs.
column 6	:	the main production version by which the hrc.devents files used in this uDST production were generated.
column 7	:	the slow production version by which the fillfile files used in this uDST production were generated.
column 8	:	the uDST production command or status; the valid commands are: go - pending production; when its turn arrives, the daemon will produce uDSTs for this fill. setup - pending production setup; when this fill's turn arrives, the daemon will execute the setup and preprocessors, but will not actually generate the uDSTs for the fill. This feature is intended to simplify the debugging of the uDST production program. hold - the fill is not and will not be processed by the daemon redo - the fill was produced by the daemon, but it should be reproduced. The daemon will delete the produced uDSTs and, by setting this field to "go" and clearing the disk (column 9) field, reschedule this fill for production. However, if there is a problem encountered while deleting the production, the daemon will by "ECFG" into this field. wipe - the fill was produced by the daemon, but the production should be deleted. When this entry is processed by the daemon, all of the files generated as part of the production (including all symlinks) will be deleted. If this deletion occurs without any problems, the daemon will update this file by putting "hold" in this field and clearing the disk (column 9) field. Otherwise, it will mark this fill as a problem by putting "ECFG" in this field. insp - the fill was produced by the daemon, but the resulting uDSTs should not be kept. Like the "wipe" directive, this command will causes the daemon to remove the uDSTs and all symlinks to the uDSTs. However, the daemon will keep the production infrastructure intact. This feature allows the production of a fill to be debugged when a problem occurs. If the deletion occurs without a hitch, the daemon will update this file by putting "hold" in this field without touching the disk (column 9) field. So, by having a disk assigned to it and being in the "hold" state, the fill is marked as ready for hand-preparation. And, finally, if there is a problem, the daemon will mark this fill as a problem by putting "ECFG" in this field. and the valid status info are: PROC - the fill is presently being produced. done - the fill has been processed by the daemon and the resulting uDSTs were made without a detected problem. fail - the fill has been processed by the daemon but there was a production problem, so the resulting uDSTs may or may not exist. ECFG - a configuration error has occurred when this fill was processed.
column 9	:	the nano DST production flags
column 10	:	the production data disk upon which the uDST production for the fill was performed and thus the disk upon which the uDSTs reside. THIS FIELD SHOULD NEVER BE TOUCHED BY HAND. If a fill has not been produced yet, this field will have contain dashes.
column 11 thru 15	:	the production data disk upon which the nanoDST production for the fill was performed and thus the disk upon which the nanoDSTs reside. THESE FIELD SHOULD NEVER BE TOUCHED BY HAND. If the nanoDSTs for a fill have not produced yet, this field will have contain dashes.

As mentioned, the daemon updates this file as it handles the production. To monitor and, to some degree, direct the daemon, this file can be examined by less or emacs or, more preferably, by using the "monitor" program described in a later section.

---

the disk.conf file

Located in the config subdirectory for a production, this file contains the list of production disks onto which the daemon is permitted to produce and thus store uDSTs. The format of this file is one line per allowed production disk. This line has the following format:

column 1	:	the disk name, e.g. /data40.
column 2	:	the number of simultaneous uDST productions which can be executed on this disk at one time. Because of the high I/O load of a uDST production, it is best to run only one production on each disk at a time.

NOTE:	unlike the fills.conf file, this file is only read when the uDST daemon is executed. So, to change the disk selections, the daemon must be killed and then, after this file has been updated, restarted.

---

the fill.conf.addon file

Placed into the config subdirectory of a production by the cron job which automatically updates the uDST production so that it stay in sync with the slow control production, this file is contains a list of the fills which either need to:

• added to the fill.conf file because they are new or
• be reprocessed because either the fill number changed or the slow control production was updated

The format of this file is identical to the fill.conf file except that the disk fields are left blank.

---

the fill.conf.kill file

Placed into the config subdirectory of a production by the cron job which automatically updates the uDST production so that it stay in sync with the slow control production, this file contains a list of the fills for which the existing uDSTs need to be removed since it is necessary to reprocessed these fills either to update the fill number or include new data available because the slow control production was rerun.

The format of this file is identical to the fill.conf.addon file except that the control field is always set to "kill".