Needs of an online production

Section 3: Additional information for online productions

For an online uDST production, it is necessary to update the [udstd_files.html#fill.conf fill.conf] file and the other input data files for the uDST production automatically as fills are collected. These duties are performed by the [production_scripts.html#update_fillconf.wrapper.sh update_fillconf.wrapper.sh] shell script which is run as a cron job four times a day. This script does the following tasks:

1.	execute the [production_scripts.html#udst_update_fill_conf.perl udst_update_fill_conf.perl] shell script to determine the necessary updates to the [udstd_files.html#fill.conf fill.conf] file so that this file is consistent with the online slow control production. These updates include adding new fills, reproducing fills whenever the slow control production has been redone, and renumbering the fills if the fill numbering is not longer in ascending time order. These changes are tranmitted to the uDST production daemon via the [udstd_files.html#fill.conf.addon fill.conf.addon] and [udstd_files.html#fill.conf.kill fill.conf.kill] files. The former file contains a list of new fill entries which should be added to the [udstd_files.html#fill.conf fill.conf] file and the later file contains a list of existing entries which should be deleted. from the [udstd_files.html#fill.conf fill.conf] file. To generate the above files, the perl script: makes a list of all of the fills which have been produced by the slow control production and, after sorting them in ascending time order, assign a fill number to each fill. checks that this fill numbering is consistent with the fill numbering in the [udstd_files.html#fill.conf fill.conf] file. If not, the fill numbers in this file need to be redone. For each misnumbered fill, the script puts an entry in the [udstd_files.html#fill.conf.kill fill.conf.kill] file to remove the old entry from the file and a "go" entry with the correct fill number in the [udstd_files.html#fill.conf.addon fill.conf.addon] file to add the new corrected entry to the file. determines the fills which were produced by the slow control production since the last invokation of this script based upon a timestamp file maintained by this script. If a fill in this list already has an entry in the [udstd_files.html#fill.conf fill.conf] file, the script will put a "redo" entry into the [udstd_files.html#fill.conf.addon fill.conf.addon] file so that, like the slow control production, the uDST production will be reexecuted on this fill. Otherwise, the fill is considered a new fill. In this case, the fill is assigned a fresh fill number and and "go" entry for it is placed into the [udstd_files.html#fill.conf.addon fill.conf.addon] file in order to add the fill to the [udstd_files.html#fill.conf fill.conf] file. and, finally, the script touchs a timestamp file (mentioned above) so that, when it is invoked again, it can determine the time of its last invokation. NOTE: If either the [udstd_files.html#fill.conf.kill fill.conf.kill] file or the [udstd_files.html#fill.conf.addon fill.conf.addon] files already exist in the /config directory, this script will not attempt to make updates. In this case, the script will exit with a non-zero error code and, subsequently, the cron job will terminate.
2.	touch the disk.conf file so that the file will be re-read this file. As a result, the disks which were marked as unused because they did not have sufficient disk space for a production will be made available for production again. This mechanism enables the daemon to adjust for changes in the disk space usage on the production disks.
3.	issue a USR2 signal to the daemon process. This signal causes the daemon to update the [udstd_files.html#fill.conf fill.conf] file (both the internal version and the "on disk" version) using the information in the [udstd_files.html#fill.conf.kill fill.conf.kill] and [udstd_files.html#fill.conf.addon fill.conf.addon] files. Also, the disk usage information will be updated using the information in the [udstd_files.html#disk.conf disk.conf] file.
4.	fetch the up-to-date run-level data quality information from the [runlist.html#runlist.logbook logrun.new] file maintained by the shift crew. At present, this file is copied from the b-machines to the SGI several times a day by the getlogrun.cron cron job executed under opa. On the SGI, the copied file is called /export/home/opa/www/logrun.new. To be read by the uDST production program, the comment field are stripped from this file by the process_logrun.new.perl script. The resulting, abbreviated file is placed into the runlist/logbook subdirectory of the production.

When these scripts are executed, the output from them are accumulated in logfiles located in the /cron subdirectory of the production. These two files are:

(1)	cronlog	-	the accumulated output from each invokation of the [production_scripts.html#update_fillconf.wrapper.sh update_fillconf.wrapper.sh] shell script.
(2)	udst_update_fill_conf.perl.log	-	the accumulated output from each invokation of the [production_scripts.html#udst_update_fill_conf.perl udst_update_fill_conf.perl] shell script.

By regularly viewing the tail of these files, one can see if the automated production is still running and, if not, what is the problem. The most likely problem is that the udst daemon is no longer running. In this case, one should restart the daemon and then, if either the [udstd_files.html#fill.conf.kill fill.conf.kill] or the [udstd_files.html#fill.conf.addon fill.conf.addon] files exist, send a USR2 signal to the daemon to process these files. This signal is sent to the daemon via the following command:

kill -USR2 {process id of the daemon}

If one fails to send this signal when the aforementioned files exist, the [production_scripts.html#udst_update_fill_conf.perl udst_update_fill_conf.perl] script will detect their presence and continue to halt the automated production.

NOTE:

At the moment, the two scripts which are required for automating production are symlinked to the /cron subdirectory of each "A" production when the [auxiliary_scripts.html#make_production_area.sh make_production_area.sh] script is executed by the production manager. So, the production manager only needs to add an crontab entry for the [production_scripts.html#update_fillconf.wrapper.sh update_fillconf.wrapper.sh] shell script and the uDST production will be executed automatically as new fills are produced by the slow control production.

---

Signal Handling by the uDST daemon

The cron job communicates with the uDST production daemon via the USR1 and USR2 signals. The USR1 signal instructs the daemon to read the [udstd_files.html#fill.conf.addon fill.conf.addon] file and update the [udstd_files.html#fill.conf fill.conf] file and read the [udstd_files.html#disk.conf disk.conf] file. The USR2 signal does the killing first and then issues a USR1 signal.