Section3

Section 3: Additional information for online productions

For an online uDST production, it is necessary to update the fill.conf file and the other input data files for the uDST production automatically as fills are collected. These duties are performed by the update_fillconf.wrapper.sh shell script which is run as a cron job four times a day. This script does the following tasks:

1.	execute the udst_update_fill_conf.perl shell script to determine the necessary updates to the fill.conf file so that this file is consistent with the online slow control production. These updates include adding new fills, reproducing fills whenever the slow control production has been redone, and renumbering the fills if the fill numbering is not longer in ascending time order. These changes are tranmitted to the uDST production daemon via the fill.conf.addon and fill.conf.kill files. The former file contains a list of new fill entries which should be added to the fill.conf file and the later file contains a list of existing entries which should be deleted. from the fill.conf file. To generate the above files, the perl script: makes a list of all of the fills which have been produced by the slow control production and, after sorting them in ascending time order, assign a fill number to each fill. checks that this fill numbering is consistent with the fill numbering in the fill.conf file. If not, the fill numbers in this file need to be redone. For each misnumbered fill, the script puts an entry in the fill.conf.kill file to remove the old entry from the file and a "go" entry with the correct fill number in the fill.conf.addon file to add the new corrected entry to the file. determines the fills which were produced by the slow control production since the last invokation of this script based upon a timestamp file maintained by this script. If a fill in this list already has an entry in the fill.conf file, the script will put a "redo" entry into the fill.conf.addon file so that, like the slow control production, the uDST production will be reexecuted on this fill. Otherwise, the fill is considered a new fill. In this case, the fill is assigned a fresh fill number and and "go" entry for it is placed into the fill.conf.addon file in order to add the fill to the fill.conf file. and, finally, the script touchs a timestamp file (mentioned above) so that, when it is invoked again, it can determine the time of its last invokation. NOTE: If either the fill.conf.kill file or the fill.conf.addon files already exist in the config directory, this script will not attempt to make updates. In this case, the script will exit with a non-zero error code and, subsequently, the cron job will terminate.
2.	touch the disk.conf file so that the file will be re-read this file. As a result, the disks which were marked as unused because they did not have sufficient disk space for a production will be made available for production again. This mechanism enables the daemon to adjust for changes in the disk space usage on the production disks.
3.	issue a USR2 signal to the daemon process. This signal causes the daemon to update the fill.conf file (both the internal version and the "on disk" version) using the information in the fill.conf.kill and fill.conf.addon files. Also, the disk usage information will be updated using the information in the disk.conf file.
4.	fetch the up-to-date run-level data quality information from the logrun.new file maintained by the shift crew. At present, this file is copied from the b-machines to the PCFarm several times a day by the getlogrun.cron cron job executed under opa. On the PCFarm, the copied file is called /production/opa/www/logrun.new. To be read by the uDST production program, the comment field are stripped from this file by the process_logrun.new.perl script. The resulting, abbreviated file is placed into the runlist/logbook subdirectory of the production.

When these scripts are executed, the output from them are accumulated in logfiles located in the cron subdirectory of the production. These two files are:

(1)	cronlog	-	the accumulated output from each invokation of the update_fillconf.wrapper.sh shell script.
(2)	udst_update_fill_conf.perl.log	-	the accumulated output from each invokation of the udst_update_fill_conf.perl shell script.

By regularly viewing the tail of these files, one can see if the automated production is still running and, if not, what is the problem. The most likely problem is that the udst daemon is no longer running. In this case, one should restart the daemon and then, if either the fill.conf.kill or the fill.conf.addon files exist, send a USR2 signal to the daemon to process these files. This signal is sent to the daemon via the following command:

kill -USR2 {process id of the daemon}

If one fails to send this signal when the aforementioned files exist, the udst_update_fill_conf.perl script will detect their presence and continue to halt the automated production.

NOTE:

At the moment, the two scripts which are required for automating production are symlinked to the cron subdirectory of each "A" production when the make_production_area.sh script is executed by the production manager. So, the production manager only needs to add an crontab entry for the update_fillconf.wrapper.sh shell script and the uDST production will be executed automatically as new fills are produced by the slow control production.

---

Signal Handling by the uDST daemon

The cron job communicates with the uDST production daemon via the USR1 and USR2 signals. The USR1 signal instructs the daemon to read the fill.conf.addon file and update the fill.conf file and read the disk.conf file. The USR2 signal does the killing first and then issues a USR1 signal.