Real Time Processing Code
Real Time (RT) processing codes connect Earthworm to the Oracle Database on a RT host. Each RT host has a chain of processing that starts with the core Earthworm system and feeds data into the CMS (a messaging system) and into the database. This section describes each of the modules used in RT processing. The simple block diagram shown below describes the layers of software in the system.
Waveforms flow into the system via whatever mechanism existed prior to installing CHEETAH. In the Southern Calif. Seismic Network (SCSN) this is via multicasted miniSEED packets and in NCSN this is via earthworm imports, and at HVO it is via Earthworm ring2coax broadcasts of TRACEBUF2 packets. Once the data are in Earthworm, they are processed using the basic Earthworm modules provided with version 7.X for event location and subnet triggering and magnitude computation. From Earthworm, three modules are used to send the data on to CHEETAH for DB storage and further action such as alarming. These modules include hyps2ps (for hypocenter CMS publishing), trig2ps (for subnet trigger CMS publishing), and ewmag2cisn (for archiving of magnitude messages directly to the DB). See the image below for a block diagram showing the basic connections to Earthworm and the Database.
CMS, the messaging system, is a CORBA based system that is written using the same software as the CISN Display and the EIDS system for reliable transport of messages over TCP/IP. The CMS stands for CHEETAH Messaging System (formerly CISN Messaging System).
The only thread not described on the RT systems in the above diagram, for simplicities sake, is the alarming thread. A diagram showing the 3 RT modules is provided below that shows the paths the alarming system takes for firing an action related to an alarm.
In a nutshell, when a hypocenter or magnitude is written to the Oracle Database, the program doing the writing, should then issue an CMS signal to the alarming thread if the new result should warrant alarming. The signal contains the unique ID that points to the event in the database known as the event ID (or evid). The alarmdec program receives the signal, looks up the event information and goes through its list of criteria for all alarms and decides which alarms to run based on how good the event location and/or magnitude of the event is. Once alarmdec decides to alarm, it sends a CMS signal to the alarmact program stating the evid and the name of the alarm.
Alarmact, short for alarm actions, then looks up the actions associated with the alarm name and writes the actions to the ALARM_ACTION database table. It then sends on a CMS signal to the final program in the chain, alarmdist to distribute the actions. alarmdist takes the action name and if the host computer is PRIMARY, it fires the actions which are simply shell scripts. There is more complexity to the actions firing, but it is described in the manual page.
Okay, why the complexity of 3 programs to fire alarms? The answer simply is flexibility. By having each program do a separate task, one can create a set of modules to handle different alarming decisions and actions if necessary. As this is deployed at Caltech, there are 3 alarmdec threads running and feeding one alarmact and one alarmdist.
Real Time Directory Layout
The RT layout is all done in the /app/rtem directory owned by the rtem "ROLE". Role is a new concept in Solaris 10 that allows a user to have restrictions on what it can do (see http://www.softpanorama.org/Solaris/Security/solaris_rbac.shtml but in the use for the CHEETAH software, the rtem role has no password and thus one cannot ssh in as rtem. Instead, users must issue an command like: sudo su - rtem to gain access to the rtem role or safer still if one wants to monitor users acting as the user rtem, then the sudo logsu rtem command should be issued and only allowed by the sys admin.
The RT layout on a typical RT host looks like this:
{lomi:~} pwd
/app/rtem
{lomi:~} ls -F
alarm/ chan-configs/ db/ ec/ mag/ rt/ tmp/ utils/
amp/ cms/ dist/ ew/ rad/ snw@ trig/ wda@
{lomi:~}
Where each directory in the first level of the /app/rtem home is a directory or a symbolic link (in the case of wda and snw). The next section briefly describes the directories listed above and what is contained within them and is followed by a detailed review of what is in each directory.
- alarm - The directory where the alarming executables (alarmdec, alarmact, alarmdist) and their configurations are contained.
- amp - The directory where ampgen executables and configurations are contained. These are for ShakeMap? data generation.
- chan-configs - The directory where channel configuration scripts, SQL, and data files are contained. The example here is from SCSN.
- cms - The directory where the CISN Messaging System is configured and executed.
- db - A directory containing SQL scripts, and master database configurations that are included by other configuration files.
- dist - A directory where distributed tar.gz and backups of the /app/rtem directory are kept.
- ec - The directory where the Event Coordinator configurations and executable are maintained.
- ew - The directory containing all of the Earthworm binaries, parameter files (configurations), and logs.
- mag - The directory where the Trimag magnitude executable and configuration are maintained.
- rad - The RealTime Amplitude Data generator executable and configuration are stored in this directory.
- rt - A common RealTime configuration and log file directory.
- snw - A symbolic link to the SeisNetWatch agents scripts that run from Earthworm and cron (Links to ew/run/params/snw)
- tmp - A temporary directory where to store and write temp files (that can be deleted). This was used during configuration at HVO and can probably be deleted.
- trig - The directory where the trigger coordinator (tc) and C++ rcg and ntrcg programs configurations and executables are stored.
- utils - A directory that contains miscellaneous helper modules like role changing scripts, conlog, and other useful utilities.
- wda - A symbolic link to the Waveform Data Area configuration scripts and related files used by ew2wda (Links to ew/run/params/wda).
alarm directory
The RT alarm directory contains the executables alarmdec, alarmact, and alarmdist in addition to a configs directory where all of their configurations are contained. The alarm dir, as does each of the RT related programs, contains alarmctl.sh which is a shell script which controls the starting and stopping of all alarming related programs. The alarming thread as it is called, can be stopped and started simply by issuing an alarmctl.sh stop or an alarmctl.sh start command.
A quick inspection of the files and directories in alarm/ is shown below:
{lomi:alarm} pwd
/app/rtem/alarm
{lomi:alarm} ls -F
alarmact* alarmctl.sh* alarmdec* alarmdist* alarms/ configs/ msgs/
{lomi:alarm} file *
alarmact: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, not stripped
alarmctl.sh: executable /usr/bin/bash script
alarmdec: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, not stripped
alarmdist: ELF 32-bit MSB executable SPARC32PLUS Version 1, V8+ Required, dynamically linked, not stripped
alarms: directory
configs: directory
msgs: directory
{lomi:alarm}
amp directory
Amplitude computation is run out of the ampgen directory (not setup at HVO, yet...).
{lomi:amp} pwd
/app/rtem/amp
{lomi:amp} ls -F
ampctl.sh* ampgen* ampgen.cfg ampgen2.cfg events/ events2/ start_ampgen*
{lomi:amp}
chan-configs directory
The chan-configs directory contains configuration files that Caltech generates from the database using SQL and a number of perl scripts. They are provided here as an example of how the configuration can be automated. The procedure at Caltech is to update a master SQL that describes the channels to be used (found in chan-configs/sql_chanloader/cisn-chan-view.sql file), store this to CVS, and then run a script which pushes the station/channel configuration from the archdb to the RT db, and run all of the scripts necessary to generate the configurations to be used. These include configurations for RT programs and Earthworm programs. The scripts are found in chan-configs/scripts and example .cfg files are found in the top level directory. These .cfg files are then symbolically linked into the needed locations so that files do not need to be moved around after creation.
By keeping the configurations in CVS, the history of the configurations on a RT system can be easily managed and monitored for changes that take place from configuration to configuration.
cms directory
The CISN Messaging System (CMS) is the communications backbone on which messages are sent between RT programs. The program is a single Java application that runs and provides services to all RT modules that publish and subscribe to messages in what are known as "named channels".
The CMS directory has several levels of structure to contain the configurations used by publishers and subscribers and also the Java app. The listing of the directories is shown below and explained in the text that follows:
{lomi:cms} pwd
/app/rtem/cms
{lomi:cms} ls -F
QWServer.init* README.obs conf/ migrate.list printqfile* qfiles/
QWServer_CMS/ check_cmsprogs.pl* local_env.sh migrate.sh* psqw*
{lomi:cms} ls -F QWServer_CMS
QWServer.jar psqw* runCubeToQWXML.bat startAllNum* stopNotif*
conf/ runAll.bat runNotif.bat startNotif* stopQWS*
doc/ runAllNum.bat runQWS.bat startQWS* storage/
idl/ runCubeToAnssEQXML runSender startStopAll* test/
log/ runCubeToAnssEQXML.bat runSender.bat startStopAllNum*
outputterdata/ runCubeToQWXML startAll* stopAll*
{lomi:cms}
The top level CMS directory contains a conf directory that contains information about the connection ports and the name of the CMS service running on a given RT host. It also contains a qfiles directory which the C++ publisher and subscriber RT programs use (through a library) to track which messages are being received and sent (sort of an intermediate holding tank for messages coming and going from CMS). The last directory contained is the QWServer_CMS which contains the configuration, scripts, logs, and storage for the Java application known as QWServer.
The QWServer_CMS directory contains its configuration in the conf/ dir, documentation in doc/, log files (that self rotate) in log, incoming messages in storage/ and output messages in outputterdata/ . The QWServer module is started from the QWServer.init script which can be issued a stop or a start argument.
db directory
ec directory
The ec/ dir contains the event coordinator configuration and executable. The Event Coordinator is responsible for taking the CMS signal from hyps2ps and writing the event hypocenter and Md magnitude (if computed) to the oracle database after waiting sufficient time for the binder_ew module to finish its work.
ew directory
The earthworm directory holds all earthworm related programs and configurations as well as the logs (which should be cleared using a crontab entry as necessary). The top level directory contains sym links to break out the earthworm configuration into versioned directories:
{lomi:ew} pwd
/app/rtem/ew
{lomi:ew} ls -l
total 8
drwxr-xr-x 4 rtem rtem 512 Jul 16 14:53 7.2
lrwxrwxrwx 1 rtem rtem 11 Jul 17 03:11 bin -> current/bin
lrwxrwxrwx 1 rtem rtem 3 Jul 17 03:10 current -> 7.2
lrwxrwxrwx 1 rtem rtem 11 Jul 17 03:11 run -> current/run
{lomi:ew} ls -F run
logs/ params/
{lomi:ew}
The params directory is the one that most rtem operators should be concerned with and the EW layout being used for HVO and Caltech has been modified to hide many of the complexities in subdirectories. Looking into the params directory, one sees the following:
{lomi:ew} cd run/params
{lomi:params} ls -F
carl/ ew_sol_sparc.bash* mag/ startstop.desc trig_id.d
earthworm.d exp/ paramsHVOPROC1_070708/ startstop_sol.d wda/
earthworm_global.d hvoinp/ pick/ statmgr.d
eq/ inp/ snw/ statmgr.desc
{lomi:params}
To the average Earthworm user, this is a simplified but atypical earthworm parameter layout. The use of subdirectories to classify the different sets of modules has significantly reduced the clutter in the top level directory which just contains a few .d files for startstop, statmgr, and the earthworm.d and earthworm_global.d. In addition, the trig_id.d is here since this cannot be moved to a subdirectory (limitations within current versions of carlsubtrig).
Each subdirectory contains the config files associated with a particular task, carl/ contains the subnet triggering configurations for carlstatrig, carlsubtrig and trig2ps. Likewise the eq/ contains the hypocenter computation components binder, eqproc, hypo2000 and hyps2ps. What is most interesting is to see how this all looks in an Earthworm status command:
{lomi:params} status
using default config file startstop_sol.d
NOTE: If next line reads "ERROR: tport_attach...", Earthworm is not running.
Sent request for status; waiting for response...
EARTHWORM SYSTEM STATUS
Hostname-OS: lomi - SunOS 5.10
Start time (UTC): Fri Oct 24 00:18:39 2008
Current time (UTC): Tue Nov 18 20:24:59 2008
Disk space avail: 1754558 kb
Ring 1 name/key/size: WAVE_RING / 1000 / 4096 kb
Ring 2 name/key/size: PICK_RING / 1010 / 1024 kb
Ring 3 name/key/size: HYPO_RING / 1015 / 1024 kb
Ring 4 name/key/size: HYPO_RING_ARC / 1110 / 1024 kb
Ring 5 name/key/size: EXPORT_RING / 1075 / 1024 kb
Ring 6 name/key/size: FILTERPICK_RING / 1085 / 1024 kb
Ring 7 name/key/size: SUBTRIG_RING / 1105 / 1024 kb
Ring 8 name/key/size: TRIG_RING / 1095 / 1024 kb
Startstop Version: v7.2 2007-12-16
Process Process Class/ CPU
Name Id Status Priority Used Argument
------- ------- ------ -------- ---- --------
startstop 29483 Alive TS/ 0 4:55 -
coaxtoring 29489 Alive TS/ 0 171:23 hvoinp/coaxtoring.d
scn2scnl 29495 Alive TS/ 0 70:48 hvoinp/scn2scnl.d
pick_ew 29496 Alive TS/ 0 127:40 pick/pick_ew.d
binder_ew 29497 Alive TS/ 0 17:39 eq/binder_ew.d
eqproc 27498 Alive TS/ 0 1:06 eq/eqproc.d
hyps2ps 29505 Alive TS/ 0 1:15 eq/hyps2ps.d
carlstatrig 29506 Alive TS/ 0 92:20 carl/carlstatrig.d
carlsubtrig 29507 Alive TS/ 0 4:10 carl/carlsubtrig.d
trig2ps 29508 Alive TS/ 0 1:17 carl/trig2ps.d
localmag 29509 Alive TS/ 0 6:30 mag/localmag.d
ew2file 29510 Alive TS/ 0 8:37 mag/ewmag2file.d
ewmag2cisn 29511 Alive TS/ 0 6:26 mag/ewmag2cisn.d
ew2wda 29512 Alive TS/ 0 44:00 wda/ew2wda.d
copystatus 29513 Alive TS/ 0 4:05 WAVE_RING HYPO_RING
copystatus 29514 Alive TS/ 0 1:03 PICK_RING HYPO_RING
copystatus 29515 Alive TS/ 0 1:03 SUBTRIG_RING HYPO_RING
copystatus 29516 Alive TS/ 0 1:02 TRIG_RING HYPO_RING
statmgr 29488 Alive TS/ 0 1:48 statmgr.d
snwclient 29517 Alive TS/ 0 20:40 snw/snwclient.d
ewagent 29518 Alive TS/ 0 163:46 snw/ewagent.d
{lomi:params}
Note how each program's .d file is referenced from the subdirectory name first. While at first this may seem complicated, it does simplify management of the .d files in the long run. We also advocate using Subversion of CVS to track the changes to the configuration files if a script is not used to generate them as discussed above in the chan-configs section.
rad directory
The rad directory contains the Reduced Amplitude Data area program's configuration. Note that this relies entirely on the ew2wda module being configured in earthworm since the rad program reads its data from the WDA. The rad program gets its data using the Waveform Data Area (WDA) shared memory area and it finds its list of channels from the database using the program and config_channel tables (using the ProgramName? from the configuration file). It writes its data to the ADA. Read the rad2 manual page for more details on that configuration. In the rad directory you will find the rad2.cfg file which contains the basic configuration information pertinent to the program's running. The rad2 program is started automatically by the rtemctl.sh script, but it can be stopped and started using the radctl.sh script with the arguments stop and start. The start scripts will automatically recreate the ADA at each startup.
Here is an example rad directory as configured at PNSN:
rtem - sisters ~/rad 582% ls -F accmon* deleteada* makeada.cfg rad2* rad2_v0.2.2* rad2.cfg Richter.tab cit/ makeada* rad_common.cfg rad2_hvo.cfg rad2_v0.2.3* radctl.sh* start_rad* rtem - sisters ~/rad 583%
Also in the rad2 directory are the makeada program and the makeada.cfg configuration file which are used to create the Amplitude Data Area (ADA). Likewise there should be a deleteada program which can remove the ADA. As mentioned above, these are run from the radctl.sh script so the user should never have to run these by hand unless experimenting. In addition to making the ADA, there is also the accmon program which can be run as follows to monitor the data flow from a given SCNL:
rtem - sisters ~/rad 576% accmon USAGE: accmon <adaKeyName> <network> <station-name> <channel> <location> rtem - sisters ~/rad 577% accmon ADA_KEY UW LTY BHE ' ' The ada configuration file is : ADA_KEY The Network name is : UW The station name is : LTY The Channel name is : BHE The Location is : Acc=0.003479 FW=1 On-Scale=1 Ml_100=1.534307 snr=3.169959 SampleTime: 2009/03/16,20:30:40.9890 Acc=0.003019 FW=1 On-Scale=1 Ml_100=1.583617 snr=3.069205 SampleTime: 2009/03/16,20:30:45.9890 Acc=0.003556 FW=1 On-Scale=1 Ml_100=1.511855 snr=3.599416 SampleTime: 2009/03/16,20:30:50.9890 Acc=0.004360 FW=1 On-Scale=1 Ml_100=1.560445 snr=3.182176 SampleTime: 2009/03/16,20:30:55.9890 Acc=0.003517 FW=1 On-Scale=1 Ml_100=1.832924 snr=4.127188 SampleTime: 2009/03/16,20:31:00.9890 Acc=0.004398 FW=1 On-Scale=1 Ml_100=1.407428 snr=2.777008 SampleTime: 2009/03/16,20:31:05.9890 Acc=0.003536 FW=1 On-Scale=1 Ml_100=1.699616 snr=3.213536 SampleTime: 2009/03/16,20:31:10.9890 Acc=0.003907 FW=1 On-Scale=1 Ml_100=1.649890 snr=3.196099 SampleTime: 2009/03/16,20:31:15.9890 Acc=0.003919 FW=1 On-Scale=1 Ml_100=1.376969 snr=2.903471 SampleTime: 2009/03/16,20:31:20.9890 Acc=0.004277 FW=1 On-Scale=1 Ml_100=1.557402 snr=2.943147 SampleTime: 2009/03/16,20:31:25.9890 Acc=0.003460 FW=1 On-Scale=1 Ml_100=1.658243 snr=3.045758 SampleTime: 2009/03/16,20:31:30.9890 Acc=0.003862 FW=1 On-Scale=1 Ml_100=1.536280 snr=2.787752 SampleTime: 2009/03/16,20:31:35.9890 Acc=0.003179 FW=1 On-Scale=1 Ml_100=1.490909 snr=2.659571 SampleTime: 2009/03/16,20:31:40.9890 Acc=0.003632 FW=1 On-Scale=1 Ml_100=1.590270 snr=3.200006 SampleTime: 2009/03/16,20:31:45.9890 where the values are the acceleration and signal to noise ratio for a given 5 second time packet of amplitudes
mag directory
The mag directory contains the trimag configuration files, start script, and executable program. The executable is named trimag and the start script is trimagctl.sh (which takes the usual arguments of start and stop).
Here is an example directory from PNSN:
rtem - sisters ~/mag 586% pwd /app/rtem/mag rtem - sisters ~/mag 587% ls -F events_fast/ Richter.tab trimag* trimag_slow.cfg trimag_v0.1.8* trimagctl.sh* events_slow/ start_trimag* trimag_fast.cfg trimag_v0.1.5* trimag_v0.1.9* rtem - sisters ~/mag 588%
rt directory
The rt/ directory contains the logs and locks directories, and other RT module specific files and information. A quick inspection below shows only 3 directories conf/, lib/, locks/, and logs/:
{lomi:rt} pwd
/app/rtem/rt
{lomi:rt} ls -F
conf/ lib/ locks/ logs/
{lomi:rt} ls conf
gcda.cfg
{lomi:rt} ls lib
TAO_1.3a.1 libTAO_CosNaming.so libTAO_PortableServer.so.1.3a.10 libxerces-c.so.27
libACE.so libTAO_CosNaming.so.1.3a.10 libTAO_Svc_Utils.so libxerces-c.so.27.0
libACE.so.5.3a.10 libTAO_IORTable.so libTAO_Svc_Utils.so.1.3a.10 libxerces-depdom.so
libTAO.so libTAO_IORTable.so.1.3a.10 libstdc++.so.2.10.0 libxerces-depdom.so.27
libTAO.so.1.3a.10 libTAO_PortableServer.so libxerces-c.so libxerces-depdom.so.27.0
{lomi:rt}
Attachments
-
c_block.png
(16.5 KB) -
added by pfriberg 5 years ago.
-
ew_block.png
(122.2 KB) -
added by pfriberg 5 years ago.
-
alarm_diag.png
(48.6 KB) -
added by pfriberg 5 years ago.
Alarming diagram
