Instructions for running the common Slow Control System (SCS)
Responsible for this page: Vladimir
Rybnikov .
Click here to find the slow
control experts for every component
What to do if the expert is not reachable? Contact
your favorite DAQ expert: see
expert shift list.
Contents
How to start the common SCS
The common SCS should
be run from the SHIFT terminal (hb-cr21).
The common SCS must be always
running. It means that if it is down no
DAQ runs with reasonable data can be taken
and there should be special instructions in
the short term plan concerning the situation
with the common SCS.
|
This chapter contains instructions how to set up all processes
required for the common SCS with all the components or part of them.
In this documentation processes are printed bold. The code
for buttons on graphical user interfaces
is this. Names of states
are printed such.
prcmanag is responsible to setup the processes
which are common for the SCS. There should be always a main prcmanag
running
in hbdaq0 ( at the moment the full RPM name
for that process is )
.
This process is launched automatically when
crcscui (SCS user interface) is started. While running the number of
prcmanag
should increase depending on the number of components included in the SCS.
Most of the processes run on hbdaq4. The
abnormal termination of one of them implies the shutdown of the system
(see trouble shouting) . In this case a list of
killing process message will appear in the error
logger window.
The user interface is called crcscui and can be
found in the usual online path.
If you are entitled
to start the common SCS follow the following procedure:
-
logon to hbshift on hb-crXXX (passwd is noted in paper logbook)
- Any cr machine is valid -
-
enter crcscui &
-
press new run on the
popped up window
-
select the run configuration 'slow'
.
-
after selecting the run components ( red
button means selected) press the key OK
and
confirm the pop-up menu
-
!!!!! ITRmust
not be included in the common SCS !!!!!
.
-
the control panel will appear
-
press MENU on the Process
Control section of the control panel (a pop-up
will appear).
-
press Create on the pop-up
to create processes.
-
wait till "state:" on the control panel
is INITIALIZED (and watch the two error
loggers windows; the white one is global and it is running permanently
in the hb-cr21 node. the yellow one
is specific to the run).
-
Press SAFE to bring
all the slow control subsystems to SAFE state.
-
Please be patient, the transition could last for few minutes.
-
Press READY in
the control panel to bring all the
slow control subsystems to READY state.
-
Please be patient, the transition could last for few minutes.
-
Now all subsystems have their high
voltages getting
On !!!
-
Check the actual values of the high
voltages for all the
subsystems by means of the graphical user interfaces belonging to the components.
-
If all the voltages have the nominal values (see slow control
instructions of the components) you are ready to start a DAQ
run
-
If you STOP
the
DAQ run
and are not going to take another one you have to switch all high
voltages Off by pressing
SAFE
in
the control
panel
-
Please be patient, the transition could last for few minutes.
-
While taking data you can watch the state and status of the
common SCS.
Top
Steps to terminate SCS run processes:
(KILL ALL PROCESSES)
-
This is needed only in case of unrecoverable
error .
-
VERY IMPORTANT = Change
state to INIT pressing the button INIT
in the control panel. Wait until the state in
the control panel changes to INIT
.
-
Please be patient, the transition could last for few minutes.
-
press MENU on the control
panel and a pop-up will appear.
-
press Terminate on the
pop-up.
Top
How to Ready/Safe/Init the system
The HERA-B SCS control Panel (crcscui)
is the top level control for the time being. It contains the following
important buttons and menus:
-
Ready button to bring
all the components slow control subsystems to the state READY
(all High Voltages On !!!).
-
Safe button to bring
all the components slow control subsystems to the state SAFE
(all
High Voltages Off).
-
Init button to bring
all the components slow control subsystems to the state INITILIZED
(the
very first state after launching the process).
-
button
MENU invokes Process
Control section that contains the following buttons:
-
create button to setup
all processes required for the common SCS
-
terminate button to stop
all processes required for the common SCS
-
terminate and exit button
to stop all processes required fro the common SCS and exit panel
-
row of the buttons with participating
components
Top
How to know that the common SCS is there and
ready
Top
NAME SERVER
This tool displays the process names
running
under the HERA-B DAQ environment. (Be extremely
careful when you use it, since you can disturb other people work).
Top
How to check the state/status of the
component /process
After the start-up of the slow control system a process state
display window appears
It consists of four fields and several buttons. In the
upper field (states/statuses filed) the states and statuses of all the
slow control processes applying the state machine and included in the run
are displayed. The field "Excluded
processes" contains the names
of the processes that have been excluded from the run., i.e. their states
don't influent on the state of the whole system ( see details in how
to exclude/include a component/process from/to the common SCS for standalone
tests) . The list of excluded processes can be updated by pressing
the title-button "Excluded processes".
The field "Started GUI"
shows
the names of the GUI processes that are registered in the slow control
run domain (currently
/RUN_slow/).
This
list can be updated by pressing the title-button "Started
GUI" All other processes
belonging in the slow control system can be seen in the field "All
other processes".
The list can be updated by means of the title-button "All
other processes".
Top
How to exclude/include a component/process from/to
the common SCS for standalone tests
If you would like to exclude a particular process from the
system for some tests, fulfill the following steps:
-
click on the name of the process in the states/statuses field
of the process state display. The field background of the chosen
process should change the background to green.
press the button "Exclude
proc" to complete the operation. IMPORTANT
!!! the system will change the
state to STOP if
all the processes of a component have been excluded. If you want to exclude
all processes for a component you have to exclude the whole component .
If you would like to exclude a component from the
system for some tests, fulfill the following steps:
-
press the button with the name of the
component on the "HERA-B SCS Control" .
a new window will appear:
-
press the button "Exclude"
to complete the operation
-
the button with the component name
will change the color for red
the names of excluded processes appear
in the the field "Excluded
processes". If
the whole componen has been excluded then the name of the branch for the
corresponding component will appear in the field.
-
to include back a component or a process
click on the process name in the field "Excluded
processes" and press
the button "Include proc".
If the whole component has been included to the system the branch and all
daughter processes will appear in the the states/statuses field of the
process state display. In case of one daughter process, it will appear
after some delay (the update time ~10 sec).
Top
Starting GUI
Two buttons "spc GUI" and
"smc GUI" should be used to start
a graphical user interface (GUI) for a process .
The name of the button "spc GUI"
changes
to " GUI " if
a process from the field "All
other processes" has been selected. To start
a GUI use the following procedure:
-
chose the process in one of the fields "States/Statuses"
,"Excluded processes" ,"All other processes"
-
to start a process specific GUI
press
the button "spc GUI" (" GUI " )
-
to start a smc GUI press
the button "smc GUI"
-
in case of starting a process specific
GUI the following window appears
-
you can chose the node to display the GUI by selecting
it from the list or by typing in the node (X-terminal) name in the
field "Node"
-
you can also chose the display (if you have more than
one) for the GUI by selecting the "Screen"
-
press "Start"
to
complete the procedure
-
in case of starting a SMC GUI the local
host is used to display it
-
you can use this interface to change the state of the corresponding
process.
To control the started GUI use the field
"Started
GUI"
Top
Terminating processes
The button "Terminate proc"
should
be used to kill a process or a GUI.
The name of the button "Terminate
proc" changes to
"Terminate GUI " if a process from
the field "Started GUI"
has
been selected. To kill a process use the following
procedure:
-
chose a process in one of the fields "States/Statuses"
,"Excluded processes" ,"Started GUI" ,"All other processes"
-
press the button "Terminate
proc" ("Terminate GUI ")
-
the following window appears
-
chose a signal for terminating the process and press "Terminate"
-
in case of SIGKILL
the
name of the process is removed from the name server
-
check the name of the terminated process
in the name server by means of the name server
panel NAMDIR
-
remove the name from the name server
if the terminated process has not been restarted by the Slow
Control System automatically that is true
for all server processes
Top
Alarm signals
By default the system produces a number of alarm signals
bound to the following events:
-
the state of the system is ERROR
(can
not be switched off)
-
the status of the system is ALARM
(can
be switched off)
-
the status of the system is UNKNOWN
(can
be switched off)
-
the status of the system is ERROR
(can
be switched off)
To switch off/on an alarm signal for a certain event
-
go to Option pull-down menu in the HERA-B
SCS Control and select traced
statuses button
-
use the corresponding check button to switch off/on the event
You can find out which process invokes the alarm by means
of the Process state display.
Top
History display
One can examine the behaviour of the slow control processes
by pressing the button "History"
in
the display state window. A process is kept in the list if its state happened
to be ERROR
or the status not NORMAL.
The history is being kept since the crcscui
start-up.
Top
How to get online errors reported
The error logger "erwin" is started by typing "erwin" at
the command line. It displays a window which contains the last some error/warning
print outs. In general an error entry looks like:
17:28:58 [Error]
evc
- Flt-id hasn't changed after 3 retries
-
The first field is the time the error
occurred at
-
The second field is the severity level
-
Inform for run progress
information
-
Warning for signs of
potential misbehavior which do not affect the data taking from a DAQ point
of view
-
Error running is seriously
effected (data most likely useless)
-
Fatal continue running
is impossible
-
next the process producing the error
message is shown
-
then a message string describing the
cause of the message
Further Documentation
Top
Trouble shooting
What to do if the state of the system is ERROR.
What
to do if the state of the system is STOP.
In most cases it happens due to the loss
of RPM messages coming from hb-vmexxx machines.
-
find out which process causes this state
-
use the procedure described in how
to check the state/status of the component /process
-
wait for 3-4 minutes until the process completes its state
transition and sends back its new state.
-
the state of the system should change.
-
if the state of the process remains STOP
than
you have to restart it.
-
for that purpose use the procedure described terminating
processes.
-
a new process will be started automatically
(some information can be seen in the yellow error window).
-
if you failed to kill the process as
described in terminating processes have
to use the online routine rmkill
-
to understand how to use this
routine enter rmkill -h
What to do if you cannot kill the whole system by means of Process
Control panel
It can
happen due to the loss of the control panel
or killing the system process manager /SYSTEM/SCS/prcmanag
What to do if the Process Control panel
is not updating
It can happen due to the loss of the synchronization with
main loop by Tcl/Tk shell.
Top
EXPERT INFORMATION.
Where to find the subdetector specific processes:
SUBDETECTOR |
NODES (PROCESSES) |
EXPERTS |
DB
|
hbda4(COMMON/DB_Monitor)
|
Antonio Amorim (3683) |
TRD
|
hb-cr14(TRD/High_Voltage_Monitor),
hb-cr14(TRD/gui_High_Voltage_Monitor)
|
on
call TRD expert |
ECAL
|
hb-vme47(ECAL/Crate_Control),
hb-cr09(ECAL/gui_Crate_Control),
hb-sun...(ECAL/Hardware_Monitor),
hb-cr09(ECAL/gui_Hardware_Monitor)
|
on
call ECAL expert |
DAQ
|
hbdaq5(DAQ/Network_Monitor),
hb-cr21(DAQ/gui_Network_Monitor)
|
Jose M. Hernandez (4853) |
Slow Control group
|
hb-vme54(COMMON/Central_Crate_Control),
hb-cr21(gui_Central_Crate_Control)
|
office: 4857 |
FARM
|
hbserv(FARM/Robot_Watch),
hbserv(FARM/gui_Robot_Watch),
hb-cr19(FARM/4LT_Monitor),
hb-cr19(FARM/gui_4LT_Monitor)
|
on
call 4LT expert |
OTR |
hb-vme31(spc_ot_fed,
spc_ot_hv,
spc_ot_asd8),
hb-cr20(gui_ot_asd8, gui_ot_fed, gui_ot_hv)
hb-con05(hv db-server) |
on
call OTR expert |
SVD |
hb-vme51(SVD/High_Voltage),
hb-vme51(SVD/Temp_Scan),
hb-cr11(SVD/Chip_Power),
hb-cr20(gui_SVD_High_Voltage,
gui_SVD_Temp_Scan,
gui_SVD/Chip_Power) |
on
call VDS expert |
RICH |
hb-vme37(RICH/High_Voltage,
RICH/Low_Voltage),
hb-cr26(RICH/gui_High_Voltage,
RICH/gui_Low_Voltage) |
on
call RICH expert |
HIPT |
hb-vme37(HIPT/High_Voltage),
hb-cr02(HIPT/gui_High_Voltage),
hb-con06(hv db-server) |
on
call HIPT expert |
MUON |
hb-vme37(hv),
hb-vme32(fed),
hb-cr04(gui_mu_hv, gui_hv_fed),
hb-con06(hv db-server) |
on
call MUON expert |
Where to find the rest of the processes:
NODE |
FUNCTION |
hbdaq0 |
main prcmanager |
hbdaq4 |
all other processes (run_watch, namservers, local prcmanagers,
branches, error logger, error window) |
What to do if I don't understand the instructions on
this page
If you feel it could adversely affect the quality of the
data being taken, contact the on-call expert. In any case, send an e-mail
to the responsible for this page with your suggestions for improvements.
Top