Minutes of the DAQ++ meeting
27-29 August, 1997
NBI Copenhagen

These minutes are incomplete. Unfortunately, I ran out of time.
The transparencies of missing talks can be found in the Hera-B
archives. MM

Present: H. Bertelsen, M. Dam, T. Fuljahn, A. Gellrich, J.D. Hansen
P. Kreuzer, I. Legrand, H. Leich, J. Li, K. Ludwig, M. Medinnis
D. Ressing, S. Scharein, S. Schmidt, U. Schwanke, A. Schwarz
R. Stdennis, A. Waananen, G. Wagner, P. Wegner, S. Xella, J. Zweizig


Discussion of "hardware aspects", Wednesday afternoon

M. Dam, A. Gellrich, H. Bertelsen, J.D. Hansen, H. Leich, K. Ludwig,
M. Medinnis, D. Ressing, P. Wegner, A. Waananen, S. Schmidt (+ ?)

1. Installation issues

The dimensions of trailer volume allocated for the 2nd and 3rd
level farms is 3.6 x 3.2 x 2.7 m3. The current plan is for 3 rows of
racks. Each row consists of 3 racks, two with length 1.29m and the
third 1.0m. The PCs will be in tower boxes. The long racks hold 24
PCs, the short racks hold 20 PCs, for a total capacity of 204
nodes. If more capacity is needed, a fourth row could be added,
although access would then become difficult.

A full 204 node farm will dissipate 10 kWatts of power. An airflow of
8000 cubic meters/hour should be sufficient for cooling. (For
comparison, a standard VME crate fan moves 1000 cubic meters/hour.)
The racks will be kept open. No enclosure is forseen.  Since, for
safety reasons, the doors to the trailer must remain open, it is
highly desirable to push filtered air into the room rather than pull
air out. K. Ludwig will check into safety implications, in particular
that the overpressure generated if the doors close is enough to cause
problems.

Since L2 and L3 nodes will not have hard disks, an uninteruptable
power supply is not strictly needed. However, since current plans are
to equip L4 nodes with hard disks and since, during startup, some L4
nodes may be located in the trailer it is desirable provide a UPS. The
total cost for 10 kWatts is of order 10 kDM.

2. CAN bus driver and farm slow control

H. Bertelsen has a working prototype of the CAN bus driver card he
designed for monitoring the health of farm nodes. The temperature of
each farm node will be monitored (via a temperature sensor probably
glued to the cpu). It will be possible to reset and power up/power
down nodes via the CAN bus.

The Siemans 80C535 CAN bus controller was chosen over the Philips
equivalent since the Philips chip is no longer being produced and it's
replacement is not yet available. A socket for the Philips chip is
also available.

NBI needs to know soon the protocol which will be used over the CAN
bus. The board cannot be finalized and the needed EPROMs cannot be
burned until the protocol is settled. 

The final card will be able to plug into either a PCI slot or an EISA
slot.

3. Linux issues

Linux distribution kit: NBI uses the Red Hat distribution kit. DESY
and Zeuthen use S.U.S.E. NBI will in any case continue to use Red Hat
because of their close ties to CERN who recommend it. Other than the
fact that Red Hat is much more widely available, there appears to be
no strong reason for choosing one over the other.

NBI currently has a 20-node Pentium Pro cluster set up and running
without problem. A system for monitoring and displaying CPU usage,
disk space and availability of nodes was written in-house.

      ================ Status reports ======================

Hardware:

  Fast control system            T. Fuljahn

Hardware: The new mother board is tested as well as one fanout and 2
cluster boards. A fiberless link board is available. System tests of
mother, fanout, cluster and one daughter are ongoing. Lab tests should
finish next week. PCBs and FPGAs for 15 boards are available. PCBs
should be stuffed by the week after next as well as a second mother
board.

Software: Software for lab tests and an FCS server are
running. Currently the server is being enhanced and a TCL/TK control
panel is being developed by a summer student.

Outlook (97): The mother board and signal distribution is working. 15
daughter boards are in production. The goal is a working system by the
end of Hera running. The bottleneck is the daughter boards. The
schedule is "very tight".

Outlook (98): The mother board, fan-out and cluster designs are
final. (There is a recently discovered problem with the mother board
concerning getting the full 64 bit bunch-crossing number onto a
SHARC. Hopefully this is solvable by reprogramming an EPLD). It is
hoped that the daughter board design is also final. Cabling has for
the most part been planned. Components for fanout, clusters and
daughters are on order. The goal is to complete the system in '98.

There was some discussion as to relative priorities of data taking and
commissioning of FCS and the new data path. Priority should be given
to commissioning of the new equipment. If possible, night and morning
shifts should be used for data taking.

The FCS will be debugged in the lab to the point that it communicates
with the FEDs. It will then be installed in the experiment and should
only be removed if there are major problems.

  SHARC board, test plans        G. Wagner

Testing of the new SHARC boards is continuing. The problems reported
at the last Hera-B meeting have all been solved. One of the reported
problems: bit errors on certain links at 80 MHz transfer rates, has
been solved but the solution has so far only been demonstrated at MSC
due to lack of time. 

Ten boards of the new design are now available at DESY and await
tests. The data path tests forseen for '97 will for the most part be
done with the old board to minimize perturbation of the existing
system.

  SHARC/PCI interface board      H. Leich

The interface has been tested and shown to work "very well" at half
speed. Full-speed has yet to be tested [This test has now been
successfully performed 3/9/97.] Seven boards have been produced. Five
of these are stuffed and tested. The remaining await delivery of the
PCI bus controller. There is no problem "in principle" to expand the
order from 40 to 100 for the middle of next year although it will have
to be checked how this fits into the Zeuthen production schedule. 

  SLT/TLT farm hardware          M. Dam

The combined SLT/TLT farm will consist of about 200 nodes, each with a
cpu, 32 (or 64) Mbytes of memory, a floppy (for booting) an ethernet
card, a cheap graphics card (for debugging), a CAN bus interface and a
SHARC interface. The preferred CPU is the Pentium II. A 266 MHz version
is currently on the market. A 300 MHz version should be available in
the fall. (The SLT benchmark runs 35% faster on the 266 MHz Pentium II
compared to a 200 MHz PentiumPro.)

Ethernet: At the June meeting, an ethernet switch manufactured by DEC
was the preferred solution. However it has since been withdrawn
because of design flaws. Another switch "NuSwitch" manufactured by
Network Peripherals is now preferred. It has similar characteristics
(apart from the design flaws!) and pricing to the DEC switch. NBI
currently has a 12xBase100Tx (12 port switch at 100 Mbit/sec) and two
1xBase100Tx->24xBase10Tx which converts from fast to "slow"
ethernet. They are so far "happy with it". They measure 42 Mbit/sec
transfer rates using TCP protocol on Linux using fast ethernet
compared to 7.2 MBit/sec for "slow" ethernet.

Question: mix fast and slow ethernet, or use only fast ethernet? The
mixed solution is cheaper (46kDM vs. 66 kDM for the system needed for
L2/L3) and should have enough bandwidth. However it leads to less
uniform hardware and fast-ethernet is likely to be better supported by
future releases of Linux. Given the marginal cost advantage, we
decided for the fast-ethernet only solution and to use hubs rather
than switches.

  System for '97 tests           M. Medinnis

Most of what was said is contained in "97 DAQ++ System evolution"
which is available via a link on the DAQ home page under the friendly
fish. 

Updates: It was decided earlier in the week to separate EVA from
"simple" EVC immediately and for JZ to adapt his "builder" to the EVC
task, which will at first run under UNIX. If the simple-EVC task is
moved into the SHARC world, it will be possible to add a simple-SLT
already at phase 2 which would allow for triggering based on ECAL
pulse heights and fit into plans of R. StDenis et al. for measurement
of pt spectra with the calorimeter (hopefully) leading to observation
of a J/psi signal.

Low(er) level software:

  RPS (was RPI) / kernel         D. Ressing

In the Hera-B scheme, Unix processes to talk each other via RPM and
SHARC processes talk to each other via RPS. L2 and L3 nodes are in
principle bilingual. Unix and SHARC processes can communicate with
each other via an RPS/RPM Bridge (RRB) which runs on VME crate
controllers. RPM has been in use for some time. RPS is now running
stably. A V0 RRB was completed in the last days and runs. 

A finite state machine (FSM) for the SHARC world is written but not
yet tested. It is needed for phase 1.

SIO (SHARC I/O), an implementation of some of the standard stdio
routines for the SHARC (notably printf) is being completed at the
moment.

Still missing is ERM (external resource management) which is primarily
needed for managing the SHARC boards global bus is still missing but
is not needed until later (next year).

Status and plans: A basic working version of RPS/kernel exists.
Complete integration with Run Control and RPM is nearly complete.
Utilities will be added as needed. The port to the new SHARC board
should be fast. The last missing piece, ERM, will come later.


  SHARC/PCI interface driver     I. Legrand

Most of what was said can be found by following the SHARC to PCI
interface driver (Iosif Legrand) from the DAQ projects page. Briefly,
a driver for the card exists which can transfer data directly to and
from the card into user space with very low latency (less than a
microsec) for and rps header and with an effective bandwidth of over
19 Mbyte/sec when operating at half speed (max possible = 20
Mbyte/sec). 

  SLT/TLT node management, slow control
                                 M. Dam

First ideas on process control were presented. The SLT and TLT nodes
will be connected through a shared fast ethernetwork. Also connected
will be a single SLT server, a single TLT server and a node to manage
the CAN bus. Each node will run a small Linux kernel, the trigger code,
a control process and some of ftp, telnet or nfs.

A simple 4-state finite state machine (EMPTY, READY, RUNNING, ERROR)
is probably adequate. Monitoring can either go via RPS and the sharc
switch or via the control process and ethernet. Node management could
be done through socket calls or, in principle, through "standard"
Hera-B software: Gizmo and/or RPM. This will be looked into.

  Data base                      D. Ressing

The "Table Based Data Base" of Mizzi is currently in use. It is a
simple client/server system with no relations. Editors based on TCL/TK
are available as is a histogramming package. It runs on all Hera-B
platforms. Two disadvantages: it uses its own TCP/IP based network
layer and has no relations.

So far, the data model is "fishy". Arte has a model defined by DD and
C/F77 source code. An abstract structure for DAQ configurations is
available via the DAQ home page. Also, formal ideas (ATLAS inspired)
have been presented . However we still have no estimates of data volumes and no
overall structure is defined.

The Bologna group is working on two projects:
(see link under the friendly fish, DAQ home page)

An interface to Arte which allows processes to store and retrieve data
from TBDB. The Arte data model is also stored in the TBDB. Only
Arte-dependent links between table rows are possible.

LEDA ("Light Environment Data base Application"): This is a relational
interface to TBDB. It provides relations between keys of table rows
and a model which can read/written from/to ASCII files. It is not Arte
compatible but can be made so.

The Lisbon group has done some preliminary comparisons of three data
bases: TBDB, My SQL, Objectivity. The current status is described 
in an html document, see link under the friendly fish, DAQ home page.
The results are under discussion.

To do: Most urgent is the development of a data model which must
happen before the system definition can be finalized. Once the
configuration model is defined, it should be implemented within LEDA
(as a test).

Subsystems:

  Event control, buffer manager status 
                                 D. Ressing

Simple-SBM is currently being developed by G. Wagner. It should be
ready next week. The RPS implementations of EVC and BMG will be done
by J.L, D.R. and G.W. in time for phase 4 (end October). The
simple-EVC needed for phase-1 will be implemented by J.Z.

  SLB data server                M. Medinnis

The event request, status request and data insert protocols of SDS are
implemented and cycling within l2simu. The range request protocol will
be implemented later since it is not needed immediately. SDS is
currently being adapted to run with simple-SBM on the SHARC.

Since we do not as yet have a configuration server which has enough
information about SHARC processes and topologies, a temporary server
(SCN == Sharc CoNfiguration server) based on l2simu routines was
implemented. So far, the only documentation is in the header file
which can be found in ~medinnis/herab/scn/v1/include/scn_msg.h on
hera-b. SCN serves info needed by RRB to set up a configuration, info,
such as routing tables to sharcs and info to the Event builder. A v0
implementation is running and tested.

  Event builder                  M. Medinnis (for F. Sun)

The current RPM version of the event builder (EVA) gets the buffer
configuration from SCN. Next it works with an SDS simulator (EVADEV)
built inside of l2simu to build events on request from the user. The
SCN dialog is tested, as with communication with SDS. Packing data
into formatted event data is nearly ready. Communicating with the
event manager (EMG) is not yet ready. Porting RPM version to RPS
should be easy. For the final version of EVA, we need to add EVC
communication and EMG connection.

  L4 control, L3/4 link          P. Wegner (for F. Sun)

  Data logger                    A. Gellrich
  4LT status and plans           A. Gellrich

The definition documents from the farm group are now available via the
farm home page. This includes "Application Software on the Farm",
"Mass Storage Management on the Farm" and "Higher Level Farm Control
(HEC) and Level 4 Farm Node ID Server". 

Several output streams from L4 are being planned: full DST, an
arbitrary number of selected DSTs, "MINI", an event directory and
monitoring, which includes calibration. Unresolved issues relating to
output files: Should events be time-ordered (not clearly necessary),
How should they be numbered (give an L4 number), what should the
typical file size be (200 MBytes is recomended by the computer
center), what is the definition of a run (e.g. "a period of time when
thing sare stable").

A directory structure for event data was proposed:
/acs/data/YEAR/EXPERIMENT/DATA_TYPE/RUN. For example:
/acs/data/97/exp03/dst/...

The components implicated in the mass storage system are the 4LT farm
nodes, the central data logger and the archiver. The various
components will be tied together with EMG buffers and RPM
messages. Hardware status: Linux PCs as L3 nodes are coming soon from
NBI. An AIX (PPC 604e) workstation from Zeuthen is setup at DESY as a
4LT node and is being looked after by T. Finnern of DESY-ZDV. Software
status: L4S (the L4 server) is complete. EVA is under
construction. Online Arte is already in use for receiving data, SVD
sparsification, datat logging and data quality checks.

  Run control, user interface    J. Zweizig
  Configuration/resource manager J. Zweizig

Existing now:

A prototype state machine now exists (GIZMO) and is being extended to
the SHARC world by D.R. A single (common) state diagram describes the
system. The state diagram needs some generalization and work on a
variable state machine will start soon (9/1?). State inference also
needs development.

The run control process currently combines configuration and control
functions. It is heavily used for test data acquisition. Some
difficulties in handling persistent processes need to be sorted out.

User interface: A control panel provides a TCL/Tk interface to zmo,
fcs, par, nam, pvc packages. It is not a shell and does not provide
for process interogation. Process specific scripts exist for the
various processes. A TCL/TK error-logger display also exists.

Development path:

New primitives need to be added: to list state machine daughters, to
list parameter names and to get error messages. Graphic design of the
run control panels is needed, This will be done iteratively.

  SHARC board monitor            Jinsheng Li

  Slow control                   J. Zweizig (for U. Uwer)

  Calibration system             U. Schwanke


Implications of "Detector optimization for the 1998 Physics run",
  T. Lohse et al.                M. Medinnis

Review of schedule, manpower for '97, status of "integration events"
                                 D. Ressing

Review of schedule, manpower for '98
                                 M. Medinnis

==================================================================

Friday, HLT algorithm developments: S. Xella

  Discussion of magnet tracker performance document
  Transparencies are available via slt/talks www page

  Discussion of silicon tracker performance document: S. Schmidt
  Transparencies are available via slt/talks www page

  SLT '97 revisited           P. Kreuzer

  TLT studies                 S. Scharein

  MC production proposal      S. Scharein

  HLT integration: discussion of input and output rates of various levels
                              all