Minutes of the Hera-B Software Strategy Meeting

Heidelberg, 9-Dec-1998


Minutes of the Hera-B Software Strategy Meeting, Heidelberg 9-Dec-1998
======================================================================

Present: Hartwig Albrecht, 
         Andreas Gellrich, 
         Thorsten Glebe,
         Natasha Ratnikova, 
         Dominik Ressing, 
         Vladimir Rybnikow, 
         Michael Schmelling,
         Rainer Wanke
         
Goals
=====
One of the goals is to allow quick realeases of the code, where
urgent changes and bug fixes can be implemented very fast. In 
general those urgent fixes will be related to the datataking.
Non-urgent changes will refer to Monte-Carlo or offline 
processing in general.
        
Reaching the goal of fast releases requires identifying and 
disentangling dependencies between software components, a
homogenuous directory structure for all components and a
central place where everything is stored. 

The following is a summary of the issues that were discussed at the 
meeting, and the decisions that were taken in order to evolve the 
existing software landscape into a uniform common hera-b environment. 

Glossary
========
A large part of the discussion centered on two basic concepts 
that constitute the hera-b software environment: packages and
projects. Although a formal definition was lacking, the 
operational definitions are roughly as follows:
 
package: A package is the smallest entity with its own 
         independent versioning. It usually is maintained 
         by a single developer. 
         Examples are marvin, ranger,...

project: A project is a collection of packages, with an 
         independent versioning for the collection. It is
         maintained by a group of people and may have a librarian
         assigned to it. Packages inside the project can depend 
         on each other as well as on external projects. Important, 
         dependencies between different projects are such, that a 
         well defined hierarchie exists. Circular dependencies must 
         be avoided.
         Examples for projects are ARTE, ONLINE, VDSLIB,...

Hera-B Software Projects
========================
The following list of projects was identified during the meeting. 
The list is still preliminary in the sense that additional 
projects may have to be defined or that some of the existing 
projects have to be split.

 ARTE
 ONLINE  (including RHP)
 VDS
 ECAL
 LEDA
 MIZZI
 FLTSIM
 SLT
 DATAMGR (GPACK, CM, L4ALOG, EVDIR)
 GLOBAL
 FARMS
 GEO

From the requirement that no circular dependencies are allowed
it follows, that some projects have to be split. The VDSLIB for 
example should be independent of ARTE. Since it also has to work 
inside ARTE, interfaces are needed. Currently those interfaces 
form part of the vdslib, which makes it depend on ARTE while ARTE 
also needs the vdslib. To break that loop the ARTE-interface 
"silarte" will become a package inside ARTE and be removed from 
the vdslib. The same scheme has to be applied to other detector 
specific projects like for example CARE.

The GLOBAL-project should collect functionalities that are of
interest to all projects. Prime candidates are hbprofile and the 
C/C++ Cernlib interfaces.

Directory Structure of Software Projects
========================================
Agreement was reached for the layout of the directory structure
of the hera-b software projects. After comparing the solutions
used by ARTE, ONLINE, VDSLIB and MIZZI, the following proposal 
has been worked out for the organization of the code that should 
be visible to the whole collaboration. No specific rules are imposed 
on development areas and private schemes in general. Note that the 
directory names and use of upper/lower cases used below is mandatory.

$HBROOT/PROJECT/pro
                dev
                version1/include/...
                        /etc/make.log, iol-files, ... 
                        /doc
                        /extras
                        (source)
                        HBBINTYPE1/lib
                                  /bin
                                  /lib.g
                                  /bin.g
                        HBBINTYPE2/lib
                                  /bin
                                  /lib.g
                                  /bin.g
                version2/include/...
                        /etc/make.log, iol-files, ... 
                        /doc
                        /extras
                        (source)
                        HBBINTYPE1/lib
                                  /bin
                                  /lib.g
                                  /bin.g
                        HBBINTYPE2/lib
                                  /bin
                                  /lib.g
                                  /bin.g

In Hamburg HBROOT=/afs/desy.de/group/hera-b. pro and dev are links 
to the respective versions. The "extras" could be scripts or examples, 
i.e. stuff that might be useful and/or specific for a particular 
project. The source should be provided either as a set of source 
code directories, as it currently is in ARTE, or simply as a tar file 
of the source code that can be consulted when trying to understand
the inner workings of the code. As discussed below, it is also needed 
for installation of the project on a remote platform. 

Softlinks
=========
One step towards achieving fast releases is the use of pro links 
for all stable versions of the software. In case of bug fixes the 
pro link can simply be moved to the next/corrected version and 
everything rebuilt by re-issuing gmake. 

The objective of "pro"-links is to point to versions that work 
together correctly with pro-versions of all other projects. If 
a piece of software is shown to work correctly with all other 
pro-versions, it can also be linked to "pro". Similarly, dev-versions 
that work together satisfactorily can be moved to "pro" together. 
The whole system should be initialized ASAP and then made to work. 
Further development then can follow the scheme outlined above.

All changes of links should be logged at a central place on WWW. 

The scheme above is a way to ensure integrity of the whole software 
by means of symbolic links, which effectively hides the actual 
version numbers. The next step is to apply this scheme to the 
hera-b software.

For the more remote future one might consider an approach where 
dependencies form part of the distribution of a package, so that 
each package knows with which versions of other packages it can 
work together. A protototype for this kind of organization is the 
the RedHat-package manager, which is publicly available for a 
large variety of platforms.


Makefiles
=========
Some general issues pertaining to makefiles have been discussed at 
the meeting. First of all, it is of course impossible to forsee
all constellations that have to be covered in practice. In general
therefore all packages will have to come with their makefiles 
which build the binaries, libraries and executables within the
respective projects.

The compiler should be specified by a variable, so that it is trivial 
to switch between different compilers in a transparent way.

All projects that a program has to link to should be defaulted to the 
pro-version in the makefile. The version string then has to be 
overwritten if a dev-version or a particular version number has to be 
used.

The recommended make is "gmake"


Compilers
=========
The default compiler for all hera-b software should be the egcs compiler. 
The problems with incompatible binary formats between egcs-output and the 
CERN libraries on the hera-b have to be solved, technically it is known 
how to proceed. Of course other compilers are admitted as well, but for 
compatibility reasons all code should be written such, that on every platform 
for that an egcs compiler is provided, it also compiles with egcs.

(A scheme still needs to be defined that also switches compiler options 
when switching  compilers.)


Distinction between different Bintypes
======================================
The HBBINTYPE variable will be used to distinguish between executables for 
different platforms. It is more flexible than the regular BINTYPE-variable 
and completely under control of hera-b. Whereever possible the value should 
be chosen identical to the value of BINTYPE.
 
   new            old

 IRIX_mips4     sgi_62
 Linux_intel    i386_linux2
 Solaris_sparc  sun4x_55
 AIX_risc       rs_aix41       
 HP-UX          hp_ux102



Support for different platforms
===============================
There are many different unix-platforms in use in hera-b. In the previous 
months it has become evident that the software group, i.e. librarians 
and developers of the different software modules, do not have sufficient 
manpower to guarantie full support for all platforms. 

The following platforms are being used within the collaboration

 SGI
 ALPHA
 HP
 AIX
 Linux-PCs
 SUN
 SHARC
 LYNX

The agreement is to always write code such that it is independent of
the specific platform it has to run on. Except for DEC-ALPHA platforms this 
seems to be feasible without major effort. There is however a certain 
priority for those platforms which are  involved in the hera-b datataking. 
For these platforms, which also are part of the DESY afs-domain, "primary" 
support is provided. "Primary support" includes full installation and 
debugging of the code. Other platforms get "secondary" support, which 
implies that the code in principle compiles and runs on these platforms, 
but that the installation has to be done by the user. Bug-fixes which are 
required for those platforms have to be communicated to the authors and 
implemented in the code.

The following list emerges:

primary support

  Linux-PCs  - Linux_intel
  SGI        - IRIX_mips4

secondary support

  AIX        - AIX_risc
  HP         - HP-UX
  SUN        - Solaris Sparc
  
not recommended, e.g. because no reference machine available from afs and 
also because of different (64-bit) processor architecture:

  ALPHA      - OSF1

It was emphazied that authors of software are asked to code in a way
that their code compiles/links/runs on ALL HERA-B platforms.


Code Distribution
=================
Despite its various well known problems, the afs-system is still the 
backbone of the hera-b code installation and distribution scheme, since
there, in a transparent way, the code can be installed for the primary 
platforms and then immediately is available for distribution.

A simple copy of the afs-directory tree holding the binaries for a 
specific version is sufficient for installation of that project. The 
distribution is required to also contain at least a tar-file with the 
respective sources. Copying and unpacking that tar-file and then typing 
(g)make should build the binary distribution on a remote machine. For 
primary supported platforms the make is required to complete successfully. 
On platforms with secondary support it is expected to work as well, but
maybe only after some bug-fixing.

Guidelines
==========
In addition to the organizational items described above, also some 
guidelines were formulated with the aim to improve transparency and
usability of the code:

 1) Archive files should use names of the type lib*.a
 2) One should aim at one header-file per archive.
    In most cases this header file would simply include 
    the other headers needed by the archive. 
 3) Recursive includes must be blocked by appropriate 
    #ifdef constructs.
 4) Environment variables are needed to define paths to
    projects that are outside the control of hera-b, e.g.
    the CERNlibs, CLHEP, MIZZI etc. The hera-b specific projects 
    are accessed by the respective makefiles by exploiting the 
    uniform naming scheme after the global path variable HBROOT.
    Versions are defaulted to pro, but can be overwritten by 
    the user.