Years of Computing in hep international Workshop on Large Scale Computing

Yüklə 470 b.
ölçüsü470 b.

20-30 years of Computing in HEP

  • International Workshop on Large Scale Computing

  • Kolkata

  • 8 February 2006

  • René Brun

  • CERN

Hardware & Software Evolution

Punched cards

Mainframes, workstations,..

, OS, Desktops & Laptops

The 3 technology laws

  • Moore's Law: formulated by Gordon Moore of Intel in the early 70's - the processing power of a microchip doubles every 18 months; corollary, computers become faster and the price of a given level of computing power halves every 18 months. (well ! Not true anymore, see later)

  • Gilder's Law: proposed by George Gilder, prolific author and prophet of the new technology age - the total bandwidth of communication systems triples every twelve months. New developments seem to confirm that bandwidth availability will continue to expand at a rate that supports Gilder's Law.

  • Metcalfe's Law: attributed to Robert Metcalfe, originator of Ethernet and founder of 3COM: the value of a network is proportional to the square of the number of nodes; so, as a network grows, the value of being connected to it grows exponentially, while the cost per user remains the same or even reduces.

  • But no laws about Software (well ! Murphy’s law)

Hardware & Compilers

Multi Core CPUs

Program Size (lines of code)

Program Size (RAM)

Time to compile

Files, Classes


Fortran to C++

From static modules to plug-ins

  • app.exe = (main.o) 1955

  • app.exe = (main.o, x.o, y.o) 1965

  • app.exe = (main.o, x.o, lib1.a, lib2.a) 1975

  • app.exe = (main.o, x.o, lib1.a,, 1985

  • app.exe = (main.o, + dyn 1995

  • app.exe = (main.o, + plug-in manager 2005

  • BOOT + URLs + local caches (interp + comp) 2015 ??

    • See my talk at CHEP

Current ROOT structure & libs

Compiled + interpreted code

  • 1980: zcedex, mini command interpreter

  • 1985: kuip/paw, command and macro interp

  • 1986: Tk/Tcl includes a GUI

  • 1984: comis, Fortran77 interpreter

  • 1994: cint, a C & C++ interpreter

  • 1998: python, OO on top of C++, Java

  • 2002: ruby, better than python?

  • 200x: BOOT (inter->code generation->compiler)

Basic types and modules

  • 1950: basic operators (trig functions part of the application)

  • 1953: trig functions in a library

  • 1954: fortran types (integer, real, hollerith). Subroutines communication only via arguments.

  • 1965: subroutines communicate via a blank common, then labeled common blocks.

  • 1975: communication via a data structure management system

  • 1980: derived types

  • 1988: Object-Oriented programming: classes

  • 1995: parametrized types, templates, STL

  • 1996: Reflexion/RTTI (Java)

Programing models

  • Procedural sequential

  • Parallelism (MPI)

  • Vectorisation

  • Shared memory

  • Multi-threading

  • Client-server

    • Statefull
    • Stateless ->web
  • Corba

  • Distributed parallel computing (asynchronous)

  • Messages. Signal/slots

Problems with Fortran

  • Abuse of common blocks.

  • Unmanageable in large programs

  • No data structures

  • No generic machine independent I/O

  • Systems like Hydra(1974), Zbook(1975),Bos(1977),Zebra(1983) designed to overcome these problems.

The Zebra system (1983)

  • Zebra = Zbook + Hydra

  • Main data structure management system used by PAW and Geant3 and also many collaborations.

  • Powerful machine independent I/O

  • FZ: sequential

  • RZ: direct access (PAW ntuples)

  • Nice Data structure documentation system, including an interactive browser DZDOC.

Zebra bank descriptor



Zebra pros/cons

  • Programming style archaic

  • Easy to overwrite data structures

  • Shared global store(s)

  • Shared global store(s)

  • Self-describing structures

  • Concept of multi-heap (constants, histograms, event,..)

  • Efficient garbage collection (division wipe)

  • Built-in efficient and machine independent I/O

  • Used by Geant3,PAW and many experiments

Geant 1,2,3,……..4

  • Geant1 1974

    • 2000 lines of Fortran 4
    • No physics, no geometry, only a bare framework
  • Geant2 1975

    • 20000 lines of Fortran 4
    • Some physics for multiple scattering, energy loss, decays, framework for geometry and tracking
  • Geant3 1980,81 1994 ------2006?

    • About 120000 lines of Fortran77 + zebra + paw
    • Electromagnetic physics
    • 4 hadronic packages (Tatina, Gheisha, Fluka, Calor)
    • Powerful geometry package including graphics
    • Hits/Digits framework
    • I/O subsystem (zebra) for all structures including geometry.
    • Used by many experiments. Still a reference!!!

Fluka  Fluka

  • Originally developed by safety protection group at CERN (stevenson) + aarnio + ranft) 1985 ?

  • Reengineered by A.Ferrari &co: Rubbia project 1990

  • Probably the best for Physics processes

  • Simple geometry

  • The reference for radiation/shielding

  • Written in fortran77

  • Interfaced with VMC (TFluka) and G4 (Flugg)


  • Started in 1994

  • Originally a flagship project for the move to C++

  • A huge investment in manpower

  • About 600000 lines of C++

  • Validation process in Atlas, CMS and LHCb

  • Physics processes getting better and better

  • But still many limitations

    • Poor interpreter (small subset callable from python)
    • No I/O interface (geometry cannot yet be made persistent)
    • Batch style graphics

The Virtual MC (1998)

Virtual Monte Carlo and ROOT Geometry

  • The ROOT geometry package (TGeo) can be used in detector simulation, reconstruction, graphics, etc.

  • TGeant3

    • Used in production – native GEANT3
    • New: TGeant3TGeo – interface to G3 using TGeo geometry
      • No modification required in the user code
      • Validated by Alice
      • Same speed or faster than TGeant3
  • TGeant4

    • Used for Geant4 physics validation – G4 native geometry built after g3tog4 conversion
    • No interface yet between G4 and ROOT geometry
      • But Andrei Gheata actively working on it (expected this spring)
  • TFluka

    • Old geometry interface using G4 geometry vis FLUGG
    • Currently a fully validated geometry interface based on TGeo
    • Validated by the Fluka team
    • At least 2 times slower than TGeant3
  • The VMC framework is currently used by Alice, Opera, Minos, NA48b,Hades, CBM and may be STAR.

PAW: a long saga

  • First version (Jan 1985) by a committee

    • Must use GKS
    • GUI based on VT100 functionality
    • No ntuples
  • June 1985: developers “abolish” the committee

  • Higz: GKS + X11

  • Row wise ntuples, then ColumnWiseNtuples (1986)

  • Frozen in 1994, but still maintained by ROOT team

Crisis: 1992 1999

Why not F90 after F77?

  • In 1989,90,91 assumption was F90

  • Some work invested in I/O with F90 (to support derived types). We could not solve this problem, because no formal way to parse the F90 module descriptors.

  • In 1992 many forces pushing towards OO

    • CHEP Annecy, web, Next
  • Crisis in Dec 1992 (at least in IT software group)

    • 1/3 in favour of f90
    • 1/3 in favour of commercial solutions
    • 1/3 in favour of C++


  • ZOO, NextPaw, Geant3.5 proposals rejected

    • ZOO: Zebra in the OO world
    • NextPaw: Paw evolution ->C->C++
    • Geant3.5: Implement geometry package in C++
  • Geant4 proposal (June 1994)

  • RD45/Objectivity project (fall 1994)

  • ROOT project starts (in NA49) (Jan 1995)


  • ROOT chooses the CINT interpreter

  • We had been attracted by Java (Object base class, many common ideas).

  • Work on object persistency based on the dictionary information (introspection).

  • Design of ROOT Trees (split mode). Comparison with Objectivity

  • LHC++ project starts (against ROOT)

    • see Jamie’s talk


  • Getting experience with OO (professional developers).

  • Most users lost in f77->C++

  • First signs of problems with Objectivity in Babar

  • FNAL RUN II chooses ROOT

    • But C++ seen as a temporary solution waiting for efficient Java at the horizon 2003.
  • ROOT : automatic I/O based on dictionary, automatic schema evolution.

Problems with commercial systems

  • Licensing

  • Deployment

  • Vendor is late to follow with compilers & OS

  • Difficult to request new functionality

  • Difficult to get good people to do support and maintenance. Programmers want to develop code.

Data Analysis Software

  • 1960: Do it yourself

  • 1968: SUMX

    • Histograms and data blocks described in input file. SUMX is the master.
  • 1973: HBOOK

    • Histogram library. User controls the event loop and the selection.
  • 1985: PAW

    • Interactive histograms/fitting. Ntuples
  • 1995: ROOT

    • Same as PAW + persistency for C++ objects. C++ interpreter
  • 2005: PROOF and GRID

    • Distributed analysis: client->Master->Workers (parallelism)



Graphics & GUI evolution

  • Plotters (eg GD3): Calcomp

  • GKS times: screen is the memory


  • X11, GL: the winners

  • From graphics attributes set in sequence to Objects

    • With PAW:
      • set color red
      • Now all primitives are red
    • With ROOT: attribute values do not depend on the order they are set => easier to write a graphics editor
  • From Callbacks Messages->Signal&Slots

  • Signal&Slots require an interpreter (see Qt and Root)

  • Scriptable GUIs (a MUST)

Graphics and GUI systems

  • Calcomp plotters 1955

  • First graphics packages (CERN GD3 1970)

  • HPLOT 1975……

  • HPLOT -> HIGZ  GD3 1978

  • HPLOT -> HIGZ US Core system (fnal) 1981

  • HPLOT -> HIGZ GKS 1983

  • HPLOT -> HIGZ  PHIGS 1985

  • HPLOT -> HIGZ  X11 1985

  • PAW ->VT100, GKS, 1985

  • PAW ->MOTIF 1991

  • ROOT -> X11 1995

  • ROOT ->Win32 1996, 2002

  • ROOT ->Qt 2002, 2006

  • ROOT -> GL 2002

Graphics and GUI systems (cont)

  • Most graphics/GUI systems that we have used have been based on International standards or de facto standards.

  • All these systems had a limited life time

    • The CORE system : 5 years
    • GKS : 10 years
    • PHIGS : < 10 years
    • X11 : > 20 years
    • MOTIF : < 8 years
    • Qt : ??
  • So far, no applications built directly on top of these systems were portable to the next generation. A new generation every 8, 10 years

ROOT GUI/graphics interoperability

ROOT GUI/graphics interoperability

Interpreters & dictionaries

Interpreter & Compiler integration

Possible Progress with Interpreters

  • Eliminate the stub interface to call C/C++ functions.

    • This is already possible in CINT with C libraries.
    • It will be possible with C++ when a standard ABI will be available, otherwise compiler&linker dependent.
  • If compiler is fast enough (eg C), use the interpreter only for organizing the top level.

  • If next C++ provides introspection, one could eliminate

    • the header files parser
    • 95 per cent of the dictionary structure in memory
  • A good argument to have the interpreted and compiled code being in the same language!

  • But WHEN ???????

Object Persistency with ROOT Object Persistency with Objectivity

Object Persistency

  • Object Persistency has been a long snake for 10 years or more.

  • Today general agreement to exploit HEP feature of having mainly read-only files and use RDBMS systems only where concurrent write access is required.

  • A lot of work spent in ROOT to understand and design an efficient object streaming system (object-wise and member-wise).

  • I/O system and query system must know each other.

OODBMS (ie Objectivity)

  • Hope:

    • Address one single object in a petabyte data base
    • Resolve all the object catalog issues
  • Reality:

    • Licensing/installation/portability problems
    • 64 bits OID did not scale above 10 terabytes
    • Request for 128 bits OID never implemented
    • Locking problems when many users in read mode.
    • Central DB mismatch with GRID
    • No automatic schema evolution (big problem)
    • No interactivity

OODBMS (ie Objectivity) (2)

  • The OODBMS evangelists (and later RDBMS) passed many wrong messages

    • Commercial data bases will save manpower
    • Commercial data bases can be used for all type of data
    • Performance is OK
  • Reality:

    • Probably more than 100 personyears invested in this exercise
    • To be compared to a few man years for ROOT I/O
    • Performance was not adequate (already spotted by ROOT/Objy comparisons early 1996)
    • Physics analysis requirements were totally ignored
    • Too much weight given to “experts in bookkeeping”

ROOT I/O principles

  • Two main I/O solutions

    • Unix-like file/directory structure with keyed objects
      • OK for histograms, geometries, mag field
    • Special Event data oriented Trees
      • With object streaming and splitting modes
      • Optimized for data analysis
  • Design targeting performance and minimum file size

  • Support for network files

  • Exploit advantages of read-only files as much as possible

  • Interface with RDBMS when locking required

ROOT Trees

Data Analysis on the GRID(s) see Fons talk

Some observations

Experience with C++

  • Very powerful but complex language.

  • Easy to make a complex system with a lot of class dependencies. Changing one class forces a recompilation of many other classes.

  • No garbage collector. Only one heap.

  • ABI(Application Binary Interface) is not yet standardized: a mess on Linux/gcc (C is OK)

  • No introspection: -> develop yours.

  • Too much coupling between data and code.

  • Templates defined statically at compilation time, ie difficult to use in an interactive environment.

  • Slow compilation if abuse of templates and STL

Missing features in C++

  • Introspection

  • Not possible to compile a class from a dictionary

  • Multi-heap (like Zebra divisions)

    • Would require a garbage collector and a Handle type like in C++/CLI from MS
  • Possibility to add one or more functions without recompiling the class, although this can be easily done in C.

  • Dynamic creation of templated types

Introspection systems

  • Meta information describing all types and functions.

  • Not necessary for languages like f77 having only basic types. I/O in f77 implemented via simple switch statements.

  • Vital for languages supporting derived types for automatic I/O, inspectors, browsers and interpreters.

  • CINT, Java, cint/root/reflex

Why not Java or Python

  • Java strong candidate in 1996->2000

  • Why experiments moved to C++?

    • Speed, Geant4, ROOT ?

Main software problems seen by large experiments

  • Move to C++ completed (well nearly!)

  • Complex experiment framework

  • Too many dependencies

  • Difficult to install (SCRAM, CMT)

  • Installation time far too long

  • The wheel is reinvented many times

  • Several unwanted features (eg Atlas Storegate)

  • Coding conventions not followed

    • A code checker is essential
  • Non documented classes and modules

A considerable amount of time is spent in installing software (up to one day for an expert).

  • A considerable amount of time is spent in installing software (up to one day for an expert).

  • Porting to a new platform is non trivial.

  • Dependency problems in case many packages must be installed.

  • Only a small subset of the software is used.

  • The installation may require a huge amount of disk space. Users are scared to download a new version.

  • This is not fitting well with the GRID concept.

  • The GRID should be used to simplify this process and not to make it more complex.


  • The fact that only a very small fraction of the total code base is used has important consequences.

  • We must turn this apparent problem into a great feature.

  • BOOT: a proposal to solve this problem.

    • see my talk at CHEP.

Spare slides

Tree Friends

File types & Access in 5.06

Typical trends with Experiments frameworks

  • A few gurus design the framework

  • In general adequate for batch processing (simulation and reconstruction).

  • But too complex for the majority of users.

  • Users find simpler individual solutions.

  • Many users work in several experiments and want to use common software.

  • Fights between groups.

  • New management structure put in place.

Experiment Frameworks Starting point

Experiment Frameworks End point

398742 PDF fortran=398729,ansic=13

  • 398742 PDF fortran=398729,ansic=13

  • 146414 PYTHIA6 fortran=140748,cpp=5413,ansic=153,pascal=100

  • 128337 HLT cpp=127601,ansic=605,sh=100,csh=31

  • 128103 ITS cpp=128010,sh=93

  • 105763 MUON cpp=105673,sh=90

  • 94548 DPMJET fortran=94267,cpp=281

  • 72400 STEER cpp=72400

  • 52443 HBTAN cpp=51260,fortran=1183

  • 51489 TPC cpp=51479,sh=10

  • 50932 PHOS cpp=50639,csh=293

  • 46176 TRD cpp=46176

  • 41998 ISAJET fortran=40483,cpp=1494,pascal=21

  • 39407 RALICE cpp=29764,ansic=9355,sh=288

  • 35916 EMCAL cpp=35410,fortran=383,csh=123

  • 31820 ANALYSIS cpp=31820

  • 27751 HERWIG fortran=27246,cpp=477,ansic=28

  • 27025 FMD cpp=27021,sh=4

  • 26667 TOF cpp=26667

  • 24258 EVGEN cpp=24258

  • 21588 HIJING fortran=21099,cpp=489

  • 20562 JETAN cpp=19687,fortran=875

  • 18344 RAW cpp=18344

  • 15232 STRUCT cpp=15232

  • 13142 PMD cpp=13142

  • 12945 RICH cpp=12945

  • 10966 FASTSIM cpp=10966

  • 10944 MONITOR cpp=10944

  • 10659 ZDC cpp=10659

Assumes BOOT already installed on your machine user@xxx.yyy.zzz

  • Assumes BOOT already installed on your machine user@xxx.yyy.zzz

  • Nothing else on the machine except the compiler (no ROOT, etc)

  • Import a ROOT file containing histograms, Trees and other classes (usecase1.root)

  • Browse contents of file

  • Draw an histogram

Yüklə 470 b.

Dostları ilə paylaş:

Verilənlər bazası müəlliflik hüququ ilə müdafiə olunur © 2022
rəhbərliyinə müraciət

    Ana səhifə