Years of Computing in hep international Workshop on Large Scale Computing

20-30 years of Computing in HEP

• International Workshop on Large Scale Computing

• Kolkata

• 8 February 2006

• René Brun

• CERN

Hardware & Software Evolution

Punched cards

Mainframes, workstations,..

, OS, Desktops & Laptops

The 3 technology laws

• Moore's Law: formulated by Gordon Moore of Intel in the early 70's - the processing power of a microchip doubles every 18 months; corollary, computers become faster and the price of a given level of computing power halves every 18 months. (well ! Not true anymore, see later)

• Gilder's Law: proposed by George Gilder, prolific author and prophet of the new technology age - the total bandwidth of communication systems triples every twelve months. New developments seem to confirm that bandwidth availability will continue to expand at a rate that supports Gilder's Law.

• Metcalfe's Law: attributed to Robert Metcalfe, originator of Ethernet and founder of 3COM: the value of a network is proportional to the square of the number of nodes; so, as a network grows, the value of being connected to it grows exponentially, while the cost per user remains the same or even reduces.

• But no laws about Software (well ! Murphy’s law)

Hardware & Compilers

Multi Core CPUs

Program Size (lines of code)

Program Size (RAM)

Time to compile

Files, Classes

Languages

Fortran to C++

From static modules to plug-ins

• app.exe = (main.o) 1955

• app.exe = (main.o, x.o, y.o) 1965

• app.exe = (main.o, x.o, lib1.a, lib2.a) 1975

• app.exe = (main.o, x.o, lib1.a, lib2.so, lib3.so) 1985

• app.exe = (main.o, libs.so) + dyn libs.so 1995

• app.exe = (main.o,libs.so) + plug-in manager 2005

• BOOT + URLs + local caches (interp + comp) 2015 ??

• See my talk at CHEP

Current ROOT structure & libs

Compiled + interpreted code

• 1980: zcedex, mini command interpreter

• 1985: kuip/paw, command and macro interp

• 1986: Tk/Tcl includes a GUI

• 1984: comis, Fortran77 interpreter

• 1994: cint, a C & C++ interpreter

• 1998: python, OO on top of C++, Java

• 2002: ruby, better than python?

• 200x: BOOT (inter->code generation->compiler)

Basic types and modules

• 1950: basic operators (trig functions part of the application)

• 1953: trig functions in a library

• 1954: fortran types (integer, real, hollerith). Subroutines communication only via arguments.

• 1965: subroutines communicate via a blank common, then labeled common blocks.

• 1975: communication via a data structure management system

• 1980: derived types

• 1988: Object-Oriented programming: classes

• 1995: parametrized types, templates, STL

• 1996: Reflexion/RTTI (Java)

Programing models

• Procedural sequential

• Parallelism (MPI)

• Vectorisation

• Shared memory

• Multi-threading

• Client-server

• Statefull
• Stateless ->web

• Corba

• Distributed parallel computing (asynchronous)

• Messages. Signal/slots

Problems with Fortran

• Abuse of common blocks.

• Unmanageable in large programs

• No data structures

• No generic machine independent I/O

• Systems like Hydra(1974), Zbook(1975),Bos(1977),Zebra(1983) designed to overcome these problems.

The Zebra system (1983)

• Zebra = Zbook + Hydra

• Main data structure management system used by PAW and Geant3 and also many collaborations.

• Powerful machine independent I/O

• FZ: sequential

• RZ: direct access (PAW ntuples)

• Nice Data structure documentation system, including an interactive browser DZDOC.

Zebra bank descriptor

Zebra DZDOC

Atlas DZDOC

Zebra pros/cons

• Programming style archaic

• Easy to overwrite data structures

• Shared global store(s)

• Shared global store(s)

• Self-describing structures

• Concept of multi-heap (constants, histograms, event,..)

• Efficient garbage collection (division wipe)

• Built-in efficient and machine independent I/O

• Used by Geant3,PAW and many experiments

Geant 1,2,3,……..4

• Geant1 1974

• 2000 lines of Fortran 4
• No physics, no geometry, only a bare framework

• Geant2 1975

• 20000 lines of Fortran 4
• Some physics for multiple scattering, energy loss, decays, framework for geometry and tracking

• Geant3 1980,81 1994 ------2006?

• About 120000 lines of Fortran77 + zebra + paw
• Electromagnetic physics
• 4 hadronic packages (Tatina, Gheisha, Fluka, Calor)
• Powerful geometry package including graphics
• Hits/Digits framework
• I/O subsystem (zebra) for all structures including geometry.
• Used by many experiments. Still a reference!!!

Fluka  Fluka

• Originally developed by safety protection group at CERN (stevenson) + aarnio + ranft) 1985 ?

• Reengineered by A.Ferrari &co: Rubbia project 1990

• Probably the best for Physics processes

• Simple geometry

• The reference for radiation/shielding

• Written in fortran77

• Interfaced with VMC (TFluka) and G4 (Flugg)

Geant4

• Started in 1994

• Originally a flagship project for the move to C++

• A huge investment in manpower

• About 600000 lines of C++

• Validation process in Atlas, CMS and LHCb

• Physics processes getting better and better

• But still many limitations

• Poor interpreter (small subset callable from python)
• No I/O interface (geometry cannot yet be made persistent)
• Batch style graphics

The Virtual MC (1998)

Virtual Monte Carlo and ROOT Geometry

• The ROOT geometry package (TGeo) can be used in detector simulation, reconstruction, graphics, etc.

• TGeant3

• Used in production – native GEANT3
• New: TGeant3TGeo – interface to G3 using TGeo geometry
• No modification required in the user code
• Validated by Alice
• Same speed or faster than TGeant3

• TGeant4

• Used for Geant4 physics validation – G4 native geometry built after g3tog4 conversion
• No interface yet between G4 and ROOT geometry
• But Andrei Gheata actively working on it (expected this spring)

• TFluka

• Old geometry interface using G4 geometry vis FLUGG
• Currently a fully validated geometry interface based on TGeo
• Validated by the Fluka team
• At least 2 times slower than TGeant3

• The VMC framework is currently used by Alice, Opera, Minos, NA48b,Hades, CBM and may be STAR.

PAW: a long saga

• First version (Jan 1985) by a committee

• Must use GKS
• GUI based on VT100 functionality
• No ntuples

• June 1985: developers “abolish” the committee

• Higz: GKS + X11

• Row wise ntuples, then ColumnWiseNtuples (1986)

• Frozen in 1994, but still maintained by ROOT team

Crisis: 1992 1999

Why not F90 after F77?

• In 1989,90,91 assumption was F90

• Some work invested in I/O with F90 (to support derived types). We could not solve this problem, because no formal way to parse the F90 module descriptors.

• In 1992 many forces pushing towards OO

• CHEP Annecy, web, Next

• Crisis in Dec 1992 (at least in IT software group)

• 1/3 in favour of f90
• 1/3 in favour of commercial solutions
• 1/3 in favour of C++

1993,1994,1995

• ZOO, NextPaw, Geant3.5 proposals rejected

• ZOO: Zebra in the OO world
• NextPaw: Paw evolution ->C->C++
• Geant3.5: Implement geometry package in C++

• Geant4 proposal (June 1994)

• RD45/Objectivity project (fall 1994)

• ROOT project starts (in NA49) (Jan 1995)

1996

• ROOT chooses the CINT interpreter

• We had been attracted by Java (Object base class, many common ideas).

• Work on object persistency based on the dictionary information (introspection).

• Design of ROOT Trees (split mode). Comparison with Objectivity

• LHC++ project starts (against ROOT)

• see Jamie’s talk

1997->2000

• Getting experience with OO (professional developers).

• Most users lost in f77->C++

• First signs of problems with Objectivity in Babar

• FNAL RUN II chooses ROOT

• But C++ seen as a temporary solution waiting for efficient Java at the horizon 2003.

• ROOT : automatic I/O based on dictionary, automatic schema evolution.

Problems with commercial systems

• Licensing

• Deployment

• Vendor is late to follow with compilers & OS

• Difficult to request new functionality

• Difficult to get good people to do support and maintenance. Programmers want to develop code.

Data Analysis Software

• 1960: Do it yourself

• 1968: SUMX

• Histograms and data blocks described in input file. SUMX is the master.

• 1973: HBOOK

• Histogram library. User controls the event loop and the selection.

• 1985: PAW

• Interactive histograms/fitting. Ntuples

• 1995: ROOT

• Same as PAW + persistency for C++ objects. C++ interpreter

• 2005: PROOF and GRID

• Distributed analysis: client->Master->Workers (parallelism)

PAW

ROOT

Graphics & GUI evolution

• Plotters (eg GD3): Calcomp

• GKS times: screen is the memory

• PHIGS

• X11, GL: the winners

• From graphics attributes set in sequence to Objects

• With PAW:
• set color red
• Now all primitives are red
• With ROOT: attribute values do not depend on the order they are set => easier to write a graphics editor

• From Callbacks Messages->Signal&Slots

• Signal&Slots require an interpreter (see Qt and Root)

• Scriptable GUIs (a MUST)

Graphics and GUI systems

• Calcomp plotters 1955

• First graphics packages (CERN GD3 1970)

• HPLOT 1975……

• HPLOT -> HIGZ  GD3 1978

• HPLOT -> HIGZ US Core system (fnal) 1981

• HPLOT -> HIGZ GKS 1983

• HPLOT -> HIGZ  PHIGS 1985

• HPLOT -> HIGZ  X11 1985

• PAW ->VT100, GKS, 1985

• PAW ->MOTIF 1991

• ROOT -> X11 1995

• ROOT ->Win32 1996, 2002

• ROOT ->Qt 2002, 2006

• ROOT -> GL 2002

Graphics and GUI systems (cont)

• Most graphics/GUI systems that we have used have been based on International standards or de facto standards.

• All these systems had a limited life time

• The CORE system : 5 years
• GKS : 10 years
• PHIGS : < 10 years
• X11 : > 20 years
• MOTIF : < 8 years
• Qt : ??

• So far, no applications built directly on top of these systems were portable to the next generation. A new generation every 8, 10 years

ROOT GUI/graphics interoperability

ROOT GUI/graphics interoperability

Interpreters & dictionaries

Interpreter & Compiler integration

Possible Progress with Interpreters

• Eliminate the stub interface to call C/C++ functions.

• This is already possible in CINT with C libraries.
• It will be possible with C++ when a standard ABI will be available, otherwise compiler&linker dependent.

• If compiler is fast enough (eg C), use the interpreter only for organizing the top level.

• If next C++ provides introspection, one could eliminate

• the header files parser
• 95 per cent of the dictionary structure in memory

• A good argument to have the interpreted and compiled code being in the same language!

• But WHEN ???????

Object Persistency with ROOT Object Persistency with Objectivity

Object Persistency

• Object Persistency has been a long snake for 10 years or more.

• Today general agreement to exploit HEP feature of having mainly read-only files and use RDBMS systems only where concurrent write access is required.

• A lot of work spent in ROOT to understand and design an efficient object streaming system (object-wise and member-wise).

• I/O system and query system must know each other.

OODBMS (ie Objectivity)

• Hope:

• Address one single object in a petabyte data base
• Resolve all the object catalog issues

• Reality:

• Licensing/installation/portability problems
• 64 bits OID did not scale above 10 terabytes
• Request for 128 bits OID never implemented
• Locking problems when many users in read mode.
• Central DB mismatch with GRID
• No automatic schema evolution (big problem)
• No interactivity

OODBMS (ie Objectivity) (2)

• The OODBMS evangelists (and later RDBMS) passed many wrong messages

• Commercial data bases will save manpower
• Commercial data bases can be used for all type of data
• Performance is OK

• Reality:

• Probably more than 100 personyears invested in this exercise
• To be compared to a few man years for ROOT I/O
• Performance was not adequate (already spotted by ROOT/Objy comparisons early 1996)
• Physics analysis requirements were totally ignored
• Too much weight given to “experts in bookkeeping”

ROOT I/O principles

• Two main I/O solutions

• Unix-like file/directory structure with keyed objects
• OK for histograms, geometries, mag field
• Special Event data oriented Trees
• With object streaming and splitting modes
• Optimized for data analysis

• Design targeting performance and minimum file size

• Support for network files

• Exploit advantages of read-only files as much as possible

• Interface with RDBMS when locking required

ROOT Trees

Data Analysis on the GRID(s) see Fons talk

Some observations

Experience with C++

• Very powerful but complex language.

• Easy to make a complex system with a lot of class dependencies. Changing one class forces a recompilation of many other classes.

• No garbage collector. Only one heap.

• ABI(Application Binary Interface) is not yet standardized: a mess on Linux/gcc (C is OK)

• No introspection: -> develop yours.

• Too much coupling between data and code.

• Templates defined statically at compilation time, ie difficult to use in an interactive environment.

• Slow compilation if abuse of templates and STL

Missing features in C++

• Introspection

• Not possible to compile a class from a dictionary

• Multi-heap (like Zebra divisions)

• Would require a garbage collector and a Handle type like in C++/CLI from MS

• Possibility to add one or more functions without recompiling the class, although this can be easily done in C.

• Dynamic creation of templated types

Introspection systems

• Meta information describing all types and functions.

• Not necessary for languages like f77 having only basic types. I/O in f77 implemented via simple switch statements.

• Vital for languages supporting derived types for automatic I/O, inspectors, browsers and interpreters.

• CINT, Java, cint/root/reflex

Why not Java or Python

• Java strong candidate in 1996->2000

• Why experiments moved to C++?

• Speed, Geant4, ROOT ?

Main software problems seen by large experiments

• Move to C++ completed (well nearly!)

• Complex experiment framework

• Too many dependencies

• Difficult to install (SCRAM, CMT)

• Installation time far too long

• The wheel is reinvented many times

• Several unwanted features (eg Atlas Storegate)

• Coding conventions not followed

• A code checker is essential

• Non documented classes and modules

A considerable amount of time is spent in installing software (up to one day for an expert).

• A considerable amount of time is spent in installing software (up to one day for an expert).

• Porting to a new platform is non trivial.

• Dependency problems in case many packages must be installed.

• Only a small subset of the software is used.

• The installation may require a huge amount of disk space. Users are scared to download a new version.

• This is not fitting well with the GRID concept.

• The GRID should be used to simplify this process and not to make it more complex.

Consequences

• The fact that only a very small fraction of the total code base is used has important consequences.

• We must turn this apparent problem into a great feature.

• BOOT: a proposal to solve this problem.

• see my talk at CHEP.

Spare slides

Tree Friends

File types & Access in 5.06

Typical trends with Experiments frameworks

• A few gurus design the framework

• In general adequate for batch processing (simulation and reconstruction).

• But too complex for the majority of users.

• Users find simpler individual solutions.

• Many users work in several experiments and want to use common software.

• Fights between groups.

• New management structure put in place.

Experiment Frameworks Starting point

Experiment Frameworks End point

398742 PDF fortran=398729,ansic=13

• 398742 PDF fortran=398729,ansic=13

• 146414 PYTHIA6 fortran=140748,cpp=5413,ansic=153,pascal=100

• 128337 HLT cpp=127601,ansic=605,sh=100,csh=31

• 128103 ITS cpp=128010,sh=93

• 105763 MUON cpp=105673,sh=90

• 94548 DPMJET fortran=94267,cpp=281

• 72400 STEER cpp=72400

• 52443 HBTAN cpp=51260,fortran=1183

• 51489 TPC cpp=51479,sh=10

• 50932 PHOS cpp=50639,csh=293

• 46176 TRD cpp=46176

• 41998 ISAJET fortran=40483,cpp=1494,pascal=21

• 39407 RALICE cpp=29764,ansic=9355,sh=288

• 35916 EMCAL cpp=35410,fortran=383,csh=123

• 31820 ANALYSIS cpp=31820

• 27751 HERWIG fortran=27246,cpp=477,ansic=28

• 27025 FMD cpp=27021,sh=4

• 26667 TOF cpp=26667

• 24258 EVGEN cpp=24258

• 21588 HIJING fortran=21099,cpp=489

• 20562 JETAN cpp=19687,fortran=875

• 18344 RAW cpp=18344

• 15232 STRUCT cpp=15232

• 13142 PMD cpp=13142

• 12945 RICH cpp=12945

• 10966 FASTSIM cpp=10966

• 10944 MONITOR cpp=10944

• 10659 ZDC cpp=10659

Assumes BOOT already installed on your machine [e-mail]

• Assumes BOOT already installed on your machine [e-mail]

• Nothing else on the machine except the compiler (no ROOT, etc)

• Import a ROOT file containing histograms, Trees and other classes (usecase1.root)

• Browse contents of file

• Draw an histogram