ROOT I/O TTree Queries CHEP 2004 René Brun / CERN Philippe Canal / Fermilab Fons Rademakers / CERN

ROOT I/OROOT I/OTTree QueriesTTree Queries

CHEP 2004CHEP 2004

René Brun / CERN René Brun / CERN Philippe Canal / Fermilab Philippe Canal / Fermilab Fons Rademakers / CERNFons Rademakers / CERN

http://root.cern.ch

September 29, 2004September 29, 2004 Conference for Computing in High Energy and NConference for Computing in High Energy and Nuclear Physicsuclear Physics

22

ContentsContents StatusStatus

OverviewOverview List of other presentationsList of other presentations

ROOT I/OROOT I/O Large FilesLarge Files Double32_tDouble32_t Foreign objectsForeign objects New interfacesNew interfaces XML back-endXML back-end Historical recap.Historical recap.

Containers Support Containers Support Mainly for STL containersMainly for STL containers SplittingSplitting TTree QueryTTree Query

TTreeTTree Auto load of TRef’ed branchesAuto load of TRef’ed branches UserInfoUserInfo CloneTreeCloneTree

TTree QueryTTree Query Calling free standing functionsCalling free standing functions RebinningRebinning Support for Support for IndexedIndexed Friends Friends Arbitrary C++ in queries Arbitrary C++ in queries

(TTree::MakeProxy)(TTree::MakeProxy)

Support for SQL back-endSupport for SQL back-end

Future PlansFuture Plans


33

Presentations and PostersPresentations and Posters [328] [328] The Next Generation Root File ServerThe Next Generation Root File Server

by Andrew ANUSHEVSKY by Andrew ANUSHEVSKY (Theatersaal: Sept 27,16:30 - 16:50)(Theatersaal: Sept 27,16:30 - 16:50)

[412] [412] XML I/O in ROOTXML I/O in ROOTby Sergey LINEV by Sergey LINEV (Brunig 1 + 2: Sept 29, 15:20 - 15:40) (Brunig 1 + 2: Sept 29, 15:20 - 15:40)

[430] [430] Global Distributed Parallel Analysis using PROGlobal Distributed Parallel Analysis using PROOF and AliEnOF and AliEn

by Fons RADEMAKERS by Fons RADEMAKERS (Theatersaal: Sept 29, 15:20 - 15:40) (Theatersaal: Sept 29, 15:20 - 15:40)

[104] [104] Authentication/Security services in the ROOT fAuthentication/Security services in the ROOT frameworkramework

by Gerardo GANIS by Gerardo GANIS (Brunig 3: Sept 29, 16:50 - 17:10)(Brunig 3: Sept 29, 16:50 - 17:10)

[169] [169] Guidelines for Developing a Good GUIGuidelines for Developing a Good GUIby Ilka ANTCHEVA by Ilka ANTCHEVA (Brunig 1+2: Sept 30,14:00 - 14:20) (Brunig 1+2: Sept 30,14:00 - 14:20)

[287] [287] Super scaling PROOF to very large clustersSuper scaling PROOF to very large clustersby Maarten BALLINTIJN by Maarten BALLINTIJN (Ballsaal: Sept 30,15:00 - 15:20) (Ballsaal: Sept 30,15:00 - 15:20)

Poster on September 29Poster on September 29

[128] [128] XTNetFile, a fault tolerant extension of ROOXTNetFile, a fault tolerant extension of ROOT TNetFile clientT TNetFile client

Poster on September 30Poster on September 30

[298] The ROOT 3-D graphics and [298] The ROOT 3-D graphics and geometry classesgeometry classes

[170] The User Interface Design in ROOT [170] The User Interface Design in ROOT [303] The ROOT Linear Algebra Package [303] The ROOT Linear Algebra Package [98] RDBC: ROOT DataBase Connectivity [98] RDBC: ROOT DataBase Connectivity [99] Interactive Data Analysis with Carrot [99] Interactive Data Analysis with Carrot

(ROOT Apache Module) (ROOT Apache Module)

http://indico.cern.ch/contributionDisplay.py?contribId=328&sessionId=7&confId=0











44

StatusStatus ROOT 4.01/02 just releasedROOT 4.01/02 just released

Production Release of 4.01 Production Release of 4.01 planned for December 2004planned for December 2004

Many improvements since Many improvements since CHEP2003CHEP2003

This talks: This talks: I/O and TTree queriesI/O and TTree queries

For other developments, see For other developments, see the other ROOT related talksthe other ROOT related talks

XROOTDXROOTD A new generation ROOT file A new generation ROOT file

serverserver Authentication Overhaul Authentication Overhaul Object Property Editor Object Property Editor

e.g.. TH1Editor, TH2Editor, e.g.. TH1Editor, TH2Editor, TGraphEditorTGraphEditor

New classes for GUINew classes for GUI GUI builderGUI builder Brand new GL viewerBrand new GL viewer Math and Stats Math and Stats

New Matrix package New Matrix package ImplementationImplementation

New functions in TMath (Now New functions in TMath (Now a namespace)a namespace)

Quadratic programmingQuadratic programming


55

TFile and TDirectoryTFile and TDirectory Very Large FilesVery Large Files

Support on all platforms for 64 bits integers via the portable typedefs Support on all platforms for 64 bits integers via the portable typedefs Long64_tLong64_t and and ULong64_tULong64_t. .

• Long long on Unix, _int64 with VC++Long long on Unix, _int64 with VC++ Support for File larger than 2Gb added in ROOT 4.00Support for File larger than 2Gb added in ROOT 4.00

• File smaller than 2Gb still readable by older version of ROOTFile smaller than 2Gb still readable by older version of ROOT Support for TTree with more than 2**31 entriesSupport for TTree with more than 2**31 entries

Double32_tDouble32_t Same as Same as Double_tDouble_t in memory in memory Same as Same as Float_tFloat_t on disk on disk Support automatic schema evolution to and from float and doubleSupport automatic schema evolution to and from float and double Warning: too many read/write cycle could result in some loss of Warning: too many read/write cycle could result in some loss of

precisionprecision


66

XML output formatXML output format

Update to the I/O classes to allow Update to the I/O classes to allow the customization of the backend.the customization of the backend.

Implemented for XMLImplemented for XML Will be used for SQL support.Will be used for SQL support.

XML files allow the interchange of XML files allow the interchange of data with applications unable to data with applications unable to read ROOT file directlyread ROOT file directly

Example:Example:

Refer to Sergey Linev’s Refer to Sergey Linev’s presentation for more detailspresentation for more details

Extract from c.xml:Extract from c.xml:

TCanvas c; h.Draw(); c.SaveAs("c.xml"); c.SaveAs("c.root");

<XmlKey name="c1" cycle="1"><Object class="TCanvas"><Version v="5"/>

<TPad version="8"><TVirtualPad version="2">

<TObject fUniqueID="0" fBits="3000008"/>

<TAttLine version="1"> <fLineColor v="1"/><fLineStyle v="1"/><fLineWidth v="1"/>

</TAttLine><TAttFill version="1">

<fFillColor v="19"/><fFillStyle v="1001"/>

</TAttFill>


77

ROOT I/O HistoryROOT I/O History Version 2.25 and olderVersion 2.25 and older

Only hand coded and generated streamer function, Schema evolution done by handOnly hand coded and generated streamer function, Schema evolution done by hand I/O requires : ClassDef, ClassImp and CINT DictionaryI/O requires : ClassDef, ClassImp and CINT Dictionary

Version 2.26Version 2.26 Automatic schema evolutionAutomatic schema evolution Use TStreamerInfo (with info from dictionary) to drive a general I/O routine.Use TStreamerInfo (with info from dictionary) to drive a general I/O routine.

Version 3.03/05Version 3.03/05 Lift need for ClassDef and ClassImp for classes not inheriting from TObjectLift need for ClassDef and ClassImp for classes not inheriting from TObject Any non TObject class can be saved inside a TTree or as part of a TObject-classAny non TObject class can be saved inside a TTree or as part of a TObject-class

Version 4.00/00Version 4.00/00 Automatic versioning of ‘Foreign’ classesAutomatic versioning of ‘Foreign’ classes

Version 4.00/08Version 4.00/08 Non TObject classes can be saved directly in TDirectoryNon TObject classes can be saved directly in TDirectory

Version 4.01/02Version 4.01/02 Large TTrees, TRef autoloadLarge TTrees, TRef autoload


88

....Bytecount(4 bytes)

0(2 bytes)checksum(4 bytes)ObjectNBytecount

0checksum

objectN+1....

Foreign ObjectsForeign Objects To save non instrumented classes:To save non instrumented classes:

Need just the data dictionaryNeed just the data dictionary Default versioning provided by a Default versioning provided by a Checksum Checksum based based

on the type and name of the persistent data memberson the type and name of the persistent data members ChecksumChecksum stored as an additional 4 bytes stored as an additional 4 bytes

ClassDef advantagesClassDef advantages The The IsAIsA function generated by function generated by ClassDefClassDef speeds up speeds up

considerably the access to the TClass for a given object.considerably the access to the TClass for a given object. The version number (2 bytes maximum) consumes less The version number (2 bytes maximum) consumes less

space on disk than the “0+checksum”space on disk than the “0+checksum”

New interface to store and retrieve object with Type SafetyNew interface to store and retrieve object with Type Safety

ptrclass *ptr = …; directory->WriteObject(ptr,"name");ptrclass *ptr; directory->GetObject("name",ptr);

0 if object absent or 0 if object absent or of wrong typeof wrong type

TBuffer


99

TClonesArrayTClonesArray

Optimization of the number of calls to new and deletesOptimization of the number of calls to new and deletes

Ability to split the collection of objects in a TTreeAbility to split the collection of objects in a TTree Improve compression and run-timeImprove compression and run-time

Ability to save object member-wiseAbility to save object member-wise Store the same data member of all the elements of the Store the same data member of all the elements of the

collections consecutivelycollections consecutively Improve compression (buffer data more homogeneous)Improve compression (buffer data more homogeneous) Improve run-time (avoid n-1 tests of the data type)Improve run-time (avoid n-1 tests of the data type)

Ability to use in TTree::Draw as a collectionAbility to use in TTree::Draw as a collection

Ability to read back without the original compiled codeAbility to read back without the original compiled code


1010

Old STL Container SupportOld STL Container SupportFor versions older than 4.00/00For versions older than 4.00/00

Collection always stored object wiseCollection always stored object wise

Nesting of STL collections was Nesting of STL collections was extremely limitedextremely limited

No splitting was possibleNo splitting was possible

STL containers stored using a STL containers stored using a generated function generated function

One generated function per actual One generated function per actual data member.data member.

Compiled version of these functions Compiled version of these functions requiredrequired for for writingwriting and also for and also for readingreading

void R__User_fList1(TBuffer &R__b, void *R__p, int) { if (R__b.IsReading()) { vector<THit> &fList1 = *(vector<THit> *)R__p; int R__n; fList1.clear(); R__b >> R__n; R__stl.reserve(R__n); for (int R__i = 0; R__i < R__n; R__i++) { THit R__t; R__t.Streamer(R__b); fList1.push_back(R__t); } } else { … writing … } }


1111

New Container SupportNew Container Support

New Abstract Interface:New Abstract Interface: TVirtualCollectionProxyTVirtualCollectionProxy Can be implemented for almost any collectionsCan be implemented for almost any collections

AllowsAllows Splitting (for collection of homogenous objects)Splitting (for collection of homogenous objects) Use in Tree Query (with automatic looping)Use in Tree Query (with automatic looping)

Will allowWill allow Member-wise streaming (as opposed to Object wise streaming)Member-wise streaming (as opposed to Object wise streaming)

AlsoAlso Arbitrary nesting of STL containersArbitrary nesting of STL containers Reading of STL containers without original code (Emulated mode)Reading of STL containers without original code (Emulated mode)

Note: as of 4.00/08 only std::vector has Proxies.Note: as of 4.00/08 only std::vector has Proxies.

Early Prototype and fundamental Concepts by Victor PerevoztchikovEarly Prototype and fundamental Concepts by Victor Perevoztchikov


1212

STL SupportSTL Support Each STL container instance now has an Each STL container instance now has an

associated TClass objectassociated TClass object

Several co-existing streaming Several co-existing streaming implementationsimplementations

Generated StreamerGenerated Streamer• For object-wise streamingFor object-wise streaming• Fully respect custom allocators and Fully respect custom allocators and

comparatorscomparators• Easier to implement and similar run-time cost Easier to implement and similar run-time cost

as a templated solutionsas a templated solutions

Templated Proxy Templated Proxy (e.g.. TVectorProxy)(e.g.. TVectorProxy)• For splitting and member-wise streaming Fully For splitting and member-wise streaming Fully

respect custom allocators and comparatorsrespect custom allocators and comparators

Emulation Proxy Emulation Proxy (e.g.. TEmulatedVectorProxy)(e.g.. TEmulatedVectorProxy)• For reading For reading withoutwithout a compiled version a compiled version• Allow easy sharing of ALL ROOT files that Allow easy sharing of ALL ROOT files that

have no custom streamers.have no custom streamers.

Why not rely only on the Emulation Why not rely only on the Emulation ProxyProxy

Implementation difficultiesImplementation difficulties• An emulation proxy acting on An emulation proxy acting on “live “live

STL object” STL object” requires a few tricks and requires a few tricks and assumptionsassumptions

memory footprint of the STL container memory footprint of the STL container object is (usually?) independent from object is (usually?) independent from the template parameterthe template parameter

List proxy would need a series of list List proxy would need a series of list of increasing fixed size content (aka. of increasing fixed size content (aka. list<char[1024]>, list<char[2048]>)list<char[1024]>, list<char[2048]>)

• Does not respect allocators and Does not respect allocators and comparatorcomparator

Templated proxy can be faster and Templated proxy can be faster and more memory efficient.more memory efficient.

The emulation layer might actually be The emulation layer might actually be implemented using alternative implemented using alternative collections (if we assume it does not collections (if we assume it does not have to deal with real objects)have to deal with real objects)


1313

Container I/O ImplementationContainer I/O Implementation Any container can be summarized by the sequence of its content’s Any container can be summarized by the sequence of its content’s

addressesaddresses

Use Use TVirtualCollection::AtTVirtualCollection::At viavia TVirtualCollection::operator[]TVirtualCollection::operator[]

ProsPros I/O Code completely independent of the collectionI/O Code completely independent of the collection

• Reduced code duplication in TStreamerInfoReduced code duplication in TStreamerInfo No run-time cost for TClonesArrayNo run-time cost for TClonesArray

ConsCons Implementation for containers with no random access iterator needs to Implementation for containers with no random access iterator needs to

cache the iterator.cache the iterator.

Member-wise implementationMember-wise implementation Member-wise/object-wise choice will be encoded in the ‘version number’ Member-wise/object-wise choice will be encoded in the ‘version number’

of the STL collectionsof the STL collections API will be provided to select member-wise or object-wise for data API will be provided to select member-wise or object-wise for data

member that are STL collectionsmember that are STL collections


1414

TTreeTTree TRef autoloadTRef autoload

Added (optional) support for the auto-loading of branches Added (optional) support for the auto-loading of branches referenced by a TRef object.referenced by a TRef object.

Generate one table of references to branches per entryGenerate one table of references to branches per entry TRef::GetObject uses this table to find and load the branch TRef::GetObject uses this table to find and load the branch

containing the referenced objectcontaining the referenced object To enable it call: To enable it call:

..

TTree::GetUserInfoTTree::GetUserInfo Used to store with the TTree any Used to store with the TTree any

user defined object(s) that is not user defined object(s) that is not depending on the entry numberdepending on the entry number

Examples:Examples:• Luminosity, Calibrations etc.Luminosity, Calibrations etc.

..

tree->BranchRef();

tree.GetUserInfo()->Add(myobject);

class Event { TClonesArray *fTracks; TRef fLastTrack;};

branch=tree.GetBranch("fLastTrack");branch->GetEntry(7); tlast = event->GetLastTrack();


1515

Copying a TTreeCopying a TTree Very flexible simple copying tools allowing cut on:Very flexible simple copying tools allowing cut on:

Number of entriesNumber of entries Number of branchesNumber of branches Selection of entries base on a FormulaSelection of entries base on a Formula Useable for both TTree and TChainUseable for both TTree and TChain

Important simplification of the interfaceImportant simplification of the interface Removed the requirement of explicitly setting the addresses forRemoved the requirement of explicitly setting the addresses for ALLALL the branches. the branches.

3 Branches 2 Branches

3 Branches

tree->SetBranchStatus(“br”,kFALSE”);newtree=tree->CloneTree();

tree->CopyTree(“fTracks.fPx<=1.2”);


1616

TTree QueriesTTree Queries

Implemented Boolean expression optimization (Implemented Boolean expression optimization (&&&& and and ||||))

Rebinning now possible from the TTree data (via new histogram Rebinning now possible from the TTree data (via new histogram editor)editor)

Improved TTree::Scan output (customization and array display)Improved TTree::Scan output (customization and array display)

Call to external functions:Call to external functions: Free standing function or class static member functionFree standing function or class static member function Compiled or interpreted with Numerical arguments and Numerical Compiled or interpreted with Numerical arguments and Numerical

return typereturn type Example:Example:

tree->Draw("TMath::Prob(var,5)");


1717

TTree QueriesTTree Queries Support for Collections Support for Collections

TTreeFormulaTTreeFormula now treats any now treats any collection class which has a collection class which has a TVirtualCollectionTVirtualCollection in the exact in the exact same way as a same way as a TClonesArrayTClonesArray

Automatically loops over the Automatically loops over the elementselements

Can access a specific elementCan access a specific element Synchronized Synchronized with other with other

collections and arrays in the collections and arrays in the formulasformulas

ConnectingConnecting several TTrees several TTrees TChain adds more entriesTChain adds more entries TTree Friends adds (virtually) more branchesTTree Friends adds (virtually) more branches Prior to ROOT 4.00/08 correlation between Prior to ROOT 4.00/08 correlation between

Friends made only by entry numberFriends made only by entry number• This is a problem if Trees have This is a problem if Trees have semanticallysemantically a a

different sequence of entriesdifferent sequence of entries Can now connect the Friend using an IndexCan now connect the Friend using an Index

• For example Run Number/Event NumberFor example Run Number/Event Number Use abstract interface TVirtualIndexUse abstract interface TVirtualIndex

• Concrete implementation: TTreeIndexConcrete implementation: TTreeIndex

1

1

1

2

2

2

1

2

2

1

2

1

Main TreeUser Treerun event run event

1

1

1

2

2

2

1

2

2

1

2

1

Indexed Main TreeUser Treerun event run event


1818

The MakeClass RevolutionThe MakeClass RevolutionCurrent Fast Analysis FrameworksCurrent Fast Analysis Frameworks

TTree::DrawTTree::Draw Fast histogrammingFast histogramming Load branch on DemandLoad branch on Demand Only simple expressionsOnly simple expressions

MakeCodeMakeCode C-Style C-Style ObsoleteObsolete

MakeClassMakeClass Flat representation of the treeFlat representation of the tree Difficulties with variable size arraysDifficulties with variable size arrays Branch loaded explicitlyBranch loaded explicitly

MakeSelectorMakeSelector Proof ReadyProof Ready Flat representation of the treeFlat representation of the tree Difficulties with variable size arraysDifficulties with variable size arrays Branch loaded explicitlyBranch loaded explicitly

Elegant Replacement for Elegant Replacement for MakeClass/MakeSelectorMakeClass/MakeSelector

Currently named MakeProxyCurrently named MakeProxy

Creates a C++ context where Creates a C++ context where branch names (including periods) branch names (including periods) can be used as variablecan be used as variable

On demand loading of branchesOn demand loading of branches

Respect/recreate the original class Respect/recreate the original class structurestructure

Array bound checkArray bound check

Use the user’s shared libraries Use the user’s shared libraries (when available)(when available)


1919

MakeProxy ExamplesMakeProxy Examples

TTree::Draw of a script:TTree::Draw of a script:

Implemented using MakeProxyImplemented using MakeProxy Enables complex loopingEnables complex looping Allow call to any C++ functions or member functions!Allow call to any C++ functions or member functions! Still provide on-demand loading of the branchesStill provide on-demand loading of the branches And … allow any arbitrary C++And … allow any arbitrary C++

Double_t hsimple() { int last = fTracks.GetLast(); for(int i=1; i < last-1; ++i) { htemp->Fill(fTracks.fPt[i]-fTracks.fPt[i-1]); } return fTracks.fPt[last] – fTracks.fPt[last-1];}

tree->Draw(“hsimple.C”);


2020

File types & Access in 4.01/xxFile types & Access in 4.01/xx

LocalFile

X.xml

RFIO Chirp

CastorDcacheLocalFile

X.root

http rootd/xrootd

Oracle

SapDb

PgSQL

MySQL

TFileTKey/TTree

TStreamerInfo

user

TSQLServerTSQLRow

TSQLResult

TTreeSQL


2121

New RDBMS interface: GoalsNew RDBMS interface: Goals

Access any RDBMS tables from TTree::DrawAccess any RDBMS tables from TTree::Draw

Create a Tree in split mode Create a Tree in split mode creating a RDBMS table creating a RDBMS table and filling it.and filling it.

The table can be processed by SQL directly.The table can be processed by SQL directly.

The interface uses the normal I/O engine, including The interface uses the normal I/O engine, including support for Automatic Schema Evolution.support for Automatic Schema Evolution.


2222

New RDBMS InterfaceNew RDBMS Interface

Current prototypeCurrent prototype Simple TTree (branch with leaf list)Simple TTree (branch with leaf list) Implemented via TSQLxxx for reading and writingImplemented via TSQLxxx for reading and writing Implemented via RDBC for reading Implemented via RDBC for reading

• See: See: http://carrot.cern.ch/~onuchin/RDBC/http://carrot.cern.ch/~onuchin/RDBC/ Should be released in December 2004.Should be released in December 2004.

Should be expanded to support branch of objectsShould be expanded to support branch of objects Need to implement a way to store and retrieve TStreamerInfo(s) Need to implement a way to store and retrieve TStreamerInfo(s)

and TProcessID(s) in the databaseand TProcessID(s) in the database Will probably use SQL binary ‘blob’ to store non-split objects.Will probably use SQL binary ‘blob’ to store non-split objects.


2323

RDBMS ExamplesRDBMS ExamplesTTreeSQL tree(const char *db,const char* uid,…)

tree.Print(), Browse, Scan, etc

tree.Draw(“var1:var2”,”varx <0”)

TTree query style converted to SQL

Connect to an existing db

TTreeSQL tree(“mysql://localhost/test”,”nobody”,”new”);

Event *event = new Event;

tree.Branch(“top”,”Event”,&event);

tree.Fill();

tree.AutoSave();

A TSQLRow is filled and sent to the server

Columns created using the normal split

algorithm.Blobs created below

split.

Create the data base on server


2424

Future Plans for I/O and TTreeFuture Plans for I/O and TTree

Implement member-wise storing for std::vector (late 2004)Implement member-wise storing for std::vector (late 2004)

Implement TVirtualCollectionProxy for each of the STL containers Implement TVirtualCollectionProxy for each of the STL containers (late 2004, early 2005)(late 2004, early 2005)

Add support for auto loading of TRef branches across treesAdd support for auto loading of TRef branches across trees

TChain, TTree Friends and IndexingTChain, TTree Friends and Indexing Add support for “Add support for “befriendingbefriending” TChain objects using an Indexed relation” TChain objects using an Indexed relation

TTree QueriesTTree Queries Allow following (transparently) Allow following (transparently) TRefTRef and and TRefArrayTRefArray


2525

SummarySummary TFile improvementTFile improvement

Large files and trees, Double32_t, XML output format.Large files and trees, Double32_t, XML output format. Support for non-instrumented classesSupport for non-instrumented classes

Enhancement in I/O and Tree Query for collectionEnhancement in I/O and Tree Query for collection Split CollectionsSplit Collections Fast histograming of (potentially) any collectionsFast histograming of (potentially) any collections Lift restrictions on STL I/OLift restrictions on STL I/O

• Nested containersNested containers• Reading without compiled codeReading without compiled code

TTreeTTree Remove stringent requirements on CloneTreeRemove stringent requirements on CloneTree Add support for auto loading of referenced objectsAdd support for auto loading of referenced objects Support for RDBMS databases back-end coming soon.Support for RDBMS databases back-end coming soon.

TTree QueriesTTree Queries Can call any functions taking numerical argumentsCan call any functions taking numerical arguments Can use arbitrary C++ and still use the branch names as variablesCan use arbitrary C++ and still use the branch names as variables TTree Friend linked by IndexTTree Friend linked by Index

ROOT I/O TTree Queries CHEP 2004 René Brun / CERN Philippe Canal / Fermilab Fons Rademakers / CERN

Documents

Transcript of ROOT I/O TTree Queries CHEP 2004 René Brun / CERN Philippe Canal / Fermilab Fons Rademakers / CERN