On the same page with Streaming

Adam Lyon

Analysis Tools - 6/04/2002

2A. Lyon (FNAL/D0) – 20020521

The Paper TrailNew Phenomena Comments on the D0 Run II data access

D0 Note 3313 (9/97, Bantly, Boehnlein, Hobbs for NP) Run I inclusive streaming: ~50% replication Disjoint classified streams based on Level 3 Filter bit,

grouped by L3 physics object. 2n-1 exclusive streams for n classifications

Stream min/zero-bias events separately (inclusive) for easy access to entire sample

Study of Possible Run II Streaming Scenarios D0 Note 3326 (10/97, Schellman, Bertram)

Studied exclusive, partially exclusive, and inclusive streaming scenarios with 100,000 Run I events. Exclusive streaming with L3 classification bits. Weighs pros and cons of each scenario

3A. Lyon (FNAL/D0) – 20020521

The Paper TrailRun II Streaming - How to Stream Events

D0 Note 3662 (6/99, Boehnlein, Fan, Schellman, de Souza)

Recipe for streaming in Run II

LogicalStream to Physical Stream Mapping [~schellma/upgrade_computing]

(2/2000 [10/2000], Schellman, Yu) Tree algorithm scheme for combining the “primitive” streams

into physical streams (2n-1 m; m << 2n-1) based on event probability to “load balance” the physical streams

Plain English Explanation of Streaming [~schellma/upgrade_computing]

(6/2001, Schellman) Short summary of previous work

4A. Lyon (FNAL/D0) – 20020521

The Paper TrailStreaming and Level 3

[http://hep.brown.edu/users/Greg/streaming/documentation.htm]

(8/2001, Hays) Combining Primitive Streams to Physical Streams -- two

example tree algorithms

Conceptual Design for Offline Data Analysis and D0 Streaming (8/2001, White) [Greg’s web page]

Offline streaming issues (SAM, Luminosity)

Stream Definition Table (Strawman Proposal) (8/2001, Landsberg) [Greg’s web page]

Streaming proposal we’d like to implement as a test case

5A. Lyon (FNAL/D0) – 20020521

Streaming (as documented) at a glance

From L2

L2 trigger(MU+JET)

L2 trigger(ELE+JET_HI)

L3 Filter Script

MU_HI (0001)

JT_HI (0100)

L3 Filter Script

ELE_LO (0010)

Primitives BitMap

(0101)

Physical Stream

MU_STRM

JT_BIG (0100)

L3 Filter assigned a “primitive” bit for the physics object it tests

Primitives bit map is OR of bits from filters of passing L3 filter scripts(2n-1) possibilities

Event is mapped (via some algorithm) to physical stream using BitMap

m << 2n-1 physical streams

6A. Lyon (FNAL/D0) – 20020521

How does L3 form the Primitive BitMap? Each L3 filter can set its bit when it runs and passes

Gives people heebee-jeebees (what if filter writer forgets to code this step)

Must OR BitMaps from passing filter scripts at the end

Use a lookup table (L3 filter script Primitives BitMap) Since all filters of a filter script must pass, if a filter script

passes, you know the resulting bit map a priori Download this information to L3 (in trigger database)

[Can this be made even better? See next slide…] L3 takes no time setting bits and it always works Can override settings easily Still must OR BitMaps from passing filter scripts at the end

to form final word

7A. Lyon (FNAL/D0) – 20020521

How to go from BitMap to Physical Stream

Run mapping algorithm in Level 3 (slide) Jon Hays already has this Must download algorithm configuration via XML file

(Where is this configuration stored? Trigger database? “Streaming database”?)

8A. Lyon (FNAL/D0) – 20020521

BitMaps Physical Streams[from Hays’ document]

Simple one-to-one:

2n-1 physical streams Always n decisions No ambiguity

Priority:

Flexible Can sometimes decide in

< n decisions Can produce few

streams, though must be careful that streams are not big

You have to think! What if Muon & Electron?

9A. Lyon (FNAL/D0) – 20020521

Exceptions (For now, the rule!) Special L1 triggers (with normal physics L3 filters)

Cosmic gap trigger Dean’s non-zero suppressed Cal trigger Hard-wire their streams in the XML

Triggers with L3 Prescale 1 (no other L3 filters) 50% of current triggers Hard wire stream

L3 Mark & Pass (no L3 Filter satisfied) Go to monitor stream (inclusive)

L2 Mark & Pass [not implemented yet]

10A. Lyon (FNAL/D0) – 20020521

Conclusions Exclusive Streaming Some (most) triggers

span more than one streamMinimize tapesA few groups win big, most groups still look at

almost all the dataHard to do inclusive studies (e.g. commissioning)Hard to determine luminosity, trigger ratesHard to re-reco a certain class of eventsLost data in low rate streams may be

devastating

TOOLS WILL BE CRITICAL TO KEEP USERS HAPPY!

11A. Lyon (FNAL/D0) – 20020521

Tools User wants to run over data from certain

triggersDetermine the streams to obtain

• Stream definitions can change over time• Needs Run database, trigger database• Must be able to work through mapping table

Determine files for those streams• SAM does this

Determine good runs (run over only these)Determine bad luminosity blocks (skip these

events) User wants luminosity for data Monitoring tools (trigger rates, verfication)

12A. Lyon (FNAL/D0) – 20020521

Tools