Big Fast Data in High-Energy Particle Physics

Andrew John Lowe

3 June 2015

About CERN

I CERN is the European Organization for Nuclear Research inGeneva Switzerland

I Worldwide communityI 21 members states (+ 2 incoming members)I Observers Turkey Russia Japan USA IndiaI About 2300 staff and 10000 users (about 5000 on-site)I Budget (2014) ~1000 MCHFI Birthplace of the World Wide Web

Particle physicsI Particle physics is the study of subatomic particles and the

fundamental forces that act between themI Present-day particle physics research represents manrsquos most

ambitious and organised effort to answer the question What isthe universe made of

I We found the Higgs boson (after a 40-year search) but manyquestions remain unanswered

I Our present theoretical model doesnrsquot explain gravity identity ofdark matter neutrino oscillations matterantimatter asymmetryof universe

I To find the Higgs boson and attempt to unlock some of theseother mysteries we needed to

I Build the worldrsquos most powerful particle accelerator and largestmachine ever constructed by humankind

I Construct incredibly sophisticated particle detectorsI Collect an enormous amount of data

The Large Hadron Collider (LHC)

LHC underground structure

I 27 km circumference 50ndash175m undergroundI Now running at 13TeV after a two-year break for upgrades

LHC Run 2 first physics at 13TeV starts TODAY

Machine back on at 1040 this morning

We are taking data at record energy NOW

CMS Control Room

ATLAS Control Room

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

About CERN

I CERN is the European Organization for Nuclear Research inGeneva Switzerland

I Worldwide communityI 21 members states (+ 2 incoming members)I Observers Turkey Russia Japan USA IndiaI About 2300 staff and 10000 users (about 5000 on-site)I Budget (2014) ~1000 MCHFI Birthplace of the World Wide Web

Particle physicsI Particle physics is the study of subatomic particles and the

fundamental forces that act between themI Present-day particle physics research represents manrsquos most

ambitious and organised effort to answer the question What isthe universe made of

I We found the Higgs boson (after a 40-year search) but manyquestions remain unanswered

I Our present theoretical model doesnrsquot explain gravity identity ofdark matter neutrino oscillations matterantimatter asymmetryof universe

I To find the Higgs boson and attempt to unlock some of theseother mysteries we needed to

I Build the worldrsquos most powerful particle accelerator and largestmachine ever constructed by humankind

I Construct incredibly sophisticated particle detectorsI Collect an enormous amount of data

The Large Hadron Collider (LHC)

LHC underground structure

I 27 km circumference 50ndash175m undergroundI Now running at 13TeV after a two-year break for upgrades

LHC Run 2 first physics at 13TeV starts TODAY

Machine back on at 1040 this morning

We are taking data at record energy NOW

CMS Control Room

ATLAS Control Room

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Particle physicsI Particle physics is the study of subatomic particles and the

fundamental forces that act between themI Present-day particle physics research represents manrsquos most

ambitious and organised effort to answer the question What isthe universe made of

I We found the Higgs boson (after a 40-year search) but manyquestions remain unanswered

I Our present theoretical model doesnrsquot explain gravity identity ofdark matter neutrino oscillations matterantimatter asymmetryof universe

I To find the Higgs boson and attempt to unlock some of theseother mysteries we needed to

I Build the worldrsquos most powerful particle accelerator and largestmachine ever constructed by humankind

I Construct incredibly sophisticated particle detectorsI Collect an enormous amount of data

The Large Hadron Collider (LHC)

LHC underground structure

I 27 km circumference 50ndash175m undergroundI Now running at 13TeV after a two-year break for upgrades

LHC Run 2 first physics at 13TeV starts TODAY

Machine back on at 1040 this morning

We are taking data at record energy NOW

CMS Control Room

ATLAS Control Room

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

The Large Hadron Collider (LHC)

LHC underground structure

I 27 km circumference 50ndash175m undergroundI Now running at 13TeV after a two-year break for upgrades

LHC Run 2 first physics at 13TeV starts TODAY

Machine back on at 1040 this morning

We are taking data at record energy NOW

CMS Control Room

ATLAS Control Room

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

LHC underground structure

I 27 km circumference 50ndash175m undergroundI Now running at 13TeV after a two-year break for upgrades

LHC Run 2 first physics at 13TeV starts TODAY

Machine back on at 1040 this morning

We are taking data at record energy NOW

CMS Control Room

ATLAS Control Room

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

LHC Run 2 first physics at 13TeV starts TODAY

Machine back on at 1040 this morning

We are taking data at record energy NOW

CMS Control Room

ATLAS Control Room

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Machine back on at 1040 this morning

We are taking data at record energy NOW

CMS Control Room

ATLAS Control Room

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

CMS Control Room

ATLAS Control Room

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

ATLAS Control Room

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

LHC Control Room

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

New data

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

LHC detector experiment CMS

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

LHC detector experiment ATLAS

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Example detector at the LHC the ATLAS detector

Weight 7000 tonnes bull 3000 km of cables bull 100 million electronicchannels mdash a big digital camera mdash focus on this experiment now

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Big machines and big dataI Why is the LHC so big

I Need to collide particles with enough energy to manifest newparticles which (if they exist) have masses beyond thoseaccessible with previous machines (E = mc2)

I Exploration of the energy frontierI Why are the LHC detector experiments so big

I Large decay length of some particles require decays happeninside detector volume where they can be recorded

I Why is our data so bigI How likely a given collision event occurs depends entirely on

quantum mechanics and is a property intrinsic to that specifictype of event

I However the rate depends on experimental variables (like beamintensity) that can be controlled

I Require huge data throughputI Parameters architectural decisions and technology choices are

driven by the physics

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Rates for different physics processes at the LHC(Rare processes at bottom frequent processes at top)

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Data challenges for ATLASI We want to study extremely rare processes

I For example the production rate of Higgs bosons at the LHC is10minus11 that of the total proton-proton interaction rate

I A high collision rate (and long runs to collect lots of data)increases our chances of observing rare processes

I Beams are composed of ldquotrainsrdquo of proton bunches that crossin the LHC detectors every 25 ns (rate = 40MHz)

I Bunches travelling close the speed of light rarr bunch separationis 75m rarr before yoursquove read out a single electronic channelfrom the 1st collision the 2nd pair of colliding bunches arealready in the detector with the 3rd pair about to enter

I There are a (Poisson) average of 23 proton-proton collisionsper bunch crossing

I These collision events are superposed that is piled-up oneupon another

I Full (zero-suppressed) event size of ATLAS is 15 MBI This would result in a data rate of 60 TBs

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Example of collision event with ldquopile-uprdquo (side view)

I There are 78 superposed proton-proton collisions in this singlebunch-crossing event ndash very messy but not uncommon

I Is one of these collisions interesting enough to trigger the readout of the detector Must decide quickly

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

The ATLAS Trigger processing Big Fast Data

I Triggering is the process whereby the detectorrsquos read-outsystem is triggered to record the data for a collision event thathas been identified as interesting

I Throwing away data in an unrecoverable way focus on fastrejection

I The Trigger is a real-time multi-stage cascade classifiercomposed of three levels each refine the trigger decision1 Radiation-hard electronics latency 2micros output rate 75 kHz2 Software-based latency 10ms output rate 3 kHz3 Software-based latency ~1 s output rate 200Hz write-out to

offline storage at 300MB s expect to store a few PByear

I Level-2 and Level-3 run in PC farm (~17000 CPU cores)

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

ATLAS Trigger Architecture

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

ATLAS Level-1 Trigger

I Hardware based radiation tolerantI Mounted on or near the detector

I Cable propagation delays limit the time available for processing

I Coarse granularity (ldquolow pixel resolutionrdquo) detector dataI Uses only fastest subdetector systemsI During processing data for multiple bunch-crossings are held in

pipelined memoriesI Identifies Regions of Interest ndash locations in the detector of

objects passing trigger thresholds

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

ATLAS Level-2 and Level-3 Triggers

I Software basedI Access to full-precision detector dataI Basic idea seeded and stepwise reconstructionI Regions of Interest from Level-1 seed processing

I Means only ~2 of the data needs to be transferred to Level-2

I The trigger software has four main componentsI The Algorithms which process the event dataI The Steering which guides and steers the algorithmic processing

of events and is responsible for the trigger decisionI The Data Manager which handles the event data during the

trigger processingI The Event Data Model which specifies the objectified

representation of the event data to be used by the algorithms

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Algorithmic processing in the ATLAS Trigger softwareI There are two types of trigger algorithm

I Feature extraction algorithms process the event data andproduce abstract physics objects (ldquofeaturesrdquo) that representcandidates for electrons muons jets and so on FEXalgorithms operate on features and produce new ones therebyrefining the event information

I Hypothesis algorithms perform a task similar to particleidentification a Hypothesis algorithm tests whether a previouslycreated feature agrees with the hypothesis of an assumedphysics object by applying selection cuts on the featurersquosproperties It can then flag the hypothesis as valid or invalid

I Algorithm sequencing is driven by a static configuration thatinforms the Steering which Algorithm must be executed in thecase that a particular (dynamic) trigger condition is active

I Configuration menu of trigger signatures wersquore interested inI Chain of algorithms can be stopped at any validation stepI Reach end of algorithm chain read out data for offline storage

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

ATLAS Level-2 and Level-3 processor farm

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

How big is our data

I LHC experiments produced ~30PB of data per year in Run 1I Run 2 (now) ~50PByearI By 2023 400PByearI A typical LHC experiment dataset has a size of tens of TB

I On my own experiment sizes are sometimes hundreds of TBI Simulated 35 PB of Monte-Carlo data with combined running

time of 18811 years

I Over the past 20 years the CERN Computer Centre hasrecorded 130PB or data ndash about 100PB in the last five years

I Bulk of data is stored on magnetic tapeI Frequently-accessed (hot) data stored in disk pool system cold

data on tape stage-in data to disk from tape on demand

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Data size comparison

From ldquoParticle physics tames big datardquo Symmetry August 2012

Physics data handling mdash CERN Computer Centre

I CERN Computer Centre hosts 11000 servers with 110000processor cores 120PB raw disk space consumes 35MW ofpower processes about 1 PB per day

Tape storage

I 106PB on tape bull 25000 tape cartridges bull 1ndash55TB eachI Cheap compact and long-lasting reliably read 30 years laterI If a tape snaps it can be spliced back together

I CERN looses only a few hundred MB of data on tape per yearI Donrsquot need power to preserve the data held on themI Safe from hackers

Data analysis on the GridI The Worldwide LHC Computing Grid consists of some 200000

processing cores and 150 petabytes of disk space distributedacross 36 countries through leased data lines

I These computer centres are arranged in ldquoTiersrdquoI Tier-0 This is the CERN Data Centre which is located in

Geneva Switzerland and also at the Wigner Research Centrefor Physics in Budapest Hungary First copy first passreconstruction distribution of data to Tier-1s (by 10 Gbpsoptical fibre private network)

I Tier-1 13 computer centres located worldwide Storage of aproportional share of data large-scale reprocessing distributionof data to Tier-2s

I Tier-2 Around 160 sites typically universities and scientificinstitutes End-user analysis and proportional share of datasimulation and reconstruction

I Users send analysis jobs to the data job runs get back resultsI Every day WLCG processes more than two million jobs

corresponding to a single PC running for more than 600 years

The Wigner Data Centre

I Inaugurated in June 2013I The Wigner Data Centre acts as a remote Tier-0 and an

extension to the CERN Data CentreI Also ensures full business continuity for the critical systems in

case of a major problem on CERNrsquos siteI 2700 servers 43000 computing cores and 72PB of storage

I Installed capacity will eventually be increased to a level similarto that at CERN

I Long distance network connection to CERN two independent100 Gbps circuits

I Bandwidth equivalent to the entire Hungarian domestic internettraffic

I The Wigner Data Centre was chosen after a tender open to all20 CERN Member States

CERN-Wigner high-bandwidth connections

Architecture of Worldwide LHC Computing Grid

Tier-0 CERN (Geneva) + Wigner RCP (Budapest)

I For experimental particle physics ROOT is the ubiquitous dataanalysis tool and has been for the last 20 years old

I Command language CINT (ldquointerpreted C++rdquo) or PythonI Small data work interactively or run macros

I Data format optimised for large data setsI Data in ROOT ldquotreerdquo (like a hierarchical database)I An entry represents an event (ie a collison)

I ldquoBranchesrdquo (electrons muons photons etc)I ldquoLeavesrdquo (energy momentum mass etc)

I Basic idea donrsquot need all of the data all of the timeI Trees in many different files can be merged into one ldquochainrdquoI Access data in chain as if it was a tree in a single fileI Big data build application with ROOT libraries run on Grid

Tape storage

I 106PB on tape bull 25000 tape cartridges bull 1ndash55TB eachI Cheap compact and long-lasting reliably read 30 years laterI If a tape snaps it can be spliced back together

I CERN looses only a few hundred MB of data on tape per yearI Donrsquot need power to preserve the data held on themI Safe from hackers

Data analysis on the GridI The Worldwide LHC Computing Grid consists of some 200000

processing cores and 150 petabytes of disk space distributedacross 36 countries through leased data lines

I These computer centres are arranged in ldquoTiersrdquoI Tier-0 This is the CERN Data Centre which is located in

Geneva Switzerland and also at the Wigner Research Centrefor Physics in Budapest Hungary First copy first passreconstruction distribution of data to Tier-1s (by 10 Gbpsoptical fibre private network)

I Tier-1 13 computer centres located worldwide Storage of aproportional share of data large-scale reprocessing distributionof data to Tier-2s

I Tier-2 Around 160 sites typically universities and scientificinstitutes End-user analysis and proportional share of datasimulation and reconstruction

I Users send analysis jobs to the data job runs get back resultsI Every day WLCG processes more than two million jobs

corresponding to a single PC running for more than 600 years

The Wigner Data Centre

I Inaugurated in June 2013I The Wigner Data Centre acts as a remote Tier-0 and an

extension to the CERN Data CentreI Also ensures full business continuity for the critical systems in

case of a major problem on CERNrsquos siteI 2700 servers 43000 computing cores and 72PB of storage

I Installed capacity will eventually be increased to a level similarto that at CERN

I Long distance network connection to CERN two independent100 Gbps circuits

I Bandwidth equivalent to the entire Hungarian domestic internettraffic

I The Wigner Data Centre was chosen after a tender open to all20 CERN Member States

CERN-Wigner high-bandwidth connections

Architecture of Worldwide LHC Computing Grid

Tier-0 CERN (Geneva) + Wigner RCP (Budapest)

I For experimental particle physics ROOT is the ubiquitous dataanalysis tool and has been for the last 20 years old

I Command language CINT (ldquointerpreted C++rdquo) or PythonI Small data work interactively or run macros

I Data format optimised for large data setsI Data in ROOT ldquotreerdquo (like a hierarchical database)I An entry represents an event (ie a collison)

I ldquoBranchesrdquo (electrons muons photons etc)I ldquoLeavesrdquo (energy momentum mass etc)

I Basic idea donrsquot need all of the data all of the timeI Trees in many different files can be merged into one ldquochainrdquoI Access data in chain as if it was a tree in a single fileI Big data build application with ROOT libraries run on Grid

Data analysis on the GridI The Worldwide LHC Computing Grid consists of some 200000

processing cores and 150 petabytes of disk space distributedacross 36 countries through leased data lines

I These computer centres are arranged in ldquoTiersrdquoI Tier-0 This is the CERN Data Centre which is located in

Geneva Switzerland and also at the Wigner Research Centrefor Physics in Budapest Hungary First copy first passreconstruction distribution of data to Tier-1s (by 10 Gbpsoptical fibre private network)

I Tier-1 13 computer centres located worldwide Storage of aproportional share of data large-scale reprocessing distributionof data to Tier-2s

I Tier-2 Around 160 sites typically universities and scientificinstitutes End-user analysis and proportional share of datasimulation and reconstruction

I Users send analysis jobs to the data job runs get back resultsI Every day WLCG processes more than two million jobs

corresponding to a single PC running for more than 600 years

The Wigner Data Centre

I Inaugurated in June 2013I The Wigner Data Centre acts as a remote Tier-0 and an

extension to the CERN Data CentreI Also ensures full business continuity for the critical systems in

case of a major problem on CERNrsquos siteI 2700 servers 43000 computing cores and 72PB of storage

I Installed capacity will eventually be increased to a level similarto that at CERN

I Long distance network connection to CERN two independent100 Gbps circuits

I Bandwidth equivalent to the entire Hungarian domestic internettraffic

I The Wigner Data Centre was chosen after a tender open to all20 CERN Member States

CERN-Wigner high-bandwidth connections

Architecture of Worldwide LHC Computing Grid

Tier-0 CERN (Geneva) + Wigner RCP (Budapest)

I For experimental particle physics ROOT is the ubiquitous dataanalysis tool and has been for the last 20 years old

I Command language CINT (ldquointerpreted C++rdquo) or PythonI Small data work interactively or run macros

I Data format optimised for large data setsI Data in ROOT ldquotreerdquo (like a hierarchical database)I An entry represents an event (ie a collison)

I ldquoBranchesrdquo (electrons muons photons etc)I ldquoLeavesrdquo (energy momentum mass etc)

I Basic idea donrsquot need all of the data all of the timeI Trees in many different files can be merged into one ldquochainrdquoI Access data in chain as if it was a tree in a single fileI Big data build application with ROOT libraries run on Grid

The Wigner Data Centre

I Inaugurated in June 2013I The Wigner Data Centre acts as a remote Tier-0 and an

extension to the CERN Data CentreI Also ensures full business continuity for the critical systems in

case of a major problem on CERNrsquos siteI 2700 servers 43000 computing cores and 72PB of storage

I Installed capacity will eventually be increased to a level similarto that at CERN

I Long distance network connection to CERN two independent100 Gbps circuits

I Bandwidth equivalent to the entire Hungarian domestic internettraffic

I The Wigner Data Centre was chosen after a tender open to all20 CERN Member States

CERN-Wigner high-bandwidth connections

Architecture of Worldwide LHC Computing Grid

Tier-0 CERN (Geneva) + Wigner RCP (Budapest)

I For experimental particle physics ROOT is the ubiquitous dataanalysis tool and has been for the last 20 years old

I Command language CINT (ldquointerpreted C++rdquo) or PythonI Small data work interactively or run macros

I Data format optimised for large data setsI Data in ROOT ldquotreerdquo (like a hierarchical database)I An entry represents an event (ie a collison)

I ldquoBranchesrdquo (electrons muons photons etc)I ldquoLeavesrdquo (energy momentum mass etc)

I Basic idea donrsquot need all of the data all of the timeI Trees in many different files can be merged into one ldquochainrdquoI Access data in chain as if it was a tree in a single fileI Big data build application with ROOT libraries run on Grid

CERN-Wigner high-bandwidth connections

Architecture of Worldwide LHC Computing Grid

Tier-0 CERN (Geneva) + Wigner RCP (Budapest)

I For experimental particle physics ROOT is the ubiquitous dataanalysis tool and has been for the last 20 years old

I Command language CINT (ldquointerpreted C++rdquo) or PythonI Small data work interactively or run macros

I Data format optimised for large data setsI Data in ROOT ldquotreerdquo (like a hierarchical database)I An entry represents an event (ie a collison)

I ldquoBranchesrdquo (electrons muons photons etc)I ldquoLeavesrdquo (energy momentum mass etc)

I Basic idea donrsquot need all of the data all of the timeI Trees in many different files can be merged into one ldquochainrdquoI Access data in chain as if it was a tree in a single fileI Big data build application with ROOT libraries run on Grid

Architecture of Worldwide LHC Computing Grid

Tier-0 CERN (Geneva) + Wigner RCP (Budapest)

I For experimental particle physics ROOT is the ubiquitous dataanalysis tool and has been for the last 20 years old

I Command language CINT (ldquointerpreted C++rdquo) or PythonI Small data work interactively or run macros

I Data format optimised for large data setsI Data in ROOT ldquotreerdquo (like a hierarchical database)I An entry represents an event (ie a collison)

I ldquoBranchesrdquo (electrons muons photons etc)I ldquoLeavesrdquo (energy momentum mass etc)

I Basic idea donrsquot need all of the data all of the timeI Trees in many different files can be merged into one ldquochainrdquoI Access data in chain as if it was a tree in a single fileI Big data build application with ROOT libraries run on Grid

I For experimental particle physics ROOT is the ubiquitous dataanalysis tool and has been for the last 20 years old

I Command language CINT (ldquointerpreted C++rdquo) or PythonI Small data work interactively or run macros

I Data format optimised for large data setsI Data in ROOT ldquotreerdquo (like a hierarchical database)I An entry represents an event (ie a collison)

I ldquoBranchesrdquo (electrons muons photons etc)I ldquoLeavesrdquo (energy momentum mass etc)

I Basic idea donrsquot need all of the data all of the timeI Trees in many different files can be merged into one ldquochainrdquoI Access data in chain as if it was a tree in a single fileI Big data build application with ROOT libraries run on Grid

LHC data flow

1 Detected by LHC experiment2 Online multi-level filtering (hardware and software)3 Transferred to CERN and Wigner Tier-0 archived and

reconstructed4 Transferred to Tier-1 sites archived reconstructed and

skimmed5 Transferred to Tier-2 sites reconstructed skimmed filtered

and analysed6 Written to locally-analysable files put on PCs7 Turned into plot in a paper

Higgs boson rarr WW signal in 2011 and 2012 data

Higgs boson rarr 4-leptons signal in 2011 and 2012 data

More information

I Data science LHC2015 WorkshopI Workshop to help foster long-term connections between the

data science and particle physics communities

I A mailing list HEP-data-sciencegooglegroupscom hasjust been created to deal with anything concerning bothparticle physics and data science in particular machine learning

I Announcementdiscussion about workshops challenges paperstools etc

I Open to all subscription by sending a mail toHEP-data-science+subscribegooglegroupscom

I Explore the CERN experiments with Google StreetviewI Explore CERNrsquos Computer Centre with Google StreetviewI ldquoProcessing LHC datardquo (short film)

Higgs boson rarr WW signal in 2011 and 2012 data

Higgs boson rarr 4-leptons signal in 2011 and 2012 data

More information

I Data science LHC2015 WorkshopI Workshop to help foster long-term connections between the

data science and particle physics communities

I A mailing list HEP-data-sciencegooglegroupscom hasjust been created to deal with anything concerning bothparticle physics and data science in particular machine learning

I Announcementdiscussion about workshops challenges paperstools etc

I Open to all subscription by sending a mail toHEP-data-science+subscribegooglegroupscom

I Explore the CERN experiments with Google StreetviewI Explore CERNrsquos Computer Centre with Google StreetviewI ldquoProcessing LHC datardquo (short film)

Higgs boson rarr 4-leptons signal in 2011 and 2012 data

More information

I Data science LHC2015 WorkshopI Workshop to help foster long-term connections between the

data science and particle physics communities

I A mailing list HEP-data-sciencegooglegroupscom hasjust been created to deal with anything concerning bothparticle physics and data science in particular machine learning

I Announcementdiscussion about workshops challenges paperstools etc

I Open to all subscription by sending a mail toHEP-data-science+subscribegooglegroupscom

I Explore the CERN experiments with Google StreetviewI Explore CERNrsquos Computer Centre with Google StreetviewI ldquoProcessing LHC datardquo (short film)

More information

I Data science LHC2015 WorkshopI Workshop to help foster long-term connections between the

data science and particle physics communities

I A mailing list HEP-data-sciencegooglegroupscom hasjust been created to deal with anything concerning bothparticle physics and data science in particular machine learning

I Announcementdiscussion about workshops challenges paperstools etc

I Open to all subscription by sending a mail toHEP-data-science+subscribegooglegroupscom

I Explore the CERN experiments with Google StreetviewI Explore CERNrsquos Computer Centre with Google StreetviewI ldquoProcessing LHC datardquo (short film)

Thanks

httpswwwlinkedincominandrewjohnlowe

Bonus slides

Data Centre statistics (2 June 2015)

1. 00
2. 01
3. 02
4. 03
5. 04
6. 05
7. 06
8. 07
9. 08
10. 09
11. 010
12. 011
13. 012
14. 013
15. 014
16. 015
17. 016
18. 017
19. 018
20. 019
21. 020
22. 021
23. 022
24. 023
25. 024
26. 025
27. 026
28. 027
29. 028
30. 029
31. 030
32. 031
33. 032
34. 033
35. 034
36. 035
37. 036
38. 037
39. 038
40. 039
41. 040
42. 041
43. 042
44. 043
45. 044
46. 045
47. 046
48. 047
49. 048
50. 049
51. 050
52. 051
53. 052
54. 053
55. 054
56. 055
57. 056
58. 057
59. 058
60. 059
61. 060
62. 061
63. 062
64. 063
65. 064
66. 065
67. 066
68. 067
69. 068
70. 069
71. 070
72. 071
73. 072
74. 073
75. 074
76. 075
77. 076
78. 077
79. 078
80. 079
81. 080
82. 081
83. 082
84. 083
85. 084
86. 085
87. 086
88. 087
89. 088
90. 089
91. 090
92. 091
93. 092
94. 093
95. 094
96. 095
97. 096
98. 097
99. 098
100. 099
101. 0100
102. 0101
103. 0102
104. 0103
105. 0104
106. 0105
107. 0106
108. 0107
109. 0108
110. 0109
111. 0110
112. 0111
113. 0112
114. 0113
115. 0114
116. 0115
117. 0116
118. 0117
119. 0118
120. 0119
121. 0120
122. 0121
123. 0122
124. 0123
125. 0124
126. 0125
127. 0126
128. 0127
129. 0128
130. 0129
131. 0130
132. 0131
133. 0132
134. 0133
135. 0134
136. 0135
137. 0136
138. 0137
139. 0138
140. 0139
141. 0140
142. 0141
143. 0142
144. 0143
145. 0144
146. 0145
147. 0146
148. 0147
149. 0148
150. 0149
151. 0150
152. 0151
153. 0152
154. 0153
155. 0154
156. 0155
157. 0156
158. 0157
159. 0158
160. 0159
161. 0160
162. 0161
163. 0162
164. 0163
165. 0164
166. 0165
167. 0166
168. 0167
169. 0168
170. 0169
171. 0170
172. 0171
173. 0172
174. 0173
175. 0174
176. 0175
177. 0176
178. 0177
179. 0178
180. 0179
181. 0180
182. 0181
183. 0182
184. 0183
185. 0184
186. 0185
187. 0186
188. 0187
189. 0188
190. 0189
191. 0190
192. 0191
193. 0192
194. 0193
195. 0194
196. 0195
197. 0196
198. 0197
199. 0198
200. 0199
201. 0200
202. 0201
203. 0202
204. 0203
205. 0204
206. 0205
207. 0206
208. 0207
209. 0208
210. 0209
211. 0210
212. 0211
213. 0212
214. 0213
215. 0214
216. 0215
217. 0216
218. 0217
219. 0218
220. 0219
221. 0220
222. 0221
223. 0222
224. 0223
225. 0224
226. 0225
227. 0226
228. 0227
229. 0228
230. 0229
231. 0230
232. 0231
233. 0232
234. 0233
235. 0234
236. 0235
237. 0236
238. 0237
239. 0238
240. 0239
241. 0240
242. 0241
243. 0242
244. 0243
245. 0244
246. 0245
247. 0246
248. 0247
249. 0248
250. 0249
251. 0250
252. 0251
253. 0252
254. 0253
255. 0254
256. 0255
257. 0256
258. 0257
259. 0258
260. 0259
261. 0260
262. 0261
263. 0262
264. 0263
265. 0264
266. 0265
267. 0266
268. 0267
269. 0268
270. 0269
271. 0270
272. 0271
273. 0272
274. 0273
275. 0274
276. 0275
277. 0276
278. anm0
279. 10
280. 11
281. 12
282. 13
283. 14
284. 15
285. 16
286. 17
287. 18
288. 19
289. 110
290. 111
291. 112
292. 113
293. 114
294. 115
295. 116
296. 117
297. 118
298. 119
299. 120
300. 121
301. 122
302. 123
303. 124
304. 125
305. 126
306. 127
307. 128
308. 129
309. 130
310. 131
311. 132
312. 133
313. 134
314. 135
315. 136
316. 137
317. 138
318. 139
319. 140
320. 141
321. 142
322. 143
323. 144
324. 145
325. 146
326. 147
327. 148
328. 149
329. 150
330. 151
331. 152
332. 153
333. 154
334. 155
335. 156
336. 157
337. 158
338. 159
339. 160
340. 161
341. 162
342. 163
343. 164
344. 165
345. 166
346. 167
347. 168
348. 169
349. 170
350. 171
351. 172
352. 173
353. 174
354. 175
355. 176
356. 177
357. 178
358. 179
359. 180
360. 181
361. 182
362. 183
363. 184
364. 185
365. 186
366. 187
367. 188
368. 189
369. 190
370. 191
371. 192
372. 193
373. 194
374. 195
375. 196
376. 197
377. 198
378. 199
379. 1100
380. 1101
381. 1102
382. 1103
383. 1104
384. 1105
385. 1106
386. 1107
387. 1108
388. 1109
389. 1110
390. 1111
391. 1112
392. 1113
393. 1114
394. 1115
395. 1116
396. 1117
397. 1118
398. 1119
399. 1120
400. 1121
401. 1122
402. 1123
403. 1124
404. 1125
405. 1126
406. 1127
407. 1128
408. 1129
409. 1130
410. 1131
411. 1132
412. 1133
413. 1134
414. 1135
415. 1136
416. 1137
417. 1138
418. 1139
419. 1140
420. 1141
421. 1142
422. 1143
423. 1144
424. 1145
425. 1146
426. 1147
427. 1148
428. 1149
429. 1150
430. 1151
431. 1152
432. 1153
433. 1154
434. 1155
435. 1156
436. 1157
437. 1158
438. 1159
439. 1160
440. 1161
441. 1162
442. 1163
443. 1164
444. 1165
445. 1166
446. 1167
447. 1168
448. 1169
449. 1170
450. 1171
451. 1172
452. 1173
453. 1174
454. 1175
455. 1176
456. 1177
457. 1178
458. 1179
459. 1180
460. 1181
461. 1182
462. 1183
463. 1184
464. 1185
465. 1186
466. 1187
467. 1188
468. 1189
469. 1190
470. 1191
471. 1192
472. 1193
473. 1194
474. 1195
475. 1196
476. 1197
477. 1198
478. 1199
479. 1200
480. 1201
481. 1202
482. 1203
483. 1204
484. 1205
485. 1206
486. 1207
487. 1208
488. 1209
489. 1210
490. 1211
491. 1212
492. 1213
493. 1214
494. 1215
495. 1216
496. 1217
497. 1218
498. 1219
499. 1220
500. 1221
501. 1222
502. 1223
503. 1224
504. 1225
505. 1226
506. 1227
507. 1228
508. 1229
509. 1230
510. 1231
511. 1232
512. 1233
513. 1234
514. 1235
515. 1236
516. 1237
517. 1238
518. 1239
519. 1240
520. 1241
521. 1242
522. 1243
523. 1244
524. 1245
525. 1246
526. 1247
527. 1248
528. 1249
529. 1250
530. 1251
531. 1252
532. 1253
533. 1254
534. 1255
535. 1256
536. 1257
537. 1258
538. 1259
539. 1260
540. 1261
541. 1262
542. 1263
543. 1264
544. 1265
545. 1266
546. 1267
547. 1268
548. 1269
549. 1270
550. 1271
551. 1272
552. 1273
553. 1274
554. 1275
555. 1276
556. 1277
557. 1278
558. 1279
559. 1280
560. 1281
561. 1282
562. 1283
563. 1284
564. 1285
565. 1286
566. 1287
567. 1288
568. 1289
569. 1290
570. 1291
571. 1292
572. 1293
573. 1294
574. 1295
575. 1296
576. 1297
577. 1298
578. 1299
579. 1300
580. 1301
581. 1302
582. 1303
583. 1304
584. 1305
585. 1306
586. 1307
587. 1308
588. 1309
589. 1310
590. 1311
591. 1312
592. 1313
593. 1314
594. 1315
595. 1316
596. 1317
597. 1318
598. 1319
599. 1320
600. 1321
601. 1322
602. 1323
603. 1324
604. 1325
605. 1326
606. 1327
607. 1328
608. 1329
609. 1330
610. 1331
611. 1332
612. 1333
613. 1334
614. 1335
615. 1336
616. 1337
617. 1338
618. 1339
619. 1340
620. 1341
621. 1342
622. 1343
623. 1344
624. 1345
625. 1346
626. 1347
627. 1348
628. 1349
629. 1350
630. 1351
631. 1352
632. 1353
633. 1354
634. 1355
635. 1356
636. 1357
637. 1358
638. 1359
639. 1360
640. 1361
641. 1362
642. 1363
643. 1364
644. 1365
645. 1366
646. 1367
647. 1368
648. 1369
649. 1370
650. 1371
651. 1372
652. 1373
653. 1374
654. 1375
655. 1376
656. 1377
657. 1378
658. 1379
659. 1380
660. 1381
661. 1382
662. 1383
663. 1384
664. 1385
665. 1386
666. 1387
667. 1388
668. 1389
669. 1390
670. 1391
671. 1392
672. 1393
673. 1394
674. 1395
675. 1396
676. 1397
677. 1398
678. 1399
679. 1400
680. 1401
681. 1402
682. 1403
683. 1404
684. 1405
685. 1406
686. 1407
687. 1408
688. 1409
689. 1410
690. 1411
691. 1412
692. 1413
693. 1414
694. 1415
695. 1416
696. 1417
697. 1418
698. 1419
699. 1420
700. 1421
701. 1422
702. 1423
703. 1424
704. 1425
705. 1426
706. 1427
707. 1428
708. 1429
709. 1430
710. 1431
711. 1432
712. 1433
713. 1434
714. 1435
715. 1436
716. 1437
717. 1438
718. 1439
719. 1440
720. 1441
721. 1442
722. 1443
723. 1444
724. 1445
725. 1446
726. 1447
727. 1448
728. 1449
729. 1450
730. 1451
731. 1452
732. 1453
733. 1454
734. 1455
735. 1456
736. 1457
737. 1458
738. 1459
739. 1460
740. 1461
741. 1462
742. 1463
743. 1464
744. 1465
745. 1466
746. 1467
747. 1468
748. 1469
749. 1470
750. 1471
751. 1472
752. 1473
753. 1474
754. 1475
755. 1476
756. 1477
757. 1478
758. 1479
759. 1480
760. 1481
761. 1482
762. 1483
763. 1484
764. 1485
765. 1486
766. 1487
767. 1488
768. 1489
769. 1490
770. 1491
771. 1492
772. 1493
773. 1494
774. 1495
775. 1496
776. 1497
777. 1498
778. 1499
779. 1500
780. 1501
781. 1502
782. 1503
783. 1504
784. 1505
785. 1506
786. 1507
787. 1508
788. 1509
789. 1510
790. 1511
791. 1512
792. 1513
793. 1514
794. 1515
795. 1516
796. 1517
797. 1518
798. 1519
799. 1520
800. 1521
801. 1522
802. 1523
803. anm1

Data Centre statistics (2 June 2015)

1. 00
2. 01
3. 02
4. 03
5. 04
6. 05
7. 06
8. 07
9. 08
10. 09
11. 010
12. 011
13. 012
14. 013
15. 014
16. 015
17. 016
18. 017
19. 018
20. 019
21. 020
22. 021
23. 022
24. 023
25. 024
26. 025
27. 026
28. 027
29. 028
30. 029
31. 030
32. 031
33. 032
34. 033
35. 034
36. 035
37. 036
38. 037
39. 038
40. 039
41. 040
42. 041
43. 042
44. 043
45. 044
46. 045
47. 046
48. 047
49. 048
50. 049
51. 050
52. 051
53. 052
54. 053
55. 054
56. 055
57. 056
58. 057
59. 058
60. 059
61. 060
62. 061
63. 062
64. 063
65. 064
66. 065
67. 066
68. 067
69. 068
70. 069
71. 070
72. 071
73. 072
74. 073
75. 074
76. 075
77. 076
78. 077
79. 078
80. 079
81. 080
82. 081
83. 082
84. 083
85. 084
86. 085
87. 086
88. 087
89. 088
90. 089
91. 090
92. 091
93. 092
94. 093
95. 094
96. 095
97. 096
98. 097
99. 098
100. 099
101. 0100
102. 0101
103. 0102
104. 0103
105. 0104
106. 0105
107. 0106
108. 0107
109. 0108
110. 0109
111. 0110
112. 0111
113. 0112
114. 0113
115. 0114
116. 0115
117. 0116
118. 0117
119. 0118
120. 0119
121. 0120
122. 0121
123. 0122
124. 0123
125. 0124
126. 0125
127. 0126
128. 0127
129. 0128
130. 0129
131. 0130
132. 0131
133. 0132
134. 0133
135. 0134
136. 0135
137. 0136
138. 0137
139. 0138
140. 0139
141. 0140
142. 0141
143. 0142
144. 0143
145. 0144
146. 0145
147. 0146
148. 0147
149. 0148
150. 0149
151. 0150
152. 0151
153. 0152
154. 0153
155. 0154
156. 0155
157. 0156
158. 0157
159. 0158
160. 0159
161. 0160
162. 0161
163. 0162
164. 0163
165. 0164
166. 0165
167. 0166
168. 0167
169. 0168
170. 0169
171. 0170
172. 0171
173. 0172
174. 0173
175. 0174
176. 0175
177. 0176
178. 0177
179. 0178
180. 0179
181. 0180
182. 0181
183. 0182
184. 0183
185. 0184
186. 0185
187. 0186
188. 0187
189. 0188
190. 0189
191. 0190
192. 0191
193. 0192
194. 0193
195. 0194
196. 0195
197. 0196
198. 0197
199. 0198
200. 0199
201. 0200
202. 0201
203. 0202
204. 0203
205. 0204
206. 0205
207. 0206
208. 0207
209. 0208
210. 0209
211. 0210
212. 0211
213. 0212
214. 0213
215. 0214
216. 0215
217. 0216
218. 0217
219. 0218
220. 0219
221. 0220
222. 0221
223. 0222
224. 0223
225. 0224
226. 0225
227. 0226
228. 0227
229. 0228
230. 0229
231. 0230
232. 0231
233. 0232
234. 0233
235. 0234
236. 0235
237. 0236
238. 0237
239. 0238
240. 0239
241. 0240
242. 0241
243. 0242
244. 0243
245. 0244
246. 0245
247. 0246
248. 0247
249. 0248
250. 0249
251. 0250
252. 0251
253. 0252
254. 0253
255. 0254
256. 0255
257. 0256
258. 0257
259. 0258
260. 0259
261. 0260
262. 0261
263. 0262
264. 0263
265. 0264
266. 0265
267. 0266
268. 0267
269. 0268
270. 0269
271. 0270
272. 0271
273. 0272
274. 0273
275. 0274
276. 0275
277. 0276
278. anm0
279. 10
280. 11
281. 12
282. 13
283. 14
284. 15
285. 16
286. 17
287. 18
288. 19
289. 110
290. 111
291. 112
292. 113
293. 114
294. 115
295. 116
296. 117
297. 118
298. 119
299. 120
300. 121
301. 122
302. 123
303. 124
304. 125
305. 126
306. 127
307. 128
308. 129
309. 130
310. 131
311. 132
312. 133
313. 134
314. 135
315. 136
316. 137
317. 138
318. 139
319. 140
320. 141
321. 142
322. 143
323. 144
324. 145
325. 146
326. 147
327. 148
328. 149
329. 150
330. 151
331. 152
332. 153
333. 154
334. 155
335. 156
336. 157
337. 158
338. 159
339. 160
340. 161
341. 162
342. 163
343. 164
344. 165
345. 166
346. 167
347. 168
348. 169
349. 170
350. 171
351. 172
352. 173
353. 174
354. 175
355. 176
356. 177
357. 178
358. 179
359. 180
360. 181
361. 182
362. 183
363. 184
364. 185
365. 186
366. 187
367. 188
368. 189
369. 190
370. 191
371. 192
372. 193
373. 194
374. 195
375. 196
376. 197
377. 198
378. 199
379. 1100
380. 1101
381. 1102
382. 1103
383. 1104
384. 1105
385. 1106
386. 1107
387. 1108
388. 1109
389. 1110
390. 1111
391. 1112
392. 1113
393. 1114
394. 1115
395. 1116
396. 1117
397. 1118
398. 1119
399. 1120
400. 1121
401. 1122
402. 1123
403. 1124
404. 1125
405. 1126
406. 1127
407. 1128
408. 1129
409. 1130
410. 1131
411. 1132
412. 1133
413. 1134
414. 1135
415. 1136
416. 1137
417. 1138
418. 1139
419. 1140
420. 1141
421. 1142
422. 1143
423. 1144
424. 1145
425. 1146
426. 1147
427. 1148
428. 1149
429. 1150
430. 1151
431. 1152
432. 1153
433. 1154
434. 1155
435. 1156
436. 1157
437. 1158
438. 1159
439. 1160
440. 1161
441. 1162
442. 1163
443. 1164
444. 1165
445. 1166
446. 1167
447. 1168
448. 1169
449. 1170
450. 1171
451. 1172
452. 1173
453. 1174
454. 1175
455. 1176
456. 1177
457. 1178
458. 1179
459. 1180
460. 1181
461. 1182
462. 1183
463. 1184
464. 1185
465. 1186
466. 1187
467. 1188
468. 1189
469. 1190
470. 1191
471. 1192
472. 1193
473. 1194
474. 1195
475. 1196
476. 1197
477. 1198
478. 1199
479. 1200
480. 1201
481. 1202
482. 1203
483. 1204
484. 1205
485. 1206
486. 1207
487. 1208
488. 1209
489. 1210
490. 1211
491. 1212
492. 1213
493. 1214
494. 1215
495. 1216
496. 1217
497. 1218
498. 1219
499. 1220
500. 1221
501. 1222
502. 1223
503. 1224
504. 1225
505. 1226
506. 1227
507. 1228
508. 1229
509. 1230
510. 1231
511. 1232
512. 1233
513. 1234
514. 1235
515. 1236
516. 1237
517. 1238
518. 1239
519. 1240
520. 1241
521. 1242
522. 1243
523. 1244
524. 1245
525. 1246
526. 1247
527. 1248
528. 1249
529. 1250
530. 1251
531. 1252
532. 1253
533. 1254
534. 1255
535. 1256
536. 1257
537. 1258
538. 1259
539. 1260
540. 1261
541. 1262
542. 1263
543. 1264
544. 1265
545. 1266
546. 1267
547. 1268
548. 1269
549. 1270
550. 1271
551. 1272
552. 1273
553. 1274
554. 1275
555. 1276
556. 1277
557. 1278
558. 1279
559. 1280
560. 1281
561. 1282
562. 1283
563. 1284
564. 1285
565. 1286
566. 1287
567. 1288
568. 1289
569. 1290
570. 1291
571. 1292
572. 1293
573. 1294
574. 1295
575. 1296
576. 1297
577. 1298
578. 1299
579. 1300
580. 1301
581. 1302
582. 1303
583. 1304
584. 1305
585. 1306
586. 1307
587. 1308
588. 1309
589. 1310
590. 1311
591. 1312
592. 1313
593. 1314
594. 1315
595. 1316
596. 1317
597. 1318
598. 1319
599. 1320
600. 1321
601. 1322
602. 1323
603. 1324
604. 1325
605. 1326
606. 1327
607. 1328
608. 1329
609. 1330
610. 1331
611. 1332
612. 1333
613. 1334
614. 1335
615. 1336
616. 1337
617. 1338
618. 1339
619. 1340
620. 1341
621. 1342
622. 1343
623. 1344
624. 1345
625. 1346
626. 1347
627. 1348
628. 1349
629. 1350
630. 1351
631. 1352
632. 1353
633. 1354
634. 1355
635. 1356
636. 1357
637. 1358
638. 1359
639. 1360
640. 1361
641. 1362
642. 1363
643. 1364
644. 1365
645. 1366
646. 1367
647. 1368
648. 1369
649. 1370
650. 1371
651. 1372
652. 1373
653. 1374
654. 1375
655. 1376
656. 1377
657. 1378
658. 1379
659. 1380
660. 1381
661. 1382
662. 1383
663. 1384
664. 1385
665. 1386
666. 1387
667. 1388
668. 1389
669. 1390
670. 1391
671. 1392
672. 1393
673. 1394
674. 1395
675. 1396
676. 1397
677. 1398
678. 1399
679. 1400
680. 1401
681. 1402
682. 1403
683. 1404
684. 1405
685. 1406
686. 1407
687. 1408
688. 1409
689. 1410
690. 1411
691. 1412
692. 1413
693. 1414
694. 1415
695. 1416
696. 1417
697. 1418
698. 1419
699. 1420
700. 1421
701. 1422
702. 1423
703. 1424
704. 1425
705. 1426
706. 1427
707. 1428
708. 1429
709. 1430
710. 1431
711. 1432
712. 1433
713. 1434
714. 1435
715. 1436
716. 1437
717. 1438
718. 1439
719. 1440
720. 1441
721. 1442
722. 1443
723. 1444
724. 1445
725. 1446
726. 1447
727. 1448
728. 1449
729. 1450
730. 1451
731. 1452
732. 1453
733. 1454
734. 1455
735. 1456
736. 1457
737. 1458
738. 1459
739. 1460
740. 1461
741. 1462
742. 1463
743. 1464
744. 1465
745. 1466
746. 1467
747. 1468
748. 1469
749. 1470
750. 1471
751. 1472
752. 1473
753. 1474
754. 1475
755. 1476
756. 1477
757. 1478
758. 1479
759. 1480
760. 1481
761. 1482
762. 1483
763. 1484
764. 1485
765. 1486
766. 1487
767. 1488
768. 1489
769. 1490
770. 1491
771. 1492
772. 1493
773. 1494
774. 1495
775. 1496
776. 1497
777. 1498
778. 1499
779. 1500
780. 1501
781. 1502
782. 1503
783. 1504
784. 1505
785. 1506
786. 1507
787. 1508
788. 1509
789. 1510
790. 1511
791. 1512
792. 1513
793. 1514
794. 1515
795. 1516
796. 1517
797. 1518
798. 1519
799. 1520
800. 1521
801. 1522
802. 1523
803. anm1