DT Shifter Training

Infrastructure 

A. Benvenuti, M.Giunta

07/07/2010

General Info• The Technical Shifter should call DT Infrastructure Expert (165507)

• 79930 given as a backup

• If you need an expert you can call:

163651 (Alberto)162355 (Marina or Daniele)

DT Safety Considerations• Overheating:

- Inside the detector (MiniCrates) due to Cooling Plant failure (MUON Circuit), affects all MC, or obstructed flow-limiter, affects two MC in adjacent sectors.- Electronic modules inside Tower or USC Racks, due to Cooling Plant failure

(Racks Circuit) that affects all racks, or a turbine failure that affects only the modules inside one rack. The most dangerous case is a turbine failure in a tower rack containing Sector Collector crates.

•Water Leaks:- Are detected by “leak detection” cables inside the detector and tower racks. Major leaks can also induce HV trips in one or more chambers.

DT expert (165507, backup: 79930) should be called for any error induced by Overheating or Water leaks •Gas Failure:

- Not critical, the DT chambers can be operated for ~ day without gas flow

DT States• ON:

- LV ON- HV ON: wires +3600V, strips +1800V, cathodes -1200V

• Standby:- LV ON- HV ON: wires +1900V, strips +1800V, cathodes -1200V

• Dormant:- LV ON- Sector Collector mezzanine boards (opto couplers) OFF- HV ON: wires +1900V, strips +1800V, cathodes -1200V

• OFF:- LV OFF- HV OFF

The DT should be in Standby mode when the Magnet is ramping.

http://cmsonline.cern.ch/portal/page/portal/CMS%20online%20system/DCS/Automation?_piref815_847251_815_847248_847248.strutsAction=%2FsubmitDetector.do%3Bjsessionid%3D898af9ec30d87aa1ed82970647a6b953ea8018662e18.e34LbhiLbx8Nay0LbxqRbN8Naxb0n6jAmljGr5XDqQLvpAe%3FSelectedDetector%3DDT

DT State vs Machine/Beam State

How to connect to PVSS from home (I)• RemoteDesktop connection to the Terminal Server:

- Connect to cerntscms01 as mudcs (usual pwd)- tunnel on lxplus → Source port: 8881, Destination: cerntscms01.cern.ch:3389- RemoteDesktop connects to→ localhost:8881

•For MAC Users: you can use CORD

How to connect to PVSS from home (II)

DT Detector Panel

1)

2)

7)

4)

5)

6)

3)

3)

Error conditions flagged by Red color on tabs and audio alarm (to be implemented):1) Cooling2) UX / USC Turbines, Smoke

alarms 3) MC

Temperatures/Communication Error

4) Water Leaks5) LV Temperatures6) Gas System7) More than 1 MC LV error8) HV Problem

6)1)

5)

8)

Yoke Circuit Flow Error• No action required

Minicrate (MUON) Flow Error• Cooling Plant Problem Call DT Expert (165507)• MC will switch OFF after 15’

Rack Circuit Flow Error• Cooling Plant Problem Call DT Expert (165507)• LV, HV switch OFF after 20’

Supply T > 18°C • Cooling Plant Problem Call DT Expert (165507)• DT-DCS shutoff in 30’

Time windows can be reset from “Reset Timeouts” button

Cooling Problem

Cooling Problem

• Tell the Technical Shifter to contact CCC and find out the time estimate to fix the problem

• Call immediately:163651 (Alberto) or162355 (Marina or Daniele)

UXC Rack Problem

Turbine is OFF (current ~ 0.4A) Call DT Expert (165507)

High Voltage USC:

• HV OFF in corresponding wheels within 300 seconds. Time window can be reset by apposite button

LV USC S4F02: No Action Required

USC Rack Problem

• Check that affected components have been switched OFF• Individual modules can be switched OFF by clicking the “ON” button• Entire Wheel can be switched ON/OFF from the “Wheel ON-OFF” panel

The LV Panel can be reached from the Detector Panel clicking the “MAO” button

Caution: Toggle Buttons if pushed will change state

MAO Panelactions described in

page 6

A drop of water in one of the leak detection cables will trigger an alarm and the corresponding buttons will turn red.

Call DT Expert (165507) No further action required from shifter

Water Leak

• A3100 Modules power the SC crates and have the largest load hence the high T.

• In Standby mode the SC Mezzanines will be off and the Temperatures will be lower.

• If the Temperature is above 40°C in several modules there is a cooling problem in the “Rack circuit”

Time Histories are available from the buttons

LV Boards Temperature Problem

Gas Alarm due to:• Stopped gas flowDT Chambers are safe without gas circulation for ~ 1day. Do NOT call expert during the night

• Communication Error Data are not stored in DataBase Call DT Expert (165507) from 08:00 to 23:00

Button becomes RED when Communication is lost

Gas Problem

HV Problem

During the day:Call DT Expert (165507)

During the night:Open ‘HV Layers Info’ panel: if you have>2 entries in the ‘Layers in STANDALONE’ columnOr>5 entries in the ‘Layers not ON’ column Call DT Expert (165507)

All Previous Problems

• Call immediately:163651 (Alberto) or162355 (Marina or Daniele)

YB+1 YB+2MiniCrates

Temperature

• If Temperature exceeds 40°C for 1 or 2 MC call Expert and it will help you to Switch OFF the MC• If high T appears in several MC it is a cooling problem

To Reach this panel

MiniCrate Temperature Problem

MiniCrate Temperature Problem

• In one MiniCrate- Switch OFF the MiniCrate (see next slide)

• In several MiniCrates- Probable Cooling Problem, call immediately:

163651 (Alberto) or162355 (Marina or Daniele)

How to switch off one Minicrate

You can choose if doing it yourself or asking the Technical Shifter to do itIf you want to do it:

- ask the Technical Shifter to Release the partition (DT+, DT- or DT0)- Take(*) the partition- Perform the action- When you’re done Release(*) the partition- Tell the Technical Shifter to Take(*) back the partition(*) see next slide to learn how to Take and Release

Give a SWITCH_OFF_LV command from the Chamber Panel

How to Take/Release a FsM node

Click the open lock and then ‘Take’• How to Take a node

• How to Release a nodeClick the lock and then ‘Release’

MiniCrates Communication Error

In case communication is lost between PVSS and the MC control program, the Wheel Banner will turn red and blinking and the chambers will turn orange.Click the “Reset” button on the wheel and send “MC_Reset” from the top node.

The LV will be switched OFF after 30’. Call DT Expert (165507)

You must “take” the “DT_Detector” to do any actionand “release” it once you are done

MiniCrate Communication Problem

• In one wheel • Restart the CCB Server for that wheel, as explained in the DT Expert

FAQ twiki: https://twiki.cern.ch/twiki/bin/view/CMS/DTexpertFAQ2009x07x27

• In ALL the wheels• Call DT Field Manager

MiniCrate with Boards Off

2.00

• Try to switch ON the MiniCrate (from DCS GUI -> Power All) once- If command fails or boards switch off again, call DT Field Manager

If the boxes are not green it means the LV channels actually tripped; in this case call the DT Field Manager

• HV and LV can be switched OFF with a fast but safe procedure from the “Emergency Stop” buttons

Automatic DCS/DSS ActionsFailure condition Agent Threshold Time Action Granularity How to suspend action

DT-DSS 200'' SMS to expertDT-DSS 10' SMS to expertDT-DSS 15' SC and MC power cut All

DCS smooth LV power off AllDT-DSS 170'' Kill LV All

DSS 180" LV power cut AllDCS smooth HV power off (+,0,-)DCS smooth LV power off one wheel

DT-DSS 6' Kill HV (+,0,-)DSS ~7' HV power cut (+,0,-)DCS smooth HV power off All

DT-DSS 6' Kill HV AllDSS ~7' HV power cut AllDSS HV power cut AllDCS 1" LV power off All

DT-DSS < 1'' Kill LV & HV AllDSS 3'' LV power cut All

MC high temperature MC 40°C ~10'' Auto power off Single MCTSC mezzanine 50°C Auto power off Single board

DT-DSS 29°C Top crate SMS to experts27°C Bottom crate SMS to experts

DT-DSS 30°C Top crate SC and MC power cut SC one wheel, MC half wheel28°C Bottom crate SC and MC power cut SC one wheel, MC half wheel

DCS smooth Rack power off Single rackDSS 1' Rack power cut Single rack

Smoke/Fire in one of S4F01<->S4F09 racks

DSS 1' Rack power cut All racks

Smoke in USC DSS Racks power cutGas flow error DCS SMS and Email

Wrong gas mixture Gas Group Turn gas offGas overpressure Release valves Restore normal pressure

MC cooling flow DCS10 l/min each wheel

or 1 m3/h for all15' LV power off All

Rack cooling flow DCS 10 m3/h 20' LV & HV power off AllRack cooling supply over temp DSS 18°C 30' LV & HV power off All

AUG Nothing can be done

PVSS - LV mainframe communication error

Power cut of LV racks

Power cut of HV racks

SC high temperature

Smoke/Fire in rack

Pause or reset the watchdog: https://twiki.cern.ch/twiki/bin/view/CMS/DTexp

ertFAQ2009x08x04Increase the latency time of the LV Kill signal:

https://twiki.cern.ch/twiki/bin/view/CMS/DTexpertFAQ2010x03x09

Increase the latency time of the HV Kill signal: https://twiki.cern.ch/twiki/bin/view/CMS/DTexp

ertFAQ2010x03x10

Same procedure as https://twiki.cern.ch/twiki/bin/view/CMS/DTexp

ertFAQ2010x03x09 but at the end unclick the "Enable" button in the Pt1000 frame

Smooth shutdown button (one per wheel)

Same procedure as https://twiki.cern.ch/twiki/bin/

view/CMS/DTexpertFAQ2010x03x10 but change "Button timeout"

Problem Granularity• HV

• LV

Failure Fraction of DT affected

1 channel 1 layer

1 A877 1 chamber

1 A876 1 sector

1 SY1527 4 or 8 sectors (in 1 or 2 wheels)

1 HV rack 1 or 2 wheels

Failure Fraction of DT affected

1 channel (A3050 or A3009) 1 chamber

1 A3050 1 or 2 chambers

1 A3009 or 1 Easy 3000S crate 1 sector

1 MAO (except S12) 1 or 2 sector (N.B. MAO S12 = 1 wheel/2 SC crates)

1 A3100 6 sectors/1 SC crate

1 LV Rack (except X2 Near) 1 quadrant (N.B. X2 Near = 1 wheel/2 SC crates)

1 Branch Controller (A1676A) half wheel

SY1527 in S4 whole detector