CERN IT Department CH-1211 Genève 23 Switzerland t DBA Experience in a multiple RAC environment DM...

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

DBA Experience in a multiple RAC environment

DM Technical Meeting, Feb 2008

Miguel Anjo et al, CERN Physics Databases

LCG

CERN IT Department

CH-1211 Genève 23


it

Databases at CERN

Physics Databases

Infrastructure

Administrative

DBA experience in a multiple RAC environment - 2

CERN IT Department

CH-1211 Genève 23


it

Motivation

• Big shops may have similar problems:– Users from different backgrounds– Power developers like to see the DB internals– Production systems are accessible by

developers– Developers/project managers want reports– Monitoring is a key for fast reaction and problem

resolution

• Share ideas with other RAC and HA environments– Some solutions may be made available– Ideas and improvements are welcome


CERN IT Department

CH-1211 Genève 23


it

Standards

• One version, same patches everywhere (using cloning from a $ORACLE_HOME image)

• Two Oracle roles – developer and application– Dev: can do DDL

• Two Oracle profiles – developer and application– Dev: limit 10 sessions, password expiration

• Three development stage– Development: 8/5 monitoring, limited space– Integration: 8/5 monitoring, dedicated RAC for tests,

patched 2 weeks before production, consultancy– Production: 24/7 monitoring, backup to tape every hour,

limited number and scheduled interventions

• Naming convention, logbook


CERN IT Department

CH-1211 Genève 23


it

HA monitoring cluster

• 2 node mid-range server (similar to production)• Runs Oracle clusterware• Connected to shared storage• Machine alias pointing to a virtual IP

• Responsible for:– monitoring – Backups– scripts repository– Oracle binaries repository (including

$ORACLE_HOME tar images)


CERN IT Department

CH-1211 Genève 23


it

User$, guser$, all$, gall$ views

• For power users that want DBA privileges

• Based on v$/gv$ views + groups viewcreate table cern_database_groups

(group_name varchar2(30),

username varchar2(30));

• View to get users from group:create view cern_all_users as select user username from dual union (select cdg2.username from cern_database_groups cdg1, cern_database_groups cdg2 where cdg1.group_name = cdg2.group_name and cdg1.username=user);


v_$session v_$open_cursor v_$session_longops v_$sqlv_$sqlarea v_$session_wait v_$db_object_cache v_$lock

v_$sgastat v_$statname v_$latch v_$latchnamev_$latch_children v_$sysstat v_$parameter v_$mystat

CERN IT Department

CH-1211 Genève 23


it

User$, guser$, all$, gall$ views

• Example simple views:create or replace view user$session as select * from v$session where username=user;

create or replace view gall$session as select * from gv$session where username in (select username from cern_all_users);

create or replace view gall$lock as select * from gv$lock where (sid,inst_id) in (select sid,inst_id from gall$session);


CERN IT Department

CH-1211 Genève 23


it

Session Manager


CERN IT Department

CH-1211 Genève 23


it

Session Manager

• Developers want to know – “what’s happening to my session?”

• PL/SQL packaged developed in-house• Allows users to use a GUI to see his

sessions• Shows SQL, DML and DDL locks• Possible to kill user sessions (buggy)

– Creates a job that runs on appropriate instance


CERN IT Department

CH-1211 Genève 23


it

Session Manager


CERN IT Department

CH-1211 Genève 23


it

Weekly reports

DBA experience in a multiple RAC environment – 11

CERN IT Department

CH-1211 Genève 23


it

Weekly reports – Main goals

• Not OEM replacement• Targeted to application owners and

experiment DBAs• Quick overview of application/services

usage– spotting behavior changes– application contacts/responsible– Preventive monitoring


CERN IT Department

CH-1211 Genève 23


it

Weekly reports - Info shown

• List projects, users and responsible• Sessions, cpu, logical and physical reads/writes

per username• Top 5 users (CPU, physical reads, writes)• Usage per service per hour normalized

– 100% ~= 1 “DB Time” hour

• Sessions per day per username• Connection errors, space usage• Very Short and long sessions

– connection pooling

• Passwords expiring and schemas not used• Queries not using bind variables


CERN IT Department

CH-1211 Genève 23


it

Weekly reports - Setup

• AUDIT_TRAIL=DB• Base information stored at

SYS.DBA_AUDIT_SESSION / SYS.AUD$• Create schema with some dictionary access• Procedure to create aggregates - DAILY,

WEEKLY, MONTHLY - per username • Materialized views for faster report

generation• PHP page to display tables


CERN IT Department

CH-1211 Genève 23


it

Wiki pages, logbook

• Procedures

• User documentation

• Logbook


CERN IT Department

CH-1211 Genève 23


it

RACMon - monitoring


CERN IT Department

CH-1211 Genève 23


it

RAC monitoring

• Based on python, bash scripts and xml• Checks host, service, ASM disks availability• Possible to disable host or RAC

– Scheduled/unscheduled interventions– Creates blackout in OEM

• Logs on database• Sends SMS after 3 failures

• Other monitoring tools– Tablespace and user quota alarms– Alert logs’ ORA- errors daily digest email– Streams chain monitoring UI


CERN IT Department

CH-1211 Genève 23


it

Service Level Status

• Monitoring for managers• Possible to get RSS feed on status changes


CERN IT Department

CH-1211 Genève 23


it

Physics Databases numbers

• ~20 RACs (2-8 nodes, 6-35TB raw space)• 220 CPUs, 440GB RAM, 300TB disk

– soon quad-cores (3-4 node RAC)

• 345 production schemas (feb 07)• 9.625 GB production data• Largest table: 3.02 billion rows (IOT, non

partitioned)• Workload

– 5.2 million sessions/week (week 3/2008)– 128 MB/s (average for week 3/2008)– 995 CPU hours/week (CPU time avg for week

3/2008)


CERN IT Department

CH-1211 Genève 23


it

Questions

• … and answers!

• Tools presented here developed mostly by– Dawid Wojcik, Eric Grancher, Jacek Wojcieszuk,

Luca Canali, Miguel Anjo, Sebastian Lopienski

[email protected]

• Thanks, [email protected]


CERN IT Department CH-1211 Genève 23 Switzerland t DBA Experience in a multiple RAC environment DM...

Documents

Transcript of CERN IT Department CH-1211 Genève 23 Switzerland t DBA Experience in a multiple RAC environment DM...