Download - Oracle 10g Waits

Transcript
  • 8/7/2019 Oracle 10g Waits

    1/55

    1

    Wait Event Enhancements

    in Oracle 10g

    Terry Sutton and Roger Schrag

    Database Specialists, Inc.www.dbspecialists.com

  • 8/7/2019 Oracle 10g Waits

    2/55

    2

    Todays Session Twelve wait event interface enhancements

    in Oracle 10g that we really like.

    Documentation gaps in some areas make thisinformation harder to come by.

    We will assume everyone is familiar withwait event concepts and the wait event

    interface in Oracle 9i or earlier. To learn more about wait events, read:

    www.dbspecialists.com/presentations.html#wait_events

  • 8/7/2019 Oracle 10g Waits

    3/55

    3

    White Paper Contains additional detail we wont have time to

    cover today.

    Includes a handy reference list of all Oracle 10gwait events, wait classes, and wait eventparameters.

    Download: www.dbspecialists.com/presentations

  • 8/7/2019 Oracle 10g Waits

    4/55

    4

    Twelve Oracle 10g Enhancements More descriptive wait event names Wait event classes v$event_name new columns v$sql / v$sqlarea new columns v$session_wait_history v$session new columns v$event_histogram

    v$system_wait_class / v$session_wait_class Active Session History (ASH) Automatic Workload Repository (AWR) Time model statistics Improved session tracing

  • 8/7/2019 Oracle 10g Waits

    5/55

    5

    More Descriptive Names Prior to Oracle 10g many wait events had

    vague names that covered many situations:

    latch free enqueue buffer busy waits

    For these waits you had to look at parametervalues to learn the wait condition.

    Oracle 10g gives the most common of thesetypes of waits more descriptive names.

  • 8/7/2019 Oracle 10g Waits

    6/55

    6

    latch free Prior to Oracle 10g: Could indicate a wait on

    any of dozens of different latches.

    Oracle 10g: The 26 most common latcheshave their own wait event. The rest continue to use the generic latch free

    wait event.

  • 8/7/2019 Oracle 10g Waits

    7/55

    7

    latch free Prior to Oracle 10gSQL> SELECT event, state, p1, p2, p3 FROM v$session_wait WHERE sid = 162;

    EVENT STATE P1 P2 P3------------- ------- ----------- ------ -----latch free WAITING 15113593728 97 5

    SQL> SELECT * FROM v$event_name WHERE name = 'latch free';

    EVENT# NAME PARAMETER1 PARAMETER2 PARAMETER3------ ---------- --------------- --------------- ---------------

    3 latch free address number tries

    SQL> SELECT name FROM v$latch WHERE latch# = 97;

    NAME--------------------cache buffers chains

  • 8/7/2019 Oracle 10g Waits

    8/55

    8

    latch free in Oracle 10gSQL> SELECT event, state2 FROM v$session_wait3 WHERE sid = 162;

    EVENT STATE

    ------------------------------ -------latch: cache buffers chains WAITING

    The more descriptive wait event name savesus the effort of decoding wait event

    parameters.

  • 8/7/2019 Oracle 10g Waits

    9/55

    9

    enqueue Prior to Oracle 10g: Could indicate a wait on

    any of a few dozen different types of

    enqueues. Oracle 10g: 184 distinct wait events replace

    the one generic enqueue wait event: Event names differentiate the enqueue type and

    sometimes even the contention type. Parameter names are more descriptive thangeneric id1 and id2.

  • 8/7/2019 Oracle 10g Waits

    10/55

    10

    enqueue Prior to Oracle 10gSQL> SELECT event, state, seconds_in_wait FROM v$session_wait WHERE sid = 96;

    EVENT STATE SECONDS_IN_WAIT----------------------------------- ------------------- ---------------enqueue WAITING 24

    SQL> SELECT sid, CHR (BITAND (p1,-16777216) / 16777215) ||2 CHR (BITAND (p1, 16711680) / 65535) enq,3 DECODE (CHR (BITAND (p1,-16777216) / 16777215) ||4 CHR (BITAND (p1, 16711680) / 65535),5 'TX', 'Transaction (RBS)',

    ...13 CHR (BITAND (p1, 16711680) / 65535)) enqueue_name,

    14 DECODE (BITAND (p1, 65535), 1, 'Null', 2, 'Sub-Share',15 3, 'Sub-Exclusive', 4, 'Share', 5, 'Share/Sub-Exclusive',16 6, 'Exclusive', 'Other') lock_mode17 FROM v$session_wait WHERE sid = 96;

    SID ENQ ENQUEUE_NAME LOCK_MODE----- ---- ------------------------------ ----------

    96 TX Transaction (RBS) Exclusive

  • 8/7/2019 Oracle 10g Waits

    11/55

    11

    enqueue in Oracle 10gSQL> SELECT event, state, seconds_in_wait FROM v$session_wait WHERE sid = 143;

    EVENT STATE SECONDS_IN_WAIT----------------------------------- ------------------- ---------------enq: TX - row lock contention WAITING 495

    Separate events for separate TX issues:

    SQL> SELECT name, parameter1, parameter2, parameter32 FROM v$event_name3 WHERE name LIKE 'enq: TX%';

    NAME PARAMETER1 PARAMETER2 PARAME------------------------------ ------------- --------------- -------------enq: TX - contention name|mode usn

  • 8/7/2019 Oracle 10g Waits

    12/55

    12

    enqueue in Oracle 10g More descriptive parameter names, too:

    SQL> SELECT name, parameter1, parameter2, parameter32 FROM v$event_name

    3 WHERE name IN ('enq: HW - contention', 'enq: SQ - contention');

    NAME PARAMETER1 PARAMETER2 PARAME------------------------------ ------------- --------------- -------------enq: HW - contention name|mode table space # blockenq: SQ - contention name|mode object # 0

  • 8/7/2019 Oracle 10g Waits

    13/55

    13

    Wait Event Classes Every wait event belongs to one of 12 classes. Classes can help point toward a root cause.Almost 70% of all wait events belong to a classcalled Other.

    Most of the wait events that occur frequentlybelong to wait event classes with helpful names.

  • 8/7/2019 Oracle 10g Waits

    14/55

    14

    Wait Event Class NamesAdministrative Idle

    Application Network

    Cluster Scheduler Commit System I/O

    Concurrency User I/O

    Configuration Other

  • 8/7/2019 Oracle 10g Waits

    15/55

  • 8/7/2019 Oracle 10g Waits

    16/55

    16

    v$event_name Three new columns show which wait class a

    wait event belongs to:

    SQL> DESCRIBE v$event_name Name Null? Type----------------------------------------- -------- -------------------EVENT# NUMBEREVENT_ID NUMBER NAME PARAMETER1 VARCHAR2(64)PARAMETER2 VARCHAR2(64)

    PARAMETER3 VARCHAR2(64) WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS

  • 8/7/2019 Oracle 10g Waits

    17/55

    17

    v$sql and v$sqlarea Six new columns give more information about

    how time was spent executing a SQL

    statement: application_wait_time concurrency_wait_time cluster_wait_time user_io_wait_time

    plsql_exec_timejava_exec_time

    Times are in microseconds.

  • 8/7/2019 Oracle 10g Waits

    18/55

    18

    v$sqlarea Example In session #1:

    SQL> UPDATE testtab SET numcol = numcol + 1 WHERE ROWNUM < 1000;

    In session #2:SQL> UPDATE testtab SET numcol = numcol + 1 WHERE ROWNUM < 1000;

    In session #1:SQL> ROLLBACK;

    In session #2:SQL> ROLLBACK;

    In session #3:SQL> UPDATE testtab SET numcol = numcol + 1;

  • 8/7/2019 Oracle 10g Waits

    19/55

    19

    v$sqlarea ExampleSQL> SELECT sql_id, application_wait_time appl,2 concurrency_wait_time concurr, user_io_wait_time user_io3 FROM v$sqlarea4 WHERE sql_text LIKE 'UPDATE testtab SET numcol%';

    SQL_ID APPL CONCURR USER_IO------------- --------- --------- -----------038m56cp4am0c 178500000 0 20000fd5mxhdbf09ny 0 10000 105040000

    SQL> SELECT sql_id, sql_text2 FROM v$sqlarea3 WHERE sql_id IN ('fd5mxhdbf09ny','038m56cp4am0c');

    SQL_ID SQL_TEXT------------- -----------------------------------------------------------038m56cp4am0c UPDATE testtab SET numcol = numcol + 1 WHERE ROWNUM < 1000fd5mxhdbf09ny UPDATE testtab SET numcol = numcol + 1

  • 8/7/2019 Oracle 10g Waits

    20/55

    20

    v$session_wait_history New view in Oracle 10g. Similar to v$session_wait, but shows the last

    ten wait events for each session. The seq# column is supposed to show the

    order in which the waits occurred, with 1 beingthe most recent wait:

    Different from seq# in v$session. Does not appear to work as documented (on our10.1.0.3 system on Solaris).

  • 8/7/2019 Oracle 10g Waits

    21/55

    21

    v$session_wait_historySQL> DESCRIBE v$session_wait_history Name Null? Type----------------------------------------- -------- --------------------------SID NUMBERSEQ# NUMBER

    EVENT# NUMBEREVENT VARCHAR2(64)P1TEXT VARCHAR2(64)P1 NUMBERP2TEXT VARCHAR2(64)P2 NUMBERP3TEXT VARCHAR2(64)P3 NUMBER

    WAIT_TIME WAIT_COUNT

  • 8/7/2019 Oracle 10g Waits

    22/55

    22

    v$session_wait_history ExampleSQL> SELECT sid, seq#, event, wait_time, p1, p2, p32 FROM v$session_wait_history3 WHERE sid = 1544 ORDER BY seq#;

    SID SEQ# EVENT WAIT_TIME P1 P2 P3--- ---- ------------------------ ---------- ------ ------ ------154 1 db file sequential read 28 4 3547 1154 2 log buffer space 18 0 0 0154 3 log buffer space 36 0 0 0154 4 db file sequential read 0 4 3559 1154 5 db file sequential read 0 4 1272 1154 6 db file sequential read 0 4 3555 1

    154 7 log buffer space 9 0 0 0154 8 db file sequential read 0 4 3551 1154 9 db file sequential read 6 4 1268 1154 10 log buffer space 8 0 0 0

  • 8/7/2019 Oracle 10g Waits

    23/55

    23

    v$session All columns from v$session_wait have been

    added to v$session:

    Saves us the effort of joining the two views.. New blocking_session, blocking_session_status

    columns: blocking_session shows the SID of the session

    holding the resource blocking this session. blocking_session_status contains VALID when

    blocking_session is populated.

  • 8/7/2019 Oracle 10g Waits

    24/55

    24

    v$session Example Prior to Oracle 10g:

    SQL> SELECT s.sid, w.state, w.event, w.seconds_in_wait siw,2 s.sql_address, s.sql_hash_value hash_value, w.p1, w.p2, w.p3

    3 FROM v$session s, v$session_wait w4 WHERE s.sid = w.sid5 AND s.sid = 154;

    Oracle 10g:

    SQL> SELECT sid, state, event, seconds_in_wait siw,

    2 sql_address, sql_hash_value hash_value, p1, p2, p33 FROM v$session4 WHERE sid = 154;

  • 8/7/2019 Oracle 10g Waits

    25/55

    25

    Another Example Why is session 154 waiting? And who is blocking

    session 154?

    SQL> SELECT sid, blocking_session, blocking_session_status block_status,2 username, event, seconds_in_wait siw3 FROM v$session4 WHERE sid = 154;

    BLOCKINGSID _SESSION BLOCK_STATUS USERNAME EVENT SIW

    --- -------- ------------ -------- ------------------------------ ---154 157 VALID TSUTTON enq: TX - row lock contention 318

  • 8/7/2019 Oracle 10g Waits

    26/55

    26

    v$event_histogram New view in Oracle 10g. Shows instance-wide summary wait event

    statistics like v$system_event, but provides await time histogram for each. Buckets: Less than 1 mS. 1 mS to 2 mS. 2 mS to 4 mS. 4 mS to 8 mS. ... and so on

  • 8/7/2019 Oracle 10g Waits

    27/55

    27

    v$event_histogramSQL> DESCRIBE v$event_histogram Name Null? Type----------------------------------------- -------- ---------------------------EVENT# NUMBEREVENT VARCHAR2(64)

    WAIT_TIME_MILLI WAIT_COUNT

  • 8/7/2019 Oracle 10g Waits

    28/55

    28

    v$event_histogram Example How significant is the row lock contention on

    our system?

    SQL> SELECT event, total_waits, time_waited, average_wait2 FROM v$system_event3 WHERE event = 'enq: TX - row lock contention';

    EVENT TOTAL_WAITS TIME_WAITED AVERAGE_WAIT----------------------------- ----------- ----------- ------------enq: TX - row lock contention 17218 2101966 122

    Does the average wait of 1.22 secondsindicated by v$system_event paint anaccurate picture?

  • 8/7/2019 Oracle 10g Waits

    29/55

    29

    v$event_histogram ExampleSQL> SELECT event, wait_time_milli, wait_count2 FROM v$event_histogram3 WHERE event = 'enq: TX - row lock contention';

    EVENT WAIT_TIME_MILLI WAIT_COUNT

    ----------------------------- --------------- ----------enq: TX - row lock contention 1 833enq: TX - row lock contention 2 635enq: TX - row lock contention 4 372enq: TX - row lock contention 8 395enq: TX - row lock contention 16 781enq: TX - row lock contention 32 3729enq: TX - row lock contention 64 3050

    enq: TX - row lock contention 128 410enq: TX - row lock contention 256 47enq: TX - row lock contention 512 46enq: TX - row lock contention 1024 37enq: TX - row lock contention 2048 3enq: TX - row lock contention 4096 6880

  • 8/7/2019 Oracle 10g Waits

    30/55

    30

    v$system_wait_class and

    v$session_wait_class New views in Oracle 10g. Show wait counts and total time waited since

    instance startup and session start. Similar to v$system_event and v$session_event,

    but summarized by wait class. Times are in centiseconds.

  • 8/7/2019 Oracle 10g Waits

    31/55

    31

    v$system_wait_class and

    v$session_wait_classSQL> DESCRIBE v$system_wait_class Name Null? Type----------------------------------------- -------- --------------------------- WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS TOTAL_WAITS NUMBERTIME_WAITED NUMBER

    SQL> DESCRIBE v$session_wait_class Name Null? Type----------------------------------------- -------- ---------------------------SID NUMBERSERIAL# NUMBER WAIT_CLASS_ID WAIT_CLASS# WAIT_CLASS TOTAL_WAITS NUMBERTIME_WAITED NUMBER

  • 8/7/2019 Oracle 10g Waits

    32/55

    32

    v$system_wait_class ExampleSQL> SELECT wait_class, time_waited2 FROM v$system_wait_class3 ORDER BY time_waited DESC;

    WAIT_CLASS TIME_WAITED

    ------------- -----------Idle 777450022System I/O 1261584User I/O 116667Configuration 116481 Application 72301Other 12432Commit 3496

    Concurrency 319 Network 1

  • 8/7/2019 Oracle 10g Waits

    33/55

    33

    Active Session History New MMNL background process samples

    v$session once each second.

    Data captured on active sessions is availablein v$active_session_history, usually for 1-4+hours.

    Automatic Workload Repository captures one

    out of every ten samples in its hourlysnapshots.

  • 8/7/2019 Oracle 10g Waits

    34/55

    34

    Active Session HistorySQL> DESCRIBE v$active_session_historyName Null? Type

    ----------------------------------------- -------- ---------------------------SAMPLE_ID NUMBER

    SAMPLE_TIME TIMESTAMP(3)SESSION_ID NUMBER

    SESSION_SERIAL# NUMBERUSER_ID NUMBER

    SQL_ID VARCHAR2(13)

    SQL_CHILD_NUMBER NUMBERSQL_PLAN_HASH_VALUE NUMBERSQL_OPCODE NUMBER

    SERVICE_HASH NUMBERSESSION_TYPE VARCHAR2(10)

    SESSION_STATE VARCHAR2(7)QC_SESSION_ID NUMBER

    QC_INSTANCE_ID NUMBEREVENT VARCHAR2(64)

    EVENT_ID NUMBEREVENT# NUMBER

    SEQ# NUMBERP1 NUMBER

    P2 NUMBERP3 NUMBER

    WAIT_TIME NUMBERTIME_WAITED NUMBER

    CURRENT_OBJ# NUMBERCURRENT_FILE# NUMBER

    CURRENT_BLOCK# NUMBERPROGRAM VARCHAR2(48)

    MODULE VARCHAR2(48)

    ACTION VARCHAR2(32)CLIENT_ID VARCHAR2(64)

  • 8/7/2019 Oracle 10g Waits

    35/55

    35

    Active Session History v$active_session_history shows detailed wait

    information, current SQL statement,

    object/file/block information, and more. When a wait that was sampled by ASH

    completes, Oracle fills in the time_waitedcolumn in the v$active_session_history row

    with the actual duration of the wait.

  • 8/7/2019 Oracle 10g Waits

    36/55

    36

    ASH is Always On You may not have to turn on tracing or begin

    monitoring v$ views when users report a

    problem, becauseA

    SH is already samplingdata. In some cases you may be able to diagnose

    and resolve a problem on first detection, even

    if you learn of the problem after the fact. You may not need to get users to reproduce theproblem.

  • 8/7/2019 Oracle 10g Waits

    37/55

    37

    Sampling versus Collecting ASH samples v$session once each second.

    Very different from extended SQL trace, which

    collects data on every wait and every OC

    I call. A session could encounter hundreds of waits inone second, and ASH will only capture data forone of them.

    Use ASH to view aggregate data on many

    sessions over a period of time. Dont use ASH to count waits, get maximum

    wait times, or look at a short time period.

  • 8/7/2019 Oracle 10g Waits

    38/55

    38

    ASH ExampleSQL> SELECT DECODE (session_state, 'WAITING', event, NULL) event,2 session_state, COUNT(*), SUM (time_waited) time_waited3 FROM v$active_session_history4 WHERE module = 'ARXENV'5 AND sample_time > SYSDATE - (2/24)

    6 GROUP BY DECODE (session_state, 'WAITING', event, NULL),7 session_state;

    EVENT SESSION_STATE COUNT(*) TIME_WAITED------------------------------ ------------- -------- -----------

    ON CPU 124 0log file sync WAITING 2 52686db file scattered read WAITING 2 28254

    db file sequential read WAITING 1 6059control file sequential read WAITING 1 9206SQL*Net break/reset to client WAITING 1 9140enq: TX - row lock contention WAITING 922 930864016

  • 8/7/2019 Oracle 10g Waits

    39/55

    39

    Automatic Workload Repository Conceptually similar to Statspack. Collects snapshots of system-wide performance

    statistics, plusA

    SHs sampling of session activity. You can generate activity reports using awrrpt.sql

    or Enterprise Manager interface. Out of the box, Oracle 10g databases collect

    snapshots hourly and retain data for seven days.

  • 8/7/2019 Oracle 10g Waits

    40/55

    40

    AWRM

    anagement Snapshot data is stored in SYS-owned tables

    in the SYSAUX tablespace.

    Use dbms_workload_repository to: Collect a snapshot on demand. Purge snapshots. Adjust snapshot interval. Adjust snapshot retention period.

    Disable AWR snapshot collection.

  • 8/7/2019 Oracle 10g Waits

    41/55

    41

    AWR versus Statspack AWR benefits:

    More tightly integrated into Oracle kernel, reducingsnapshot collection overhead.

    Snapshots include sampling ofASH data. Data collected by AWR is accessible via views with

    names starting DBA_HIST. DBA_HIST views are documented.

    Statspack has been updated to be 10gaware.

  • 8/7/2019 Oracle 10g Waits

    42/55

    42

    TimeM

    odel Statistics New concept in Oracle 10g. Two new views provide a more detailed

    picture of how Oracle spends time: Separates out background and user processes. Provides data not previously available, such as

    how much time was spent in PL/SQL execution.

    Times are in microseconds.

  • 8/7/2019 Oracle 10g Waits

    43/55

  • 8/7/2019 Oracle 10g Waits

    44/55

    44

    TimeM

    odel Statistics Some important notes about these statistics:

    Statistics do not include background processesunless background appears in the statistic name.

    DB time shows elapsed time spent in Oraclecalls. (Basically CPU time plus non-idle waits.)

  • 8/7/2019 Oracle 10g Waits

    45/55

    45

    TimeM

    odel Statistics ExampleSQL> SELECT stat_name, value / 1000000 seconds FROM v$sys_time_model2 ORDER BY seconds DESC;

    STAT_NAME SECONDS------------------------------------------------ ----------DB time 80970.190sql execute elapsed time 75057.271

    DB CPU 44448.628 background elapsed time 29333.160PL/SQL execution elapsed time 8824.538 background cpu time 5170.311 parse time elapsed 1270.147hard parse elapsed time 838.068PL/SQL compilation elapsed time 176.731sequence load elapsed time 112.334connection management call elapsed time 44.644failed parse elapsed time 11.946hard parse (sharing criteria) elapsed time 5.579hard parse (bind mismatch) elapsed time 4.610failed parse (out of shared memory) elapsed time 0.000Java execution elapsed time 0.000inbound PL/SQL rpc elapsed time 0.000

  • 8/7/2019 Oracle 10g Waits

    46/55

  • 8/7/2019 Oracle 10g Waits

    47/55

    47

    Tracing Prior to Oracle 10g Tracing sessions in Oracle 9i and earlier had

    annoyances: dbms_support usually missing or not installed. dbms_system.set_ev not officially supported. ALTER SESSION SET EVENTS to set 10046

    event has a syntax hard (for some of us) toremember.

  • 8/7/2019 Oracle 10g Waits

    48/55

    48

    Easier Tracing in Oracle 10g Control tracing of your own session:

    EXECUTE dbms_monitor.session_trace_enable (waits=>TRUE, binds=>TRUE);EXECUTE dbms_monitor.session_trace_disable ();

    Control tracing of another session:

    EXECUTE dbms_monitor.session_trace_enable (session_id=>153, waits=>TRUE);EXECUTE dbms_monitor.session_trace_disable (session_id=>153);

    Tracing sessions in Oracle 10g is easier becausedbms_monitor is provided and installed in everyOracle 10g database by default.

  • 8/7/2019 Oracle 10g Waits

    49/55

    49

    Tracing Prior to Oracle 10g Straightforward if session being traced uses

    dedicated server connection.

    Tracing session using shared serverarchitecture leads to multiple trace files thathave to be merged manually.

    Tracing an end-user session that does not

    map to one Oracle session (eg: connectionpooling or multiplexing) is not practical.

  • 8/7/2019 Oracle 10g Waits

    50/55

    50

    New Tracing Features in 10g Tracing can be enabled for sessions with a

    specific client identifier:EXECUTE dbms_session.set_identifier ('client identifier');

    EXECUTE dbms_session.clear_identifier ();

    Tracing can be enabled for sessions with specificservice / module / action combination:EXECUTE dbms_application_info.set_module ('module name', 'action name');EXECUTE dbms_application_info.set_action ('action name');

    Trace data split over multiple trace files can bemerged into one trace file for TKPROF processing:$ trcsess output= [clientid=] [service=] [module=] [action=] [session=]

  • 8/7/2019 Oracle 10g Waits

    51/55

    51

    Connection Pooling Example Application server connection pooling code:EXECUTE dbms_session.set_identifier ('session_id from application table');...do the work for this end user session...EXECUTE dbms_session.clear_identifier ();

    Trace the end-user session assigned session_id12345 in the applications sessions table:

    SQL> EXECUTE dbms_monitor.client_id_trace_enable -> (client_id=>'12345', waits=>TRUE, binds=>TRUE);

    Consolidate trace data into one trace file suitablefor use by TKPROF:

    $ cd $ORACLE_BASE/admin/$ORACLE_SID/udump$ trcsess output=mytracefile.trc clientid=12345

  • 8/7/2019 Oracle 10g Waits

    52/55

    52

    Twelve Oracle 10g Enhancements More descriptive wait event names Wait event classes v$event_name new columns

    v$sql / v$sqlarea new columns v$session_wait_history v$session new columns v$event_histogram v$system_wait_class / v$session_wait_class Active Session History (ASH) Automatic Workload Repository (AWR) Time model statistics Improved session tracing

  • 8/7/2019 Oracle 10g Waits

    53/55

  • 8/7/2019 Oracle 10g Waits

    54/55

    54

    White Paper Contains additional detail we didnt have time to

    cover today.

    Includes a handy reference list of all Oracle 10gwait events, wait classes, and wait eventparameters.

    Download: www.dbspecialists.com/presentations

  • 8/7/2019 Oracle 10g Waits

    55/55

    55

    Contact InformationTerry Sutton and Roger Schrag

    Database Specialists, Inc.

    388M

    arket Street, Suite 400San Francisco, CA 94111

    Tel: 415/344-0500Toll-free: 888/648-0500

    [email protected]@dbspecialists.com