DEEP DIVE INTO 12c MATCH RECOGNIZE - Oracle...Do You Need An ALWAYS TRUE Pattern Variable? • Some...

Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.|

LastUpdatedJune2017–Version12

GettingdeepinsidetheSQLpatternmatchingprocessinDatabase12c

DEEPDIVEINTO12cMATCH_RECOGNIZE

–@ASQLBarista

–oracle-big-data.blogspot.com


WhoAmI

• KeithLaker• SeniorPrincipalProductManager,AnalyticSQL• Oracle

• Twitter:@ASQLBarista• Blog:http://oracle-big-data.blogspot.com• Email:[email protected]

2


Purposeofthispresentationisto……

Makeyousuccessfulwith12cpatternmatching

3


Agenda

SQLPatternMatching–quickrecap

Usingbuilt-inmeasurestounderstandyourpattern

Greedyvs.reluctantquantifiers

Outputs:ALLROWSvs.ONEROW

Outputs:Unmatchedvs.emptymatches

SKIPtowhereexactly?

Preparingtorunaquery

1

2

3

4

5

6

7

12

Predicates:how,whereandwhen

Minimizingnumberofsorts

Understandingstatemachines

StatemachinesandtheOptimizer

Whatisbacktracking?

Recommendationsfortesting

Summary

8

9

10

11

14

13

4


AppleiBook+PDF• Introduction• Usecases• Yourfirstsimplepattern

– SyntaxofMATCH_RECOGNIZE– Gettingstarted– FoundationsofMATCH_RECOGNIZE– Goingalittledeeper– SearchingforV-shapedpatterns– Listingyouroutputcolumns

AvailableshortlyviaOTNAnalyticSQLpageandiBookStore5


AppleiBook+PDF• Introductionandrecap• Usingbuilt-inmeasures• Greedyvs.reluctantquantifiers• Emptymatchesandunmatchedrows

• Skiptowhere?• Statemachinesandbacktracking• Runningaquery

– explainplans,MVs,usingpredicates…

AvailableshortlyviaOTNAnalyticSQLpageandiBookStore6

Copyright©2017,Oracleand/oritsaffiliates.Allrightsreserved.| 7

QuickrecapBackground


MysampleTICKERdataset

8

0

8

15

23

30

2-Apr-15 4-Apr-15 6-Apr-15 8-Apr-15 10-Apr-15 12-Apr-15 14-Apr-15 16-Apr-15 18-Apr-15 20-Apr-15

ACMEGLOBEXOSCORP


Tutorialsforeachsectiononlivesql.us.oracle.com

9

http://livesql.oracle.comisaCOMPLETELYFREEservicethatgivesyouaccesstotheverylatestversionofOracleDatabase

Sampleschemasarepreloaded,youcanaddyourowntables,writeyourownscripts,saveyourscripts,runtutorialsetc

http://livesql.us.oracle.com

http://livesql.oracle.com



QuickoverviewofMATCH_RECOGNIZEsyntax

RECAP:UsingMATCH_RECOGNIZEforPatternMatching


FindingVshapedpatternsinatickerstreamdataset

SELECT symbol, tstamp, price, first_down, first_price, last_up, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, FINAL LAST(up.tstamp) AS last_up, FINAL LAST(up.price) as last_price ALL ROWS PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))ORDER BY symbol, tstamp;

11

Usingnew12cMATCH_RECOGNIZEtosearchforpatterns

NewMATCH_RECGONIZEclausewithtableTICKERasinput


Overviewof12cMATCH_RECOGNIZEsyntax


12

Partitioningandorderingyourdata

Orderedandpartitionedstreamofrows




13

Describingyourpattern

Variablenamesandoperators





14

Definingyourpattern



Variabledefinition




15

DefiningvaluestoreturntoSELECTstatement


Measures


Variabledefinition




16

DetermininghowmuchdetailtoreturntoSELECTstatement

Controllingoutput:levelofdetail


GettingConsistentResults• ORDERBYclauseisoptionalinsyntax(andANSIproposal)

• MydataisalreadysortedsoIdon’tneedORDERBY-correct?– TemptingtoignoreORDERBYclauseandassumedatawillbecorrectlyordered–WithoutORDERBY,consistentresultsarenotguaranteed!

• AlwaysincludeORDERBYclause– IforderoftworowsinapartitionisnotdeterminedbyORDERBYresults(non-uniqueorderbykey),theresultwillbenon-deterministic

– Ifyouhavenonuniqueorderbykeyswithinpartition,consideraddingadditionalorderbycolumnstomakeorderbyuniqueanddeterministic

– IfOraclecansuppresstheorderbythenitwilldoso!

17


MostquerieswilluseONE ROW PER MATCHsyntax….SELECT symbol, start_tstamp, end_tstamp, match_numFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’OSCORP’;

18


…whichproducesasummaryreport

Notveryusefulforshowinghowyourpatternisbeingmatchedasrowsareprocessed.Howdoyoudebugapatternmatchingquery?

Allowsuseofbuilt-inmeasures:MATCH_NUMBER()andCLASSIFIER()19


OtherquerieswilluseALL ROWS PER MATCHsyntax….SELECT symbol, tstamp, price, start_tstamp, end_tstamp, match_num, classifierFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, MATCH_NUMBER() AS match_num, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’GLOBEX’;

20

NOTE:ALWAYSTRUESTRTvariablehasbeenremoved,checkstartingdateforpatterncomparedwithreportonpreviouspage


…whichproducesamoredetailedreport

Moreusefulforshowinghowyourpatternisbeingmatchedsoperfectfordebuggingapatternmatchingquery.

AllowsadditionalmeasuressuchasCLASSIFIER()andMATCH_NUMBER()toprovidemorefeedback

21


…moredetailedreportshowmatchedrowsand…

MATCHEDrows

22


…moredetailedreportalsoshowsunmatchedrows

UNMATCHEDrows

23


Howtodetermineifyouneedanalwaystruepatternvariable

RECAP:ALWAYSTRUEvariables


DoYouNeedAnALWAYSTRUEPatternVariable?• Someexamplesonlivesql.oracle.comuseALWAYSTRUEpatternvariable

– e.gsearchingforV-shapedpatternsintickerdata…

PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price))

NOTE:thereisnodefinitionforSTRTwithinDEFINEclause• WhatisthepointofanALWAYSTRUEevent?

– Needtocapturecaptureastartingrowtoenableacomparisontest.i.e>=PREV()– WithoutALWAYSTRUEpatternvariablematchwillprobablystart1rowlaterthanexpected

25



SearchingForV-PatternwithALWAYSTRUEPatternVariableSELECT symbol, tstamp, price, start_tstamp, end_tstamp, match_num, classifierFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, MATCH_NUMBER() AS match_num, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’GLOBEX’;

26


PatternStartsMatchingV-PatternatRow3

27

Notethatthepatternstartsonrow3

Row1:matchedtoSTRT

Row2:price12isgreaterthanprice11,Row1-DOWNcannotbematched.Patternfails.

RestartmatchingatRow2:Row2:matchedtoSTRT

Row3:price13isgreaterthanprice12,Row2-DOWNcannotbematched.Patternfails.

RestartmatchingatRow3:Row3:matchedtoSTRT

Row4:price12islessthanprice13,Row3-DOWNismatched.Patterncontinues…

continuesmatchingV-patternuptorow15

1

2

3

4

…

5

6


SearchingForV-PatternwithoutALWAYSTRUEPatternVariableSELECT symbol, tstamp, price, start_tstamp, end_tstamp, match_num, classifierFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, MATCH_NUMBER() AS match_num, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’GLOBEX’;

28

NOTE:ALWAYSTRUESTRTvariablehasbeenremoved,checkstartingdateforpatterncomparedwithreportonpreviouspage


PatternNowStartsMatchingV-PatternatRow4

29

Notethatthepatternnowstartsonrow4

Row1:previousrowisnotvalid-DOWNcannotbematched

Row2:price12isgreaterthanprice11,Row1-DOWNcannotbematched

Row3:price13isgreaterthanprice12,Row2-DOWNcannotbematched

Row4:price12islessthanprice13,Row3-DOWNismatched

ContinuesmatchingV-patternuptorow15

1

2

3

4

…


Usingbuilt-indebuggingtoolstohelpyouunderstandthepatternmatchingprocess

MATCH_NUMBER()CLASSIFIER()

Built-inMeasures


Twokeybuilt-inmeasurestohelpwithdebugging1. MATCH_NUMBER()

– Returnsanintegertoshowwhichrowsaremembersofwhichmatch– Assignsthesamenumbertoeachrowofaspecificmatch– Forinstance,alltherowsinthefirstmatchfoundinarowpatternpartitionareassignedthematchnumbervalueof1

– Notethatmatchnumberingstartsoveragainat1ineachrowpatternpartition

1. CLASSIFIER()– Showswhichrowsmaptowhichvariable

31


Examplecodeusingtwobuilt-inmeasures

SELECT symbol, tstamp, price, mn, pattern, first_down, first_price, last_up, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES MATCH_NUMBER() AS mn, CLASSIFIER() as pattern, FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, FINAL LAST(up.tstamp) AS last_up, FINAL LAST(up.price) as last_price ALL ROWS PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))ORDER BY symbol, tstamp;

32

FindingVshapedpatternsinatickerstream


1.MATCH_NUMBER()• MATCH_NUMBERassignsthesamenumbertoeachrowofaspecificmatch– Firstmatchofcompletepatternfoundinapartitionassignedmatch_number()valueof1

– Nextmatchgetsvalueof2,etc.

• Notethatmatchnumberingstartsoveragainat1ineachrowpatternpartition

33

PART

ITION

1PA

RTITION

2


2.CLASSIFIER()• CLASSIFIER()showswhichrowsmaptowhichvariable:STRT,DOWNorUP

• Inthisexample,– rows1,7,11,15maptovariableSTRT– Rows2,8,12,16maptovariableDOWN– remainingrowsmaptovariableUP

34


What’stheimpactofusingONEROWPERMATCH?SELECT symbol, tstamp, price, mn, pat, first_down, first_price, last_up,last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES price as price, tstamp as tstamp, MATCH_NUMBER() AS mn, CLASSIFIER() as pat, FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, FINAL LAST(up.tstamp) AS last_up, FINAL LAST(up.price) as last_price ONE ROW PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))ORDER BY symbol, tstamp;

35


What’stheimpactofusingONEROWPERMATCH?

36

Note:CLASSIFER()canbeusedwithONE ROW PER MATCHButonlylastpatternvariableisreturned(inthiscaseUP)-sonotreallyveryuseful


UsingaggregateswithCLASSIFIER()SELECT symbol, tstamp, price, mn, pat, first_down, first_price, last_up,last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES price as price, tstamp as tstamp, MATCH_NUMBER() AS mn, FIRST(CLASSIFIER()) as f_pat, LAST(CLASSIFIER()) as l_pat FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, FINAL LAST(up.tstamp) AS last_up, FINAL LAST(up.price) as last_price ONE ROW PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))ORDER BY symbol, tstamp;

37


UsingaggregateswithCLASSIFIER()

ORA-62507: illegal use of MATCH_NUMBER or CLASSIFIER in MATCH_RECOGNIZE clause

62507. 00000 - "illegal use of MATCH_NUMBER or CLASSIFIER in MATCH_RECOGNIZE clause"*Cause: An attempt was made to use MATCH_NUMBER or CLASSIFIER in the MATCH_RECOGNIZE clause. CLASSIFIER and MATCH_NUMBER can only be used in the MEASURES clause. CLASSIFIER can only be used if the query is ALL ROWS PER MATCH. CLASSIFIER and MATCH_NUMBER cannot be used inside aggregates/FIRST/LAST/PREV/NEXT.

*Action: Modify the query and retry the operation.

38

Note-notactuallytrue.Onlythelastpatternvariablethatismappedisshownintheoutput!


CanIuseMATCH_NUMBERinDEFINEClause?

• Shortansweris:Yes• Longansweris:Itdepends…• Followingispossible:

PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price) and match_number() = 1), UP AS (price >= PREV(price))

•

39


UsingMATCH_NUMBER()=1

40


CanIuseMATCH_NUMBERinDEFINEClause?• Followingispossiblebutdoesnotfindanymatches:

PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price) and match_number() = 2), UP AS (price >= PREV(price))

41


UsingMATCH_NUMBER()=2->Norowsreturned

42


CanYouStopSearchingAfterN’thMatch?• Canyoustopsearchingafterthefirstorsecondorn’thmatch?

AfteramatchisfoundIwouldlikematch_recognizetostopsearching-Iwantatmostonematchperpartition.Idon’twanttofilterbyMATCH_NUMBER()inanouterquery-thatistoowasteful(or,insomecases,Imayknowinadvancethatthereisatmostonematchperpartition,andIdon’twantmatch_recognizetowastetimesearchingformorematcheswhichIknowdon'texist).

• CanMATCH_RECOGNIZEdothisusingMATCH_NUMBER?–Shortansweris:NO.–Longansweris:StillNO.

43


Thefullresultsetwithnofiltering• WanttouseMATCH_RECOGNIZEtoonlyreturnthefirstmatchwithineachpartition– Rowsboundedbyredboxesshouldnotappearintheoutput

• CanMATCH_RECOGNIZEdothisusingMATCH_NUMBER?–Shortansweris:NO.–Longansweris:StillNO.

44


WillThisQueryStopAfter1stMatch?SELECT symbol,tstamp, price, match_number, classifier, first_x, last_yFROM tickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(y.tstamp) AS last_y, MATCH_NUMBER() AS match_number, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS AFTER MATCH SKIP PAST LAST ROW PATTERN (strt X+ Y) DEFINE X AS (price <= PREV(price)), Y AS (price >= PREV(price)))WHERE match_number = 1ORDER BY symbol, match_number, tstamp asc;

45


LooksPromising….onlyfirstmatchisreturnedforeachsymbol

46


BUThavewesavedanyprocessing?• Thatistosay:didMATCH_RECOGNIZEstopsearchingformatchesafterthefirstmatchwasfound?

NO!• Checkingtheexplainplanwecanseethatall60rowsfromourtablewhereprocessed:

47


GoingDeeper,ALotDeeperThereisalotmoreinformationonthistopicinthefollowingblogpost:• BlogPost…MATCH_RECOGNIZE:CanIuseMATCH_NUMBER()asafilter?

• https://oracle-big-data.blogspot.co.uk/2017/03/matchrecognize-can-i-use-matchnumber-as.html

48

https://oracle-big-data.blogspot.co.uk/2017/03/matchrecognize-can-i-use-matchnumber-as.html














Usinggreedyandreluctantquantifiersinyourpatterndefinition

GreedyQuantifiers


DefiningPATTERNs• PATTERN componentisusedtospecifyregularexpressions• Regularexpressionisbuiltfromvariablenamesandoperators

–Operatorscanbeconcatenation,grouping,alternation,permutes,quantifiers,…– Alargelibraryofbuilt-onquantifiersisavailable– Regularexpressionsareamazinglypowerfulanddeeplyexpressive

• Whatisaregularexpression?– aregularexpression(sometimescalledarationalexpression)isasequenceofcharactersthatdefineasearchpattern,mainlyforuseinpatternmatchingwithstrings,orstringmatching,i.e."findandreplace”-likeoperations

50

Wikipedia:https://en.wikipedia.org/wiki/Regular_expression


Whatisaregularexpression?• Regularexpressionsusedtospecifyasetofstrings(tokensandquantifiers)requiredforaparticularpurpose

• Quantifierafteratokenorgroupspecifieshowoftenthatprecedingelementisallowedtooccur

• Mostcommonquantifiersare:–Questionmark,indicateszerooronematch– Asterisk,indicatesneedforzeroormorematches– Plussign,indicatesneedforoneormorematches

• Oracle'sregularexpressionsareslightlydifferent– RowpatternvariablesaredefinedbyBooleanconditionsratherthancharactersorsetsofcharacters

51


Quantifiersusedin PATTERN clause

• POSIX basic and extended quantifiers:

– * 0 or more matches – + 1 or more matches – ? 0 or 1 match – {n} exactly n matches – {n,} n or more matches – {n, m} at least n but not more than m (inclusive) matches – {, m} at most m (inclusive) matches

52


Howtousequantifiers

• The following are examples of using quantifiers: – A? matches 0 or 1 iteration of variable A – A* matches 0 or more iterations of variable A – A+ matches 1 or more iterations of variable A – A{3,6} matches 3 to 6 iterations of variable A – A{,4} matches 0 to 4 iterations of variable A

• A is defined in the DEFINE component of the MATCH_RECOGNIZE clause – For example: A AS (price <= PREV(price))

53


• Pattern quantifiers are referred to as greedy – Attempt to match as many instances as possible of the regular expression on

which they are applied

• Reluctant quantifiers use a question mark ? as additional suffix – Attempt to match as few instances as possible of the regular expression on

which they are applied

• Convert greedy to reluctant quantifier by adding additional “?” – Examples: ?? or *? or +? or {n, m }?

Greedyandreluctantquantifiers

54


Example–usinggreedyquantifiers

SELECT symbol, tstamp, price, mn, pattern, first_down, first_price, last_up, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES MATCH_NUMBER() AS mn, CLASSIFIER() as pattern, FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, LAST(up.tstamp) AS last_up, LAST(up.price) as last_price ALL ROWS PER MATCH PATTERN (strt down+ up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))WHERE symbol = 'ACME’ORDER BY symbol, tstamp;

55

FindingVshapedpatternsinatickerstreamusingplus-signgreedyquantifier

Usingthe“+”greedyquantifier


Usinggreedyquantifiers

56

MatchingtovariableDOWNtakesprecedenceoverUPonrow130

8

15

23

30

Conflict:horizontalareacouldbemappedto

DOWNorUP

Result:GreedyDOWNmatchesasmany

instancespossibletoDOWNbeforematchingto

UP


Example-usingareluctantquantifier

SELECT symbol, tstamp, price, mn, pattern, first_down, first_price, last_up, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES MATCH_NUMBER() AS mn, CLASSIFIER() as pattern, FIRST(strt.tstamp) AS first_down, FIRST(strt.price) as first_price, LAST(up.tstamp) AS last_up, LAST(up.price) as last_price ALL ROWS PER MATCH PATTERN (strt down+? up+) DEFINE down AS (price <= PREV(price)), up AS (price >= PREV(price)))WHERE symbol = 'ACME’ORDER BY symbol, tstamp;

57

Usingquestion-marktoindicatereluctantquantifier

WhathappensifDOWNuses?tomakequantifier

‘reluctant’...


Usingareluctantquantifiers

58

MatchingtovariableUPtakesprecedenceoverDOWNonrow130

8

15

23

30

RecordsmatchestoUPbeforeconsidering

reluctantDOWNagainafterhavingfoundonematchofDOWNalready(pattern

satisfied)


Returningonlytherowsyouareinterestedin...withemptymatchesvs.omitemptymatchesvs.unmatchedrowsvs.!

OUTPUT:ONEROWvs.ALLROWS...


MostquerieswilluseONE ROW PER MATCHsyntax….SELECT symbol, start_tstamp, end_tstamp, match_numFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH PATTERN (STRT DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’OSCORP’;

60


…whichproducesasummaryreport

Notveryusefulforshowinghowyourpatternisbeingmatchedasrowsareprocessed.Howdoyoudebugapatternmatchingquery?

Allowsuseofbuilt-inmeasures:MATCH_NUMBER()andCLASSIFIER()61


OtherquerieswilluseALL ROWS PER MATCHsyntax….SELECT symbol, tstamp, price, start_tstamp, end_tstamp, match_num, classifierFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(up.tstamp) AS end_tstamp, MATCH_NUMBER() AS match_num, CLASSIFIER() AS classifier ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (DOWN+ UP+) DEFINE DOWN AS (price <= PREV(price)), UP AS (price >= PREV(price)))WHERE symbol = ’GLOBEX’;

62


…whichproducesamoredetailedreport

Moreusefulforshowinghowyourpatternisbeingmatchedsoperfectfordebuggingapatternmatchingquery.

AllowsadditionalmeasuressuchasCLASSIFIER()andMATCH_NUMBER()toprovidemorefeedback

63


Whatisthedifferencebetweenanunmatchedrowandanemptymatch?

OUPTUT:EMPTYvs.UNMATCHEDROWS


EmptyMatchesvs.UnmatchedRowsTwooutputoptionsforMATCH_RECOGNIZE• ONEROWPERMATCH:eachmatchproducesonesummaryrow.Thisisthedefault.

• ALLROWSPERMATCH:amatchspanningmultiplerowswillproduceoneoutputrowforeachrowinthematch– ALLROWSPERMATCHSHOWEMPTYMATCHES<-notethatthisisthedefault– ALLROWSPERMATCHOMITEMPTYMATCHES– ALLROWSPERMATCHWITHUNMATCHEDROWS

65


EmptyMatchesvs.UnmatchedRowsHowdoyougetan“emptymatch”?• Determinedbythetypeofquantifierusedaspartofthepatterndefinition

– Currentlypossibletogetemptymatcheswithoutquantifiers• usingapatternlikePATTERN(^|A)

– Possibletogetunmatchedrowswithoutusingquantifiers

• Bychangingthequantifieritispossibletoproducethesimilarresultusingbothsetsofkeywords:

– ALLROWSPERMATCHSHOWEMPTYMATCHES<-notethatthisisthedefault– ALLROWSPERMATCHWITHUNMATCHEDROWS

66


EmptyMatchesSELECT. . .FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp), LAST(down.tstamp), match_number() AS match_num, classifier() AS classifier ALL ROWS PER MATCH SHOW EMPTY MATCHES PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

67

Emptymatches:classifierisNULLeventhoughmatch_number()returnsavalue


Whatisanemptymatch?• Emptymatchisarowthatdoesnotmapexplicitlytoapatternvariable

– inpreviousexample–DOWN

• Usuallyresultofusingspecificquantifier:*(asterisk).– GiventhatDOWNvariablecanbematchedzeroormoretimesthereisopportunityforanemptymatchtooccur.

• Emptymatchdoesinfacthaveastartingrow– itisassignedasequentialmatchnumber,basedontheordinalpositionofitsstartingrow

68


EmptyMatchesandMATCH_NUMBERSELECT. . .FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp), LAST(down.tstamp), match_number() AS match_num, classifier() AS classifier ALL ROWS PER MATCH OMIT EMPTY MATCHES PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

69

Notethatmatch_numberisnotcontiguous


ImpactofEmptyMatchesonAnalyticsSELECT *FROM tickerMATCH_RECOGNIZE (PARTITION BY symbol ORDER BY tstampMEASURES FIRST(down.tstamp) AS start_time, LAST(down.tstamp) AS end_time, COUNT(*) AS counter_all, avg(down.price) as avg_price, MATCH_NUMBER() as mn, CLASSIFIER() as patALL ROWS PER MATCH SHOW EMPTY MATCHESPATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)))WHERE symbol = 'GLOBEX';

70

NotethatCOUNTandAVGonlyfunctionwhereCLASSIFIERreturnsavalue


SummaryofEmptyMatchesThreemainspointstorememberwhenyourpatternpermitsthistypeofmatching:1. ValueofMATCH_NUMBER()issequentialmatchnumberoftheemptymatch2. AnyCOUNTorsimilarfunction(MAX,MIN,AVGetc.)willreturn03. Anyotheraggregate,rowpatternnavigationoperation,orordinaryrowpatterncolumn

referenceisnull

• Thedefaultisalwaystoreturnemptymatches– Determinefromstarthowyouwanttomanagetheserows:

• includethem(SHOWEMPTYMATCHES)• excludethem(OMITEMPTYMATCHES).

– BecarefulifyouareusingMATCH_NUMBER()withintheDEFINEsectionaspartofaformulabecauseemptymatchesincrementtheMATCH_NUMBER()counter.

71


ViewingUnmatchedRows• Alwaysusefultoviewthecompleteresultset

– atleastwhenyouarerunningyourcodeagainsttestdatasets.

• Gettingalltheinputrowsintoyouroutputisrelativelyeasybecauseyoujustneedtoincludethephrase:– ALLROWSPERMATCHWITHUNMATCHEDROWS.

• OtherthanfortestingpurposesIcan’tthinkofagoodusecaseforusingthisinproductionsomakesureyoucheckyourcodebeforesubmittingyourproduction-readycodetoyourDBA.

72


RECAP:Quantifiersusedin PATTERN clause

• POSIX basic and extended quantifiers:

– * 0 or more matches – + 1 or more matches – ? 0 or 1 match – {n} exactly n matches – {n,} n or more matches – {n, m} at least n but not more than m (inclusive) matches – {, m} at most m (inclusive) matches

73


RECAP:Howtousequantifiers

• The following are examples of using quantifiers: – A? matches 0 or 1 iteration of variable A – A* matches 0 or more iterations of variable A – A+ matches 1 or more iterations of variable A – A{3,6} matches 3 to 6 iterations of variable A – A{,4} matches 0 to 4 iterations of variable A

• A is defined in the DEFINE component of the MATCH_RECOGNIZE clause – For example: A AS (price <= PREV(price))

74


EmptyMatchesandUnmatchedRowsInSameQuery?SELECT symbol, tstamp, price, mnm, nmr, cls FROM ticker MATCH_RECOGNIZE( PARTITION BY symbol ORDER BY tstamp MEASURES match_number() AS mnm, count(*) AS nmr, classifier() AS cls ALL ROWS PER MATCH SHOW EMPTY MATCHES PATTERN ((^A*)|A+) DEFINE A AS price > 11) WHERE symbol = 'GLOBEX' ORDER BY 1, 2;

75


EmptyMatchesandUnmatchedRowsInSameQuery?SELECT symbol, tstamp, price, mnm, nmr, cls FROM ticker MATCH_RECOGNIZE( PARTITION BY symbol ORDER BY tstamp MEASURES match_number() AS mnm, count(*) AS nmr, classifier() AS cls ALL ROWS PER MATCH OMIT EMPTY MATCHES PATTERN ((^A*)|A+) DEFINE A AS price > 11) WHERE symbol = 'GLOBEX' ORDER BY 1, 2;

76


EmptyMatchesandUnmatchedRowsInSameQuery?

SELECT symbol, tstamp, price, mnm, nmr, cls FROM ticker MATCH_RECOGNIZE( PARTITION BY symbol ORDER BY tstamp MEASURES match_number() AS mnm, count(*) AS nmr, classifier() AS cls ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN ((^A*)|A+) DEFINE A AS price > 11) WHERE symbol = 'GLOBEX' ORDER BY 1, 2;

77

UnmatchedrowEmptymatch


UsingjustasimpleSELECT*canhavesurprisingresults!

OUTPUT:WHICHCOLUMNSARERETURNED


NotListingColumnsToBeReturned• SELECT * … isverytemptingbutcanhaveconsequences• Manyexamplesonlivesql.oracle.com,blogsandusergrouppresentationsusingSELECT *syntax

• Recommendyoualwayslistthecolumnstobereturned–Willpreventyourreport/appreturningmorecolumnsthanexpected

• Followingexamplesexplainimpactofnotspecifyingcolumns

79


UsingSELECT *:NoPARTITIONBY,ORDERBY,MEASURESSELECT *FROM ticker MATCH_RECOGNIZE ( ONE ROW PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

80

ORA-30732: table contains no user-visible columns30732. 00000 - "table contains no user-visible columns"*Cause: An attempt was made to query on a system table which has no user-visible columns.*Action: Do not query on a system table that has no user-visible columns.


UsingSELECT *:NoPARTITIONBY,ORDERBY,MEASURESSELECT *FROM ticker MATCH_RECOGNIZE ( ALL ROWS PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Queryreturnsjustcolumnsfromsourcetable(ticker)

Note:resultisinfactwrong!

81


WhatHappensAfterNewColumnsAddedToSource?• AddnewcolumnstodemotableTICKER

ALTER TABLE TICKER ADD opening_price NUMBER

• NowrunqueriesusingSELECT * …

• Newcolumnisautomaticallyincludedinresultset

82


UsingSELECT * withPARTITIONBYSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ALL ROWS PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Notedifferentresultcomparedtopreviousslidebutthisresultreliesondatabeingincorrectorder

83


UsingSELECT * withPARTITIONBYandORDERBYSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp ALL ROWS PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

Notesameresultaspreviousslidebutresultisnowdeterministic

84


UsingSELECT * withPARTITIONBYandORDERBYSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp ONE ROW PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

OnlyPARTITIONBYcolumnisreturned

85


UsingSELECT * withALLROWSPERMATCHSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(down.tstamp) as end_tstamp, match_number() AS match_num, classifier() AS classifier ALL ROWS PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

86

Noteorderingofcolumns:1) Partitionbycolumn2) Orderbycolumn3) Measurecolumns4) Allothercolumnsfromsourcetable


UsingSELECT * withONEROWPERMATCHSELECT *FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(down.tstamp) AS start_tstamp, LAST(down.tstamp) as end_tstamp, match_number() AS match_num, classifier() AS classifier ONE ROW PER MATCH PATTERN (DOWN*) DEFINE DOWN AS (price <= PREV(price)) ) WHERE symbol = 'GLOBEX';

87

Noteorderingofcolumns:1) Partitionbycolumn2) Orderbycolumn3) Measurecolumns

Allothercolumnsfromsourcetableexcluded


Summary:ExplicitlyListRequiredColumns

• DON’TUSESELECT * FROM…• Columnsincludedinoutputwillvaryanddependson:

–Typeofreportoutputspecified:• ALLROWSPERMATCH• ONEROWPERMATCH

–UseofPARTITIONBYandORDERBYclauses–UseofMEASURESclause

• Columnsaddedtosourcetablemaysuddenlyappearinoutputwhichcouldcauseapplicationerrors

88


Wheretostartsearchingforthenextpattern...therearesomanyplacestochoosefrom!

AFTERMATCHSKIPTO...


Skipping-basicsyntax• AFTER MATCH SKIP TO NEXT ROW

– Resumepatternmatchingattherowafterthefirstrowofthecurrentmatch.

• AFTER MATCH SKIP PAST LAST ROW [DEFAULT]– Resumepatternmatchingatthenextrowafterthelastrowofthecurrentmatch.

• AFTER MATCH SKIP TO FIRST pattern_variable– Resumepatternmatchingatthefirstrowthatismappedtothepatternvariable.

• AFTER MATCH SKIP TO LAST pattern_variable– Resumepatternmatchingatthelastrowthatismappedtothepatternvariable.

• AFTER MATCH SKIP TO pattern_variable – ThesameasAFTERMATCHSKIPTOLASTpattern_variable.

90


Match#1Complete:Rows5-10

91

MATCH #1 STARTS HERE

MATCH #1 ENDS HERE


NEXTROW:RestartSearchingForNextMatchAtRow6

92

• AFTER MATCH SKIP TO NEXT ROW

Scanningfornextmatch


LASTROW:RestartSearchingForNextMatchAtRow11

93


• AFTER MATCH SKIP PAST LAST ROW


Defaultprocessing


FIRSTpattern_variable:RestartSearchingForNextMatchAtRow6

94



• AFTER MATCH SKIP TO FIRST DOWN



LASTpattern_variable:RestartSearchingForNextMatchAtRow10

95




• AFTER MATCH SKIP TO LAST UPScanningfornextmatch


pattern_variable:RestartSearchingForNextMatchAtRow10

96




• AFTER MATCH SKIP TO LAST UP

• AFTER MATCH SKIP TO UP



Lookingforshapesandcontrollingskipping

97

MultipleOverlappingW-ShapesintheTickerData


LookingformultipleW-shapesusingAFTERMATCHSKIP…

SELECT *FROM Ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_w, LAST(z.tstamp) AS end_w ONE ROW PER MATCH AFTER MATCH SKIP PAST LAST ROW PATTERN (STRT x+ y+ w+ z+)DEFINE x AS x.price <= PREV(x.price), y AS y.price >= PREV(y.price), w AS w.price <= PREV(w.price), z AS z.price >= PREV(z.price) ) MRWHERE symbol='ACME'ORDER BY symbol, MR.start_w;

98

Findingonly1W-shape



99

Findingonly1W-shape

AFTER MATCH SKIP PAST LAST ROWResumesearchingfornextpatternhere,meanswecan’tfindanothercompleteW-shape



SELECT *FROM Ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_w, LAST(z.tstamp) AS end_w ONE ROW PER MATCH AFTER MATCH SKIP TO LAST Y PATTERN (STRT x+ y+ w+ z+)DEFINE x AS x.price <= PREV(x.price), y AS y.price >= PREV(y.price), w AS w.price <= PREV(w.price), z AS z.price >= PREV(z.price) ) MRWHERE symbol='ACME'ORDER BY symbol, MR.start_w;

100

Finding2W-shapes


OverlappingW-Shapes–rowsprocessedmultipletimes

101

Finding2w-shapes

Rows10-Apr-11to14-Apr-11areprocessedagainduringsearchforpattern#2


Lookingforshapesandcontrollingskipping

102

MultipleOverlappingW-ShapesintheTickerData



SELECT *FROM Ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_w, LAST(z.tstamp) AS end_w ONE ROW PER MATCH AFTER MATCH SKIP TO NEXT ROW PATTERN (STRT x+ y+ w+ z+) DEFINE x AS x.price <= PREV(x.price), y AS y.price >= PREV(y.price), w AS w.price <= PREV(w.price), z AS z.price >= PREV(z.price) ) MRWHERE symbol='ACME'ORDER BY symbol, mr.start_w;

103

Finding3W-shapes



104

Finding3W-shapes

AFTER MATCH SKIP TO NEXT ROWResumesearchingfornextpatternattherowafterthefirstrowofthecurrentmatch.Therefore,wefind3completeW-shapes


HowdoestheoptimizerpreparetorunaquerycontainingaMATCH_RECOGNIZEclause?

Preparingtorunaquery


OptimizingMATCH_RECOGNIZE

CostoftheMATCH_RECOGNIZEevaluationisbasedon:1. Costofsortinginputdata

– executionofstatemachinerequiresdataispartitionedandorderedaccordingtothepartitionbyandorderbykeys

– Ifincomingdataisnotcorrectlyorderedthensortingofthedataisneededandsortingcostisadded

2. Costofexecutingthestatemachine– thiscostisaffectedbymanyfactorssuchaspatternregularexpression(andwhetherfinitestatemachineisbuilt),predicatesdefiningvariables,matchoptions,AFTERMATCHSKIPtooption.

106

Costing



• Optimizerattemptstoeliminatesortrequestedbyaquerybyselectinganaccessmethodwhichreturnsordereddata– forexampleindexaccessorsort-merge-joinaccess.

• Optimizercompares:1. costoftheoptimalplanitfoundsofarincludingthesortcostand2. costofplanwithorderpreservingaccessmethod– Obviouslyitselectsthecheapestone!

• Ifsortcanbeeliminated,optimizerdoesnotadditscosttothecostofqueryblock.

107

Sortelimination



• Calculationaffectedbymanyfactors:– patternregularexpression– variabledefiningconditions– rowspermatchoption+aftermatchskiptooption

• Gettinganaccurateestimateisnotfeasible.– ALLROWSPERMATCH-cardinalityestimatewillbesettothatofrowpatterninputtable

–ONEROWPERMATCH-cardinalityestimatewillbesettoestimateoftotalnumberofdistinctPARTITIONBYvaluesoutrowpatterninputtable

108

CalculatingCardinality


Howarethestandardquerytransformationsimplemented?• ThereareanumberofquerytransformationsthatareaffectedbyMATCH_RECOGIZEsuchas:

109

• ViewMerging• Un-nesting

• Pushingjoinpredicates• Pushingpredicates• Predicatemovearound• Predicatepullup

• Grouppruning• Groupbyplacement• Materializedviewrewrite


Howarethestandardquerytransformationsimplemented?• ViewMerging

– ParserexpandsaMATCH_RECOGNIZEtableexpressionintoaninlineviewqueryblock• MATCH_RECOGNIZEeviewqueryblockinvolvespatternrecognitionevaluationonly

– ViewmergingisdisabledforMATCH_RECOGNIZEviewqueryblock– IfMATCH_RECOGNIZEisappliedonaview,thenmergingofinputviewwithitsouterqueryblock(i.e.MATCH_RECOGNIZEviewqueryblock)isalsodisabled

– BUTnotethat,ifMATCH_RECOGNIZEclauseappliesonaview,thenallthequerytransformationscurrentlyavailablearestillavailablewithinview

• Un-nesting– NotdoneforsubqueriescontainingMATCH_RECOGNIZEclause

110


Howarethestandardquerytransformationsimplemented?• Pushingjoinpredicates

– Similartowindowfunctions,pushingofjoinpredicatesintomatchrecognizeview(cost-based)willbeenabledprovidedjoinkeysareonpartitionbykeys

• Pushingpredicates– Predicateswillbepushedonthepartitionbykeysandwillfilterentirepartitions

• PredicateMoveAround– Predicateswillbeexportedfromtheun-mergeablematchrecognizeviewonlyiftheyareonthepartitionbykeys

111


Howarethestandardquerytransformationsimplemented?• GroupPruningforGroupingSets

– notbeaffectedasaqueryblockcannotcontaingrouping

• GROUPBYplacement– groupbyplacementtransformationwillbedisabledforviewsusingMATCH_RECOGNIZE

• Materializedviewsrewrite– IfMVhasaMATCH_RECOGNIZECLAUSEthenonlyfullorpartialtextmatchrewritewillbesupported

112


MaterializedViews-somerestrictions• Conventionalfastrefreshwillbedisabled• PCTrefreshisONLYfeasibleandsupportedIF:

–MVandbasetablesarepartitionedandthepartitioningcolumnsareasubsetofthepartition-bykeysofMATCH_RECOGNIZEclause

113


Overview

NewKeywordsinExplainPlans


NewKeyWordsinExplainPlans• Fournewkeywordsrelatingtopatternmatchingthatwillappearinyourexplainplan:

1. MATCHRECOGNIZE2. SORT3. BUFFER4. DETERMINISTICFINITEAUTO

• Importantkeywordintermsofperformance–deterministicfiniteautostatemachinesdeliverfasterprocessing!

115


PREDICATESWhen,where,howandwhyaretheyappliedwhenusingMATCH_RECOGNIZE?


WhenexactlyistheWHEREclauseprocessed?

SELECT symbol, first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 )) WHERE symbol EXISTS ('ACME', 'OSCORP');

117

ApplyingpredicatestoPARTITIONBYcolumn


ApplyingpredicatestoORDERBYcolumn

SELECT symbol, first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z tstamp AS tstamp ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 )) WHERE symbol IN (‘ACME', 'OSCORP') AND tstamp BETWEEN ’01-Apr-11’ AND ’09-Apr-11’;

118


Applyingpredicatestoothercolumns

SELECT symbol, first_x, last_z FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z, tstamp AS tstamp, price as price ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 )) WHERE symbol IN ('ACME', 'OSCORP') AND tstamp BETWEEN ’01-Apr-11' and '09-Apr-11’ AND price > 0;

119


SORTINGWhendoessortinghappenandhowtominimizenumberofsortsinyourexplainplan!


Newkeywordsinexplainplans• Fournewkeywordsrelatingtopatternmatchingthatwillappearinyourexplainplan:

1. MATCHRECOGNIZE

2. SORT

3. BUFFER

4. DETERMINISTICFINITEAUTO

121


Sortvs.Bufferkeywordsinexplainplan• Mostexample/sampleexplainplansshowSORTkeyword

– Indicatesdataissortedduringinitialstageofprocessing

• SORTkeywordreplacedwithBUFFERwhen…– Applyingpredicatesandanindexisavailablethatprovidesthecorrectsortorder– Nopredicatesappliedbutindexavailablethatprovidesthecorrectsortorder

• Createanindexonthecolumnssymbolandtstamp(PARTITIONBYandORDERBYcolumns)

CREATE INDEX ticker_tstamp_idx ON ticker(symbol, tstamp)

122


Sortvs.BufferkeywordsinexplainplanSELECT symbol, first_x, last_z FROM tickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z, tstamp AS tstamp, price as price ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 ));WHERE symbol IN (‘ACME', 'OSCORP')AND tstamp BETWEEN '01-Apr-11' AND '09-Apr-11')

123


Sortsvs.Buffers

• Usingpredicatestofilteronsymbol• SORTkeywordwithintheMATCH_RECOGNIZElinehasdisappeared• IndexisabletoprovideacorrectlyorderedrowsetintoMATCH_RECOGNIZE• MATCH_RECOGNIZElineisshowingthekeywordBUFFER

– Indicatesnoadditionalsortingisbeingappliedtodataasitflowsintomatchingprocess

124


Sortvs.Bufferkeywordswithnopredicates

125

• Nopredicates-indexwillactuallybeused• MATCH_RECOGNIZEissmartenoughtospotthatindexisusefulandcanprovidetheorderingneededformatchingprocessing– IndexprovidescorrectsortorderandexplainplanswitchestousingBUFFERkeyword


Sortvs.BufferkeywordswithnopredicatesSELECT symbol, first_x, last_z FROM tickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z, tstamp AS tstamp, price as price ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 ));

126


Sortsvs.Bufferswithnopredicates

127


…andwhyyouneedtocareaboutthem!

UnderstandingStateMachines


MATCH_RECOGNIZEandStateMachines• CompilationphaseofMATCH_RECOGNIZEgeneratesafinitestatemachine• WhatisanFSM?Afinite-statemachine(FSM)…isamathematicalmodelofcomputation....itisconceivedasanabstractmachinethatcanbeinoneofafinitenumberofstates.Themachineisinonlyonestateatatime…changesfromonestatetoanotherwheninitiatedbyatriggeringeventorcondition;thisiscalledatransition.AparticularFSMisdefinedbyalistofitsstates,andthetriggeringconditionforeachtransition.

Referencefromwikipedia-https://en.wikipedia.org/wiki/Finite-state_machine

129

http://en.wikipedia.org/wiki/Finite-state_machine



https://en.wikipedia.org/wiki/Finite-state_machine


TurnstileStateMachine

130

• Hastwostates:LockedandUnlocked• Twoeventsaffectitsstate:

– Puttingacoinintheslot(coin)– Pushingthearm(push)

• Lockedstate,pushingonthearmhasnoeffect• PuttingacoininshiftsthestatefromLockedtoUnlocked– Puttingadditionalcoinsinhasnoeffect;

• Pushingthroughthearms,givingapushinput,shiftsthestatebacktoLockedImagescourtesyofwikipedia


MATCH_RECOGNIZEandStateMachines• StatemachineisconstructedusinginformationinPATTERNandDEFINEcomponents:

• Statemachinerepresentedbyadirectedgraphcalledastatediagram– Eachstateisrepresentedbyanode(circle)– Edges(arrows)showtransitionsfromonestatetoanother.

• Labeledwiththeevent(condition)thattriggerstransition.

– Events(conditions)thatdon'tcauseachangeofstatearerepresentedbyacirculararrowreturningtotheoriginalstate.

131

0Start State1 State3State2

event

eventState4

Precedenceistoreadfromtop->down

eventevent event


StateMachineforourpattern….• GraphrepresentationofpatterntosearchforV’sinstocktickerdata

• PATTERN(STRTDOWN+?UP+)

– NotetheprecedenceofUPoverDOWNforreluctantquantifierDOWN

132

0Start State1(strt)

State3(up)

State2(down)

down

up

downstrt up

eventprice<=PREV(down.price)

price<=PREV(down.price)

price>=PREV(up.price)price>=PREV(up.price)

stateedge


Reviewofnewpatternmatchingkeywordsintheexplainplan

MATCH_RECOGNIZEandtheOptimizer


Newkeywordsinexplainplans• Fournewkeywordsrelatingtopatternmatchingthatwillappearinyourexplainplan:

1. MATCHRECOGNIZE

2. SORT

3. BUFFER

4. DETERMINISTICFINITEAUTO

134


MATCH_RECOGNIZEplansbasedonStateMachines• CompilationphaseofMATCH_RECOGNIZEgeneratesafinitestatemachine• DetailsofPATTERNcomponentdetermineifstatemachineis:1. DeterministicFiniteAuto(DFA)

• Eachofitstransitionsisuniquelydeterminedbyitssourcestateandevent• DFAusesanefficientalgorithmwithoutbacktracking,runsinlineartime

2. NondeterministicFiniteAuto(NFA)• NextstateofanNFAdependsnotonlyonthecurrentevent,butalsopossiblyonanarbitrary

numberofsubsequentevents• NFAimplementsbacktracking+otheroptimizationtechniques

135


MATCH_RECOGNIZEplansbasedonStateMachinesExplainplanindicateswhichalgorithmisused:

136


BuiltADeterministicFiniteStateMachine…SELECT symbol, start_tstamp, end_tstamp, match_numFROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_tstamp, LAST(UP.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH AFTER MATCH SKIP TO LAST UP PATTERN (STRT DOWN UP) DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)) WHERE symbol= ‘ACME’ ;

137


BuiltADeterministicFiniteStateMachine…

SELECT symbol, start_tstamp, end_tstamp, match_numFROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES strt.tstamp AS start_tstamp, LAST(down.tstamp) AS b_tstamp, LAST(up.tstamp) AS e_tstamp ONE ROW PER MATCH AFTER MATCH SKIP TO LAST UP PATTERN (STRT DOWN UP) DEFINE down AS price < PREV(price), up AS price > PREV(price)) WHERE symbol= ‘ACME’ ;

138

Nobacktrackingandrunsinlineartime...


ManagedtobuildDeterministicFiniteStateMachine…SELECT symbol, start_tstamp, end_tstamp, match_numFROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_tstamp, LAST(UP.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH AFTER MATCH SKIP TO LAST UP PATTERN (STRT DOWN UP*) DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)) WHERE symbol= ‘ACME’ ;

139


BuiltADeterministicFiniteStateMachine…• KeyattributeofaDeterministicFiniteAuto:

1. Eachofitstransitionsisuniquelydeterminedbyitssourcestateandevent PATTERN (STRT DOWN UP*) DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)

• Therefore,theDFAstatemachineisabletouseanefficientalgorithm(nobacktracking)andrunsinlineartime

140


Didn’tmanagetobuildDeterministicFiniteStateMachine…SELECT symbol, start_tstamp, end_tstamp, match_numFROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES STRT.tstamp AS start_tstamp, LAST(UP.tstamp) AS end_tstamp, match_number() AS match_num ONE ROW PER MATCH AFTER MATCH SKIP TO LAST UP PATTERN (STRT DOWN* UP*) DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)) WHERE symbol= ‘ACME’ ;

141


CouldNotBuildADeterministicFiniteStateMachine…• KeyreasonwhyitisnotpossibletobuildDeterministicFiniteAutostatemachine:

1. NextstateofanNFAdependsnotonlyonthecurrentevent,butalsopossiblyonanarbitrarynumberofsubsequentevents

PATTERN (STRT DOWN* UP*)DEFINE DOWN AS price < PREV(price), UP AS price > PREV(price)

• Therefore:1. NFAimplementsbacktracking2. Butotheroptimizationtechniquesimplementedtoensuregoodperformance

142


Non-deterministicstatemachine

SELECT *FROM TickerMATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES strt.tstamp AS start_tstamp, LAST(down.tstamp) AS b_tstamp, LAST(up.tstamp) AS e_tstamp ONE ROW PER MATCH AFTER MATCH SKIP TO LAST up PATTERN (strt down* up*) DEFINE down AS price <= PREV(price), up AS price >= PREV(price)) WHERE symbol= ‘ACME’ ;

143

Determinismunknown,backtrackinginplace,runsinnon-lineartime...


WhatisBacktrackingWhyshouldIcareaboutbacktracking?


Extendingthepatterntotestfinalpricevs.initialpriceSELECT symbol, mn, tstamp, pattern, price, first_price, last_priceFROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES MATCH_NUMBER() AS mn, FIRST(strt.tstamp) AS first_x, FIRST(strt.price) AS first_price, LAST(z.tstamp) AS last_z, last(z.price) AS last_price, classifier() AS pattern ALL ROWS PER MATCH WITH UNMATCHED ROWS PATTERN (STRT X+ Y+ W+ Z+ AVGP) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price)), AVGP AS (last(z.price) < strt.price*1.5)) ;

145


Comparingresultsoffirstandsecondstatement

146

Howaretherowsofdatanowprocessedthroughthenon-deterministicstatemachine?

BasicPattern

ModifiedpatternwithAVGPasfinalpartofpattern


Comparingresultsoffirstandsecondstatement

1. ROW2isnotmatchedanymoretoalways-trueeventSTRT2. STRTvariablenowmatchedatrow3.3. W-patternstillendsatROW8

4. ROW9isnowmappedtovariableAVGP

147

Backtrackinginaction!

12

34


(STRT X+ Y+ W+ Z+ AVGP)

0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z

Currentstate Rowevaluated Event Eventmet Newstate

0 1 strt Y 1



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


0 1 strt Y 1

1 2 X N 1

• Patternhasfailedtomatch• Movepointertonextrow:Row2• Startapplyingstatemachinetotestforpattern



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


0 1 strt Y 1



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


0 2 strt Y 1

1 3 X Y 2



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3

3 7 W Y 4



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3

3 7 W Y 4

4 8 W N 4

4 8 Z Y 5



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3

3 7 W Y 4

4 8 W N 4

4 8 Z Y 5

5 9 Z N 5



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

2 6 X N 2

2 6 Y Y 3

3 7 Y N 3

3 7 W Y 4

4 8 W N 4

4 8 Z Y 5

5 9 Z N 5

5 9 Avgp(2–8) N FAIL



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X Y 2

BACK

TRAC

KING

STAR

TED



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X Y 2

2 5 X IGNORE 2

2 5 Y N FAIL

BACK

TRAC

KING

STAR

TED



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y 2

2 4 X IGNORE 2

2 4 X N FAIL

BACK

TRAC

KING

STAR

TED



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


1 2 strt Y 1

2 3 X Y FAILBA

CKTR

ACKING

STAR

TED



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


0 2 Rowskipped

0 3 Strt Y 1

• Can’tbacktrackanyfurtherbecausetherearenomorematchesforX

• Patternhasfailedtomatch!• Movepointertonextrow:Row3• Startapplyingstatemachinetotestforpattern



0Start State1(strt)

State3(y)

State2(x)

State4(w)

x y

wxstrt

yState5

(z)State6(avgp)

z

avgp

w

z


0 3 strt Y 1


Patterncompletes:startsRow3,endsRow9

167


Backtracking• Non-deterministicstatemachinecapturesstateateachrowatpatternevaluationtimeandpushesdetailsintostack– Backtrackingsimplywalksbackthroughthestack,lookingforpossiblere-evaluation

• Movingforwardweputmoreandmorerowsintothestack– Repeatedwithineachpartition

• Dependingoncomplexityofpatternthiscanbecomememory-consuming– ChancetorunoutofPGA (ORA-30009) forlarge,complexpatternmatchingstatements

– Circumventsuchsituationsbyallocatingmorememoryor…– Considersimplifyyourpattern!

168


NAMINGBLOCKSKeepingyourcodeneatlyandclearlydefinedforthenextdeveloperwhohastoworkonit…


Nameyourblocks-notmandatory,justhelpful• Nameyourblocks• ExplicitlymarktheMATCH_RECOGNIZEblock– ReferenceinSELECTclauseandpredicates

• Makeslifealoteasierforpersontryingtoupdateyourcode

170

SELECT mr.symbol, mr.first_x, mr.last_z FROM ticker t MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES FIRST(x.tstamp) AS first_x, LAST(z.tstamp) AS last_z, tstamp AS tstamp ONE ROW PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price) AND z.tstamp - FIRST(x.tstamp) <= 7 )) mr WHERE mr.symbol IN ('ACME', 'OSCORP') AND mr.tstamp BETWEEN '01-Apr-11' AND '09-Apr-11';


Don’trelyonDEFAULTSorexcludeoptionalkeywords!

171

• Manykeywords/clauseshavedefaultvalues– AFTERMATCHSKIP….defaultisPASTLASTROW– ALLROWSPERMATCH….DefaultisSHOWEMPTYMATCHES

• Somekeywords/clausesareoptional– PARTITIONBYandORDERBY:don’texpectsourcedatatobeincorrectorder!–MEASURES:mostpatternscontainusefuldatapointssoaddadditionalmeasures

• MATCH_NUMBERandCLASSIFIERareusefulasdebuggingtoolsbutprobablyneedtoberemoved


TESTINGYOURPATTERNBuilddatasetsthattestallpossibilitiesformatchingyourpatterntoavoidunexpectedresults


Youwillneedthreedatasetsfortesting…

173

1

2

3

Datasetcontainingnomatchessoyoucanmakesurethatyourpatterndoesnotunexpectedlyfindpatterns

DatasetcontainingpartialmatchessoyoucanmakesurethatyourpatterndoesnotconsumeallyourPGAresourcesandgenerateanORA-3009error.Checkyourgreedy/reluctantquantifiersarematchingasexpected

Datasetcontainsatleastonecompletematch(preferablyalotmore)foryourpattern


Summary

SQLPatternMatchingisapowerfultool

Howtousemostimportantkeywords

KeythingstoconsiderbeforeyourunaMATCH_RECOGINZEquery

Howstatemachinesworkandimpactofbacktracking

GoanduseSQLPatternMatchingtoyouradvantage!

✔

✔

✔

✔

✔

174


TryMATCH_RECOGNIZEtodayusinglivesql.us.oracle.com

175

Largeselectedofpatternmatchingscriptsandtutorialsonlivesql.us.oracle.com




Nowyouareready……

..tobesuccessfulwithSQLpatternmatching

176

DEEP DIVE INTO 12c MATCH RECOGNIZE - Oracle...Do You Need An ALWAYS TRUE Pattern Variable? • Some...

Documents

Transcript of DEEP DIVE INTO 12c MATCH RECOGNIZE - Oracle...Do You Need An ALWAYS TRUE Pattern Variable? • Some...