Backend to Frontend: When database optimization affects the full stack
-
Upload
markus-winand -
Category
Technology
-
view
344 -
download
1
Transcript of Backend to Frontend: When database optimization affects the full stack
![Page 1: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/1.jpg)
© 2015 by Markus WinandiStockPhoto/Mitshu
From Backend to FrontendWhen database optimization affects the full stack
@MarkusWinand@SQLPerfTips@ModernSQL
![Page 2: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/2.jpg)
© 2015 by Markus Winand
Full Stack, Really?
![Page 3: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/3.jpg)
© 2015 by Markus Winand
Full Stack, Really?
![Page 4: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/4.jpg)
© 2015 by Markus Winand
What Users Want are Used to
![Page 5: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/5.jpg)
© 2015 by Markus Winand
What Users Want are Used to
Let’sclick here
![Page 6: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/6.jpg)
© 2015 by Markus Winand
User Clicks on Page 2
GimmePage
2!
Gimme Rows11 to
20
![Page 7: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/7.jpg)
SQL for “Gimme Rows 11 to 20”
se!e"# * $%om news whe%e #op&" ' 1234 o%de% b( da#e des", &d des" o!!se" 10 #$m$" 10
![Page 8: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/8.jpg)
Inside the Database: Worst Case
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'20) So%# Me#hod: "op-N heapso&" Memo&(: 19)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
![Page 9: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/9.jpg)
Inside the Database: Worst Case
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'30) So%# Me#hod: "op-N heapso&" Memo&(: 20)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
![Page 10: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/10.jpg)
Inside the Database: Worst Case
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'40) So%# Me#hod: "op-N heapso&" Memo&(: 22)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
![Page 11: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/11.jpg)
Inside the Database: Worst Case
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'10000) So%# Me#hod: ex"e&na# me&*e D$s): 1200)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
![Page 12: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/12.jpg)
Inside the Database: Worst Case
Speed depends on the number of matched rows and the page fetched.
Fetching the last pagecan take considerablylonger than fetchingthe first page.
![Page 13: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/13.jpg)
Improvement 1: Indexed order by
se!e"# * $%om news whe%e #op&" ' 1234 o%de% b( da#e des", &d des" o$$se# 10 !&m&# 10;
%&ea"e $ndex on news ("op$%, da"e, $d)
A single index to support the whe%e and o%de% b( clauses.
![Page 14: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/14.jpg)
Improvement 1: Indexed order by
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: (#op&" ' 0)
![Page 15: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/15.jpg)
Improvement 1: Indexed order by
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'20) Index *ond: (#op&" ' 0)
![Page 16: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/16.jpg)
Improvement 1: Indexed order by
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'30) Index *ond: (#op&" ' 0)
![Page 17: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/17.jpg)
Improvement 1: Indexed order by
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'40) Index *ond: (#op&" ' 0)
![Page 18: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/18.jpg)
Improvement 1: Indexed order by
L&m&# (a"#ua! %ows'10)-> So%# (a%"ua# &ows'10000) So%# Me#hod: ex"e&na# me&*e D$s): 1200)B -> B&#map Heap S"an (a"#ua! %ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (a"#ua! %ows'10000) Index *ond: (#op&" ' 1234)
![Page 19: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/19.jpg)
Improvement 1: Indexed order by
Fetching the first page is not affected by theBase-Set size!
Fetching the next pageis also faster. However, the database might revert to the other execution plan when browsing to the end.
![Page 20: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/20.jpg)
© 2013 by Markus Winand
It’s not just about performance
![Page 21: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/21.jpg)
© 2015 by Markus Winand
Offsets Yield Unstable Results
![Page 22: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/22.jpg)
© 2015 by Markus Winand
Offsets Yield Unstable Results
![Page 23: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/23.jpg)
© 2015 by Markus Winand
Offsets Yield Unstable Results
![Page 24: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/24.jpg)
© 2013 by Markus Winand
Why don’t we ask for the next rows after...
![Page 25: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/25.jpg)
© 2015 by Markus Winand
Asking for “Next After” in SQL
![Page 26: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/26.jpg)
© 2015 by Markus Winand
Asking for “Next After” in SQL
![Page 27: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/27.jpg)
© 2015 by Markus Winand
Asking for “Next After” in SQL
![Page 28: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/28.jpg)
© 2015 by Markus Winand
Asking for “Next After” in SQL
![Page 29: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/29.jpg)
© 2015 by Markus Winand
Asking for “Next After” in SQL
![Page 30: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/30.jpg)
© 2015 by Markus Winand
Asking for “Next After” in SQL
![Page 31: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/31.jpg)
SQL For “Next After” (Key-Set Pagination)
se!e"# * $%om news whe%e #op&" ' 1234 and (da"e, $d) < (?, ?) o%de% b( da#e des", &d des" !&m&# 10 Look Ma,
no offset!
![Page 32: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/32.jpg)
Key-Set Pagination: Worst Case
![Page 33: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/33.jpg)
Key-Set Pagination: Worst Case
L&m&# (a%"ua# &ows'10)-> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (&ows'10000) Re"he") *ond: (#op&" ' 1234) -> B&#map Index S"an (&ows'10000) Index *ond: (#op&" ' 1234)
![Page 34: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/34.jpg)
Key-Set Pagination: Worst Case
L&m&# (a"#ua! %ows'10) -> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (a%"ua# &ows'9990) Rows Remo+ed b( F$#"e&: 10 (new &n 9.2) -> B&#map Index S"an (a%"ua# &ows'10000) Index *ond: (#op&" ' 1234)
![Page 35: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/35.jpg)
Key-Set Pagination: Worst Case
L&m&# (a"#ua! %ows'10) -> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (a%"ua# &ows'9980) Rows Remo+ed b( F$#"e&: 20 (new &n 9.2) -> B&#map Index S"an (a%"ua# &ows'10000) Index *ond: (#op&" ' 1234)
![Page 36: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/36.jpg)
Key-Set Pagination: Worst Case
L&m&# (a"#ua! %ows'10) -> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (a%"ua# &ows'9970) Rows Remo+ed b( F$#"e&: 30 (new &n 9.2) -> B&#map Index S"an (a%"ua# &ows'10000) Index *ond: (#op&" ' 1234)
![Page 37: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/37.jpg)
Key-Set Pagination: Worst Case
L&m&# (a"#ua! %ows'10) -> So%# (a%"ua# &ows'10) So%# Me#hod: "op-N heapso&" Memo&(: 18)B -> B&#map Heap S"an (a%"ua# &ows'10) Rows Remo+ed b( F$#"e&: 9990 (new &n 9.2) -> B&#map Index S"an (a%"ua# &ows'10000) Index *ond: (#op&" ' 1234)
![Page 38: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/38.jpg)
Key-Set Pagination w/o Index for order by
Always needs to retrieve the full base set, but the top-n sort buffer needs to hold only 10 rows.
The response time remains the same even whenbrowsing to the last page.And the memory footprintis very low!
![Page 39: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/39.jpg)
Key-Set Pagination: Best Case
se!e"# * $%om news whe%e #op&" ' 1234 and (da#e, &d) < (?, ?) o%de% b( da#e des", &d des" !&m&# 10;
%&ea"e $ndex on news ("op$%, da"e, $d)
![Page 40: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/40.jpg)
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: (#op&" ' 1234)
![Page 41: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/41.jpg)
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 23456)))
![Page 42: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/42.jpg)
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 34567)))
![Page 43: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/43.jpg)
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 45678)))
![Page 44: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/44.jpg)
Key-Set Pagination: Best Case
L&m&# (a%"ua# &ows'10)-> Index S"an Ba")wa%d (a%"ua# &ows'10) Index *ond: ((#op&" ' 1234) AND (ROW(d#, &d) < ROW(‘...’, 56789)))
![Page 45: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/45.jpg)
Key-Set Pagination: Best Case
Successively browsing back doesn’t slow down.
Neither the size of thebase set(*) nor the fetched page number affects the response time.
(*) the index tree depth still affects the response time.
![Page 46: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/46.jpg)
ComparisonO
ffset
Key
-Set
Worst Case Best Case
![Page 47: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/47.jpg)
© 2015 by Markus Winand
Too good to be true?
Key-Set Pagination has serious limitations:
‣You cannot directly navigate to arbitrary pages‣because you need the values from the previous page
‣Bi-directional navigation is possible but tedious‣you need to reverse the o%de% b( direction and RV comparison
‣Tooling...???
![Page 48: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/48.jpg)
© 2015 by Markus Winand
Required Tooling for Key-Set Pagination
![Page 49: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/49.jpg)
© 2015 by Markus Winand
Required Tooling for Key-Set Pagination
Proper Row-Values Support.
Don’t use OFFSET.NoSQL is
also Also for NoSQL
![Page 50: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/50.jpg)
© 2015 by Markus Winand
Required Tooling for Key-Set Pagination
Proper Row-Values Support.
Don’t use OFFSET.
Offer Infinite Scrolling.
Don’t think in pages.
Ask the right question.
![Page 51: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/51.jpg)
OFFSET is EVIL
http://use-the-index-luke.com/no-offset
![Page 52: Backend to Frontend: When database optimization affects the full stack](https://reader030.fdocuments.in/reader030/viewer/2022032502/55b83424bb61ebe6488b4625/html5/thumbnails/52.jpg)
© 2015 by Markus Winand
About Markus Winand
Tuning developers forhigh SQL performance
Training & co (one-man show): winand.at
Geeky blog: use-the-index-luke.com
Author of: SQL Performance Explained