Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file...

62
Lecture 10: Buffer Manager and File Organization Lecture 10

Transcript of Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file...

Page 1: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

Lecture10:BufferManagerandFile

Organization

Lecture10

Page 2: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

Today’sLecture

1. Recap:BufferManager

2. ReplacementPolicies

3. FilesandRecords

2

Lecture10

Page 3: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

1. BufferManager

3

Lecture10>Section1

Page 4: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

Whatyouwilllearnaboutinthissection

1. BufferPool

2. BufferManager

4

Lecture10>Section1

Page 5: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

TheBuffer(Pool)

Lecture10>Section1

Disk

MainMemory

Buffer(Pool)• Abuffer isaregionofphysicalmemoryusedtostoretemporarydata

• Inthislecture:aregioninmainmemoryusedtostoreintermediatedatabetweendiskandprocesses

• Keyidea:Reading/writingtodiskisslow-needtocachedata!

Page 6: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

6

• DatamustbeinRAMforDBMStooperateonit!• Can’tkeepalltheDBMSpagesinmainmemory

• BufferManager:Efficientlyusesmainmemory• Memorydividedintobufferframes:slotsforholdingdiskpages

Upperlevels:• releasepageswhendone

• indicateifapageismodified

DB

MAINMEMORY

DISK

diskpage

freeframe

PageRequestsfromHigherLevelsBUFFERPOOL

choiceofframedictatedbythereplacementpolicy

BufferManagementinaDBMS

Lecture10>Section1

Page 7: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

7

BufferManager

Lecture10>Section1

• Bookkeepingperframe:• Pincount:#usersofthepageintheframe• Pinning :Indicatethatthepageisinuse• Unpinning :Releasethepage,andalsoindicateifthepageisdirtied

• Dirtybit :Indicatesifchangesmustbepropagatedtodisk

2requestorswanttomodifythesamepage?Handledbytheconcurrencycontrolmanager(usinglocks)

Page 8: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

8

BufferManager

Lecture10>Section1

•WhenaPageisrequested:• Inbufferpool->returnahandletotheframe.Done!• Incrementthepincount

• Notinthebufferpool:• Chooseaframeforreplacement(Onlyreplacepageswithpincount==0)

• Ifframeisdirty,writeittodisk• Readrequestedpageintochosenframe• Pinthepageandreturnitsaddress

Canyoutellthe#currentusersofapageintheBP?PinCount!

Page 9: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

2. ReplacementPolicies

9

Lecture10>Section2

Page 10: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

Whatyouwilllearnaboutinthissection

1. ReplacementPolicy

2. LRUandClock

3. SequentialFlooding

10

Lecture10>Section2

Page 11: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

• Howdowechooseaframeforreplacement?• LRU(LeastRecentlyUsed)• Clock• MRU(MostRecentlyUsed)• FIFO,random,…

• Thereplacementpolicyhasbigimpacton#ofI/O’s(dependsontheaccesspattern)

11

Bufferreplacementpolicy

Lecture10>Section2

Page 12: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

12

LRU

Lecture10>Section2

• usesaqueue ofpointerstoframesthathavepincount=0

• apagerequestusesframesonlyfromthehead ofthequeue

• whenathepincountofaframegoesto0,itisaddedtotheendofthequeue

Page 13: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

13

LRUExample

Lecture10>Section2

Bufferpoolwith4frames

Page 14: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

14

LRUExample

Lecture10>Section3

Page 15: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

15

LRUExample

Lecture10>Section2

Page 16: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

16

LRUExample

Lecture10>Section2

Page 17: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

17

LRUExample

Lecture10>Section2

Whichpageisevictednext?

Page 18: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

18

LRUExample

Lecture10>Section2

Page 19: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

19

Clock

Lecture10>Section2

• VariantofLRUwithlowermemoryoverhead• TheN framesareorganizedintoacycle• Eachframehasareferencedbitthatissetto1whenpincountbecomes0

• Acurrentvariablepointstoaframe• Whenaframeisconsidered:

• Ifpincount>0,incrementcurrent• Ifreferenced=1,setto0andincrement• Ifreferenced=0andpincount=0,choosethepage

Page 20: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

20

ClockExample

Lecture10>Section2

Referencedbit

Page 21: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

21

ClockExample

Lecture10>Section2

Page 22: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

22

ClockExample

Lecture10>Section2

Page 23: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

23

ClockExample

Lecture10>Section2

Page 24: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

24

ClockExample

Lecture10>Section2

Page 25: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

25

ClockExample

Lecture10>Section2

Page 26: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

26

ClockExample

Lecture10>Section2

Page 27: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

27

ClockExample

Lecture10>Section2

Page 28: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

28

ClockExample

Lecture10>Section2

Page 29: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

29

ClockExample

Lecture10>Section2

Page 30: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

30

ClockExample

Lecture10>Section2

Page 31: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

31

ClockExample

Lecture10>Section2

Page 32: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

32

ClockExample

Lecture10>Section2

Page 33: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

33

ClockExample

Lecture10>Section2

Page 34: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

34

SequentialFlooding

Lecture10>Section2

• NastysituationcausedbyLRUpolicy+repeatedsequentialscans• #bufferframes<#pagesinfile• eachpagerequestcausesanI/O!!• MRUmuchbetterinthissituation

Page 35: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

35

SequentialFloodingExample

Lecture10>Section2

NestedLoop

Page 36: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

36

SequentialFloodingExample

Lecture10>Section2

NestedLoop

Page 37: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

37

SequentialFloodingExample

Lecture10>Section2

NestedLoop

Page 38: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

38

SequentialFloodingExample

Lecture10>Section2

NestedLoop

Page 39: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

39

SequentialFloodingExample

Lecture10>Section2

NestedLoop

Page 40: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

40

SequentialFloodingExample

Lecture10>Section2

NestedLoop

Page 41: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

41

SequentialFloodingExample

Lecture10>Section2

NestedLoop

Page 42: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

42

SequentialFloodingExample

Lecture10>Section2

NestedLoop

LRUhappenstoevictexactlythepagewhichwe

willneednext!!!

Page 43: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

43

SequentialFlooding

Lecture10>Section2

• NastysituationcausedbyLRUpolicy+repeatedsequentialscans• #bufferframes<#pagesinfile• eachpagerequestcausesanI/O!!• MRUmuchbetterinthissituation

Page 44: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

3. FilesandRecords

44

Lecture10>Section3

Page 45: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

Whatyouwilllearnaboutinthissection

1. FileOrganization

2. PageOrganization

3. BONUS:ColumnStores

45

Lecture10>Section3

Page 46: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

ManagingDiskSpace

46

Lecture10>Section3

PageorblockisOKforI/O,buthigherlevelsoperateonrecords,andfilesofrecords.

• Thediskspaceisorganizedintofiles

• Filesaremadeupofpages

• Pagescontainrecords

Page 47: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

FileOperations

47

Lecture10>Section3

• Thediskspaceisorganizedintofiles

• Filesaremadeupofpages

• Pagescontainrecords

Fileoperations:

• insert/delete/modifyrecord• readaparticularrecord(specifiedusingtherecordid)• scanallrecords(possiblywithsomeconditionsontherecordstoberetrieved)

Page 48: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

FileOrganization:Unordered(Heap)Files

48

Lecture10>Section3

• Simplestfilestructurecontainsrecordsinnoparticularorder.

• Asfilegrowsandshrinks,diskpagesareallocatedandde-allocated.• Tosupportrecordleveloperations,wemust:

• keeptrackofthepages inafile: pageid(pid)• keeptrackoffreespaceonpages• keeptrackoftherecords onapage:recordid(rid)• Manyalternativesforkeepingtrackofthisinformation

• Operations:create/destroyfile,insert/deleterecord,fetcharecordwithaspecifiedrid,scanallrecords

Page 49: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

HeapFileasaList

49

Lecture10>Section3

HeaderPage

DataPage

DataPage

DataPage

Fullpages

DataPage

DataPage

DataPage

Pageswithfreespace

• (heapfilename,headerpageid)recordedinaknownlocation• Eachpagecontainstwopointers plusdata:Pointer=PageID(pid)• Pagesinthefreespacelisthave“some”freespace

Q:Whathappenswithvariablelengthrecords?

A:Allpagesaregoingtohavefreespace,butmaybewewillhavetogothroughalotofthem

beforewefindonewithenoughspace.

Page 50: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

HeapFileasaPageDirectory

50

Lecture10>Section3

• Eachentryforapagekeepstrackof:• isthepagefreeorfull?• howmanyfreebytesare?

• Wecannowlocatepagesfornewtuplesfaster!

DataPageN

DataPage2

DataPage1

Headerpage

DIRECTORY

Page 51: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

ManagingDiskSpace

51

Lecture10>Section3

• Filesmadeupofpages

• andpagescontainrecords

• Butfileoperationsareonrecords:

Fileoperations:

• insert/delete/modifyrecord• readaparticularrecord(specifiedusingtherecordid)• scanallrecords(possiblywithsomeconditionsontherecordstoberetrieved)

Page 52: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

PageOrganization:PageFormats

52

Lecture10>Section3

• Apageiscollectionofrecords

• Slottedpageformat• Apageisacollectionofslots• Eachslotcontainsarecord

• rid=<pageid,slotnumber>

• Therearemanyslottedpageorganizations

• Weneedtohavesupportfor:• search,insert,deleterecordsonapage

Page 53: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

53

Slot1Slot2

SlotN

...

N

PACKED

FreeSpace

numberofrecords

PageFormats:FixedLengthRecords

Lecture10>Section3

Packedorganization: N recordsarealwaysstoredinthefirstN slots.

Movingrecordschangesrid!Maynotbeacceptable.

Recordid=<pageid,slot#>

Page 54: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

54

PageFormats:FixedLengthRecords

Lecture10>Section3

UnpackedOrganization: useabitmap tolocaterecordsinthepage.

Recordid=<pageid,slot#>

FreeSpace ...

M10...M...321

UNPACKED,BITMAP

Slot1Slot2

SlotN

SlotM

11

numberofslots

Page 55: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

55

PageFormats:VariableLengthRecords

Lecture10>Section3

Book-keeping0, 70

-1,0

560,90

-1, 0

120,40 670,

50012345

Slot directoryFree SpacePointer

• Directorygrowsbackwards!• Moverecordsonsamepage;

ridunchanged!Goodforfixed-lengthrecordstoo.

Rid=?Rid= (11, 1)Page num = 11

Delete a record? Slot Entry: Offset, record length Number of slots

Page 56: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

56

PageFormats:VariableLengthRecords

Lecture10>Section3

• Deletion:• offsetissetto-1

• Insertion:• useanyavailableslot• ifnospaceisavailable,reorganize

• rid remainsunchangedwhenwemovetherecord(sinceitisdefinedbytheslotnumber)

Page 57: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

57

PageFormats:VariableLengthRecords

Lecture10>Section3

Book-keeping0, 70

-1,0

560,90

-1, 0

120,40 670,

50012345

Slot directoryFree SpacePointer

• Directorygrowsbackwards!• Moverecordsonsamepage;

ridunchanged!Goodforfixed-lengthrecordstoo.

Rid=?Rid= (11, 1)Page num = 11

Delete a recordOffset is set to -1

Slot Entry: Offset, record length Number of slots

Page 58: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

58

• Allrecordsonthepagearethesamelength• Informationaboutfieldtypessameforallrecordsinafile;storedinsystem catalogs.

Baseaddress(B)

L1 L2 L3 L4

F1 F2 F3 F4

Address=B+L1+L2

RecordFormats:FixedLength

Lecture10>Section3

Page 59: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

59

Twoalternativeformats(#fieldsisfixed):

ArrayofOffsets

4 $ $ $ $

FieldCount

F1F2F3F4

Fielddelimiter(specialsymbol)

n Secondalternativeoffersdirectaccesstoi’th fieldn Efficientstorageofnullsn Smalldirectoryoverhead.

n Issueswithgrowingrecords!n changesinattributevalue,add/dropattributes

n Recordslargerthanpages

RecordFormats:VariableLength

Lecture10>Section3

Useanarrayofintegeroffsetsinthebeginning

Page 60: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

ColumnStores:Motivation

• Consideratable:• Foo(aINTEGER,bINTEGER,cVARCHAR(255),…)

• Andthequery`:• SELECTa FROMFooWHEREa>10

• WhathappenswiththepreviousrecordformatintermsofthebytesthathavetobereadfromtheIOsubsystem?

60

Lecture10>Section3

Page 61: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

• Storedata“vertically”• Contrastthatwitha“row-store”thatstoresalltheattributesofatuple/recordcontiguously• Thepreviousrecordformatsare“rowstores”

61

111 212 Itwasacoldmorning222 222 Warmandsunnyhere333 232 Articwinterconditions444 242 Tropicalweather

111 212 Itwasacoldmorning222 222 Warmandsunnyhere333 232 Articwinterconditions444 242 Tropicalweather

File1 File2 File3

ColumnStores:Motivation

Lecture10>Section3

Eachfileisasetofpages.Columnscanbestoredincompressedform

Page 62: Lecture 10: Buffer Manager and File Organization · Lecture 10 > Section 3 •Simplest file structure contains records in no particular order. •As file grows and shrinks, disk pages

• Arethereanydisadvantagesassociatedwithcolumnstores?

62

1. Updatesareslower

2. Retrievingbackmorethanoneattributecanbeslower,e.g.QuerieslikeSELECT*areslower

ColumnStores:Motivation

Lecture10>Section3