Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You...
Transcript of Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You...
![Page 1: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/1.jpg)
EverythingYouAlwaysWantedtoKnowAboutSynchronization
butWereAfraidtoAsk
TudorDavid,Rachid Guerraoui andVasileios TrigonakisEcole Polytechnique Federale deLausanne(EPFL)
Haksu Lim,Luis,Hwanjin Jeong
2016-04-18
![Page 2: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/2.jpg)
Multi-Core
• Multi-coreisusedinmanysystems
• Thennumberofcore↑,Performance↑?
2
NO
Synchronizationisoneofthebiggestscalabilitybottlenecks
![Page 3: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/3.jpg)
Synchronization
• Whydoesweuse?▪ Concurrentaccesstoshareddata
▪ Toensuretheorderlyexecution
• Whyissynchronizationbottleneck?▪ Hardware
▪ Synchronizationalgorithm
▪ Applicationcontext
▪ Workload3
Focusingthis
![Page 4: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/4.jpg)
CacheCoherence
• Multi-coresystemhaveaseparatecacheforeachcore▪ Writeoperationbreakconsistencyamongcaches
• Cachecoherence▪ Tomaintaincachesofacommonmemoryresource
4
![Page 5: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/5.jpg)
CacheCoherenceprotocols
Modified Shared
Invalid
Write
Busread
Write ReadBuswrite Buswrite
ReadBusRead
ReadWrite
• MSIprotocol
![Page 6: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/6.jpg)
CacheCoherenceProtocols
• MESIprotocol§ Addedexclusivestate
− Nootherhasacopyofthiscacheline
§ Reducedexpensive invalidateoperation
• MOESIprotocol§ Addedownedstate
− Thiscachelinehasbeenmodifiedbuttheremightbemoresharedcopyonothercore
§ Reducedexpensivewriteoperationtomemory
![Page 7: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/7.jpg)
CacheCoherenceExample
• Acquiringlockprocess
Mod Held=1State Data
Cache
Processor
Acq(lock);
Mod Held=1State Data
Cache
Processor
Acq(lock);
Sharedmemory(held=0)
Read-ExclusiveUpdate Invalidate
Inval
Sharedmemory(held=1)
![Page 8: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/8.jpg)
Whattodealwith
• HardwareProcessors§ Multi-sockets
– AMDOpteron• 4x6172– 48cores
– IntelXeon• 8xE7-8867L– 80cores
§ Single-sockets– SunNiagara2
• 8cores
– Tilera TILE-Gx36• 36cores
• Synchronization layer§ Concurrentsoftware
– Hashtable,etc.
§ Primitives– Lock,etc.
§ Atomicoperations– Compare&swap,etc.
§ Cachecoherence– Load&store
8
![Page 9: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/9.jpg)
Hardware-LevelAnalysis
9
![Page 10: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/10.jpg)
LocalAccesses
10
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
Opteron Zeon
• Withinsocket:40ns • Withinsocket:20– 40ns
![Page 11: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/11.jpg)
RemoteAccesses
11
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
CC
CC
CC
CC
C C
Opteron Zeon
• Withinsocket:40ns
• Perhop:+40ns
• Withinsocket:20– 40ns
• Perhop:+50ns
![Page 12: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/12.jpg)
OperationLatency– MultiSocket
12
7.5x
3x
CrossingsocketsisakillerUpto7.5xmoreexpensive
![Page 13: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/13.jpg)
Single-SocketProcessors
13
C
C
C
C
C
C
C
C
C C C C C C
C C C C C C
C C C C C C
C C C C C C
C C C C C C
C C C C C C
NiagaraTilera
• Equidistantfromthecache
• Uniform:23ns
• Nonuniform
• 1hop:40ns
• Perhop:+2ns
![Page 14: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/14.jpg)
OperationLatency– SingleSocket
14
0.5x
Uniformisexpectedtoscalebetter,Thenon-uniformisaffectedbothdistanceand
thenumberofinvolvedcores
![Page 15: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/15.jpg)
AtomicOperations– MultiSockets
• Veryfastsingle-threadperformance▪ Butdropsontwoormorecoresanddecreasesfurtherwhenthereiscross-socketcommunication
15
Opteron Xeon
![Page 16: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/16.jpg)
AtomicOperations– SingleSockets
• Lowersingle-threadthroughput▪ Butscaletoamaximumvalue
16
Niagara Tilera
![Page 17: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/17.jpg)
Software-LevelAnalysis
17
![Page 18: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/18.jpg)
AnalysisScope
• 9Locks▪ Spinlocks
– Testandtest-and-setlock(TTAS),Ticketlock
▪ Queuebasedlock– Arraybasedlock,CLHlock,MCSlock
▪ Hierarchicallock– HierarchicalCLHlock,Hierarchicalticketlock
▪ Mutex
• Concurrentsoftware▪ Hashtable
18
![Page 19: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/19.jpg)
TicketLock
19
Lock Nextticket :Nowserving:
AcquiredTicket :0
10
AcquiringTicket :1
AcquiringTicket :2
AcquiringTicket :3
AcquiringTicket :4
Spin Spin Spin Spin
![Page 20: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/20.jpg)
TicketLock
20
Lock Nextticket :Nowserving:
Release
21
AcquiringTicket :1
AcquiringTicket :2
AcquiringTicket :3
AcquiringTicket :4
Spin Spin SpinSpin
![Page 21: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/21.jpg)
CLHLock
21
tail false
Acquiring
![Page 22: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/22.jpg)
CLHLock
22
tail false
Acquired
true
prev
reference
![Page 23: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/23.jpg)
CLHLock
23
tail false
Acquired
true
prev
reference
false
Acquiring
Spin
![Page 24: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/24.jpg)
CLHLock
24
tail false
Unlock
false
prev
reference
false
Acquiring
Spin
![Page 25: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/25.jpg)
CLHLock
25
tail falsefalse
prev
reference
true
Acquired
![Page 26: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/26.jpg)
HierarchicalLock
26
C
C
C
C
C
C
C
C
![Page 27: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/27.jpg)
HierarchicalLock
27
C
C
C
C
C
C
C
C
• NUMAawarelock▪ Usinglocalcacheforlock
![Page 28: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/28.jpg)
LocksMicrobenchmark
• InitializeNlocks&Tthreads
• Eachthreadrepeatedly▪ ChoosesonelockoutofNatrandom▪ Acquiresthelock▪ Readsandwritestheprotecteddata▪ Releases thelock
• Repeatwith9differentlockalgorithms▪ spinlocks,queue-based, hierarchical,mutex
• Reportthebesttotalthroughput
28
![Page 29: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/29.jpg)
LocksonMultiSockets
29X:Y,X:thescalabilityoverthesingle-threadexecution
Y:thebest-performance lock
Highcontention(4locks) Lowcontention(128locks)
Multisocketsprovidelimitedscalabilityduetohigherlatenciesofremoteaccess
![Page 30: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/30.jpg)
LocksonSingleSockets
30X:Y,X:thescalabilityoverthesingle-threadexecution
Y:thebest-performance lock
Highcontention(4locks) Lowcontention(128locks)
Complexlocksaregenerallythebestunderextremecontention,Simplelocksperformbetterunderlowcontention
![Page 31: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/31.jpg)
HashTable– bestlocks
31
Simplelocksarepowerful25/32
Highcontention Lowcontention
![Page 32: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/32.jpg)
Conclusion
• Crossingsocketsisakiller▪ Upto7.5xmoreexpensivecommunication
• Intra-socketuniformitymatters
• Simplelocksarepowerful▪ Betterin25outof32data-pointsonahashtable
32
![Page 33: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/33.jpg)
ExtraSlides
33
![Page 34: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/34.jpg)
Hardware-LevelAnalysis
• Multisocketprocessor▪ Localaccesslatency
▪ Remoteaccesslatency
• Singlesocketprocessor▪ Intra-socketaccesslatency
34
![Page 35: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/35.jpg)
KeyObservations
• Crossingsocketsisakiller
• Intra-socketuniformitydoesmatter
• Loadsandstorescanbeasexpensiveasatomicoperations
• Simplelocksarepowerful
35
![Page 36: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/36.jpg)
HighContention
• Multi-socket,singlelock
36
![Page 37: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/37.jpg)
HighContention
• Single-socket,singlelock
37
![Page 38: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/38.jpg)
LowContention
• Multi-socket,512locks
38
![Page 39: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/39.jpg)
LowContention
• Single-socket,512locks
39
![Page 40: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/40.jpg)
HashTableonMultiSockets
40
Highcontention(12buckets) Lowcontention(512buckets)
• Using80%get,10%put,and10%remove
![Page 41: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/41.jpg)
HashTableonSingleSockets
41
Highcontention(12buckets) Lowcontention(512buckets)
• Using80%get,10%put,and10%remove
![Page 42: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/42.jpg)
TheScalableCommutativityRule:DesigningScalableSoftwareforMulticoreProcessors
AustinT.Clements,M.Frans Kaashoek,Nickolai Zeldovich,RobertT.Morris,andEddieKohler†
MITCSAILand†HarvardUniversitySOSP2013
-Presentedby-Luis,Haksu, Hwanjin
![Page 43: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/43.jpg)
Background• Evaluatingscalabilityofmulticoresoftware:
• Focuseffortonrealissues.
• DifferentWorkloads?
• Highercore#?
• Criticalbottlenecks?
• Mightnotonlybeimplementation.
• Toolatefordesign-lvl solutions?
43
Workload
Testwithmorecores
AnalyzeScalability
Findbottlenecks
Fixthebottlenecks
![Page 44: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/44.jpg)
Approach• Inshared-memorymulticoreprocessorwith~MESIcoherentcache,acorecanscalereadsandwritesithascachedexclusivelyandscalereadsthatarecachedinsharedmode.
• Operationsscaleifimplementations haveconflict-freememoryaccess.
• Considerscalabilityearlierintheprocess->softwareinterface.▪ Beforeimplementation.
▪ Beforehardware.
▪ Findscalabilityproblemsearlier->solvethemearlier.
![Page 45: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/45.jpg)
TheScalableCommutativityRule“Wheneverinterfaceoperationscommute,theycanbeimplementedinawaythatscales.”
BasedonSIMCommutativity
• State-dependent: Contextofsystem,op.arguments,andconcurrentop.NOTallstateswillcommute.
• Interface-based: Independentofimplementation,justsameresultingstate.
• Monotonic: foranyreorderinginaprefixsequenceofoperationstheregioniscommutative.
![Page 46: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/46.jpg)
Formalexplanationoftherule• Asystemexecutesactions (invocationorresponse).
• Invocation:Systemcall.Response:result.
• Aseriesofactionsformsahistory.
• Theruleonlyconsiderswell-formed histories(oneoutstandinginvocationatanypointperthread,andeachthreadshistoryforminvocation-responsesequence.
![Page 47: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/47.jpg)
Formalexplanationoftherule• Aspecification (closesetofwell-formed histories)distinguishesifahistoryis“correct”, defining theinterface.
• Ie.UNIXgetpid()
• Commutativity =orderofoperations irrelevant• Asetofactionsarecommutativewhenthespecification isindifferent totheexecutionorderofthatset.• ForH=
![Page 48: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/48.jpg)
Formalexplanationoftherule• SI-commutation(forY):
• Xputsthesystemintodesiredstate.• SwitchingYforY’(reorderofY)requiresthatthereturnvaluesofYarevalidregardlessoforder.• ZForcesthattheresultsfromthereorderingof(Y)donotaffectfutureoperations.• Howeverthisisnon-monotonic,thatisforsomeprefixreorderingtheregionmightnotbecommutative
![Page 49: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/49.jpg)
Formalexplanationoftherule
• SIM-commutation(forY):
• WhenforanyprefixP ofsomereorderingofY.
• P SIcommutesinX||P.
• SIMcommutativity isinterfacebased=evaluatesconsequencesofexecutionorderusingonlyspecification.
![Page 50: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/50.jpg)
Designingcommutativeinterface• ApplytheruletoPOSIXresultsininsights:
• Decomposecompoundoperations
fork()
NewP
Copymemst
fdsignal
exec()
Replacememst
fdsignal
posix_spawn()
NewP
Loadimage
![Page 51: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/51.jpg)
Designingcommutativeinterface• EmbraceSpecificationnon-determinism
open()
Allocatefd
Returnsmallest fd
![Page 52: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/52.jpg)
Designingcommutativeinterface• Weakordering
message socketmessagemessage
![Page 53: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/53.jpg)
Pipe-SIGPIE
Designingcommutativeinterface• Releaseresourcesasynchronously.▪ Operationshaveglobaleffectsvisibleuponreturn.▪ Goodforusableinterfacebutstrictforops.Thatreturnresources.
▪ Nocommutewithlastclose()ofareadfd.Musttrackno.ofreadfd.
fd
![Page 54: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/54.jpg)
AnalyzinginterfacedesignusingCOMMUTER
• Understandingcommutativityofcomplexinterfaceisnottrivial.
• Developanimplementationthatdoesn’tsharewhenoperationscommuteincreasesdifficulty
• AutomatedtoolnamedCOMMUTER
Commutativityconditions
ANALIZER TESTGEN MTRACEPythonmodel
Testcases Sharedcachelines
Implementation
![Page 55: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/55.jpg)
AnalyzinginterfacedesignusingCOMMUTER
ANALYZER
• Inputpythonsymbolicmodelofinterface.
• Findsconditions inwhichthemodelcommutes.
• Outputscommutativityconditions: argumentsandstates.
• Symbolicmodelenablesfocusonexternalbehavior.
Commutativityconditions
ANALYZER TESTGEN MTRACEPythonmodel
Testcases Sharedcachelines
Implementation
![Page 56: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/56.jpg)
AnalyzinginterfacedesignusingCOMMUTER
TESTGEN
• Input:Commutativityconditions.
• Convertintotestcases.
• Specifyconcretevaluesforeverysymbolicvariableinthemodel.
• ProduceactualCtestcasecode.
• Testcasecode:statesetup+functionstorun.
• Pathcoverage– codepath.
• Conflictcoverage– accesspattern.
Commutativityconditions
ANALIZER TESTGEN MTRACEPythonmodel
Testcases Sharedcachelines
Implementation
![Page 57: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/57.jpg)
AnalyzinginterfacedesignusingCOMMUTER
MTRACE
• Runthetestcasesonarealimplementation.
• Onviolationofcommutativity ruleitreportswhatvariableswheresharedandthecodethataccessed them.
• Runsonqemu anstartslogforeachtestcase.
Commutativityconditions
ANALIZER TESTGEN MTRACEPythonmodel
Testcases Sharedcachelines
Implementation
![Page 58: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/58.jpg)
Implementation• PrototypeofCOMMUTER
▪ (ANALYZERandTESTGEN)=3,050linesofpython.
▪ MTRACE1,594linesofcodechangesinqemu.
▪ Modify612linesofcodeoflinux.
▪ 2,865linesofC++codetomadeaprogramthatprocessthelogfiles.
![Page 59: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/59.jpg)
Findingscalabilityopportunities
• Modeled18POSIXfilesystemandvirtualmemorysystemcallsinCOMMUTER.
• EvaluateLinuxkernel3.8scalability.
• Developscalablefileandvirtualmemorysystem.
• COMMUTERgenerated13,664testcases.
• Runningthetestcases8minutes.
![Page 60: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/60.jpg)
ComparingScalability• ForLinuxkernelOutof13,664testcases4,257werenotconflictfree.
• Commoncases:sharedreferencecount,coarsegrainedlocks.
![Page 61: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/61.jpg)
ComparingScalabilityFollowcommutativitydesignprinciplesandimplementontopofsv6:
• in-memoryfilesystemcalledScaleFS
• virtualmemorysystemcalledRadixVM
![Page 62: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/62.jpg)
ComparingScalabilityCOMMUTERpointedout• Layerscalability:Useofdatastructuresthatsatisfycommutativityrulesuchas;radixarray,hashtabledetc.
• DeferWork:Lazyresourcerelease.Batchreferencecountreconciliation.
• Precedepessimismwithoptimism:Checkfirstthenacquirelock.
• Don’treadunlessnecessary.
![Page 63: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/63.jpg)
Performanceevaluation• 80core machine,eight 2.4Ghz10core IntelE7-8870and256GBRAM.
• Each 30MBsocketL3cacheis shared by 10cores.
• Nohardwareprefetcher.
• CompareLinux3.5.7(UbuntuQuantal)Vs.Sv6
• Singlecore baseline
![Page 64: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/64.jpg)
Microbenchmark:statbench
• Scalability offstat.
• Create singlefilethat n/2cores fstat().
• Other n/2core linktonewname then unlink
![Page 65: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/65.jpg)
Microbenchmark:openbench
• Scalability ofopen.
• Nthreads concurrently openandclose per-threadfiles
![Page 66: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/66.jpg)
Microbenchmark:Mailserver
• Morerealworld workload.
• Separate comm proc.
• Roughly like qmail.
• Mailclient with nthreads continuosly deliver emailby spawning andfeeding mail-enqueue.
![Page 67: Everything You Always Wanted to Know About ...csl.skku.edu/uploads/ECE5658S16/pr7.pdfEverything You Always Wanted to Know About Synchronization but Were Afraid to Ask Tudor David,](https://reader035.fdocuments.in/reader035/viewer/2022070217/61220096b85723408f22b68b/html5/thumbnails/67.jpg)
Conclusion• The newruleenables design for scalability design.
• +scalable implemention ==+perfomance (ALWAYS???)
• Casespecific,what totunefor?
• Toolsgives hint about commutative ruleimplementation feasibility but it wont clearly specify how toachieve this.