MapR, Implications for Integration
-
Upload
trihug -
Category
Technology
-
view
2.407 -
download
1
description
Transcript of MapR, Implications for Integration
04/11/2023 © MapR Confidential 1
MapR, Implications for Integration
CHUG – August 2011
04/11/2023 © MapR Confidential 2
Outline
• MapR system overview• Map-reduce review• MapR architecture• Performance Results• Map-reduce on MapR
• Architectural implications• Search indexing / deployment• EM algorithm for machine learning• … and more …
04/11/2023 © MapR Confidential 3
Map-Reduce
Input Output
Shuffle
04/11/2023 © MapR Confidential 4
Bottlenecks and Issues
• Read-only files• Many copies in I/O path• Shuffle based on HTTP• Can’t use new technologies• Eats file descriptors
• Spills go to local file space• Bad for skewed distribution of sizes
04/11/2023 © MapR Confidential 5
MapR Areas of Development
Map Re-
duce
Storage Service
s
Ecosystem
HBase
Management
04/11/2023 © MapR Confidential 6
MapR Improvements
• Faster file system• Fewer copies• Multiple NICS• No file descriptor or page-buf competition
• Faster map-reduce• Uses distributed file system• Direct RPC to receiver• Very wide merges
04/11/2023 © MapR Confidential 7
MapR Innovations
• Volumes• Distributed management• Data placement
• Read/write random access file system• Allows distributed meta-data• Improved scaling• Enables NFS access
• Application-level NIC bonding• Transactionally correct snapshots and mirrors
04/11/2023 © MapR Confidential 8
MapR's Containers
Each container contains Directories & files Data blocks
Replicated on servers No need to manage
directly
Files/directories are sharded into blocks, whichare placed into mini NNs (containers ) on disks
Containers are 16-32 GB segments of disk, placed on nodes
04/11/2023 © MapR Confidential 9
Container locations and replication
CLDB
N1, N2
N3, N2
N1, N2
N1, N3
N3, N2
N1
N2
N3
Container location database (CLDB) keeps track of nodes hosting each container
04/11/2023 © MapR Confidential 10
MapR ScalingContainers represent 16 - 32GB of data
Each can hold up to 1 Billion files and directories 100M containers = ~ 2 Exabytes (a very large cluster)
250 bytes DRAM to cache a container 25GB to cache all containers for 2EB cluster
But not necessary, can page to disk Typical large 10PB cluster needs 2GB
Container-reports are 100x - 1000x < HDFS block-reports Serve 100x more data-nodes Increase container size to 64G to serve 4EB cluster
Map/reduce not affected
04/11/2023 © MapR Confidential 11
MapR's Streaming Performance
Read Write0
250
500
750
1000
1250
1500
1750
2000
2250
Read Write0
250
500
750
1000
1250
1500
1750
2000
2250
HardwareMapRHadoopMB
persec
Tests: i. 16 streams x 120GB ii. 2000 streams x 1GB
11 x 7200rpm SATA 11 x 15Krpm SAS
Higher is better
04/11/2023 © MapR Confidential 12
Terasort on MapR
1.0 TB0
10
20
30
40
50
60
3.5 TB0
50
100
150
200
250
300
MapRHadoop
Elapsed time (mins)
10+1 nodes: 8 core, 24GB DRAM, 11 x 1TB SATA 7200 rpm
Lower is better
04/11/2023 © MapR Confidential 13
HBase on MapR
Recordsper
second
Higher is betterZipfian Uniform0
5000
10000
15000
20000
25000
MapRApache
YCSB Random Read with 1 billion 1K records10+1 node cluster: 8 core, 24GB DRAM, 11 x 1TB 7200 RPM
04/11/2023 © MapR Confidential 14
# of files (m)
Rate
(file
s/se
c)
Op: - create file - write 100 bytes - close
Notes:
- NN not replicated
- NN uses 20G DRAM
- DN uses 2G DRAM
Out of box
Tuned
Small Files (Apache Hadoop, 10 nodes)
04/11/2023 © MapR Confidential 15
MUCH faster for some operations
# of files (millions)
CreateRate
Same 10 nodes …
04/11/2023 © MapR Confidential 16
What MapR is not
• Volumes != federation• MapR supports > 10,000 volumes all with
independent placement and defaults• Volumes support snapshots and mirroring
• NFS != FUSE• Checksum and compress at gateway• IP fail-over• Read/write/update semantics at full speed
• MapR != maprfs
04/11/2023 © MapR Confidential 17
New Capabilities
04/11/2023 © MapR Confidential 18
NFS mounting models
• Export to the world• NFS gateway runs on selected gateway hosts
• Local server• NFS gateway runs on local host• Enables local compression and check summing
• Export to self• NFS gateway runs on all data nodes, mounted
from localhost
04/11/2023 © MapR Confidential 19
Export to the world
NFSServerNFS
ServerNFSServerNFS
ServerNFSClient
04/11/2023 © MapR Confidential 20
Client
NFSServer
Local server
Application
Cluster Nodes
04/11/2023 © MapR Confidential 21
ClusterNode
NFSServer
Universal export to self
Task
Cluster Nodes
04/11/2023 © MapR Confidential 22
ClusterNode
NFSServer
Task
ClusterNode
NFSServer
Task
ClusterNode
NFSServer
Task
Nodes are identical
04/11/2023 © MapR Confidential 23
Application architecture
• So now we have a hammer
• Let’s find us some nails!
04/11/2023 © MapR Confidential 24
Sharded text Indexing
MapReducer
Input documents
Localdisk Search
EngineLocal
disk
Clustered index storage
Assign documents to shards
Index text to local disk and then copy index to
distributed file store
Copy to local disk typically required before
index can be loaded
04/11/2023 © MapR Confidential 25
Sharded text indexing
• Mapper assigns document to shard• Shard is usually hash of document id
• Reducer indexes all documents for a shard• Indexes created on local disk• On success, copy index to DFS• On failure, delete local files
• Must avoid directory collisions • can’t use shard id!
• Must manage and reclaim local disk space
04/11/2023 © MapR Confidential 26
Conventional data flow
MapReducer
Input documents
Localdisk Search
EngineLocal
disk
Clustered index storage
Failure of a reducer causes garbage to accumulate in the
local disk
Failure of search engine requires
another download of the index from clustered storage.
04/11/2023 © MapR Confidential 27
SearchEngine
Simplified NFS data flows
MapReducer
Input documents
Clustered index storage
Failure of a reducer is cleaned up by
map-reduce framework
Search engine reads mirrored index directly.
Index to task work directory via NFS
04/11/2023 © MapR Confidential 28
Simplified NFS data flows
MapReducer
Input documents
SearchEngine
Mirrors
SearchEngine
Mirroring allows exact placement
of index data
Aribitrary levels of replication also possible
04/11/2023 © MapR Confidential 29
How about another one?
04/11/2023 © MapR Confidential 30
K-means
• Classic E-M based algorithm• Given cluster centroids,• Assign each data point to nearest centroid• Accumulate new centroids• Rinse, lather, repeat
04/11/2023 © MapR Confidential 31
Aggregatenew
centroids
K-means, the movie
Assignto
Nearestcentroid
Centroids
Input
04/11/2023 © MapR Confidential 32
But …
04/11/2023 © MapR Confidential 33
Averagemodels
Parallel Stochastic Gradient Descent
Trainsub
model
Model
Input
04/11/2023 © MapR Confidential 34
Updatemodel
Variational Dirichlet Assignment
Gathersufficientstatistics
Model
Input
04/11/2023 © MapR Confidential 35
Old tricks, new dogs
• Mapper• Assign point to cluster• Emit cluster id, (1, point)
• Combiner and reducer• Sum counts, weighted sum of points• Emit cluster id, (n, sum/n)
• Output to HDFS
Read fromHDFS to local disk by distributed cache
Written by map-reduce
Read from local disk from distributed cache
04/11/2023 © MapR Confidential 36
Old tricks, new dogs
• Mapper• Assign point to cluster• Emit cluster id, (1, point)
• Combiner and reducer• Sum counts, weighted sum of points• Emit cluster id, (n, sum/n)
• Output to HDFSMapR FS
Read fromNFS
Written by map-reduce
04/11/2023 © MapR Confidential 37
Poor man’s Pregel
• Mapper
• Lines in bold can use conventional I/O via NFS
37
while not done: read and accumulate input models for each input: accumulate model write model synchronize reset input formatemit summary
04/11/2023 © MapR Confidential 38
Click modeling architecture
Featureextraction
anddown
sampling
Input
Side-data
Datajoin
SequentialSGD
Learning
Map-reduce
Now via NFS
04/11/2023 © MapR Confidential 39
Click modeling architecture
Map-reduceMap-reduce
Featureextraction
anddown
sampling
Input
Side-data
Datajoin
SequentialSGD
Learning
Map-reduce cooperates
with NFSSequential
SGDLearning
SequentialSGD
Learning
SequentialSGD
Learning
04/11/2023 © MapR Confidential 40
And another…
04/11/2023 © MapR Confidential 41
??
Hybrid model flow
Map-reduce
Map-reduce
Feature extraction and
down sampling
SVD(PageRank)(spectral)
DeployedModel
Down stream
modeling
04/11/2023 © MapR Confidential 42
04/11/2023 © MapR Confidential 43
Map-reduceSequential
Hybrid model flow
Feature extraction and
down sampling
SVD(PageRank)(spectral)
DeployedModel
Down stream
modeling
04/11/2023 © MapR Confidential 44
And visualization…
04/11/2023 © MapR Confidential 45
Trivial visualization interface
• Map-reduce output is visible via NFS
• Legacy visualization just works
$ R> x <- read.csv(“/mapr/my.cluster/home/ted/data/foo.out”)> plot(error ~ t, x)> q(save=‘n’)
04/11/2023 © MapR Confidential 46
Conclusions
• We used to know all this• Tab completion used to work• 5 years of work-arounds have clouded our
memories
• We just have to remember the future