Apache REEF - stdlib for big data
-
Upload
sergiy-matusevych -
Category
Software
-
view
6.617 -
download
3
Transcript of Apache REEF - stdlib for big data
![Page 1: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/1.jpg)
![Page 2: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/2.jpg)
a library to simplify and unify the lower layersof big data systems on modern resource managers.
![Page 3: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/3.jpg)
• One cluster used by all workloads
(interactive, batch, streaming, …)
• Resources are handed out as
containers
Container is slice of a machine
Fixed RAM, CPU, I/O, …
• Examples:
Apache Hadoop YARN
Apache Mesos
Google Borg
Resource Managers
![Page 4: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/4.jpg)
Resource Managers
Enable true multi-tenancy…
Many workloads:Streaming, Batch, Interactive…
Many users:Production, Ad-Hoc, Experiments…
…but, only for sophisticated apps
![Page 5: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/5.jpg)
Resource Managers
Enable true multi-tenancy…
Many workloads:Streaming, Batch, Interactive…
Many users:Production, Ad-Hoc, Experiments…
…but, only for sophisticated apps
Fault tolerance
![Page 6: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/6.jpg)
Resource Managers
Enable true multi-tenancy…
Many workloads:Streaming, Batch, Interactive…
Many users:Production, Ad-Hoc, Experiments…
…but, only for sophisticated apps
Fault tolerance
![Page 7: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/7.jpg)
Resource Managers
Enable true multi-tenancy…
Many workloads:Streaming, Batch, Interactive…
Many users:Production, Ad-Hoc, Experiments…
…but, only for sophisticated apps
Fault tolerancePreemption
![Page 8: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/8.jpg)
Resource Managers
Enable true multi-tenancy…
Many workloads:Streaming, Batch, Interactive…
Many users:Production, Ad-Hoc, Experiments…
…but, only for sophisticated apps
Fault tolerancePreemptionElasticity
![Page 9: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/9.jpg)
Example 1:
SQL / MapReduce
Elasticity
Fault tolerance
σπ
⋈
![Page 10: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/10.jpg)
Example 1:
SQL / MapReduce
Elasticity
Fault tolerance
σπ
⋈⋈⋈ ⋈⋈
![Page 11: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/11.jpg)
Example 1:
SQL / MapReduce
Elasticity
Fault tolerance
π
⋈⋈⋈ ⋈⋈
![Page 12: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/12.jpg)
Example 2:
Machine learning
Fault tolerance
Elasticity
Iterative computations
![Page 13: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/13.jpg)
Example 3:
Graph processing
Fault tolerance
Elasticity
Iterative computations
Low latency communication
![Page 14: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/14.jpg)
Silos
Silos are hard to buildEach duplicates the samemechanisms under the hood
In practice, silos form pipelines• In each step:
Read from and write to HDFS
• Synchronize on complete data between steps
Slow!
σπ
⋈
![Page 15: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/15.jpg)
Fun new problems,boring old ones
Control flow setupMaster / Slave
MembershipVia heartbeats
Task Submission
Inter-Task Messaging
…
σπ
⋈
![Page 16: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/16.jpg)
REEF
BreadthMechanism over Policy
Avoid silosRecognize the needfor different models
But: allow them to be composed
RM portabilityMake app independent from low-level Resource Manager APIs.
Bridge the JVM/CLR/… divideDifferent parts of the computation can be in either of them.
Resource Manager and DFS := Cluster OS
REEF := stdlib
![Page 17: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/17.jpg)
![Page 18: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/18.jpg)
![Page 19: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/19.jpg)
![Page 20: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/20.jpg)
REEF Control Flow
Yarn ( ) handles resource management (security, quotas, priorities)
Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc…
+ 3
![Page 21: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/21.jpg)
REEF Control Flow
Yarn ( ) handles resource management (security, quotas, priorities)
Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc.
+ 3
![Page 22: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/22.jpg)
REEF Control Flow
Yarn ( ) handles resource management (security, quotas, priorities)
Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc…
REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks
( , , , , , , etc…) to use the same cached state.
πσ
+ 3
![Page 23: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/23.jpg)
REEF Control Flow
Yarn ( ) handles resource management (security, quotas, priorities)
Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc…
REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks
( , , , , , , etc…) to use the same cached state.
πσ
![Page 24: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/24.jpg)
REEF Control Flow
Yarn ( ) handles resource management (security, quotas, priorities)
Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc…
REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks
( , , , , , , etc…) to use the same cached state.
πσ
![Page 25: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/25.jpg)
REEF Control Flow
Yarn ( ) handles resource management (security, quotas, priorities)
Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc…
REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks
( , , , , , , etc…) to use the same cached state.
πσ
![Page 26: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/26.jpg)
REEF Control Flow
Yarn ( ) handles resource management (security, quotas, priorities)
Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc…
REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks
( , , , , , , etc…) to use the same cached state.
πσ
σ σσ
![Page 27: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/27.jpg)
REEF Control Flow
Yarn ( ) handles resource management (security, quotas, priorities)
Per-job Drivers ( ) request resources, coordinate computations, and handle events: faults, preemption, etc…
REEF Evaluators ( ) hold hardware resources, allowing multiple Tasks
( , , , , , , etc…) to use the same cached state.
πσ
![Page 28: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/28.jpg)
![Page 29: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/29.jpg)
![Page 30: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/30.jpg)
Hello World
1. Client submits the Driver allocation request to the Resource Manager ( )
$…
![Page 31: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/31.jpg)
Hello World
1. Client submits the Driver allocation request to the Resource Manager ( )
2. Once started, Driver ( ) requests one Evaluator container from YARN
$…
![Page 32: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/32.jpg)
Hello World
1. Client submits the Driver allocation request to the Resource Manager ( )
2. Once started, Driver ( ) requests one Evaluator container from YARN
3. Driver submits HelloWorldTask to the newly allocated Evaluator
$…
![Page 33: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/33.jpg)
Hello World
1. Client submits the Driver allocation request to the Resource Manager ( )
2. Once started, Driver ( ) requests one Evaluator container from YARN
3. Driver submits a HelloWorld Task to the newly allocated Evaluator
4. HelloWorld Task prints a greeting to the log and quits
$…
LOG.log(Level.INFO, "Hello REEF!");
![Page 34: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/34.jpg)
Hello World
1. Client submits the Driver allocation request to the Resource Manager ( )
2. Once started, Driver ( ) requests one Evaluator container from YARN
3. Driver submits a HelloWorld Task to the newly allocated Evaluator
4. HelloWorld Task prints a greeting to the log and quits
5. Driver receives CompletedTasknotification, releases the Evaluator and quits $…
![Page 35: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/35.jpg)
public final class HelloTask implements Task {
@Injectprivate HelloTask() {}
@Overridepublic byte[] call(final byte[] memento) {LOG.log(Level.INFO, "Hello REEF!");return null;
}}
![Page 36: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/36.jpg)
public final class StartHandler implements EventHandler<StartTime> {@Overridepublic void onNext(final StartTime startTime) {requestor.submit(EvaluatorRequest.newBuilder()
.setNumber(1).setMemory(64).setNumberOfCores(1).build());}
}
public final class EvaluatorAllocatedHandlerimplements EventHandler<AllocatedEvaluator> {
@Overridepublic void onNext(final AllocatedEvaluator allocatedEvaluator) {allocatedEvaluator.submitTask(TaskConfiguration.CONF
.set(TaskConfiguration.IDENTIFIER, "HelloREEFTask")
.set(TaskConfiguration.TASK, HelloTask.class)
.build();}
}
![Page 37: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/37.jpg)
final Configuration runtimeConfig = YarnRuntimeConfiguration.CONF.build();
final Configuration driverConfig = DriverConfiguration.CONF.set(DriverConfiguration.DRIVER_IDENTIFIER, "HelloREEF").set(DriverConfiguration.GLOBAL_LIBRARIES,
EnvironmentUtils.getClassLocation(HelloDriver.class)).set(DriverConfiguration.ON_DRIVER_STARTED, HelloDriver.StartHandler.class).set(DriverConfiguration.ON_EVALUATOR_ALLOCATED,
HelloDriver.EvaluatorAllocatedHandler.class).build();
DriverLauncher.getLauncher(runtimeConfig).run(driverConfig, JOB_TIMEOUT);
![Page 38: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/38.jpg)
TangCommand = ‘ls’
Error:container-487236457-02.stderr:NullPointerException at:java…eval():1234ShellTask.helper():546ShellTask.onNext():789 YarnEvaluator.onNext():12
Configuration is hard
- Errors often show up at runtime only
- State of receiving process is
unknown to the configuring process
![Page 39: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/39.jpg)
TangCommand = ‘ls’
Error:Unknown parameter "Command"Missing required parameter "cmd"
Configuration is hard
- Errors often show up at runtime only
- State of receiving process is
unknown to the configuring process
Our approach:
- Use Dependency Injection
- Configuration is pure data
Early static and dynamic checks
![Page 40: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/40.jpg)
Tangcmd = ‘ls’
ShellTask
Evaluator
Error:Required instanceof EvaluatorGot ShellTask
Configuration is hard
- Errors often show up at runtime only
- State of receiving process is
unknown to the configuring process
Our approach:
- Use Dependency Injection
- Configuration is pure data
Early static and dynamic checks
![Page 41: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/41.jpg)
Tangcmd = ‘ls’
ShellTask
Task
YarnEvaluator
Evaluator
Configuration is hard
- Errors often show up at runtime only
- State of receiving process is
unknown to the configuring process
Our approach:
- Use Dependency Injection
- Configuration is pure data
Early static and dynamic checks
![Page 42: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/42.jpg)
Wake:Events + I/O
Event-basedprogramming and remoting
API: A static subset of Rx
- Static checking of event flows
- Aggressive JVM event inlining
Implementation: “SEDA++”
- Global thread pool
- Thread sharing where possible
![Page 43: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/43.jpg)
![Page 44: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/44.jpg)
Distributed Shell
1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and nnumber of Evaluators to run it on
$…
![Page 45: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/45.jpg)
Distributed Shell
1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and nnumber of Evaluators to run it on
2. Once started, Driver requests nEvaluators from YARN
$…
![Page 46: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/46.jpg)
Distributed Shell
1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and nnumber of Evaluators to run it on
2. Once started, Driver requests nEvaluators from YARN
3. Driver submits ShellTask with cmdto each Evaluator #!
$…
#!
#!
![Page 47: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/47.jpg)
Distributed Shell
1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and nnumber of Evaluators to run it on
2. Once started, Driver requests nEvaluators from YARN
3. Driver submits ShellTask with cmdto each Evaluator
4. Each ShellTask runs the command, logs its stdout, and quits
#!
$…
#!
#!
![Page 48: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/48.jpg)
Distributed Shell
1. Client submits Driver configuration to YARN runtime. Configuration has cmd shell command, and nnumber of Evaluators to run it on
2. Once started, Driver requests nEvaluators from YARN
3. Driver submits ShellTask with cmdto each Evaluator
4. Each ShellTask runs the command, logs its stdout, and quits
5. Driver receives CompletedTasknotifications, releases Evaluators and quits when all Evaluators are gone
$…
![Page 49: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/49.jpg)
@Injectprivate ShellTask(@Parameter(Command.class) final String command) {
this.command = command;}
@Overridepublic byte[] call(final byte[] memento) {
final String result = CommandUtils.runCommand(this.command);LOG.log(Level.INFO, result);return CODEC.encode(result);
}
![Page 50: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/50.jpg)
@NamedParameter(doc="Number of evaluators", short_name="n", default_value="1")public final class NumEvaluators implements Name<Integer> {}
@NamedParameter(doc="The shell command", short_name="cmd")public final class Command implements Name<String> {}
![Page 51: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/51.jpg)
public final class EvaluatorAllocatedHandlerimplements EventHandler<AllocatedEvaluator> {
@Overridepublic void onNext(final AllocatedEvaluator allocatedEvaluator) {
final JavaConfigurationBuilder taskConfigBuilder =tang.newConfigurationBuilder(TaskConfiguration.CONF
.set(TaskConfiguration.IDENTIFIER, "ShellTask")
.set(TaskConfiguration.TASK, ShellTask.class)
.build());
taskConfigBuilder.bindNamedParameter(Command.class, command);
allocatedEvaluator.submitTask(taskConfigBuilder.build());}
}
![Page 52: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/52.jpg)
final JavaConfigurationBuilder driverConfig =tang.newConfigurationBuilder(DriverConfiguration.CONF.set(DriverConfiguration.DRIVER_IDENTIFIER, "DistributedShell").set(DriverConfiguration.GLOBAL_LIBRARIES,
EnvironmentUtils.getClassLocation(ShellDriver.class)).set(DriverConfiguration.ON_DRIVER_STARTED, ShellDriver.StartHandler.class).set(DriverConfiguration.ON_EVALUATOR_ALLOCATED,
ShellDriver.EvaluatorAllocatedHandler.class).build());
new CommandLine(driverConfigBuilder).registerShortNameOfClass(Command.class).registerShortNameOfClass(NumEvaluators.class).processCommandLine(args);
DriverLauncher.getLauncher(runtimeConfig).run(driverConfig.build(), JOB_TIMEOUT);
![Page 53: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/53.jpg)
YARN Example: 1333 lines of code
![Page 54: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/54.jpg)
YARN Example: 1333 lines of code
Simple REEF Application: 83 lines
94%
![Page 55: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/55.jpg)
YARN Example: 1333 lines of code
Simple REEF Application: 83 lines
Interactive Fault-Tolerant
Web Application on REEF: 361 lines
27%
![Page 56: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/56.jpg)
•
•
•
•
•
![Page 57: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/57.jpg)
![Page 58: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/58.jpg)
Group Communication
Also: Collective Communication
Communications with many participants
Contrast: Peer-to-peer communication
Most commonly used interface: MPI
![Page 59: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/59.jpg)
Broadcast
MechanismThe sender (S) sends a value
All receivers (R) receive the identical value
Use caseDistribute a model
Distribute a descent direction for line search
OptimizationsTrees to distribute the data
Be mindful of the topology of machines
Do peer-to-peer sends
…
S
![Page 60: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/60.jpg)
Reduce
MechanismThe senders (S) each send a value
Those values are aggregated
The receiver (R) receives the aggregate total
Use caseAggregate gradients, losses
OptimizationsAggregation trees
Pipelining
![Page 61: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/61.jpg)
![Page 62: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/62.jpg)
![Page 63: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/63.jpg)
// We can send and receive any Java serializable data, e.g. jBLAS matricesprivate final Broadcast.Sender<DoubleMatrix> modelSender;private final Broadcast.Receiver<DoubleMatrix[]> resultReceiver;
// Broadcast the model, collect the results, repeat.do {
this.modelSender.send(modelSlice);
// ...final DoubleMatrix[] result = this.resultReceiver.reduce();
} while (notConverged(modelSlice, prevModelSlice));
![Page 65: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/65.jpg)
![Page 66: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/66.jpg)
http://github.com/cmssnu/dolphin
http://github.com/Microsoft-CISL/TensorFactorization-LDA
![Page 68: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/68.jpg)
![Page 69: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/69.jpg)
![Page 70: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/70.jpg)
Start with a random 𝑤0
Until convergence:
Compute the gradient
𝜕𝑤 =
𝑥,𝑦 𝜖 𝑋
𝑤, 𝑥 − 𝑦
Apply gradient and regularizer to the model
𝑤𝑡+1 = 𝑤𝑡 − 𝜂 𝜕𝑤 + 𝜆
Data parallel in X
Reduce
Needed by Partitions
Broadcast
![Page 71: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/71.jpg)
On REEF
Driver requests Evaluators
![Page 72: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/72.jpg)
On REEF
Driver requests Evaluators
Driver sends Tasks to load & parse data
![Page 73: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/73.jpg)
On REEF
Driver requests Evaluators
Driver sends Tasks to load & parse data
Driver sends ComputeGradient and master Tasks
![Page 74: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/74.jpg)
On REEF
Driver requests Evaluators
Driver sends Tasks to load & parse data
Driver sends ComputeGradient and master Tasks
Computation commences in sequence of Broadcast and Reduce
![Page 75: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/75.jpg)
![Page 76: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/76.jpg)
![Page 77: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/77.jpg)
Start with a random 𝑤0
Until convergence:
Compute the gradient
𝜕𝑤 =
𝑥,𝑦 𝜖 𝑋
2 𝑤, 𝑥 − 𝑦
Apply gradient to the model
𝑤𝑡+1 = 𝑤𝑡 − 𝜕𝑤
Not having some machines means
training on a (random) subset of X
![Page 78: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/78.jpg)
On REEF
First iteration
![Page 79: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/79.jpg)
On REEF
Second Iteration
![Page 80: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/80.jpg)
On REEF
End state
![Page 81: Apache REEF - stdlib for big data](https://reader034.fdocuments.in/reader034/viewer/2022042706/587065b41a28ab48378b4ed5/html5/thumbnails/81.jpg)