Enterprise Deep Learning with DL4J
-
Upload
josh-patterson -
Category
Data & Analytics
-
view
105 -
download
2
Transcript of Enterprise Deep Learning with DL4J
Presenter: Josh Patterson
Past
Research in Swarm Algorithms
TVA / NERC
Cloudera
Today
Patterson Consulting
Skymind (Advisor)
[email protected] / @jpatanooga
WHAT IS DEEP LEARNING?
“Cooper: [When Cooper tries to reconfigure TARS] Humour 75%.TARS: 75%. Self destruct sequence in T minus 10, 9, 8...Cooper: Let's make it 65%.TARS: Knock, knock.”
--- Interstellar
Automated Feature Engineering• Deep Learning can be thought of as workflows for
automated feature construction– Where previously we’d consider each stage in the
workflow as unique technique• Many of the techniques have been around for years– But now are being chained together in a way that
automates exotic feature engineering• As LeCunn says:– “machines that learn to represent the world”
These are the features learned at each neuron in a Restricted Boltzmann Machine (RBMS)
These features are passed to higher levels of RBMs to learn more complicated things.
Part of the “7” digit
Deep Learning Architectures• Deep Belief Networks
– Most common architecture• Convolutional Neural Networks
– State of the art in image classification• Recurrent Networks
– Timeseries • Recursive Networks
– Text / image– Can break down scenes
What is DL4J?• “The Hadoop of Deep Learning”
– Command line driven– Java, Scala, and Python APIs
• ASF 2.0 Licensed• Java implementation
– Parallelization / GPU support• Runtime Neutral
– Local– Hadoop / YARN– Spark– AWS
• https://github.com/deeplearning4j/ deeplearning4j
14
DL4J and Parallelization
Model
Training Data
Worker 1
Master
Partial Model
Global Model
Worker 2
Partial Model
Worker N
Partial Model
Split 1 Split 2 Split 3
…
Traditional Serial TrainingModern Parallel Engine
(Hadoop / Spark)
DL4J Suite of Tools• DL4J
– Main library for deep learning• Canova
– Vectorization library• ND4J
– Linear Algebra framework – Swappable backends (JBLAS, GPUs)
• Arbiter– Model evaluation and testing platform
Canova for Command Line Vectorization
• Library of tools to take – Audio– Video– Image– Text– CSV data
• And convert the input data into vectors in a standardized format– Adaptable with custom input/output formats
• Open Source, ASF 2.0 Licensed– https://github.com/deeplearning4j/Canova– Part of DL4J suite
Building Deep Learning Workflows• Our terminology in data science has gotten more exotic
– But its still about gather, cleaning, visualizing, and feature construction of data
• We need to get data from a raw format into a baseline raw vector– Which is why Canova exists
• Deep Learning is not just classification– But an automated feature construction pipeline
• Together, DL4J and Canova give us the base workflow
Modeling UCI Data: Iris
• We need to vectorize the data– Possibly with some per column transformations– Which is why Canova exists• To feed raw data into a form DL4J can consume
• We then need to build a deep learning model over the data– We’ll use the DL4J lib to do this
Vectorization with Canova• Setup the configuration file• Setup the schema transforms for the input
CSV data• Generate the SVMLight vector data as the
output – with the command line interface
Canova Configuration
input.header.skip=falseinput.statistics.debug.print=falseinput.format=org.canova.api.formats.input.impl.LineInputFormat
input.directory=src/test/resources/csv/data/uci_iris_sample.txtinput.vector.schema=src/test/resources/csv/schemas/uci/iris.txtoutput.directory=/tmp/iris_unit_test_sample.txt
output.format=org.canova.api.formats.output.impl.SVMLightOutputFormat
Canova Vector Schema
@RELATION UCIIrisDataset@DELIMITER ,
@ATTRIBUTE sepallength NUMERIC !NORMALIZE @ATTRIBUTE sepalwidth NUMERIC !NORMALIZE @ATTRIBUTE petallength NUMERIC !NORMALIZE @ATTRIBUTE petalwidth NUMERIC !NORMALIZE @ATTRIBUTE class STRING !LABEL
Canova CLI./bin/canova vectorize -conf /tmp/iris_conf.txt File path already exists, deleting the old file before proceeding...Output vectors written to: /tmp/iris_svmlight.txt
DL4J from the Command Linebin/dl4jUsage:
dl4j [command] [params] Commands:
train build a deep learning model test test a deep learning model predict score new records against a deep learning
model
DL4J Configuration Fileinput.format=org.canova.api.formats.input.impl.SVMLightInputFormat
input.directory=src/test/resources/data/irisSvmLight.txt
model.config=src/test/resources/model_dbn.json
output.directory=/tmp/
JSON Model Config{"sparsity":0.0,"useAdaGrad":true,"lr":0.10000000149011612,"corruptionLevel":0.30000001192092896,"numIterations":1000,"momentum":0.5,"l2":0.0,"useRegularization":false,"momentumAfter":null,"resetAdaGradIterations":-1,"numLineSearchIterations":100,"dropOut":0.0,"applySparsity":false,"weightInit":"VI","optimizationAlgo":"CONJUGATE_GRADIENT","lossFunction":"RECONSTRUCTION_CROSSENTROPY","concatBiases":false,"constrainGradientToUnitNorm":false,"seed":123,"nIn":0,"nOut":0,"activationFunction":"sigmoid","visibleUnit":"BINARY","hiddenUnit":"BINARY","k":1,"weightShape":[0,0],"filterSize":[2,2,2,2],"numFeatureMaps":2,"featureMapSize":[2,2],"stride":[2,2],"kernel":5,"batchSize":0,"minimize":false,"rng":"org.apache.commons.math3.random.MersenneTwister","layerFactory":"org.deeplearning4j.nn.layers.factory.PretrainLayerFactory,org.deeplearning4j.models.featuredetectors.rbm.RBM","stepFunction":"org.deeplearning4j.optimize.stepfunctions.DefaultStepFunction","renderWeightIterations":-1,"dist":"org.apache.commons.math3.distribution.NormalDistribution\t{SQRT2=1.4142135623730951, mean=0.0, solverAbsoluteAccuracy=1.0E-9, DEFAULT_INVERSE_ABSOLUTE_ACCURACY=1.0E-9, standardDeviation=0.1, logStandardDeviationPlusHalfLog2Pi=-1.3836465597893728, serialVersionUID=8589540077390120676}","listeners":["org.deeplearning4j.optimize.listeners.ScoreIterationListener\t{printIterations=10, log=Logger[org.deeplearning4j.optimize.listeners.ScoreIterationListener]},"]}
Skymind as DL4J Distribution
• Just as Redhat was to Linux– A distribution of Linux with enterprise grade packaging
• Just as Cloudera was to Hadoop– A distribution of Apache Hadoop with enterprise grade
packaging• Skymind is to DL4J
– A distribution of DL4J (+tool suite) with enterprise grade packaging