Neural Networks with Google TensorFlow
-
Upload
darshan-patel -
Category
Software
-
view
4.452 -
download
12
Transcript of Neural Networks with Google TensorFlow
![Page 1: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/1.jpg)
Neural Networks with
Google TensorFlowDarshan Patel
Northeastern University
![Page 2: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/2.jpg)
• Overview:
1) Computer Vision Tasks2) Convolution Neural Network (CNNs) Architecture3) CNNs using Google TensorFlow4) Google TensorBoard
Neural Networks with Google TensorFlow
![Page 3: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/3.jpg)
Computer VisionTasks
![Page 4: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/4.jpg)
Source : http://cs231n.stanford.edu
![Page 5: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/5.jpg)
![Page 6: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/6.jpg)
![Page 7: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/7.jpg)
![Page 8: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/8.jpg)
Source : http://googleresearch.blogspot.com/2014/09/building-deeper-understanding-of-images.html
![Page 9: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/9.jpg)
Source : http://googleresearch.blogspot.com/2014/09/building-deeper-understanding-of-images.html
![Page 10: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/10.jpg)
![Page 11: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/11.jpg)
![Page 12: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/12.jpg)
ConvolutionNeural
Networks(CNNs/ConvNets)
![Page 13: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/13.jpg)
• Mathematical Definition:A function derived from two given functions by integration that expresses how the shape of one is modified by the other
What is Convolution?
![Page 14: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/14.jpg)
Neural Networks
![Page 15: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/15.jpg)
Neural Networks - Forward Pass
![Page 16: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/16.jpg)
Neural Networks - Back Propagation
Source : http://cs231n.github.io
![Page 17: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/17.jpg)
• ConvNet architectures make the explicit assumption that the inputs are images, which allows us to encode certain properties into the architecture.
• This assumption makes the forward function more efficient to implement and vastly reduces the amount of parameters in the network.
How CNN/ConvNets is different?
![Page 18: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/18.jpg)
Cont. How CNN/ConvNets is different?
Source : http://cs231n.github.io/convolutional-networks
![Page 19: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/19.jpg)
LeNet-5 1990
Yann LeCunDirector of AI Research at Facebook Handwritten Digits Classification
![Page 20: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/20.jpg)
LeNet-5 Architecture
![Page 21: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/21.jpg)
AlexNet Architecture - ImageNet 2012
![Page 22: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/22.jpg)
![Page 23: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/23.jpg)
![Page 24: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/24.jpg)
![Page 25: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/25.jpg)
LeNet-5 Architecture
![Page 26: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/26.jpg)
Layers in ConvNets
1. Convolution Layer2. ReLU (Activation) Layer3. Pooling Layer4. Fully Connected Layer
![Page 27: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/27.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
![Page 28: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/28.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
![Page 29: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/29.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
![Page 30: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/30.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
![Page 31: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/31.jpg)
![Page 32: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/32.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
![Page 33: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/33.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
![Page 34: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/34.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
Convolution Layer
![Page 35: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/35.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
Convolution Layer
![Page 36: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/36.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
Convolution Layer
![Page 37: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/37.jpg)
Source : http://cs231n.stanford.edu/slides/
Activation Layer
![Page 38: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/38.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
![Page 39: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/39.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
Pooling Layer
![Page 40: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/40.jpg)
Source : http://cs231n.stanford.edu/slides/winter1516_lecture7.pdf
Pooling Layer
![Page 41: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/41.jpg)
Fully Connected Layer
![Page 42: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/42.jpg)
![Page 43: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/43.jpg)
• A ConvNet architecture is a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)
• There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)
• Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function
• Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don't)
ConvNets Summary
![Page 44: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/44.jpg)
![Page 45: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/45.jpg)
![Page 46: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/46.jpg)
• Second generation Machine Learning system, followed by DistBelief
• TensorFlow grew out of a project at Google, called Google Brain, aimed at applying various kinds of neural network machine learning to products and services across the company.
• An open source software library for numerical computation using data flow graphs
• Used in following projects at Google1. DeepDream2. RankBrain3. Smart ReplyAnd many more..
Google TensorFlow
![Page 47: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/47.jpg)
Data Flow Graph• Data flow graphs describe mathematical computation
with a directed graph of nodes & edges
• Nodes in the graph represent mathematical operations.
• Edges represent the multidimensional data arrays (tensors) communicated between them.
• Edges describe the input/output relationships between nodes.
• The flow of tensors through the graph is where TensorFlow gets its name.
![Page 48: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/48.jpg)
Google TensorFlow Basic Elements
• Tensor• Variable• Operation• Session• Placeholder• TensorBoard
![Page 49: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/49.jpg)
• TensorFlow programs use a tensor data structure to represent all data
• Think of a TensorFlow tensor as an n-dimensional array or list
In the following example, c, d and e are symbolic Tensor Objects, where as result is a numpy array
Tensor
![Page 50: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/50.jpg)
1. Constant Value Tensors2. Sequences3. Random Tensors
Tensor Types
![Page 51: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/51.jpg)
Constant Value Tensors
![Page 52: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/52.jpg)
Sequence Tensors
![Page 53: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/53.jpg)
Random Tensors
![Page 54: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/54.jpg)
• In-memory buffers containing tensors• Initial value defines the type and shape of the variable.• They must be explicitly initialized and can be saved to disk during and after
training.
Variable
![Page 55: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/55.jpg)
• An Operation is a node in a TensorFlow Graph• Takes zero or more Tensor objects as input, and produces zero or
more Tensor objects as output.
• Example:c = tf.matmul(a, b) Creates an Operation of type "MatMul" that takes tensors a and b as input,
and produces c as output.
Operation
![Page 56: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/56.jpg)
• A class for running TensorFlow operations• InteractiveSession is a TensorFlow Session for use in interactive contexts, such as
a shell and Ipython notebook.
Session & Interactive Session
![Page 57: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/57.jpg)
• A value that we'll input when we ask TensorFlow to run a computation.
Placeholder
![Page 58: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/58.jpg)
TensorBoard : Visual Learning
![Page 59: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/59.jpg)
MNIST Dataset
![Page 60: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/60.jpg)
MNIST Dataset
![Page 61: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/61.jpg)
LeNet-5 Architecture
![Page 62: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/62.jpg)
Load MNIST Data
![Page 63: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/63.jpg)
Load MNIST Data
Start a session
![Page 64: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/64.jpg)
PlaceholdersDynamic Size
![Page 65: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/65.jpg)
Placeholders
Weight/Filter & Bias
![Page 66: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/66.jpg)
Convolution and PoolingStride of one
Max Pooling over 2x2 blocksStride of two
![Page 67: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/67.jpg)
First Convolution Layer including ReLU
It will consist of convolution, followed by max poolingFilter/Patch Dimension
Number of Input Channels
Number of Output Channel
Number of Output Channel
![Page 68: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/68.jpg)
Second Convolution Layer including ReLU
It will consist of convolution, followed by max pooling
![Page 69: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/69.jpg)
Fully Connected Layer
• Reshape the tensor from the pooling layer into a batch of vectors• Multiply by a weight matrix, add a bias, and apply a ReLU
![Page 70: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/70.jpg)
Dropout
• To reduce over fitting, we will apply dropout before the readout layer. • Dropout is an extremely effective, simple and recently introduced regularization technique by
Srivastava et al. in “Dropout: A Simple Way to Prevent Neural Networks from Overfitting” that complements the other methods (L1, L2, maxnorm).
Source : http://cs231n.github.io/neural-networks-2/
![Page 71: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/71.jpg)
Dropout
• We create a placeholder for the probability that a neuron's output is kept during dropout.
• This allows us to turn dropout on during training, and turn it off during testing.
• While training, dropout is implemented by only keeping a neuron active with some probability pp (a hyperparameter)
![Page 72: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/72.jpg)
Readout Layer
• Finally, we add a softmax layer, just like for the one layer softmax regression.
![Page 73: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/73.jpg)
Train and Evaluate the Model
InitializeAllVariables
Training
Accuracy
Testing
OptimizerLoss Function
![Page 74: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/74.jpg)
• TensorBoard operates by reading TensorFlow events files, which contain summary data that you can generate when running TensorFlow.
TensorBoard
![Page 75: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/75.jpg)
• TensorBoard operates by reading TensorFlow events files, which contain summary data that you can generate when running TensorFlow.
• First, create the TensorFlow graph that we'd like to collect summary data from, and decide which nodes should be annotated with summary operation.• For example,
• For MNIST digits CNNs, we'd like to record how the learning rate varies over time, and how the objective function is changing
• We’d like to record distribution of gradients or weights
TensorBoard
![Page 76: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/76.jpg)
TensorBoardGraph Representation
Graph Representation
Histogram Summary
![Page 77: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/77.jpg)
TensorBoard
![Page 78: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/78.jpg)
TensorBoard
![Page 79: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/79.jpg)
![Page 80: Neural Networks with Google TensorFlow](https://reader035.fdocuments.in/reader035/viewer/2022062311/586e73c81a28ab99598b55ad/html5/thumbnails/80.jpg)