TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow...
Transcript of TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow...
![Page 2: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/2.jpg)
2
![Page 3: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/3.jpg)
Agenda
Why TensorFlow
Graphs and Sessions
Linear Regression
tf.data
word2vec
Structuring your model
Managing experiments
3
![Page 4: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/4.jpg)
Why TensorFlow?
● Flexibility + Scalability● Popularity
4
![Page 5: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/5.jpg)
import tensorflow as tf
5
![Page 6: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/6.jpg)
Graphs and Sessions
6
![Page 7: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/7.jpg)
Data Flow Graphs
TensorFlow separates definition of computations from their execution
Graph from TensorFlow for Machine Intelligence 7
![Page 8: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/8.jpg)
Data Flow Graphs
Phase 1: assemble a graph
Phase 2: use a session to execute operations in the graph.
8Graph from TensorFlow for Machine Intelligence
![Page 9: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/9.jpg)
Data Flow Graphs
Phase 1: assemble a graph
Phase 2: use a session to execute operations in the graph.
9Graph from TensorFlow for Machine Intelligence
This might change in the future with eager mode!!
![Page 10: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/10.jpg)
What’s a tensor?
10
![Page 11: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/11.jpg)
What’s a tensor?
An n-dimensional array
0-d tensor: scalar (number)
1-d tensor: vector
2-d tensor: matrix
and so on
11
![Page 12: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/12.jpg)
Data Flow Graphs
import tensorflow as tf
a = tf.add(3, 5)
Visualized by TensorBoard
12
![Page 13: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/13.jpg)
Data Flow Graphs
import tensorflow as tf
a = tf.add(3, 5)
Why x, y?
TF automatically names the nodes when you don’t explicitly name them. x = 3y = 5
Visualized by TensorBoard
13
![Page 14: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/14.jpg)
Data Flow Graphs
import tensorflow as tf
a = tf.add(3, 5)
Nodes: operators, variables, and constantsEdges: tensors
Tensors are data.TensorFlow = tensor + flow = data + flow (I know, mind=blown)
14
Interpreted?
5
3
a
![Page 15: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/15.jpg)
Data Flow Graphs
import tensorflow as tf
a = tf.add(3, 5)
print(a)
>> Tensor("Add:0", shape=(), dtype=int32)(Not 8)
15
5
3
a
![Page 16: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/16.jpg)
How to get the value of a?
Create a session, assign it to variable sess so we can call it later
Within the session, evaluate the graph to fetch the value of a
16
![Page 17: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/17.jpg)
How to get the value of a?
Create a session, assign it to variable sess so we can call it later
Within the session, evaluate the graph to fetch the value of a
import tensorflow as tf
a = tf.add(3, 5)
sess = tf.Session()
print(sess.run(a))
sess.close()
The session will look at the graph, trying to think: hmm, how can I get the value of a, then it computes all the nodes that leads to a. 17
![Page 18: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/18.jpg)
How to get the value of a?
Create a session, assign it to variable sess so we can call it later
Within the session, evaluate the graph to fetch the value of a
import tensorflow as tf
a = tf.add(3, 5)
sess = tf.Session()
print(sess.run(a))
sess.close()>> 8
8
The session will look at the graph, trying to think: hmm, how can I get the value of a, then it computes all the nodes that leads to a. 18
![Page 19: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/19.jpg)
How to get the value of a?
Create a session, assign it to variable sess so we can call it later
Within the session, evaluate the graph to fetch the value of a
import tensorflow as tf
a = tf.add(3, 5)
sess = tf.Session()
with tf.Session() as sess:
print(sess.run(a))
sess.close()8
19
![Page 20: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/20.jpg)
tf.Session()
A Session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated.
20
![Page 21: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/21.jpg)
tf.Session()
A Session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated.
Session will also allocate memory to store the current values of variables.
21
![Page 22: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/22.jpg)
More graph
x = 2
y = 3
op1 = tf.add(x, y)
op2 = tf.multiply(x, y)
op3 = tf.pow(op2, op1)
with tf.Session() as sess:
op3 = sess.run(op3)
Visualized by TensorBoard
22
![Page 23: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/23.jpg)
Subgraphs
Because we only want the value of pow_op and pow_op doesn’t depend on useless, session won’t compute value of useless→ save computation
useless
add_op mul_op
pow_op
23
x = 2
y = 3
add_op = tf.add(x, y)
mul_op = tf.multiply(x, y)
useless = tf.multiply(x, add_op)
pow_op = tf.pow(add_op, mul_op)
with tf.Session() as sess:
z = sess.run(pow_op)
![Page 24: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/24.jpg)
SubgraphsPossible to break graphs into several chunks and run them parallelly across multiple CPUs, GPUs, TPUs, or other devices
Example: AlexNet
Graph from Hands-On Machine Learning with Scikit-Learn and TensorFlow 24
![Page 25: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/25.jpg)
Distributed ComputationTo put part of a graph on a specific CPU or GPU:
# Creates a graph.
with tf.device('/gpu:2'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], name='b')
c = tf.multiply(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print(sess.run(c))
25
![Page 26: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/26.jpg)
26
![Page 27: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/27.jpg)
Why graphs
1. Save computation. Only run subgraphs that lead to the values you want to fetch.
27
![Page 28: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/28.jpg)
Why graphs
1. Save computation. Only run subgraphs that lead to the values you want to fetch.
2. Break computation into small, differential pieces to facilitate auto-differentiation
28
![Page 29: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/29.jpg)
Why graphs
1. Save computation. Only run subgraphs that lead to the values you want to fetch.
2. Break computation into small, differential pieces to facilitate auto-differentiation
3. Facilitate distributed computation, spread the work across multiple CPUs, GPUs, TPUs, or other devices
29
![Page 30: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/30.jpg)
Why graphs
1. Save computation. Only run subgraphs that lead to the values you want to fetch.
2. Break computation into small, differential pieces to facilitate auto-differentiation
3. Facilitate distributed computation, spread the work across multiple CPUs, GPUs, TPUs, or other devices
4. Many common machine learning models are taught and visualized as directed graphs
A neural net graph from Stanford’s CS224N course
30
![Page 31: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/31.jpg)
TensorBoard
31
![Page 32: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/32.jpg)
import tensorflow as tf
a = tf.constant(2, name='a')
b = tf.constant(3, name='b')
x = tf.add(a, b, name='add')
with tf.Session() as sess:
print(sess.run(x))
Your first TensorFlow program
32
![Page 33: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/33.jpg)
import tensorflow as tf
a = tf.constant(2, name='a')
b = tf.constant(3, name='b')
x = tf.add(a, b, name='add')
writer = tf.summary.FileWriter('./graphs', tf.get_default_graph())
with tf.Session() as sess:
# writer = tf.summary.FileWriter('./graphs', sess.graph)
print(sess.run(x))
writer.close() # close the writer when you’re done using it
Visualize it with TensorBoard
33
Create the summary writer after graph definition and before running your session
‘graphs’ or any location where you want to keep your event files
![Page 34: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/34.jpg)
Go to terminal, run:
$ python [yourprogram].py
$ tensorboard --logdir="./graphs" --port 6006
Then open your browser and go to: http://localhost:6006/
Run it
34
6006 or any port you want
![Page 35: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/35.jpg)
35
![Page 36: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/36.jpg)
Constants, Sequences, Variables, Ops
36
![Page 37: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/37.jpg)
import tensorflow as tf
a = tf.constant([2, 2], name='a')b = tf.constant([[0, 1], [2, 3]], name='b')x = tf.multiply(a, b, name='mul')
with tf.Session() as sess:
print(sess.run(x))
# >> [[0 2]
# [4 6]]
Constants
37
Broadcasting similar to NumPy
![Page 38: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/38.jpg)
Tensors filled with a specific valuetf.zeros([2, 3], tf.int32) ==> [[0, 0, 0], [0, 0, 0]]
# input_tensor is [[0, 1], [2, 3], [4, 5]]
tf.zeros_like(input_tensor) ==> [[0, 0], [0, 0], [0, 0]]
tf.fill([2, 3], 8) ==> [[8, 8, 8], [8, 8, 8]]
Similar to NumPy
38
![Page 39: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/39.jpg)
Constants as sequences
tf.lin_space(start, stop, num, name=None) tf.lin_space(10.0, 13.0, 4) ==> [10. 11. 12. 13.]
tf.range(start, limit=None, delta=1, dtype=None, name='range')tf.range(3, 18, 3) ==> [3 6 9 12 15]
tf.range(5) ==> [0 1 2 3 4]
NOT THE SAME AS NUMPY SEQUENCES
Tensor objects are not iterable
for _ in tf.range(4): # TypeError
39
![Page 40: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/40.jpg)
Randomly Generated Constants
tf.random_normal
tf.truncated_normal
tf.random_uniform
tf.random_shuffle
tf.random_crop
tf.multinomial
tf.random_gamma
40
![Page 41: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/41.jpg)
Randomly Generated Constants
tf.set_random_seed(seed)
41
![Page 42: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/42.jpg)
TensorFlow integrates seamlessly with NumPy
tf.int32 == np.int32 # ⇒ True
Can pass numpy types to TensorFlow ops
tf.ones([2, 2], np.float32) # ⇒ [[1.0 1.0], [1.0 1.0]]
For tf.Session.run(fetches): if the requested fetch is a Tensor , output will be a NumPy ndarray.
sess = tf.Session()
a = tf.zeros([2, 3], np.int32)
print(type(a)) # ⇒ <class 'tensorflow.python.framework.ops.Tensor'>
a_out = sess.run(a)
print(type(a)) # ⇒ <class 'numpy.ndarray'>
TF vs NP Data Types
42
![Page 43: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/43.jpg)
● Python native types: TensorFlow has to infer Python type
Use TF DType when possible
43
![Page 44: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/44.jpg)
● Python native types: TensorFlow has to infer Python type● NumPy arrays: NumPy is not GPU compatible
Use TF DType when possible
44
![Page 45: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/45.jpg)
Not trainable
What’s wrong with constants?
45
![Page 46: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/46.jpg)
my_const = tf.constant([1.0, 2.0], name="my_const")
with tf.Session() as sess:
print(sess.graph.as_graph_def())
Constants are stored in graph definition
46
![Page 47: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/47.jpg)
This makes loading graphs expensive when constants are big
Constants are stored in graph definition
47
![Page 48: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/48.jpg)
This makes loading graphs expensive when constants are big
Only use constants for primitive types.
Use variables or readers for more data that requires more memory
Constants are stored in graph definition
48
![Page 49: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/49.jpg)
Variables# create variables with tf.Variable
s = tf.Variable(2, name="scalar")
m = tf.Variable([[0, 1], [2, 3]], name="matrix")
W = tf.Variable(tf.zeros([784,10]))
# create variables with tf.get_variable
s = tf.get_variable("scalar", initializer=tf.constant(2))
m = tf.get_variable("matrix", initializer=tf.constant([[0, 1], [2, 3]]))
W = tf.get_variable("big_matrix", shape=(784, 10), initializer=tf.zeros_initializer())
49
![Page 50: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/50.jpg)
You have to initialize your variablesThe easiest way is initializing all variables at once:with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
50
Initializer is an op. You need to execute it within the context of a session
![Page 51: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/51.jpg)
You have to initialize your variablesThe easiest way is initializing all variables at once:with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
Initialize only a subset of variables:with tf.Session() as sess:
sess.run(tf.variables_initializer([a, b]))
51
![Page 52: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/52.jpg)
You have to initialize your variablesThe easiest way is initializing all variables at once:with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
Initialize only a subset of variables:with tf.Session() as sess:
sess.run(tf.variables_initializer([a, b]))
Initialize a single variableW = tf.Variable(tf.zeros([784,10]))
with tf.Session() as sess:
sess.run(W.initializer)
52
![Page 53: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/53.jpg)
Eval() a variable# W is a random 700 x 100 variable object
W = tf.Variable(tf.truncated_normal([700, 10]))
with tf.Session() as sess:
sess.run(W.initializer)
print(W)
>> Tensor("Variable/read:0", shape=(700, 10), dtype=float32)
53
![Page 54: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/54.jpg)
tf.Variable.assign()W = tf.Variable(10)
W.assign(100)
with tf.Session() as sess:
sess.run(W.initializer)
print(W.eval()) # >> ????
54
![Page 55: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/55.jpg)
tf.Variable.assign()W = tf.Variable(10)
W.assign(100)
with tf.Session() as sess:
sess.run(W.initializer)
print(W.eval()) # >> 10
55
Ugh, why?
![Page 56: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/56.jpg)
tf.Variable.assign()W = tf.Variable(10)
W.assign(100)
with tf.Session() as sess:
sess.run(W.initializer)
print(W.eval()) # >> 10
56
W.assign(100) creates an assign op.That op needs to be executed in a session to take effect.
![Page 57: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/57.jpg)
tf.Variable.assign()W = tf.Variable(10)
W.assign(100)
with tf.Session() as sess:
sess.run(W.initializer)
print(W.eval()) # >> 10
--------
W = tf.Variable(10)
assign_op = W.assign(100)
with tf.Session() as sess:
sess.run(W.initializer)
sess.run(assign_op)
print(W.eval()) # >> 100
57
![Page 58: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/58.jpg)
Placeholder
58
![Page 59: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/59.jpg)
A TF program often has 2 phases: 1. Assemble a graph 2. Use a session to execute operations in the graph.
A quick reminder
59
![Page 60: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/60.jpg)
A TF program often has 2 phases: 1. Assemble a graph 2. Use a session to execute operations in the graph.
⇒ Assemble the graph first without knowing the values needed for computation
Placeholders
60
![Page 61: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/61.jpg)
A TF program often has 2 phases: 1. Assemble a graph 2. Use a session to execute operations in the graph.
⇒ Assemble the graph first without knowing the values needed for computation
Analogy:Define the function f(x, y) = 2 * x + y without knowing value of x or y. x, y are placeholders for the actual values.
Placeholders
61
![Page 62: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/62.jpg)
Why placeholders?
We, or our clients, can later supply their own data when they need to execute the computation.
62
![Page 63: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/63.jpg)
Placeholders
tf.placeholder(dtype, shape=None, name=None)
# create a placeholder for a vector of 3 elements, type tf.float32
a = tf.placeholder(tf.float32, shape=[3])
b = tf.constant([5, 5, 5], tf.float32)
# use the placeholder as you would a constant or a variable
c = a + b # short for tf.add(a, b)
with tf.Session() as sess:
print(sess.run(c)) # >> ???
63
![Page 64: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/64.jpg)
Placeholders
tf.placeholder(dtype, shape=None, name=None)
# create a placeholder for a vector of 3 elements, type tf.float32
a = tf.placeholder(tf.float32, shape=[3])
b = tf.constant([5, 5, 5], tf.float32)
# use the placeholder as you would a constant or a variable
c = a + b # short for tf.add(a, b)
with tf.Session() as sess:
print(sess.run(c)) # >> InvalidArgumentError: a doesn’t an actual value
64
![Page 65: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/65.jpg)
Supplement the values to placeholders using a dictionary
65
![Page 66: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/66.jpg)
Placeholders
tf.placeholder(dtype, shape=None, name=None)
# create a placeholder for a vector of 3 elements, type tf.float32
a = tf.placeholder(tf.float32, shape=[3])
b = tf.constant([5, 5, 5], tf.float32)
# use the placeholder as you would a constant or a variable
c = a + b # short for tf.add(a, b)
with tf.Session() as sess:
print(sess.run(c, feed_dict={a: [1, 2, 3]})) # the tensor a is the key, not the string ‘a’
# >> [6, 7, 8]
66
![Page 67: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/67.jpg)
Placeholders
tf.placeholder(dtype, shape=None, name=None)
# create a placeholder for a vector of 3 elements, type tf.float32
a = tf.placeholder(tf.float32, shape=[3])
b = tf.constant([5, 5, 5], tf.float32)
# use the placeholder as you would a constant or a variable
c = a + b # short for tf.add(a, b)
with tf.Session() as sess:
print(sess.run(c, feed_dict={a: [1, 2, 3]}))
# >> [6, 7, 8]
Quirk:shape=None means that tensor of any shape will be accepted as value for placeholder.
shape=None is easy to construct graphs and great when you have different batch sizes, but nightmarish for debugging
67
![Page 68: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/68.jpg)
Placeholders
tf.placeholder(dtype, shape=None, name=None)
# create a placeholder of type float 32-bit, shape is a vector of 3 elements
a = tf.placeholder(tf.float32, shape=[3])
# create a constant of type float 32-bit, shape is a vector of 3 elements
b = tf.constant([5, 5, 5], tf.float32)
# use the placeholder as you would a constant or a variable
c = a + b # Short for tf.add(a, b)
with tf.Session() as sess:
print(sess.run(c, {a: [1, 2, 3]}))
# >> [6, 7, 8]
Quirk:shape=None also breaks all following shape inference, which makes many ops not work because they expect certain rank.
68
![Page 69: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/69.jpg)
Placeholders are valid ops
tf.placeholder(dtype, shape=None, name=None)
# create a placeholder of type float 32-bit, shape is a vector of 3 elements
a = tf.placeholder(tf.float32, shape=[3])
# create a constant of type float 32-bit, shape is a vector of 3 elements
b = tf.constant([5, 5, 5], tf.float32)
# use the placeholder as you would a constant or a variable
c = a + b # Short for tf.add(a, b)
with tf.Session() as sess:
print(sess.run(c, {a: [1, 2, 3]}))
# >> [6, 7, 8]
69
![Page 70: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/70.jpg)
What if want to feed multiple data points in?
You have to do it one at a time
with tf.Session() as sess:
for a_value in list_of_values_for_a:
print(sess.run(c, {a: a_value}))
70
![Page 71: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/71.jpg)
Extremely helpful for testingFeed in dummy values to test parts of a large graph
71
![Page 72: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/72.jpg)
Linear Regression in TensorFlow
72
![Page 73: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/73.jpg)
Model the linear relationship between:● dependent variable Y● explanatory variables X
73
![Page 74: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/74.jpg)
74Visualization made by Google, based on data collected by World Bank
![Page 75: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/75.jpg)
X: birth rateY: life expectancy
190 countries
75
World Development Indicators dataset
![Page 76: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/76.jpg)
Find a linear relationship between X and Yto predict Y from X
76
Want
![Page 77: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/77.jpg)
Model
Inference: Y_predicted = w * X + b
Mean squared error: E[(y - y_predicted)2]
77
![Page 78: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/78.jpg)
Interactive Coding
birth_life_2010.txt
78
![Page 79: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/79.jpg)
Interactive Coding
linreg_starter.py
birth_life_2010.txt
79
![Page 80: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/80.jpg)
Phase 1: Assemble our graph
80
![Page 81: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/81.jpg)
Step 1: Read in data
I already did that for you
81
![Page 82: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/82.jpg)
Step 2: Create placeholders for inputs and labels
tf.placeholder(dtype, shape=None, name=None)
82
![Page 83: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/83.jpg)
Step 3: Create weight and bias
tf.get_variable(
name,
shape=None,
dtype=None,
initializer=None,
…
) 83
No need to specify shape if using constant initializer
![Page 84: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/84.jpg)
Step 4: Inference
Y_predicted = w * X + b
84
![Page 85: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/85.jpg)
Step 5: Specify loss function
loss = tf.square(Y - Y_predicted, name='loss')
85
![Page 86: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/86.jpg)
Step 6: Create optimizer
opt = tf.train.GradientDescentOptimizer(learning_rate=0.001)
optimizer = opt.minimize(loss)
86
![Page 87: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/87.jpg)
Phase 2: Train our model
Step 1: Initialize variables
Step 2: Run optimizer
(use a feed_dict to feed data into X and Y placeholders)
87
![Page 88: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/88.jpg)
Write log files using a FileWriter
writer = tf.summary.FileWriter('./graphs/linear_reg', sess.graph)
88
![Page 89: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/89.jpg)
See it on TensorBoard
Step 1: $ python linreg_starter.py
Step 2: $ tensorboard --logdir='./graphs'
89
![Page 90: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/90.jpg)
90
![Page 91: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/91.jpg)
91
![Page 92: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/92.jpg)
tf.data
92
![Page 93: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/93.jpg)
Placeholder
Pro: put the data processing outside TensorFlow, making it easy to
do in Python
Cons: users often end up processing their data in a single thread
and creating data bottleneck that slows execution down.
93
![Page 94: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/94.jpg)
Placeholder
data, n_samples = utils.read_birth_life_data(DATA_FILE)
X = tf.placeholder(tf.float32, name='X')
Y = tf.placeholder(tf.float32, name='Y')
…with tf.Session() as sess:
…# Step 8: train the model
for i in range(100): # run 100 epochs
for x, y in data:
# Session runs train_op to minimize loss
sess.run(optimizer, feed_dict={X: x, Y:y})
94
![Page 95: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/95.jpg)
tf.data
Instead of doing inference with placeholders and feeding in data
later, do inference directly with data
95
![Page 96: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/96.jpg)
tf.data
tf.data.Dataset
tf.data.Iterator
96
![Page 97: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/97.jpg)
Store data in tf.data.Dataset
● tf.data.Dataset.from_tensor_slices((features, labels))
● tf.data.Dataset.from_generator(gen, output_types, output_shapes)
97
![Page 98: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/98.jpg)
Store data in tf.data.Dataset
tf.data.Dataset.from_tensor_slices((features, labels))
dataset = tf.data.Dataset.from_tensor_slices((data[:,0], data[:,1]))
98
![Page 99: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/99.jpg)
Store data in tf.data.Dataset
tf.data.Dataset.from_tensor_slices((features, labels))
dataset = tf.data.Dataset.from_tensor_slices((data[:,0], data[:,1]))
print(dataset.output_types) # >> (tf.float32, tf.float32)
print(dataset.output_shapes) # >> (TensorShape([]), TensorShape([]))
99
![Page 100: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/100.jpg)
Can also create Dataset from files
● tf.data.TextLineDataset(filenames)
● tf.data.FixedLengthRecordDataset(filenames)
● tf.data.TFRecordDataset(filenames)
100
![Page 101: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/101.jpg)
tf.data.Iterator
Create an iterator to iterate through samples in Dataset
101
![Page 102: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/102.jpg)
tf.data.Iterator
● iterator = dataset.make_one_shot_iterator()
● iterator = dataset.make_initializable_iterator()
102
![Page 103: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/103.jpg)
tf.data.Iterator
● iterator = dataset.make_one_shot_iterator()Iterates through the dataset exactly once. No need to initialization.
● iterator = dataset.make_initializable_iterator()Iterates through the dataset as many times as we want. Need to initialize with each epoch.
103
![Page 104: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/104.jpg)
tf.data.Iterator
iterator = dataset.make_one_shot_iterator()
X, Y = iterator.get_next() # X is the birth rate, Y is the life expectancy
with tf.Session() as sess:
print(sess.run([X, Y])) # >> [1.822, 74.82825]
print(sess.run([X, Y])) # >> [3.869, 70.81949]
print(sess.run([X, Y])) # >> [3.911, 72.15066]
104
![Page 105: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/105.jpg)
tf.data.Iterator
iterator = dataset.make_initializable_iterator()
...
for i in range(100):
sess.run(iterator.initializer)
total_loss = 0
try:
while True:
sess.run([optimizer])
except tf.errors.OutOfRangeError:
pass
105
![Page 106: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/106.jpg)
Handling data in TensorFlow
dataset = dataset.shuffle(1000)
dataset = dataset.repeat(100)
dataset = dataset.batch(128)
dataset = dataset.map(lambda x: tf.one_hot(x, 10))
# convert each element of dataset to one_hot vector
106
![Page 107: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/107.jpg)
Does tf.data really perform better?
107
![Page 108: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/108.jpg)
Does tf.data really perform better?
With placeholder: 9.05271519 seconds
With tf.data: 6.12285947 seconds
108
![Page 109: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/109.jpg)
Should we always use tf.data?
● For prototyping, feed dict can be faster and easier to write (pythonic)● tf.data is tricky to use when you have complicated preprocessing or multiple
data sources● NLP data is normally just a sequence of integers. In this case, transferring the
data over to GPU is pretty quick, so the speedup of tf.data isn't that large
109
![Page 110: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/110.jpg)
How does TensorFlow know what variables to update?
110
![Page 111: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/111.jpg)
Optimizers
111
![Page 112: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/112.jpg)
Optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(loss)
_, l = sess.run([optimizer, loss], feed_dict={X: x, Y:y})
112
![Page 113: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/113.jpg)
Optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(loss)
_, l = sess.run([optimizer, loss], feed_dict={X: x, Y:y})
Session looks at all trainable variables that loss depends on and update them
113
![Page 114: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/114.jpg)
Optimizer
Session looks at all trainable variables that optimizer depends on and update them
114
![Page 115: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/115.jpg)
Trainable variables
tf.Variable(initial_value=None, trainable=True,...)
115
Specify if a variable should be trained or notBy default, all variables are trainable
![Page 116: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/116.jpg)
List of optimizers in TF
tf.train.GradientDescentOptimizer
tf.train.AdagradOptimizer
tf.train.MomentumOptimizer
tf.train.AdamOptimizer
tf.train.FtrlOptimizer
tf.train.RMSPropOptimizer
...
116
Usually Adam works out-of-the-box better than SGD
![Page 117: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/117.jpg)
word2vec skip-gramin TensorFlow
![Page 118: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/118.jpg)
118
Embedding Lookup
Illustration by Chris McCormick
![Page 119: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/119.jpg)
119
Embedding Lookup
Illustration by Chris McCormick
tf.nn.embedding_lookup(params, ids, partition_strategy='mod', name=None, validate_indices=True, max_norm=None)
![Page 120: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/120.jpg)
120
Negative sampling vs NCE
● Negative sampling is a simplified model of Noise Contrastive Estimation (NCE)● NCE guarantees approximation to softmax. Negative sampling doesn’t
![Page 121: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/121.jpg)
121
NCE Loss
tf.nn.nce_loss( weights, biases, labels, inputs, num_sampled, num_classes, …)
![Page 122: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/122.jpg)
Interactive Coding
word2vec_utils.py
word2vec_starter.py
122
![Page 123: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/123.jpg)
Embedding visualization
![Page 124: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/124.jpg)
Interactive Coding
word2vec_visualize.py
124
![Page 125: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/125.jpg)
Visualization from Chris Olah’s blog125
Visualize vector representation of anything
![Page 126: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/126.jpg)
126
![Page 127: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/127.jpg)
127
Name scope
TensorFlow doesn’t know what nodes should be grouped together, unless you tell it to
![Page 128: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/128.jpg)
128
Name scope
Group nodes together with tf.name_scope(name)
with tf.name_scope(name_of_that_scope):
# declare op_1
# declare op_2
# ...
![Page 129: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/129.jpg)
129
Name scopewith tf.name_scope('data'): iterator = dataset.make_initializable_iterator() center_words, target_words = iterator.get_next()
with tf.name_scope('embed'): embed_matrix = tf.get_variable('embed_matrix', shape=[VOCAB_SIZE, EMBED_SIZE], ...) embed = tf.nn.embedding_lookup(embed_matrix, center_words)
with tf.name_scope('loss'): nce_weight = tf.get_variable('nce_weight', shape=[VOCAB_SIZE, EMBED_SIZE], ...) nce_bias = tf.get_variable('nce_bias', initializer=tf.zeros([VOCAB_SIZE])) loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weight, biases=nce_bias, …)
with tf.name_scope('optimizer'): optimizer = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(loss)
![Page 130: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/130.jpg)
130
TensorBoard
![Page 131: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/131.jpg)
131
Variable scope
Name scope vs variable scope
tf.name_scope() vs tf.variable_scope()
![Page 132: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/132.jpg)
132
Variable scope
Name scope vs variable scope
Variable scope facilitates variable sharing
![Page 133: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/133.jpg)
133
Variable sharing: The problemdef two_hidden_layers(x):
w1 = tf.Variable(tf.random_normal([100, 50]), name='h1_weights')
b1 = tf.Variable(tf.zeros([50]), name='h1_biases')
h1 = tf.matmul(x, w1) + b1
w2 = tf.Variable(tf.random_normal([50, 10]), name='h2_weights')
b2 = tf.Variable(tf.zeros([10]), name='2_biases')
logits = tf.matmul(h1, w2) + b2
return logits
![Page 134: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/134.jpg)
134
Variable sharing: The problemdef two_hidden_layers(x):
w1 = tf.Variable(tf.random_normal([100, 50]), name='h1_weights')
b1 = tf.Variable(tf.zeros([50]), name='h1_biases')
h1 = tf.matmul(x, w1) + b1
w2 = tf.Variable(tf.random_normal([50, 10]), name='h2_weights')
b2 = tf.Variable(tf.zeros([10]), name='2_biases')
logits = tf.matmul(h1, w2) + b2
return logits
logits1 = two_hidden_layers(x1)
logits2 = two_hidden_layers(x2)
What will happen if we make these two calls?
![Page 135: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/135.jpg)
135
Sharing Variable: The problem
Two sets of variables are created.
You want all your inputs to use the same weights and biases!
![Page 136: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/136.jpg)
136
tf.get_variable()
tf.get_variable(<name>, <shape>, <initializer>)
If a variable with <name> already exists, reuse it
If not, initialize it with <shape> using <initializer>
![Page 137: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/137.jpg)
137
tf.get_variable()def two_hidden_layers(x):
assert x.shape.as_list() == [200, 100]
w1 = tf.get_variable("h1_weights", [100, 50], initializer=tf.random_normal_initializer())
b1 = tf.get_variable("h1_biases", [50], initializer=tf.constant_initializer(0.0))
h1 = tf.matmul(x, w1) + b1
assert h1.shape.as_list() == [200, 50]
w2 = tf.get_variable("h2_weights", [50, 10], initializer=tf.random_normal_initializer())
b2 = tf.get_variable("h2_biases", [10], initializer=tf.constant_initializer(0.0))
logits = tf.matmul(h1, w2) + b2
return logits
logits1 = two_hidden_layers(x1)
logits2 = two_hidden_layers(x2)
![Page 138: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/138.jpg)
138
def two_hidden_layers(x):
assert x.shape.as_list() == [200, 100]
w1 = tf.get_variable("h1_weights", [100, 50], initializer=tf.random_normal_initializer())
b1 = tf.get_variable("h1_biases", [50], initializer=tf.constant_initializer(0.0))
h1 = tf.matmul(x, w1) + b1
assert h1.shape.as_list() == [200, 50]
w2 = tf.get_variable("h2_weights", [50, 10], initializer=tf.random_normal_initializer())
b2 = tf.get_variable("h2_biases", [10], initializer=tf.constant_initializer(0.0))
logits = tf.matmul(h1, w2) + b2
return logits
logits1 = two_hidden_layers(x1)
logits2 = two_hidden_layers(x2)
ValueError: Variable h1_weights already exists, disallowed. Did you mean to set reuse=True in VarScope?
tf.get_variable()
![Page 139: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/139.jpg)
139
def two_hidden_layers(x):
assert x.shape.as_list() == [200, 100]
w1 = tf.get_variable("h1_weights", [100, 50], initializer=tf.random_normal_initializer())
b1 = tf.get_variable("h1_biases", [50], initializer=tf.constant_initializer(0.0))
h1 = tf.matmul(x, w1) + b1
assert h1.shape.as_list() == [200, 50]
w2 = tf.get_variable("h2_weights", [50, 10], initializer=tf.random_normal_initializer())
b2 = tf.get_variable("h2_biases", [10], initializer=tf.constant_initializer(0.0))
logits = tf.matmul(h1, w2) + b2
return logits
with tf.variable_scope('two_layers') as scope:
logits1 = two_hidden_layers(x1)
scope.reuse_variables()
logits2 = two_hidden_layers(x2)
Put your variables within a scope and reuse all variables within that scope
tf.variable_scope()
![Page 140: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/140.jpg)
140
tf.variable_scope()
Only one set of variables, all within the variable scope “two_layers”
They take in two different inputs
![Page 141: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/141.jpg)
141
tf.variable_scope()
tf.variable_scope implicitly creates a name scope
![Page 142: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/142.jpg)
142
def two_hidden_layers(x):
assert x.shape.as_list() == [200, 100]
w1 = tf.get_variable("h1_weights", [100, 50], initializer=tf.random_normal_initializer())
b1 = tf.get_variable("h1_biases", [50], initializer=tf.constant_initializer(0.0))
h1 = tf.matmul(x, w1) + b1
assert h1.shape.as_list() == [200, 50]
w2 = tf.get_variable("h2_weights", [50, 10], initializer=tf.random_normal_initializer())
b2 = tf.get_variable("h2_biases", [10], initializer=tf.constant_initializer(0.0))
logits = tf.matmul(h1, w2) + b2
return logits
with tf.variable_scope('two_layers') as scope:
logits1 = two_hidden_layers(x1)
scope.reuse_variables()
logits2 = two_hidden_layers(x2)
Reusable code?
![Page 143: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/143.jpg)
143
def fully_connected(x, output_dim, scope):
with tf.variable_scope(scope, reuse=tf.AUTO_REUSE) as scope:
w = tf.get_variable("weights", [x.shape[1], output_dim], initializer=tf.random_normal_initializer())
b = tf.get_variable("biases", [output_dim], initializer=tf.constant_initializer(0.0))
return tf.matmul(x, w) + b
def two_hidden_layers(x):
h1 = fully_connected(x, 50, 'h1')
h2 = fully_connected(h1, 10, 'h2')
with tf.variable_scope('two_layers') as scope:
logits1 = two_hidden_layers(x1)
logits2 = two_hidden_layers(x2)
Layer ‘em up
Fetch variables if they already exist
Else, create them
![Page 144: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/144.jpg)
144
Layer ‘em up
![Page 145: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/145.jpg)
Manage Experiments
![Page 146: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/146.jpg)
tf.train.Saversaves graph’s variables in binary files
146
![Page 147: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/147.jpg)
tf.train.Saver.save(sess, save_path, global_step=None...)tf.train.Saver.restore(sess, save_path)
147
Saves sessions, not graphs!
![Page 148: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/148.jpg)
# define modelmodel = SkipGramModel(params)
# create a saver objectsaver = tf.train.Saver()
with tf.Session() as sess:for step in range(training_steps):
sess.run([optimizer])
# save model every 1000 stepsif (step + 1) % 1000 == 0:
saver.save(sess, 'checkpoint_directory/model_name',global_step=step)
148
Save parameters after 1000 steps
![Page 149: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/149.jpg)
# define modelmodel = SkipGramModel(params)
# create a saver objectsaver = tf.train.Saver()
with tf.Session() as sess:for step in range(training_steps):
sess.run([optimizer])
# save model every 1000 stepsif (step + 1) % 1000 == 0:
saver.save(sess, 'checkpoint_directory/model_name',global_step=step)
149
Specify the step at which the model is saved
![Page 150: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/150.jpg)
global_step = tf.Variable(0, dtype=tf.int32, trainable=False, name='global_step')
150
Global step
Very common in TensorFlow program
![Page 151: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/151.jpg)
global_step = tf.Variable(0, dtype=tf.int32, trainable=False,
name='global_step')
optimizer = tf.train.AdamOptimizer(lr).minimize(loss, global_step=global_step)
151
Global step
Need to tell optimizer to increment global step
This can also help your optimizer know when to decay learning rate
![Page 152: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/152.jpg)
152
Your checkpoints are saved in checkpoint_directory
![Page 153: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/153.jpg)
153
tf.train.Saver
Only save variables, not graph
Checkpoints map variable names to tensors
![Page 154: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/154.jpg)
154
tf.train.SaverCan also choose to save certain variables
v1 = tf.Variable(..., name='v1')
v2 = tf.Variable(..., name='v2')
You can save your variables in one of three ways:
saver = tf.train.Saver({'v1': v1, 'v2': v2})
saver = tf.train.Saver([v1, v2])
saver = tf.train.Saver({v.op.name: v for v in [v1, v2]}) # similar to a dict
![Page 155: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/155.jpg)
saver.restore(sess, 'checkpoints/name_of_the_checkpoint')
e.g. saver.restore(sess, 'checkpoints/skip-gram-99999')
155
Restore variables
Still need to first build graph
![Page 156: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/156.jpg)
# check if there is checkpointckpt = tf.train.get_checkpoint_state(os.path.dirname('checkpoints/checkpoint'))
# check if there is a valid checkpoint pathif ckpt and ckpt.model_checkpoint_path: saver.restore(sess, ckpt.model_checkpoint_path)
156
Restore the latest checkpoint
1. checkpoint file keeps track of the latest checkpoint
2. restore checkpoints only when there is a valid checkpoint path
![Page 157: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/157.jpg)
tf.summaryWhy matplotlib when you can summarize?
157
![Page 158: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/158.jpg)
Visualize our summary statistics during our trainingtf.summary.scalartf.summary.histogramtf.summary.image
158
tf.summary
![Page 159: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/159.jpg)
with tf.name_scope("summaries"): tf.summary.scalar("loss", self.loss) tf.summary.scalar("accuracy", self.accuracy) tf.summary.histogram("histogram loss", self.loss) summary_op = tf.summary.merge_all()
159
Step 1: create summaries
merge them all into one summary op to make managing them easier
![Page 160: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/160.jpg)
loss_batch, _, summary = sess.run([loss, optimizer,
summary_op])
160
Step 2: run them
Like everything else in TF, summaries are ops. For the summaries to be built, you have to run it in a session
![Page 161: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/161.jpg)
writer.add_summary(summary, global_step=step)
161
Step 3: write summaries to file
Need global step here so the model knows what summary corresponds to what step
![Page 162: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/162.jpg)
tf.summary.scalar("loss", self.loss)tf.summary.histogram("histogram loss", self.loss)summary_op = tf.summary.merge_all()
saver = tf.train.Saver() # defaults to saving all variables
with tf.Session() as sess: sess.run(tf.global_variables_initializer()) ckpt = tf.train.get_checkpoint_state(os.path.dirname('checkpoints/checkpoint')) if ckpt and ckpt.model_checkpoint_path: saver.restore(sess, ckpt.model_checkpoint_path)
writer = tf.summary.FileWriter('./graphs', sess.graph) for index in range(10000): ... loss_batch, _, summary = sess.run([loss, optimizer, summary_op]) writer.add_summary(summary, global_step=index)
if (index + 1) % 1000 == 0: saver.save(sess, 'checkpoints/skip-gram', index)
162
Putting it together
![Page 163: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/163.jpg)
See summaries on TensorBoard
163
![Page 164: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/164.jpg)
164
Scalar loss
![Page 165: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/165.jpg)
165
Histogram loss
![Page 166: TensorFlow! An introduction tovedio.leiphone.com/5b32eff053cd7.pdf · 2018-06-27 · TensorFlow integrates seamlessly with NumPy tf.int32 == np.int32 # ⇒ True Can pass numpy types](https://reader034.fdocuments.in/reader034/viewer/2022050523/5fa6bb6cd7978e3410398f84/html5/thumbnails/166.jpg)
166
Toggle run to compare experiments