Developing Neural Networks using Visual StudioJames McCaffreyMicrosoft Research2-401

AgendaSlide 1: What types of problems does a neural network solve?Slide 2: What exactly is a neural network?Slide 3: How does a neural network actually work?Slide 4: Understanding activation functions.Slide 5: Alternatives to neural networks.Slide 6: Understanding neural network training.Slide 7: Neural network over-fitting.Slide 8: Developing with Visual Studio.Slide 9: Summary and resources.

What type of problems does a neural network solve?

Training data

Independent variables/predictors/attributes/regressors/x-values

“The thing to classify (predict)”/ dependent variable/y

What is a neural network?38

51,000

M

Pres

3.8

5.1

-1.0

0.0

1.0

0.0

0.43

Age

Income

Sex

0.37

Dem0.20

Religion

Politics

input hidden output

The perceptron building block0.10

0.30

0.20 ??

W0 = 4.0

W1 = -5.0

W2 = 6.0

b = 2.0 a. (0.1)(4.0) + (0.2)(-5.0) + (0.3)(6.0) = 1.2b. 1.2 + 2.0 = 3.2c. Activation(3.2) = 0.73d. ?? = 0.73

Technical note: most neural network literature treats the bias as a weight that has a dummy input which is always a 1.0 value.

Four most common activation functionsLogistic sigmoidOutput between [0, 1]y = 1.0 / (1.0 + e–x)

Hyperbolic tangentOutput between [-1, +1]y = tanh(x) = (ex – e-x) / (ex + e-x)

Heaviside stepOutput either 0 or 1if (x < 0) then y = 0 else if (x >= 0) then y = 1

SoftmaxOutputs between [0, 1] and sum to 1.0y = (e-xi) / Σ (e-xj)

Alternatives to neural networksThe six main alternatives to using a neural network1. Linear regression: assumes data can be modeled as

y = ax1 + bx2 + . . + k2. Logistic regression: assumes data can be modeled as

y = 1.0 / ( 1.0 + e-(ax1 + bx2 + . . + k) )3. Naive Bayes: assumes input data are all independent, output is binary.4. Decision trees: do not work well for complex data, assumes binary

output.5. Adaptive boosting: relatively new and effectiveness not well

understood, assumes binary output.6. Support vector machines: extremely complex implementation, assumes

binary output.

Alternatives to neural networksNeural networks pros and consPro: can model any underlying math equation!Pro: can handle multinomial output without resorting to tricks.Con: moderate complexity, requires lots of training data.Con: must pick number hidden nodes, activation functions, input/output encoding, error definition.Con: must pick training method, training “free parameters,” (and over-fitting defense strategy).

TrainingBack-propagationFastest technique.Does not work with Heaviside activation.Requires “learning rate” and “momentum.”

Genetic algorithmSlowest technique.Generally most effective.Requires “population size,” “mutation rate,” “max generations,” “selection probability.”

Particle swarm optimizationGood compromise.Requires “number particles,” “max iterations,” “cognitive weight,” “social weight.”

Avoiding model over-fittingWhat is it?Symptom: Model is great on predicting existing data, but fails miserably on new data.Roulette example: red, red, black, red, red, black, red, red, black, red, red, ??A serious problem for all classification/prediction techniques, not just neural networks.

Five most common techniquesUse lots of training data.Train-Validate-Test (early stop when error on validate set begins to increase).K-fold cross validation.Repeated sub-sampling validation.Jittering: deliberately adding noise data to make over-fitting impossible.Quite a few exotic techniques also available (weight penalties, Bayesian learning, etc.).

Weka

TLC

TLC API

Visual Studio

Demo: the famous Iris problem

SummaryExisting neural network tools are difficult or impossible to integrate into a software system.Commercial and Open Source API libraries work well for some machine learning tasks but are extremely limited for neural networks.To develop neural networks using Visual Studio you must understand seven core concepts: feed-forward, activation, data encoding, error, training, free parameters, and over-fitting.Once the concepts are mastered, implementation with Visual Studio is not difficult (but not easy either).

ResourcesConcepts:ftp://ftp.sas.com/pub/neural/FAQ.html#questions

Weka:http://www.cs.waikato.ac.nz/ml/weka/

Custom C#:http://msdn.microsoft.com/en-us/magazine/jj190808.aspx

Special enhanced demo code for Build 2013 attendees:http://www.quaetrix.com/Build2013.html

Thank You!Developing neural networks using Visual Studio.

2013 Build ConferenceJune 25–28, 2013San Francisco, CA

Dr. James McCaffreyMicrosoft Researchjammc@microsoft.com

Session 2-401

Device distribution starts after sessions conclude today (approximately 6:00pm) in the Big Room, Hall D. If you choose not to pick up your devices tonight, distribution will continue for the duration of the conference at Registration in the North Lobby.

Get your goodies

Acer Iconia W3, Surface Pro, and Surface Type Cover

Evaluate this session

Scan this QR code to evaluate this session and be automatically entered in a drawing to win a prize!

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.