Outline

• Class organization

• Artificial Neural networks– Introduction– History of Artificial Neural Networks– Applications

• Survey

04/12/23 CAP 5615: Introduction to Neural Networks 2

Programming Approaches

• Programming approaches– Abstract the problem– Design an algorithm to solve the problem

• An algorithm is a precise step-by-step procedure to solve the problem

– Implement the algorithm

• Artificial neural network approaches for solving problems are different than typical programming approaches in computer science

04/12/23 CAP 5615: Introduction to Neural Networks 3

An Example

• No hands across America– Sponsored by Delco Electronics, AssistWare

Technology, and Carnegie Mellon University– Navlab 5 drove from Pittsburgh, PA to San

Diego, CA, using the RALPH computer program. – The trip was 2849 miles of which 2797 miles

were driven automatically with no hands • Which is 98.2%

04/12/23 CAP 5615: Introduction to Neural Networks 4

An Example – cont.

04/12/23 CAP 5615: Introduction to Neural Networks 5

An Example – cont.

04/12/23 CAP 5615: Introduction to Neural Networks 6

An Example – cont.

• RALPH – Rapidly Adapting Lateral Position Handler– Uses video images to determine the location of

the road ahead and the appropriate steering direction to keep the vehicle on the road

– It uses a multi-layer perceptron with backpropagation as the learning algorithm

04/12/23 CAP 5615: Introduction to Neural Networks 7

An Example – cont.

04/12/23 CAP 5615: Introduction to Neural Networks 8

Artificial Neural Network Approaches

• Artificial neural network approaches– Motivated by the biological counterpart

• The brain, which has about 1011 neurons

• All biological neural functions including learning and memory are stored in the neurons and the connections between them

• The brain is also called “biological neural network”

– Try to “learn” from examples

04/12/23 CAP 5615: Introduction to Neural Networks 9

Artificial Neural Networks – cont.

• Inspiration from neuroscience– The brain consists of 1011 highly interconnected

neurons

04/12/23 CAP 5615: Introduction to Neural Networks 10

Artificial Neural Networks – cont.

• Each neuron is a complicated process– The best available mathematical model is called

Hodgkin-Huxley Model

• We are not interested in recreating the neurons but trying to abstract their main functionalities

04/12/23 CAP 5615: Introduction to Neural Networks 11

McCulloch and Pitts Model

• McCulloch and Pitts Model– It is a simple model as a binary threshold unit– The model neuron first computes a weighted sum

of its inputs– It outputs one if the weighted sum is above a

threshold and zero otherwise

otherwise 0

0 if 1(x)

))(()1(

x

tnwtnj

ijiji

04/12/23 CAP 5615: Introduction to Neural Networks 12

McCulloch and Pitts Model – cont.

wi1

win

04/12/23 CAP 5615: Introduction to Neural Networks 13

McCulloch and Pitts Model – cont.

• McCulloch and Pitts networks (consisting of multiple layers of McCulloch and Pitts neurons) are Turing-machine equivalent in turns of computation capability– In other words, a synchronous assembly of

McCulloch-Pitts neurons can perform any computation that an ordinary digital computer can, though they can be slow or cumbersome.

04/12/23 CAP 5615: Introduction to Neural Networks 14

Digital Computers

• All digital computers consist of digital gates to control the dataflow and perform computation– How many different types of logic gates do we

need in order to be able to build any computer?

04/12/23 CAP 5615: Introduction to Neural Networks 15

McCulloch and Pitts Model – cont.

04/12/23 CAP 5615: Introduction to Neural Networks 16

History of Artificial Neural Networks

• General theories of learning– Late 19th and early 20th centuries– Interdisciplinary in physics, psychology, and

neurophysiology– Hermann Helmholtz, Ernst Mack, Ivan Pavlov

04/12/23 CAP 5615: Introduction to Neural Networks 17

History of Artificial Neural Networks – cont.

• McCulloch and Pitts Model– 1940s– Artificial neural networks in principle could

compute any arithmetic and logical functions• Origin of the modern neural networks

– Warren McCulloch and Walter Pitts– McCulloch and Pitts network cannot learn

• A network must be designed for each function

• For computers, we write a program for each function

04/12/23 CAP 5615: Introduction to Neural Networks 18

History of Artificial Neural Networks – cont.

• Hebbian learning rule– 1949– A mechanism for learning in biological neurons

• The first learning algorithm

– Donald Hebb

04/12/23 CAP 5615: Introduction to Neural Networks 19

History of Artificial Neural Networks – cont.

• Perceptron– 1950s– A network can learn to solve pattern recognition

problems– Frank Rosenblatt– Unfortunately, it only solve a limited class of

problems• Linear separability

04/12/23 CAP 5615: Introduction to Neural Networks 20

History of Artificial Neural Networks – cont.

• Widrow-Hoff learning rule– 1960s– Similar to perceptron

• Widely used in adaptive signal processing

• Most long distance phone lines use Widrow-Hoff for echo cancellation

04/12/23 CAP 5615: Introduction to Neural Networks 21

History of Artificial Neural Networks – cont.

• Minsky and Papert’s book– 1969– Pointed out the intrinsic limitations of

Perceptrons and neural networks at that time• The neural network area became quiet for almost

twenty years

04/12/23 CAP 5615: Introduction to Neural Networks 22

History of Artificial Neural Networks – cont.

• New Stage– 1980s and 1990s– New powerful computers– Two new concepts

• Statistical mechanics to analyze neural network behaviors

• Backpropagation algorithm– Overcome the limitation of perceptron

04/12/23 CAP 5615: Introduction to Neural Networks 23

History of Artificial Neural Networks – cont.

• Current stage– A standard tool to solve many practical problems– Theoretical and statistical foundations of neural

networks– New neural network architectures and learning

algorithms• Support vector machines

• Adaboost

04/12/23 CAP 5615: Introduction to Neural Networks 24

Applications

• Aerospace

• Banking

• Defense

• Financial

• Medical

• Securities

• Transportation

• See http://www.calsci.com/

04/12/23 CAP 5615: Introduction to Neural Networks 25

Financial Forecasting

• Use neural networks to predict stock market trends

• Use neural networks to perform credit scoring

• Use neural networks to perform fraud detection

• See http://www.calsci.com/Applications.html#Stock%20Applications

04/12/23 CAP 5615: Introduction to Neural Networks 26

Online Neural Network Resources

• Neural Java– Neural network tutorial with Java

• Neural network resources– http://www.makhfi.com/resources.htm

04/12/23 CAP 5615: Introduction to Neural Networks 27

Topics to Be Covered

• Basic neural network architectures and learning algorithms– Perceptron– Multiple layer perceptron and backpropagation– Associate learning– Competitive networks– Hopfield network– These topics will be on the exams

04/12/23 CAP 5615: Introduction to Neural Networks 28

Topics to Be Covered – cont.

• Advanced topics– Will NOT be on the exams but very important to

have a comprehensive understanding of the area– Radial basis neural networks– Support vector machines– AdaBoost