Introduction & Fundamentals

• Part I: Introduction

• Part II: Fundamental Concepts

• Part III: Classification Lab

Part I: IntroductionWhat Is The Problem? Our world is full of data. After collection and

organization, data, if we are lucky, becomes information. In today's interconnected world, information exists in electronic form that can be stored and transmitted instantly. Challenge is to understand, integrate, and apply information to generate useful knowledge (“actionable intelligence”)

Are we drowning in data/information but starved for knowledge??

Data, Data Everywhere…….

How do we extract knowledge from noisy mass of data ?

Every source of data, from process and product manufacturing to medical research to activity in financial markets to patient examinations, to the billions of consumer and business purchase transactions that occur every day, is influenced by “other data” from the surrounding environment. Our world is a noisy and messy source of data - virtually nothing is known with certainty. Knowledge is based on data analysis that accommodates uncertainty.

Empirical Models that Learn

What Is The Solution? Interpretation requires data acquisition, cleaning

(preparing the data for analysis), analysis, and presentation in a way that permits knowledgeable decision making and action. Key is to extract information about data from relationships buried within the data itself.

Tools and Technology Human brain is most powerful pattern recognition engine ever invented, however, it is not very good at serially processing huge quantities of discrete data.

Enter New Breed of Processor: Artificial Neural Networks

• Instead of programming computational system to do specific tasks, teach system how to perform task

• To do this, generate Artificial Intelligence System- AI• Empirical model which can rapidly and accurately

find the patterns buried in data that reflect useful knowledge

• One case of these AI models is neural networks• AI systems must be adaptive – able to learn from

data on a continuous basis

Artificial Intelligence

• Artificial Intelligence techniques such as Neural networks, genetic algorithms and fuzzy logic are among the most powerful tools available for detecting and describing subtle relationships in massive amounts of seemingly unrelated data.

• Neural networks can learn and are actually taught instead of being programmed.

• Teaching mode can be supervised or unsupervised

• Neural Networks learn in the presence of noise

Question: What are Artificial Neural Networks?

Answer:

………

………

………

………

Output Layer

Hidden Layer 2

Hidden Layer 1

Input Buffer

Biological Basis of ANNs

• Animals exhibit intelligence……………….

• Biological neural networks…………..

• Human beings can benefit from simulation of biological neural networks on computers. These are Artificial Neural Networks (ANN)

• Artificial Neural Networks (ANN) are……….

• ANN’s represent an attempt to simulate……

Biological Basis of ANNs

• ANNs have many names: connectionist systems, neural nets, neurocomputers, parallel distributed processing systems, machine learning algorithms, etc

• Each neuron is linked to its neighbors with varying coefficients of connectivity that represent the strengths of these connections

• Learning…………..

What are Neural Networks Used For?

2 Basic Types of Learning

Neural Nets

Supervised Learning

UnsupervisedLearning

Types of Problems

• Mathematical Modeling (Function Approximation)

• Classification• Clustering• Forecasting• Vector Quantization• Pattern Association• Control• Optimization

Mathematical Modeling (Function Approximation)

• Modeling – often mathematical relationship between two sets of data unknown analytically

• No closed form expression available• Empirical data available defining output parameters for each input• Data is often noisy• Need to construct a mathematical model that correctly generates

outputs from inputs (See fig 1.22 on page 29)• Approx carried out using ………………………….• Network learns ………………………………………….

• Trained Neural Net can be substituted for ………………………….• Fast computation, reasonable approximation

Classification

• Assignment of objects to specific class

• Given a database of objects and classes of those objects

• Deduce ……………………………

• Create a classifier that will ………………..

Clustering

• Grouping together objects similar to one another• Usually based on some “distance” measurement

in object parameter space• Objects and distance relationships available• No prior info on classes or groupings• Objects clustered based on ………………..

• Clustering may precede ………………………• Similar to statistical k-nearest neighbor

clustering method

Forecasting

• Prediction of future events based on history• Laws underlying behavior of system sometimes

hidden; too many related variables to handle• Trends and regularities often masked by noise• Prediction system must be able to ……………….

• Time series forecasting – special case of ………. • Weather, Stock market indices, machine

performance

Vector Quantization

• Kohonen classifier – most well known• Object space divided into several connected

regions• Objects classified based on proximity to regions• Closest region or node is “winner”• Form of compression of high dimensional input

space• Successfully used in many geological and

environmental classification problems where input object characteristics often unknown

Pattern Association

• Auto-associative systems useful when incoming data is a corrupted version of actual object e.g. face, handwriting

• Corrupt input sample should trigger ……………

• Require a response which …………………

• May require several iterations of repeated modification of input

• Will be discussed under ……………………..

Control• Manufacturing, Robotic and Industrial machines

have complex relationships between input and output variables

• Output variables define state of machine• Input variables define machine parameters

determined by operation conditions, time and human input

• System may be static or dynamic• Need to map inputs to outputs for stable smooth

operation• Examples include chemical plants, truck backup,

robot control

Optimization• Requirement to improve system performance or

costs subject to constraints• Maximize or Minimize …………………….• “Terrain” of objective function typically very

…………………………………..• Large number of …………….. affecting

objective function (high …………… of problem)

• Design variables often subject to ……………….• Lots of local ………………………..• Neural nets can be used to find global optima

(Ch 7)

Now for some Practical Applications !

Neural networks have performed successfully where other methods have not, predicting system behavior, recognizing and matching complicated, vague, or incomplete data patterns. Apply ANNs to pattern recognition, interpretation, prediction, diagnosis, planning, monitoring, debugging, repair, instruction, control

• GOTCHA!• Biomedical Signal Processing• Biometric Identification• System Reliability • Business• Spiral Inductor Modeling• Target Tracking

Neural Network Applications

• Common use for neural networks is to project what will most likely happen - demand prediction. Can help in setting ……………………. For example, hospital emergency rooms, communications systems, power distribution, consumer goods manufacture and storage, …………

• Extremely successful in categorization, pattern recognition. System classifies object under investigation (e.g. an illness, a pattern, a picture, a chemical compound, a word, the financial profile of a customer) into one of numerous possible categories. This triggers …………………………………

GOTCHA!

• GOTCHA………………………….

• Current surveillance and reconnaissance systems (S&R) structured to observe huge areas, attempt to detect movement of hostile forces.

• “Forensic” approach: …………………………………….• • USAF Command and Control Battlelab (C2B): use past S&R imagery

to locate an explosion, run data back in time to identify the vehicle (or object) which carried munitions, “lock” onto vehicle & backtrack to locate significant portions of path – assembly areas, passenger pick-up, arming site, and any other spot with intelligence value

• Technology: imagery, net-centric gathering & sharing of data, target identification, …………………………..

• Collaboration of information ………………………..

Biomedical Signal ProcessingBiometric Identification

• “Instant Physician” developed using neural net

• Net presented with a set of symptoms, medical records

• Output is best diagnosis and treatment

• Finger prints never change. Bifurcations or “Minutae” ………………………………………

• Minutiae-based techniques find minutiae points and map their relative placement on the finger

• Large volumes of fingerprints are collected and stored everyday in a wide range of applications including forensics, access control, and driver license registration

• Automatic recognition of people based on fingerprints requires ………………………….

• FBI database contains 70 million fingerprints!

System Control & Reliability

• Backing Up a truck to a loading dock is a difficult problem for a novice, easy for an experienced driver

• Very difficult problem mathematically• Can train a neural net to

…………………….

• Automobile airbags can do serious damage …………………………

• Accelerometer MEMS are …………….

• System reliability continuously assessed & failure pre-empted by correct interpretation of data from accelerometers

Business• Mortgage Risk Assessment –

reduces delinquency rates• Inputs include years of employment,

# of dependents, property info, income, loan-to-value-ratio

• Output is ……………………….

• Prediction of of behavior of stock market indices

• Requires knowledge of ……………..

• Time series forecasting• Short and long term predictions

Spiral Inductor Modeling

• In today’s portable wireless communications market, demand is for low cost, low power dissipation, high frequency IC building blocks that incorporate spiral inductors on the silicon substrate

• Challenge: ………………………• Empirical models widely reported based on actual measurements but

non-predictive and do not permit re-design of inductor layout • Neural network approach serves as basis for ………………… …….

and permits ………………………. from post-optimization inductor circuit-level parameters

• Ilumoka& Park, Proc. SSST 04, Georgia Tech, Atlanta GA, March 2004

Automatic Target Tracking & Recognition

• Algorithms for automated tracking & recognition of targets have recently been identified by the National Critical Technologies Panel as of critical importance to mission of the Department of Defense.

• Automated target recognition, localization, and tracking in the presence of ……………. is an important signal processing problem

• Algorithms have been developed for …………………………… such as those that occur during active jamming, non-cooperative maneuvering & complex battlefield scenarios of the future

MUTUAL FUNDS: NEURAL NETWORKS versus REGRESSION ANALYSIS

• For neural networks to be successful, they must outperform methods currently being used in the marketplace.

• Mutual funds are basically ……………………………………. …. Mutual funds have become a major force on Wall Street over the past few years. They function much like an individual security and their prices should reflect all public information. Relationships between …………………………………………………. are very hard to forecast. For years, regression analysis has been a popular tool investors have used to forecast …………………… of mutual funds.

• Investors know that neural networks might be able to pinpoint these relationships better than old methods.

• Predictions made for Net Asset Value using 15 economic variables as inputs showed that neural networks were 40% better as tools for forecasting: …………………………….(difference between actual and forecasted NAV) was ………. for neural nets as compared to …….. for regression.

• Important reason for superior performance of neural networks is its ………….. It was able to look at all aspects of relationships, whereas regression analysis was ………………………………………………

• .

Historical Perspective

Origin of ANNs is neurobiological research in the early 20th century

Several fronts of attack:• Neurobiologists: How do nerve cells behave

when stimulated by an electric current?• Psychologists: How is learning accomplished

by animals?• Mathematicians: How can we apply gradient

descent to neuron learning?• McCulloch & Pitts – 1st math model of neuron• Learning rules devp by Hebb (1945),

Rosenblatt (1958), Widrow-Hoff (1961)• Several Limitations encountered• See pages 5-7 for chronological history

Neurons

• Biological neurons are ………………………. receiving & sending signals across synapses to other neurons via tree-like dendrites at both ends (see fig 1.3, page 8)

• Artificial neurons are ……………………….. (fig 1.4, page 10) in which inputs are weighted and summed to produce a weighted sum output net

• Activation function f is ……………………………… f(net) corresponding to firing frequency of the biological neuron

• Several different activations are possible including ………………

• Although neural networks have great potential, there is still a long way to go. Complex neural networks have less than the brain power of a ………………. (100,000 neurons). Human brains contain about …………………….. neurons.

• Neural network software sales annually exceed $2 billion because they offer ………………………………..

Part II: Fundamental Concepts

_ a _a _ _ _ m

What is a conceptual framework that permits investigation of phenomena in a field of enquiry ?

Answer

……………………..

Important ANN Parameters

1. Architecture (or Topology)2. Learning Rule3. Paradigm: Combination of Architecture & Learning Rule – complete

neural network model - emulates ………………………….

……………….. ………………..

PARADIGM

Architecture• What is architecture? ……………………..

• Single node, single layer insufficient for practical problems; require multiple nodes connected by excitatory (positive) or inhibitory (negative) weights

• 3 types of nodes: ………..(receive external inputs), ………. (generate external outputs) , ……….. (no interaction with external environment)

• Nodes often partitioned into layers (layered nets); intra-layer connections may be prohibited (acyclic) – see fig 1.13, 1.14 page 19

• Feedforward nets – ……………………….

• Recurrent nets – ……………………………

MLP Feedforward

Modular NetART

Hopfield Net

LVQ

BAM

Hamming

Learning Rule• What is a learning rule or algorithm ?• Learning is process by which neural

net adapts itself to stimulus in order to produce a desired response

• Learning rule is …………………….

• Just as individuals learn differently, neural network have different learning rules

• Learning may be Supervised or Unsupervised

• Supervised learning requires that when the input stimuli are applied, the desired output is known a priori

Backpropagation

Competitive

…………….Correlation

………………..

ParadigmsExamples of Paradigms• Adaptive Resonance Theory (ART) (………………………………)• Modular Neural Networks MNN (……………………………………)• Learning Vector Quantization (……………………)

Backpropagation

Modular Architecture

Modular Neural Network Paradigm successfully applied to Spiral Inductor Modeling

Course Objective

To understand, successfully apply and evaluate neural network paradigms for problems in science, engineering and business

Supervised Learning

Select Neural Net …………… & …………………..

Present………. to neural net

Supply ……………………

Train neural net to learn ………………………………

Test or Verify that network has learned and can ……………. well

………………….. network

Will begin with supervised learning. Procedure is:

Biological Neural Net Example : Human Eye

• Eye is ………………. of brain – powerful bio-electrochemical computer• Light enters thru ……and focuses on …….. (similar to photographic film)• Retina is dense matrix of photoreceptors – ……………………………….• Rods – form ……… images in dim light 100X more sensitive than cones• Cones handle …………….., 4X faster than rods in response to light• Rods, Cones convert light to electric signals, total 130million, 6% cones• Highest conc of cones is in ……….., 1.5mm diameter, 2000 cones• Retinal neurons arranged in layers receive electric signals via synapses• Pre-processing of image takes place at retinal level in neuron network• Signals arrive at …………….(1 million), axons of which form ………….• Optic nerve fibers terminate in lateral geniculate nucleus LGN in brain

lens

retina

fovea

opticnerve

conesrods

Visual Pathways

Optic Nerves from L & R eye to R & L LGN

conesrods

Layers of retinal neurons

Left

Right

Questions on Biological Example

• Q1: What is basic building block of nervous system?

• Q2: What are basic parts of neuron?

• Q4:How can eye be adjusted to large differences in light intensity? (e.g. sun to star is 10billion range

• Q3: If cones were absent from retina, how would color pictures be perceived?

Botanical Application Example:Iris Flower Classification

Part III: Classification Lab

Botanical Application Example:Iris Flower Classification

• 3 species of Iris – SETOSA, VERSICOLOR, VIRGINICA• Each flower has parts called PETALS & SEPALS• Length and Width of sepal & petal can be used to

determine iris type• Data collected on large number of iris flowers • For example, in one flower petal length=6.7mm and

width=4.3mm also sepal length=22.4mm & sepal width =62.4mm. Iris type was SETOSA

• Neural net will be trained to determine specie of iris for given set of petal and sepal width and length

Iris training and testing data:

Sepal Length Sepal Width Petal Length Petal Width Iris Class

0.224 0.624 0.067 0.043 Setosa

0.749 0.502 0.627 0.541 Veracolor

0.557 0.541 0.847 1.000 Virginica

0.110 0.502 0.051 0.043 Setosa

0.722 0.459 0.663 0.584 Veracolor

0.776 0.416 0.831 0.831 Virginica

0.196 0.667 0.067 0.043 Setosa

0.612 0.333 0.612 0.584 Veracolor

0.612 0.416 0.812 0.875 Virginica

0.055 0.584 0.067 0.082 Setosa

0.557 0.541 0.627 0.624 Veracolor

0.165 0.208 0.592 0.667 Virginica

0.027 0.376 0.067 0.043 Setosa

0.639 0.376 0.612 0.498 Veracolor

0.667 0.208 0.812 0.710 Virginica

0.306 0.710 0.086 0.043 Setosa

0.196 0.000 0.424 0.376 Veracolor

0.612 0.502 0.694 0.792 Virginica

0.137 0.416 0.067 0.000 Setosa

Iris Flower Classification• Since output is non-numeric, will use a 3bit binary code

to specify output• 1 0 0 represents SETOSA• 0 1 0 represents VERSICOLOR• 0 0 1 represents VIRGINICA• Columns 1-4 rep sepal L, W and petal L, W in mmX0.01• Sample data below

0.224 0.624 0.067 0.043 1 0 00.749 0.502 0.627 0.541 0 1 00.557 0.541 0.847 1 0 0 10.11 0.502 0.051 0.043 1 0 00.722 0.459 0.663 0.584 0 1 00.776 0.416 0.831 0.831 0 0 10.196 0.667 0.067 0.043 1 0 0