Symbolic Rules Extraction From Trained Neural Networks

26/5/201226/5/2012 Koushal KumarKoushal Kumar 11

Symbolic Rules Symbolic Rules Extraction From Extraction From Trained Neural Trained Neural

Networks Networks

Koushal Kumar

M .Tech CSE

Mob: +918968939621

What are Artificial Neural Networks?What are Artificial Neural Networks?

• Artificial Neural Networks are powerful computational systems consisting of many simple processing elements connected together to perform tasks analogously to biological brains.

• They are massively parallel, which makes them efficient, robust, fault tolerant and noise tolerant

• They can learn from training data and generalize to new situations..

The limitation of Neural NetworkThe limitation of Neural Network

The major criticism against Neural Network is that The major criticism against Neural Network is that

decision given by neural networks is difficult to decision given by neural networks is difficult to

understand by a human being. Reasons for this areunderstand by a human being. Reasons for this are

Knowledge in Neural Networks are stored as real Knowledge in Neural Networks are stored as real

valuesvalues

parameters (weights and bias) of networksparameters (weights and bias) of networks

Neural Networks are unable to explain its internal Neural Networks are unable to explain its internal

processing how they come to particular decisionprocessing how they come to particular decision

This behavior makes Neural Networks Black Box in This behavior makes Neural Networks Black Box in

NatureNature

Continue..Continue..

Rules extraction From Rules extraction From Neural Networks.Neural Networks.

So to overcome the Black Box nature of Neural So to overcome the Black Box nature of Neural

Networks we need to extract rules from Neural Networks we need to extract rules from Neural

Networks so that the user can gain a better Networks so that the user can gain a better

understanding of the decision process. understanding of the decision process.

following types of rules can be extracted from following types of rules can be extracted from

neural networksneural networks

I) M OF N types rulesI) M OF N types rules

II) Fuzzy rulesII) Fuzzy rules

continue..continue..

III) III) IF THEN RULESIF THEN RULES

IV) Decision RulesIV) Decision Rules

V) First order logic rulesV) First order logic rules

From all above types of rules IF THEN RULES From all above types of rules IF THEN RULES

and Decision rules are easy to understand and Decision rules are easy to understand

then others kind of rules.then others kind of rules.

J48 Algorithm for extracting J48 Algorithm for extracting decision treesdecision trees

• J48 is an algorithm used to generate a decision tree.J48 is an algorithm used to generate a decision tree.• Developed by quinlan and most widely used decision Developed by quinlan and most widely used decision

tree induction algorithm.tree induction algorithm.• It is based upon greedy search approach i.e select the It is based upon greedy search approach i.e select the

best attribute and never looks back to reconsider early best attribute and never looks back to reconsider early choices.choices.

• It select the best attribute according to its entropy It select the best attribute according to its entropy value.value.

• More preference will be given to that attribute which More preference will be given to that attribute which has more value of entropy.has more value of entropy.

Before normalizationBefore normalization

After normalizationAfter normalization

MATLAB SimulatorMATLAB Simulator

• Matlab stands for matrix laboratory.• It integrate computation, visualization, and

programming in an easy-to-use environment.• MATLAB is a package that has been purpose-

designed to make computations easy, fast and reliable.

• Matlab can be used in math and computation, algorithm development, simulation purposes.

• MATLAB is a powerful system that can plot graphs and perform a large variety of calculations with numbers.

Weka simulatorWeka simulator

• WEKA is abbreviation of Waikato Environment for Knowledge Analysis.

• Weka is open source simulator with machine learning algorithms.

• The Weka workbench contains a collection of visualization tools and algorithms for data analysis and predictive modelling..

Training data set and its HistogramTraining data set and its Histogram

Histogram of input target valuesHistogram of input target values

Histogram of testing set as InputsHistogram of testing set as Inputs

Histogram of testing set as Target valuesHistogram of testing set as Target values

Neural networks Training with BPANeural networks Training with BPA

Best Training Performance curveBest Training Performance curve

Gradient Performance Curve After Training & Gradient Performance Curve After Training & TestingTesting

Data import window in wekaData import window in weka

J48 algorithm Output (pruned)J48 algorithm Output (pruned)

Results from pruned TreeResults from pruned Tree

J48 algorithm Output (Unpruned)J48 algorithm Output (Unpruned)

Unpruned Tree ResultsUnpruned Tree Results

Results of classification from Decision TreeResults of classification from Decision Tree

Decision Tree Extracted From Exp no 7Decision Tree Extracted From Exp no 7

The Following Rule Set Is Obtained From the above Decision The Following Rule Set Is Obtained From the above Decision

TreeTree I. Applying Remove Redundancy ConditionsI. Applying Remove Redundancy Conditions IF Children ≥ 1 AND Children >2AND Children >3 THEN Marital status =YES

Children ≥ 1 is more specific than Children >3 and Children > 2. So we remove all such conditions

Rule 1:

a) IF Current_act = NO AND Age ≤ 48.0 AND Sex = FEMALE AND Children ≤ 0 THEN Region Town

b) IF AGE > 48.0 AND Region Suburban AND Current_act = NO then Pep = NO

II. For every pair decision trees Remove redundancy rules. For example

Rule 1: IF Age ≤60 AND Salary ≤ 3500 AND Pep = NO THEN Mortage = YES

Rule 2: IF Age ≤ 50 AND Salary ≤ 3500 AND Pep = NO THEN Mortage = YES

New Rule: IF Age ≤ 50 AND Salary ≤ 3500 AND Pep = NO THEN Mortage =YES

III. Remove more specific rules. The rules with a condition set which is a superset of

another rule should be removed. For example

Continue….Continue….

• Rule 1: IF Age ≤ 60 AND Region = Rural AND Saving_ act = YES THEN Pep = NO

• Rule 2: IF Age <= 60 AND Children <= 1 AND Region =Rural AND saving act = YES THEN Pep= NO

• Rule 3: IF Region = Rural AND saving _ act =YES THEN Pep = NO

• New Rule: IF Region = Rural AND saving _ act =YES THEN Pep = NO

Comparison of J48 algorithm with Others ClassifiersComparison of J48 algorithm with Others Classifiers

Graphical comparisons of algorithmsGraphical comparisons of algorithms

Data Flow diagram of J48 algorithmData Flow diagram of J48 algorithm

Paper PublishedPaper Published

• ““Extracting Explanation From Artificial Neural Networks” is Extracting Explanation From Artificial Neural Networks” is published in International Journal of Computer Science and published in International Journal of Computer Science and Information Technologies.Information Technologies.

• ““Advanced Applications of Neural Networks and Artificial Advanced Applications of Neural Networks and Artificial Intelligence: A Review” has been selected in International journal of Intelligence: A Review” has been selected in International journal of information technologies and computer science and it will published information technologies and computer science and it will published on May June volume of journal. on May June volume of journal.

• Seminar: Published Research paper on “Symbolic Rules Extraction Seminar: Published Research paper on “Symbolic Rules Extraction From Trained Neural Networks” in two days UGC Sponsored From Trained Neural Networks” in two days UGC Sponsored National Seminar on Social Implications of Artificial Intelligence National Seminar on Social Implications of Artificial Intelligence organized in KMV College in Jalandhar. organized in KMV College in Jalandhar.

ReferencesReferences

• Olcay Boz AT & T Labs”Converting a trained neural network to Olcay Boz AT & T Labs”Converting a trained neural network to a decision tree DecText-Decision Tree Extractor.a decision tree DecText-Decision Tree Extractor.

• J.T Yao Dept of computer sci university of Regina “Knowledge J.T Yao Dept of computer sci university of Regina “Knowledge extracted from Trained neural networks whats next?extracted from Trained neural networks whats next?

• Simon Haykin ”Neural networks a Comprehensive foundation” Simon Haykin ”Neural networks a Comprehensive foundation” Pearson education(second edition).Pearson education(second edition).

• R. Davis, B.G. Buchanan, and E. Shortcliff, “Production Rules R. Davis, B.G. Buchanan, and E. Shortcliff, “Production Rules as aas a

• Representation for a Knowledge Based Consultation Progra”,Representation for a Knowledge Based Consultation Progra”,

• Artificial Intelligence, 1977, vol. 8(1), pp.15-45.Artificial Intelligence, 1977, vol. 8(1), pp.15-45.

• Knowledge Extraction from the Neural ‘Black Box’ in Knowledge Extraction from the Neural ‘Black Box’ in Ecological Monitoring Journal of Industrial and Systems Ecological Monitoring Journal of Industrial and Systems Engineering Vol. 3, No. 1, pp 38-55 Spring 2009Engineering Vol. 3, No. 1, pp 38-55 Spring 2009

Symbolic Rules Extraction From Trained Neural Networks

Education

Transcript of Symbolic Rules Extraction From Trained Neural Networks

Seerupa Das -- Dynamic on-line Clustering and State Extraction an Approach to Symbolic Learning

SensOrchestra: Collaborative Sensing for Symbolic Location …hengtzec/papers/mobicase10_se... · 2010-11-21 · Sensor Module Data Pre-processing Bluetooth Scan Image Feature Extraction

Symbolic Logic Outline - University of Oklahomacs1313.ou.edu/symbolic_logic_lesson.pdf · Symbolic Logic Lesson CS1313 Spring 2018 1 Symbolic Logic Outline 1. Symbolic Logic Outline

SYMBOLIC REGRESSION OF IMPLICIT EQUATIONS - Genetic … · Symbolic Regression of Implicit Equations 75 be trained using sensor data from intact operation only, but will be required

Symbolic Logic

Symbolic Washer

Symbolic Head

NEURAL NETWORKS AND GENETIC ALGORITHMS · Knowledge extraction by forming symbolic rules from the internal parameters of neural networks is now becoming an accepted technique for

Symbolic interaction

Rules and Feature Extraction for Microcalcifications ...Rules and Feature Extraction for Microcalcifications Detection in Digital Mammograms Using Neuro-Symbolic Hybrid Systems and

Knowledge Extraction From Trained Neural Networks

Developmental trajectories of symbolic and non- symbolic number

Efﬁcient Symbolic Differentiation for Graphics Applications · Efﬁcient Symbolic Differentiation for Graphics Applications Brian Guenter Microsoft Research ... Symbolic differentiation

Project-Team CORTEX Neuromimetic Intelligence · Keywords: data analysis, knowledge extraction, neuro-symbolic integration, pre-processing, visualization. Artiﬁcial neural networks

Limited Warrantiespdf.lowes.com/warrantyguides/190402003065_warranty.pdf · 2020-02-10 · extraction cleaning performed by a trained qualified carpet care professional, at least

Darnton Symbolic

EASy Summer 2006Non-Symbolic AI lec 41 Non-Symbolic AI lecture 4 A major difference between Symbolic and Non-Symbolic AI approaches is in modelling, or.

Soil Vapor Extraction, Air Sparging and Aquifer Depression ...€¦ · remediation system • Identified necessary system modifications to overcome site conditions changes • Trained

Web view2-3 word sentences. Symbolic play and potty trained. Primitive Reflexes. Reflex. Eliciting; Description; Moro. Sudden head extension. ... Erb’s palsy - upper N root injury

Text Mining, Information and Fact Extraction Part 1: Introduction and Symbolic Techniques Marie-Francine Moens Department of Computer Science Katholieke.