Master thesis: Using Machine Learning Methods for Evaluating the Performance Metrics
24
An Academic presentation by Dr. Nancy Agnes, Head, Technical Operations, Tutors India Group www.tutorsindia.com Email: [email protected] PERFORMANCE EVALUATION METRICS FOR MACHINE- LEARNING BASED DISSERTATION
-
Upload
TutorsIndia -
Category
Education
-
view
4 -
download
0
description
Evaluation metric plays an important role in obtaining the best possible classifier in the classification training. Thus, choosing an appropriate evaluation metric is an essential key for obtaining a selective and best possible classifier. The associated evaluation metrics have been reviewed systematically that are specifically considered as a discriminator for optimizing a classifier. In general, many possible classifiers use accuracy as a measure to classify the optimal solution during the classification evaluation. Thus, the measurement device that measures the performance of a classifier is considered as the evaluation metric. Different metrics are used to evaluate various characteristics of the classifier induced by the classification method.
Contact:
www.tutorsindia.com
[email protected]
(WA): +91-8754446690
(UK): +44-1143520021
Transcript of Master thesis: Using Machine Learning Methods for Evaluating the Performance Metrics
An Academic presentation by Dr. Nancy Agnes, Head, Technical Operations, Tutors India Group www.tutorsindia.comEmail: [email protected]
PERFORMANCE EVALUATIONMETRICS FOR MACHINE-LEARNING BASEDDISSERTATION
Abstract
Introduction
Evaluation of Machine Learning
Performance measures of ML
Bayesian Inference
Recommended Algorithms
Future Topics
Conclusion
OUTLINE
Today's Discussion
Abstract Evaluation metric plays an important role in obtaining the bestpossible classifier in the classification training.
Thus, choosing an appropriate evaluation metric is anessential key for obtaining a selective and best possibleclassifier.
The associated evaluation metrics have been reviewedsystematically that are specifically considered as a discriminatorfor optimizing a classifier.
In general, many possible classifiers use accuracy as a measureto classify the optimal solution during the classificationevaluation.
Contd...
Thus, the measurement device that measures the performance of a classifier isconsidered as the evaluation metric.
Different metrics are used to evaluate various characteristics of the classifier inducedby the classification method.
Contd...
Contd...
IntroductionAn important aspect of the Machine Learning process isperformance evaluation.
The right choice of performance metrics is one of the mostsignificant issues in evaluating performances.
It is also a complex task. Therefore, it should be performedcautiously in order for the machine learning application to bereliable.
Accuracy is used to assess the predictive capability of amodel on the testing samples.
Machine learning and data mining are the fields that use this major metric.
Another alternate metric that has been used in pattern recognition and machinelearning is the ROC curve.
Thus, there are many performance metrics that have been developed for assessingthe performance of ML algorithms. 1
EvaluationofMachineLearning
The evaluation of categorized tasks is usually done by dividingthe data set into a training data set and a testing data set.
The machine learning method is then trained on the first setof data, while the testing data set calculates the performanceindicators to assess the quality of the algorithm.
ML algorithm’s common issue lies in accessing the limitedtesting and training data.
Thus, overfitting can be a serious issue when assessing theseprograms. In order to tackle this problem, a common methodis, to employ an X-Fold Cross-Validation.
Contd...
The cross-Validation method describes the process of dividing the entire data setinto X parts and employing each set consecutively as the test data set while mergingthe other sets to the training data.
Then the performance indicators are normalized overall validation processes.
There is no ideal performance indicator for every topic that concerns the evaluationof machine learning algorithms since every method has its own flaws andadvantages. 3
Contd...
Image source: Evaluating Learning Algorithms 8
Performancemeasures ofML
The performance of a classification problem can bemeasured easily using this metric.
Here, the output can be of two or more classes. Aconfusion matrix is a table with two dimensions i.e.,“Actual” and “Predicted” and also, both the dimensionshave “True Positives (TP)”, “True Negatives (TN)”, “FalsePositives (FP)”, “False Negatives (FN)”
A. CONFUSION MATRIX
Contd...
Contd...
Accuracy is a metric to measure the accuracy of the model.
Accuracy = Correct Predictions / Total Predictions
Accuracy is the simplest performance metric.
B. ACCURACY
Contd...
Precision is the ratio of True Positives (TP) and the total positive predictions.
The recall is a True Positive Rate. All the positive points that are predicted positive areexplained here.
The mean of precision and recall is termed as F measure.
C. PRECISION & RECALL
Contd...
ROC is a plot between True Positive Rate and False Positive Rate that is estimated bytaking several threshold values of probability scores from the reverse sorted list givenby a model.
D. ROC & AUC
The recent development in machine learning has led many ITprofessionals to focus mainly on accelerating associatedworkloads, especially in machine learning.
However, in the case of unsupervised learning, the Bayesianmethod often works better than machine learning with a limited orunlabelled data set, and can influence informative priors, and alsohave interpretable approaches.
Bayesian inference model has become the most popular andaccepted model over the years as it is a huge compliment tomachine learning.
BayesianInference
Contd...
Some recent revolutionizing research in machine learning accepts Bayesiantechniques like generative Bayesian neural networks (BNN), adversarial networks(GAN), and variational autoencoder.
Contd...
Through visual assessment, it has been proved thatnaive Bayes was the most successful algorithm forevaluating programming performance.
Many detailed analyses were carried out statistically tofind out if there were any considerable differencesbetween the estimated accuracy of each of thealgorithms.
This is important as involved parties may prefer forchoosing an algorithm that they would like to executeand must know if the use of such algorithm(s) wouldresult in a significantly lower performance evaluation.
RecommendedAlgorithms
The analysis identified that all of the ML algorithms, naive Bayes had comparablybest performance evaluation and thus could be used to assess the performance ofML dissertation.
Naive Bayes has been recommended as the best choice for predicting programperformance. 5
Performance measurement has become an emergingfield during the last decades.
Organizations have many motives for usingperformance measures but the most crucial one wouldbe that they increase productivity when utilized properly.
a technique to support performance enhancement inindustrial operations.
1. EVALUATING AND MODIFYING PERFORMANCEMEASUREMENT SYSTEMS.
2. PERFORMANCE ENHANCEMENT
Future Topics
Contd...
The main of this research is to: Build and assess a method that supportsperformance enhancement in industrial operations.
This is performed through many case studies and literature research.
The outcome is a systematically evaluated method for Performance Improvement.
prioritizing performance measures
The main aim is to decrease costs and boost the profitability of organizations tothrive in the market of competition.
3. DETERMINING PERFORMANCE MEASURES OF THE SUPPLY CHAIN
Contd...
Many organizations use the performance measurement (PM) method to supportoperational management and strategic management processes.
This is chiefly important as it leads to modifications in organization strategy and PMsystems.
Approaches are dynamic naturally, while the current measurement systems arepredictable and stable.
Merging strategies with measurement methods is absurd and has created issues fororganizations as the strategic framework modifies.
4. A CURRENT STATE ANALYSIS TECHNIQUE FOR PERFORMANCEMEASUREMENT METHODS.
5. DYNAMIC PERFORMANCE MEASUREMENT METHODS: A FRAMEWORK FORORGANIZATIONS
Contd...
Improving the evaluation performance of an emergingworkload, the most proficient way is to make use of existingsystems.
Another important research implemented is generic Bayesianframeworks for GPUs.
As of now, Bayesian inference is considered the bestcombination of algorithm and hardware platform forperformance evaluation.
Performance evaluation aims to approximate thegeneralization accuracy of a model in future unknown data.
Conclusion
Contd...
In future research, research work can be carried out to improve the evaluation metricseven further.
It would be better to test those metrics on various Machine Learning cloud services to assess the services, to check how easy it is to use the metrics, and what type ofdata can be obtained using the metrics.
Research work must be carried out in this direction to build a framework that wouldhelp in prioritizing the metrics and identify a set of conditions to join results fromvarious metrics. 6
