Performance Evaluation Metrics for Machine-Learning Based Dissertation
24
An Academic presentation by Dr. Nancy Agnes, Head, Technical Operations, Tutors India Group www.tutorsindia.com Email: [email protected] PERFORMANCE EVALUATION METRICS FOR MACHINE- LEARNING BASED DISSERTATION
-
Upload
TutorsIndia -
Category
Education
-
view
3 -
download
0
description
Evaluation metric plays an important role in obtaining the best possible classifier in the classification training. Thus, choosing an appropriate evaluation metric is an essential key for obtaining a selective and best possible classifier. The associated evaluation metrics have been reviewed systematically that are specifically considered as a discriminator for optimizing a classifier. In general, many possible classifiers use accuracy as a measure to classify the optimal solution during the classification evaluation. Thus, the measurement device that measures the performance of a classifier is considered as the evaluation metric. Different metrics are used to evaluate various characteristics of the classifier induced by the classification method.
Contact:
www.tutorsindia.com
[email protected]
(WA): +91-8754446690
(UK): +44-1143520021
Transcript of Performance Evaluation Metrics for Machine-Learning Based Dissertation
An Academic presentation byDr. Nancy Agnes, Head, Technical Operations, Tutors India Group www.tutorsindia.comEmail: [email protected]
PERFORMANCE EVALUATION METRICS FOR MACHINE- LEARNING BASED DISSERTATION
Abstract
Introduction
Evaluation of Machine Learning
Performance measures of ML
Bayesian Inference
Recommended Algorithms
Future Topics
Conclusion
OUTLINE
Today's Discussion
Abstract
Evaluation metric plays an important role in obtaining the best possible classifier in the classification training.
Thus, choosing an appropriate evaluation metric is an essential key for obtaining a selective and best possible classifier.
The associated evaluation metrics have been reviewed systematically that are specifically considered as a discriminator for optimizing a classifier.
In general, many possible classifiers use accuracy as a measure to classify the optimal solution during the classification evaluation.
Contd...
Thus, the measurement device that measures the performance of a classifier is considered as the evaluation metric.
Different metrics are used to evaluate various characteristics of the classifier induced by the classification method.
Contd...
Introduction An important aspect of the Machine Learning process is
performance evaluation.
The right choice of performance metrics is one of the most significant issues in evaluating performances.
It is also a complex task. Therefore, it should be performed cautiously in order for the machine learning application to be reliable.
Accuracy is used to assess the predictive capability of a model on the testing samples.
Contd...
Machine learning and data mining are the fields that use this major metric.
Another alternate metric that has been used in pattern recognition and machine learning is the ROC curve.
Thus, there are many performance metrics that have been developed for assessing the performance of ML algorithms. 1
Evaluation of Machine Learning
The evaluation of categorized tasks is usually done by dividing the data set into a training data set and a testing data set.
The machine learning method is then trained on the first set of data, while the testing data set calculates the performance indicators to assess the quality of the algorithm.ML algorithm’s common issue lies in accessing the limited testing and training data.
Thus, overfitting can be a serious issue when assessing these programs. In order to tackle this problem, a common methodis, to employ an X-Fold Cross-Validation.
Contd...
The cross-Validation method describes the process of dividing the entire data setinto X parts and employing each set consecutively as the test data set while merging the other sets to the training data.
Then the performance indicators are normalized overall validation processes.
There is no ideal performance indicator for every topic that concerns the evaluation of machine learning algorithms since every method has its own flaws and advantages. 3
Contd...
Image source: Evaluating Learning Algorithms 8
Performance measures of ML
The performance of a classification problem can be measured easily using this metric.
Here, the output can be of two or more classes. A confusion matrix is a table with two dimensions i.e., “Actual” and “Predicted” and also, both the dimensions have “True Positives (TP)”, “True Negatives (TN)”, “False Positives (FP)”, “False Negatives (FN)”
A. CONFUSION MATRIX
Contd...
Contd...
Accuracy is a metric to measure the accuracy of the model.
Accuracy = Correct Predictions / Total Predictions
Accuracy is the simplest performance metric.
B. ACCURACY
Contd...
Precision is the ratio of True Positives (TP) and the total positive predictions.
The recall is a True Positive Rate. All the positive points that are predicted positive are explained here.
The mean of precision and recall is termed as F measure.
C. PRECISION & RECALL
Contd...
ROC is a plot between True Positive Rate and False Positive Rate that is estimated by taking several threshold values of probability scores from the reverse sorted list given by a model.
D. ROC & AUC
The recent development in machine learning has led many IT professionals to focus mainly on accelerating associated workloads, especially in machine learning.
However, in the case of unsupervised learning, the Bayesian method often works better than machine learning with a limited or unlabelled data set, and can influence informative priors, and also have interpretable approaches.
Bayesian inference model has become the most popular and accepted model over the years as it is a huge compliment to machine learning.
Bayesian Inference
Contd...
Some recent revolutionizing research in machine learning accepts Bayesian techniques like generative Bayesian neural networks (BNN), adversarial networks (GAN), and variational autoencoder.
Through visual assessment, it has been proved that naive Bayes was the most successful algorithm for evaluating programming performance.
Many detailed analyses were carried out statistically to find out if there were any considerable differences between the estimated accuracy of each of the algorithms.
This is important as involved parties may prefer for choosing an algorithm that they would like to execute and must know if the use of such algorithm(s) would result in a significantly lower performance evaluation.
Recommended Algorithms
Contd...
The analysis identified that all of the ML algorithms, naive Bayes had comparably best performance evaluation and thus could be used to assess the performance of ML dissertation.
Naive Bayes has been recommended as the best choice for predictingprogram performance. 5
1.EVALUATING AND MODIFYINGPERFORMANCE MEASUREMENT SYSTEMS.
Performance measurement has become an emerging field during the last decades.
Organizations have many motives for using performance measures but the most crucial one would be that they increase productivity when utilized properly.
2. PERFORMANCE ENHANCEMENT
a technique to support performance enhancement in industrial operations.
Future Topics
Contd...
The main of this research is to: Build and assess a method thatsupports performance enhancement in industrial operations.
This is performed through many case studies and literature research.
The outcome is a systematically evaluated method for Performance Improvement.
3. DETERMINING PERFORMANCE MEASURES OF THE SUPPLY CHAIN
prioritizing performance measures
The main aim is to decrease costs and boost the profitability oforganizations to thrive in the market of competition.
Contd...
4.A CURRENT STATE ANALYSIS TECHNIQUE FORPERFORMANCE MEASUREMENT METHODS.
Many organizations use the performance measurement (PM) method tosupport operational management and strategic management processes.
This is chiefly important as it leads to modifications in organization strategy and PM systems.
5.DYNAMIC PERFORMANCE MEASUREMENT METHODS: A FRAMEWORKFOR ORGANIZATIONS
Approaches are dynamic naturally, while the current measurement systemsare predictable and stable.
Merging strategies with measurement methods is absurd and has created issues for organizations as the strategic framework modifies.
Improving the evaluation performance of an emerging workload, the most proficient way is to make use of existing systems.
Another important research implemented is generic Bayesian frameworks for GPUs.
As of now, Bayesian inference is considered the best combination of algorithm and hardware platform for performance evaluation.
Performance evaluation aims to approximate the generalization accuracy of a model in future unknown data.
Conclusion
Contd...
In future research, research work can be carried out to improve the evaluation metrics even further.
It would be better to test those metrics on various Machine Learning cloud services to assess the services, to check how easy it is to use the metrics, and what type of data can be obtained using the metrics.
Research work must be carried out in this direction to build a framework that would help in prioritizing the metrics and identify a set of conditions to join results from various metrics. 6
Contd...
CONTACT US
UNITED KINGDOM+44-1143520021
INDIA+91-4448137070
![ExploitMeter: Combining Fuzzing with Machine Learning for ...ghyan/papers/pac17.pdfassess software security, such as attack surface metrics [22], vulnerability density metrics [7],](https://static.fdocuments.in/doc/165x107/5ff735dfd7541b59ee41ca21/exploitmeter-combining-fuzzing-with-machine-learning-for-ghyanpaperspac17pdf.jpg)
