“I hereby declare that I have read through this report ...eprints.utem.edu.my/19980/1/Fruit...
Transcript of “I hereby declare that I have read through this report ...eprints.utem.edu.my/19980/1/Fruit...
“I hereby declare that I have read through this report entitle “Fruit Sorting By Machine
Vision Technique” and found that it has comply the partial fulfilment for awarding the
degree of Bachelor of Mechatronics Engineering.
Signature : ..........................................................................................
Supervisor’s Name : ..........................................................................................
Date : ..........................................................................................
FRUIT SORTING BY MACHINE VISION TECHNIQUE
KHOR CHIN WE
A report submitted in partial fulfilment of the requirements for the degree of
Bachelor of Mechatronics Engineering
Faculty of Electrical Engineering
UNIVERSITI TEKNIKAL MALAYSIA MELAKA
2016
iii
I declare that this report entitle “Fruit Sorting By Machine Vision Technique” is the
result of my own research except as cited in the references. The report has not been
accepted for any degree and is not concurrently submitted in candidature of any other
degree.
Signature : ..........................................................................................
Name : ..........................................................................................
Date : ..........................................................................................
KHOR CHIN WE
2/6/2016
iv
To my beloved mother and father
v
ACKNOWLEDGEMENT
First, I would like to thank to my supervisor, Pn. Nursabillilah Mohd Ali who
accept me as her FYP student to carry out her research with the title of ‘Fruit Sorting by
Machine Vision’. Pn. Nursabillilah gives me a lot of valuable advices and guides me
throughout the whole semester. Without her guidance and persistent help, I would not be
able to complete this Final Year Project Report 1 and Final Year Project Report 2.
Besides, special thanks my Machine Vision subject lecturer, Dr. Muhammad
Herman bin Jamaluddin who spend time to teach us patiently and provide us a lot of useful
information. Also, my grateful thanks to those all my friends who are willing to help me in
the understanding of image processing subject. Last, not forget to give my great
appreciation my beloved parents who mentally support and encourage me. Without helps
of particular that I mentioned above, I would face many difficulties in completing my Final
Year Report 1 and Final Year Report 2.
vi
ABSTRACT
Nowadays, automation system such as machine vision system is widely used in
many industrial. Fruit sorting by machine vision technique is a potential automation system
which is able to obtain their target in term of productivity and quality. However, there are
several problems such as camera resolution, illumination and performance that need to be
considered for using the machine vision technique. Hence, the objectives of this project is
to design a high accuracy algorithm that is able to classify apples based on color and size.
Furthermore, the performance of fruit sorting system is needed to be evaluated and
optimized by manipulating the illumination, thresholding value and camera distance.
MATLAB is an analysis tool that has been used in this research project. There are three
experiments such as HSV value comparison, thresholding value performance comparison
and the different camera distance performance comparison. The experiments are aim to
define the tolerance limits of hue value and number of pixels. The tolerance limits are used
for the purpose of categorizing the type of apple. At the end of the experiment, the analysis
and discussion explained about the performances of each variation of parameters. The
problems such as low illumination condition and the limitation of different thresholding
value and distance of camera are described as well in the analysis and discussion. As
conclusion, the performance in term of accuracy and precision of the designed algorithm
has been evaluated and optimized at the end of this project.
vii
ABSTRAK
Pada masa kini, sistem automasi seperti sistem penglihatan mesin digunakan
semakin luas dalam pelbagai industri. Sistem pembahagian buah-buahan dengan
mengunakan teknik penglihatan mesin adalah satu sistem automasi yang berpotensi dan
bermampu untuk mendapatkan meningkatkan produktiviti dan kualiti dalam sesebuah
industri. Walau bagaimanapun, terdapat beberapa masalah seperti resolusi kamera,
pencahayaan dan prestasi yang perlu dipertimbangkan semasa menggunakan teknik
penglihatan mesin. Oleh itu, projeck penyelidik ini bertujuan untuk merekabentuk sebuah
algoritma yang berketepatan tinggi dan mampu membezakan epal berdasarkan warna dan
saiz. Tambahan pula, prestasi sistem pembahagian buah-buahan dapat dioptimumkan
dengan memanipulasi pencahayaan, nilai ambang dan jarak kamera. MATLAB adalah alat
analisis yang telah digunakan dalam projek penyelidikan ini. Terdapat tiga eksperimen
seperti HSV perbandingan, prestasi perbandingan dan jarak camera perbandingan telah
dilaksanakan untuk menentukan batas-batas toleransi nilai warna dan bilangan piksel. Had
toleransi digunakan untuk mengkategorikan jenis-jenis epal. Bahagian analisis dan
perbincangan telah menjelaskan bahawa tentang prestasi setiap perubahan parameter.
Masalah-masalah seperti pencahayaan rendah, had nilai ambang yang berbeza dan jarak
kamera berbeza telah diterangkan serta dalam analisis dan perbincangan. Kesimpulannya,
prestasi dari segi ketepatan algoritma telah dinilai dan dioptimumkan pada akhir projek ini.
viii
TABLE OF CONTENTS
CHAPTER TITLE PAGE
ACKNOWLEDGEMENT v
ABSTRACT vi
ABSTRAK vii
TABLE OF CONTENTS viii
LIST OF TABLES xi
LIST OF FIGURES xii
LIST OF APPENDICES xv
1 INTRODUCTION 1
1.1 Introduction 1
1.2 Motivation 2
1.3 Problem Statement 4
1.4 Objectives 5
1.5 Scopes 5
1.6 Thesis Organization 6
2 LITERATURE REVIEW 7
2.1 Fruit Sorting and Grading System 7
2.2 Background of Machine Vision 8
2.3 Components of Machine Vision System 9
2.3.1 Lighting 9
ix
2.3.2 Frame Grabber 10
2.3.3 Software - MATLAB 11
2.3.4 Cameras 11
2.3.5 Image Processing 12
2.3.6 Example of Image Processing Algorithm 18
2.4 Summary of Image Processing Technique 23
3 METHODOLOGY 24
3.1 Description of Research Project 25
3.2 Description of Fruit Sorting System Flow 27
3.3 Component Used 29
3.3.1 Digital Camera 29
3.3.2 Microcontroller (Arduino UNO) 30
3.3.3 DC Gear Motor (SPG30) 31
3.3.4 Single Channel 5V Relay 32
3.3.5 Light Bulb 33
3.4 Materials Used 34
3.4.1 Apples 34
3.5 Prototype of Fruit Sorting System 34
3.6 Analysis Tools 36
3.6.1 MATLAB 36
3.7 Experiment Setup 37
3.7.1 Experiment 1:Comparison of HSV 37
3.7.2 Experiment 2:Thresholding Value Comparison 38
3.7.3 Experiment 3:Distance of Camera 40
4 RESULT AND DISCUSSION 41
4.1 Experiment 1: Comparison of HSV 41
x
4.1.1 Precision Analysis 41
4.1.2 Performance Evaluation 45
4.1.3 Hue Tolerance Setting 46
4.2 Experiment 2: Thresholding Value Comparison 50
4.2.1 Image Thresholding Analysis 50
4.2.2 Performance Evaluation 52
4.3 Experiment 3: Comparison of Distance of Camera 55
4.3.1 Number of Pixels Tolerance Setting 55
4.3.2 Performance Evaluation 61
5 CONCLUSION AND RECOMMENDATION 62
5.1 Conclusion 62
5.2 Recommendation 63
REFERENCES 64
APPENDICES 67
xi
LIST OF TABLES
TABLE TITLE PAGE
2.1 Characteristics of CCD and CMOS Cameras [19] 12
2.2 Summary of Image Processing Technique 23
3.1 Properties of Labeled Regions [46] 31
4.1 Results in different TH value 51
4.2 Detectability (Red Apple) in different TH value 52
4.3 Detectability (Green Apple) in different TH value 53
xii
LIST OF FIGURES
FIGURE TITLE PAGE
1.1 The number of robots in use in the five largest global markets,
projected to 2017 [3]
2
1.2 The efficiency of implementing manual facilities and automated
facilities [4]
3
2.1 Automated fruit sorting and grading system [16] 10
2.2 Image Processing Main Steps [5] 12
2.3 Brightness matrix adjustment [19] 13
2.4 Contrast matrix adjustment [19] 14
2.5 Cropping Image in matrix mode [19] 15
2.6 Image Zooming in matrix mode [19] 15
2.7 Rotation of Image in matrix mode [19] 16
2.8 Image Dilation in matrix mode [19] 17
2.9 Image Erosion in matrix mode [19] 17
2.10 Image captured from mobile device is converted into L*a*b 18
2.11 Typical segmentation techniques (a) Thresholding, (b) edge-based
segmentation and (c) region based segmentation [5] age
19
2.12 Standard deviation measurement superimposed on each object in
the im
19
2.13 Pipeline of detection algorithm [14] 20
2.14 Image extracting flow of fruit sorting system [13] 21
xiii
2.15 Block diagram of thresholding process [5] 22
3.1 Research project flow chart 26
3.2 Fruit Sorting System flow chart 28
3.3 CMOS Camera 29
3.4 Arduino UNO 30
3.5 DC Gear Motor 31
3.6 Single Channel 5V Relay 32
3.7 Fluorescent light bulbs 33
3.8 Incandescent light bulbs 33
3.9 Type of apples 34
3.10 Conveyer belt 35
3.11 Inspection box 35
3.12 Assembled prototype 35
3.13 MATLAB Computer Software 36
3.14 Experiment setup 37
3.15 Installation of light bulb 38
3.16 Tracking process 39
3.17 Distance variation 40
4.1 HSV color space in a conical object 42
4.2 Low precision (left) and high precision (high) 43
4.3 Standard deviation in different illuminations 44
4.4 Overall performance in different illuminations 45
4.5 Control limit of hue value (Red) 46
4.6 Control limit of hue value (Green) 47
4.7 Rotten and yellowish red apples 48
4.8 Ripe apple versus fresh apple 49
xiv
4.9 Red and Green apples hue value region 51
4.10 Accuracy of red apple detection 54
4.11 Accuracy of green apple detection 54
4.12 Number of pixels in camera position 20cm (Small apple) 56
4.13 Number of pixels in camera position 20cm (Big apple) 56
4.14 Number of pixels in camera position 25cm (Small apple) 57
4.15 Number of pixels in camera position 25cm (Big apple) 57
4.16 Number of pixels in camera position 30cm (small apple) 58
4.17 Number of pixels in camera position 30cm (Big apple) 58
4.18 Number of pixels in camera position 35cm (Small apple) 59
4.19 Number of pixels in camera position 35cm (Big apple) 59
4.20 Number of pixels in camera position 40 cm (Small apple) 60
4.21 Number of pixels in camera position 40 cm (Big apple) 60
4.22 Overall performance in different camera positions 61
xv
LIST OF APPENDICES
APPENDIX TITLE PAGE
A Program code 69
B Gantt chart for FYP1 and FYP2 73
C Table of Result 74
D Publication 81
1
CHAPTER 1
1. INTRODUCTION
This chapter review the introduction of automation fruit sorting system. The
motivation, problem statement, objective and scope about this research project are discussed
in this chapter. Besides, the thesis organization is descripted at the last part of this chapter.
1.1 Introduction
There are several handling steps of fruit process in fruit processing industries. The
fruit after collecting from farm, it will undergo the process of washing, sorting, grading,
packing, transporting and storage before going to sell in market [1]. Fruit sorting and grading
can be considered as the heaviest processing task compared to the other steps. Normally,
more time and effort will be invested in this process. This process can be either operated in
manual sorting method or automatic sorting method. Manual fruit sorting is a method
operated by human in order to identify and separate fruit manually while automatic fruit
sorting is a method can be operated by using the technique of machine vision. One of the
advantages of machine vision is high precision [2]. Machine vision can be carried to get
measurements without any touch on the surface on objects, so there is no damage or
dangerous on the fragile parts. Besides that, machine vision can operate continuously and the
operating and maintenance costs are low. The high flexibility of visual system becomes the
reason for people to choose machine vision system rather than optical sensors.
2
1.2 Motivation
Manpower contributed by labour is flexible and fast. Meanwhile, machine vision is
type of automatic system which is dependable and precise. Many manufacturing industries
have tried to implement the automated system to reduce the use of manpower.
The Figure 1.1 shows the number of robots in use in the five largest global markets.
From the graph, the demand of robots from 4 countries except Japan shows the trend of
increase from year 2012 to 2017. According to the German-based International Federation of
Robotics, China is already the world’s largest market for industrial robots [3]. These trends
will keep increasing in the next few years. Hence, the trend obviously indicate that automatic
fruit sorting method is more preferable choice for fruit industrial compared to manual sorting
method.
Figure 1.1: The number of robots in use in the five largest global markets, projected to
2017 [3]
3
The Figure 1.2 represents the differences in term of profit, cost, and sales between
manual operating system and automated system. The sales of both systems are the same but
the operating cost of automated system decrease drastically while the profit increases by time.
The trends clearly indicate that most of the industries choose automation system rather that
manual system.
Figure 1.2: The efficiency of implementing manual facilities and automated facilities
[4]
The reason of using machine vision system that can be able to increase the profit is
high productivity. However, some industries hire more labours in order to seek and achieve
higher productivity but normally manual fruit sorting causes unwanted problems that may
directly influence the production result. Hence, training is basically to be provided for labours
in order to produce skillful, experience and quality worker but high investment cost on
labours will highly reduce a company profit.
Machine vision technique is a reliable system to be implemented in fruit sorting for
fruit industries. It is time-saving and high efficient system which is able to significantly hit
the production target in fruit industries. The designed system not only can operate
continuously day and night without any tiredness, the production output quality is also hugely
trustable. The capability of fruit sorting by machine vision is truly high compare to human
vision.
4
Besides, the visual results generally are highly dependent on labours mental and
physical fitness. Humans are not able to have fully concentration while making many
decisions within a certain time. Human decisions are variable and difficult to be standardized.
In addition, the task of visualizing fruit is quite tedious. It probably causes the problems of
difficulty to hire workers and labours turnover. Therefore, fruit sorting by using the technique
of machine vision is recommended system to be implemented.
1.3 Problem Statement
Machine vision technology is highly used in factory automation. It is a system that
camera captures products’ picture and interprets them to find defects and categorize them.
Nowadays, camera has become common in our life. Camera has been transformed from
analogue to digital camera and even the mobile phone has the function of high quality camera.
Thus, with this huge change of camera, it has become a selective device to detect object or
feature.
There are thousand types of camera in the market. Different cameras have different
image processing quality. Camera quality is a constraint that should be considered in machine
vision system. Normally, high Mega-Pixels of camera will cause a challenge in having too
much information. In real system operating time, high resolution camera will cause
unacceptable processing time [5]. Large image sizes require longer time to compute and
collect data from camera. Thus, selecting an appropriate type of camera based on the
requirement is able to improve a vision system in term of efficiency.
Besides, another problem should be overcome is the illumination conditions that
surround the camera. The colour detection and ability to distinguish an object are can be
affected by the surrounding environment [5]. The surrounding lighting condition should be
standardized in order to get precise analysis result. In addition, reflection of light on an object
causes analysis errors during image processing.
For an algorithm design, the accuracy is important in measuring the size and colour of
apples. A good algorithm is able to reduce the percentage of the errors such as systematic
5
error and random error from happening. Thus, the result consistency is highly maintained
with a good design of image processing algorithm.
1.4 Objectives
The objectives of the research project are:
1. To design and develop an algorithm for fruit classification based on size and colour.
2. To evaluate the system performance for fruit sorting and classification in term of
accuracy and precision.
3. To optimize the system performance by manipulating the illumination, thresholding value
and distance of camera.
1.5 Scopes
The scopes of the research project are listed as the following:
1. Apple is chosen as the analysis material for this fruit sorting system.
2. Algorithms are developed based on the size and colour of apple.
3. Algorithms are designed by using MATLAB computer software.
4. Analysis is based on the data of static image.
6
1.6 Thesis Organization
This report is the study about the fruit sorting by using the machine vision technique.
In this report, the chapter 1 will cover about the motivation and problem statement for
designing an image processing algorithm of fruit sorting system. Besides that, chapter 1 will
state about the objective and scope. For chapter 2, it will cover the theoretical background
and basic principle of machine vision. The previous related research of automatic fruit sorting
system and machine vision are also covered in chapter 2 and the methods and techniques in
designing an image processing algorithm of fruit sorting system will be discussed in chapter 3.
Finally, the result and conclusion will be covered in the last two chapters which are chapter 4
and chapter 5.
7
CHAPTER 2
2. LITERATURE REVIEW
This chapter presents the overview about the preview research project which related
to fruit sorting and grading system. Furthermore, the methods and theories of image
processing which related to fruit sorting process are reviewed and analyzed. The constraints
of applying machine vision technique are described in this chapter as well.
2.1 Fruit Sorting and Grading System
The demand of fruit based on the quality is getting high in market. Sorting and
grading are the important process to maintain the quality of fruit [6]. Many machine vision
systems have been proposed for agricultural applications such as fruit grading, fruit maturity
analyzing and fruit sorting. Machine vision has been used in the research of mango inspection
and grading system, mango maturity prediction and vegetable automation sorting by machine
vision [7]. The fruits such as citrus fruits, apple and strawberry have been used.
A distributed computer machine vision system for automated inspection and grading
of fruits was presented to control the quality of fruits [8]. The automation system is designed
for food industries so that the quality of food is able to meet the international standards.
Generally, maturity of fruit needs experts to be justified. Manual sorting is time cosuming,
laborious process and diffult to judge by human according to fruit maturity. Maturity
prediction system is used to predict the maturity of harvested mango [9]. The system is
design to solve the problem of manual sorting system.
8
For some small agro-industrial companies, it is burdensome to procure a high cost
sorting machine in order to maintain and increase their product quality. A portable smart
sorting and grading machine by using computer vision was presented to solve the problem of
heavy built-up system [9].
Grading and sorting system by using machine vision is very useful in maintaining the
consistency, uniformity and depletion of time. An integral automation industrial fruit and
vegetable system and the research is aim to meet the market standards, classified according to
quality levels based on the maturity, size, weight, density and skin defect [10]. To analyzing
the image, algorithm should be designed. There are different kinds of fruits that can be
different by applying the image processing technique. Image processing is widely used in
different agriculture products such as apple, tomatoes, mango, strawberry, cherries, orange,
lemon and banana was presented [11]. Besides, the application analysis of machine vision
technology in agricultural inspection consist various kind of problems [12]. The problems
such as surrounding light intensity should be taken serious to get a precise result.
2.2 Background of Machine Vision
Machine Vision (MV) is the technology and method that provide digital imaging
information from the environment and use it for analysis purpose. The analysis method by
using MV can be applied in many fields such as industrial inspection, medical visualization,
law enforcement and artistic effect. MV is highly demanded in many fields due to speed,
consistency and rest period. Machine Vision can process hundreds or thousands pieces of
information in a few seconds. Whereas human operative system cannot process information
as fast as machine vision based system. Besides that, the consistency of output process by
human is truly unstable compared to machine vision system. Humans are not able to work 24
hours per day but machine vision system can operate continuously with stop unless
maintenance.
Machine Vision is the method about digital processing image. Digital processing
image define as the process from a given input image in order to obtain a required output
image. There are basically 3 categories of digital image processing which are image
processes, image analysis and computer vision. For image processes, it is used in the
9
application of camera. The camera will capture a scene and the internal processor will
process the data of input image. The image will undergo noise removal, sharpening,
brightness adjustment and etc. Finally, the output image will be stored in the memory and it
has been ready for photo printing. For image analysis, the image after capturing or recording,
it will be undergoes the analysis process. The analysis process is basically running based on
the features, shape and texture of an object. The application of image analysis can be used in
object recognition and segmentation. Computer vision is basically the high level image
process which is used the computer system and digital processing image system. For this
digital processing image category, it needs to design a set of program based on requirements
and used for scene understanding and autonomous navigation. Normally, this kind of
technique has been applied for face recognition, fingerprint system and object tracking
system.
2.3 Components of Machine Vision System
The basic components of a machine vision system consist of a camera, a computer
and a lighting system [13]. The computer should be equipped with an image acquisition
board. Besides, computer software is needed for transmitting electronic signals to computers,
acquiring images and performing storage and processing of the images.
2.3.1 Lighting
The lighting unit selection is dependable for different applications. With a suitable-
chosen lighting system, recognition and analysis process will be taken easier in the optimal
way [14]. Also, eliminating the tedious image processing procedures is one of the reasons to
choose a best lighting system.
Due to the high sensitivity of of image processing to illumination, constant
environment conditions is important in achieving a robust performance of algorithm. Hence,
methods of controlling the environment conditions is to prevent the camera and objects from