Normalization of Vehicle License Plate Images Based on ...

Acta Polytechnica Hungarica Vol. 17, No. 6, 2020

– 193 –

Normalization of Vehicle License Plate Images

Based on Analyzing of Its Specific Features for

Improving the Quality Recognition

Aizhan Tlebaldinova

S. Amanzholov East Kazakhstan State University

Kazakhstan Str. 55, 070004 Ust-Kamenogorsk, Kazakhstan

e-mail: [email protected]

Natalya Denissova, Olga Baklanova

D. Serikbayev East Kazakhstan State Technical University

Faculty of Information Technology

A. K. Protazanov Str. 69, 070004 Ust-Kamenogorsk, Kazakhstan

e-mail: {NDenisova, OBaklanova}@ektu.kz

Iurii Krak

Taras Shevchenko National University of Kyiv

Volodymyrska Str. 60, 01033 Kyiv, Ukraine

e-mail: [email protected]

György Györök

Óbuda University, Alba Regia Technical Faculty

Budai út 45, H-8000 Székesfehérvár, Hungary

[email protected]

Abstract: This paper presents technique for recognizing license plates structured

characters of the Republic of Kazakhstan. This technique includes methods for converting

the geometric-topological characteristics of license plates and the method for classifying

alphanumeric characters by using cluster analysis. Developed modified algorithm for

character recognition based on methods of contour analysis and template method with the

addition of proposed transformations.

https://e.mail.ru/[email protected]

http://science.univ.kiev.ua/en/researchgroups/[email protected]

A. Tlebaldinova et al. Normalization of Vehicle License Plate Images Based on Analyzing of Its Specific Features for Improving the Quality Recognition

– 194 –

Keywords: image processing; geometric-topological characteristics; contour analysis;

character segmentation; character recognition

Introduction

Nowadays there are a lot of recognition systems for license plates of vehicles, that

are characterized by rapid response time and high recognition rate even if

automobiles move at high speed. However, in order to provide continuous up and

running of such systems, special expensive hardware is required. To purchase the

equipment of this type is not always reasonable in case vehicles speed is not high.

This relates to gasoline service stations, parking areas, storefronts, internal

development roads and garage co-ops, etc. The demand for researches and

development of such technologies for solving problems of this level made it

necessary to develop methods and models adopted both for detection of special

structures on an image and for analysis of structured symbols for identification of

text information reflected on registered license plates.

Generally, methodologies [1-5] used for the development of license plates

recognition systems can vary due to different conditions of their operation and

peculiarities of the national numbering system. However, on the one hand, most

such recognition systems have acommon structure that realizes standard

information technology. The technology, as a rule, consists of the following steps:

image generation, image preprocessing, object localization, image segmentation,

and recognition. On the other hand, at present there are some methods of image

characteristic points detection – points (areas) that possess high local information

content [6-10]. Such points of interest for many methods are stable enough to

photometric and geometric image distortions including irregular brightness

variations, shift, angling, scale conversion, view distortion. The initial stage of

recognition task is selecting of characteristic points on an image. Main advantage

of characteristic points used for such tasks is relative simplicity and rate of their

identification. Besides, sometimes it isn’t always possible to distinguish other

characteristic features (sharp outlines or areas), as for characteristic points they

can be identified in the vast majority of cases. As a consequence, it is possible to

replace stages of image formation and preprocessing for object localization with

the method of object area detection used according to characteristic points, and

only after that to provide steps for identification by the common geometric and

morphological methods, knowing the information about the object structure.


– 195 –

1 Problem Formulation

The goal of the work is to study the provided stages of data conversion for the

realization of the informational system of license plate recognition in the Republic

of Kazakhstan (RK). The information technology includes basic stages:

localization, preprocessing of the localized object, segmentation, and recognition.

Localization is a very challenging stage in the task of license plate recognition. A

license plate by itself is rather informative due to sufficient visibility of the

information provided on it. Generally, inverse colors are used for characters and

backgrounds. As for a vehicle and surrounding changing background they are

rather many-coloured and vary due to brightness variations, shifts, angling, scale

conversion, and view distortions. Methods of image characteristic points detection

are suggested for these very conditions: SIFT [11, 12], Speeded-Up Robust

Features (SURF) [11, 13], Histogram of Oriented Gradients (HOG) [14, 15],

Local Binary Patterns (LBP) [16] and others.

In order to locate vehicle license plates the authors have suggested to use the

contour analysis method that enables to define the shape of the image entirely. It

also contains all the required information for their identification according to the

shape. Such an approach enables not considering internal points of an image and

thus reducing the amount of processed information. As a consequence, it can

facilitate the work of the system in real-time mode.

Contour implies a number of pixels that separate the object from the background.

Freeman Chain Code will be taken as the method of coding contours that are

applied for the representation of boundaries. They are represented by the sequence

of straight lines segments of different lengths and directions. 4- and 8-side grid is

the basis of this representation. The length of every segment is determined by the

resolution of the grid, and directions are set by the selected code. In order to

represent all the directions in 4-side grid, 2 bits is enough, bur 3 bits is required

for 8-side grid of chain code.

Such an approach enables moving from two-dimensional objects to their one

dimensional (vector) description, i.e. development of chain code can be

considered the procedure of image vectorization.

The most evident method of getting contours from an image is the Canny edge

detector [17]. Any methods of binary image acquisition can be taken for this use:

threshold transformation, object selection according to colour, Canny algorithm

applies 4 filters for detecting horizontal, vertical, diagonal lines, as lines can be in

different directions on an image. It results in a binary image containing lines.

When the localized license plate is preprocessed, it is converted as a single line.

License plates specific feature is that they are of different formats (single-line or

two-line) and they have areas of different colours. So it is more convenient to

convert them to the standard one-line type of the same grey colour for further

processing. Character’s recognition is the final stage in the procedure of license


– 196 –

plate recognition. For this purpose, the images containing license plate symbol

generated in the result of preprocessing and segmentation are analyzed on the

basis of number plate topological features.

2 License Plate Localization

Contour analysis approximates the data in the image to simple geometric shapes.

Thus, this method allows filtering the obtained contours by the signs of the ratio of

sides, area, perimeter, angle, etc.

The general sequence of actions of the algorithm is as follows: the input of the

algorithm comes pre-processed image. Then, it is checked for binarization of the

image. Using the binarized image, contours are selected on the image.

To find the contour of the license plate it is necessary to filter all found contours

by criteria reflecting the characteristics of the license plate in comparison with

other objects of the world. According to the scheme of the algorithm of contour

analysis for the localization of license plates should be filtered by the following

characteristics of the contour:

- in form, i.e. the contour should contain only 4 vertices, pairwise-parallel oppo-

site sides and angles between adjacent vectors close to 90°, and also have a length

in relation to the width, satisfying the values of proportions;

- by size, i.e. the contour must meet the requirements of the minimum and

maximum area of the content area in order to be able to extract data (symbol

images) of sufficient information (sufficient size).

As a result, the algorithm highlights with a red border all areas in which the

license plate can be placed.

In case of full compliance, the circuit is added to the list of detected license plates

of the vehicle.

2.1 License Plate Preprocessing

Preprocessing includes methods of license plates geometric and topological

characteristics transformation. It is required for the conversion of license plates of

all types to standard single-line type of the same grey colour.

According to RK license plate standards [18-20] license plates of the Republic of

Kazakhstan differ significantly from each other both in size (520x112 mm,

280x202 mm, 240x202 mm, 288x202 mm, 260x242 mm), and in the number of

lines (single-line, two-line). In order to transform two-line number plates into a

standardized single-line type, the following algorithm is provided in Figure 1.


– 197 –

Figure 1

The transformation algorithm two-line case to a one-line license plate

When a number plate localization is finished, the characteristics of the found

objects are additionally estimated. These operations are required for the

conversion of two-line license plates into one-line license plates. A license plate

filling ratio (1) and aspect ratio (2) are taken as permanent characteristics.

(1)

(2)

where Sobj - the area of the found object, hbar – the height of a license plate, wbar –

the width of a license plate.

The value Zcoef for a license plate will be equal to the number known in advance

as the value of aspect ratio ZAspRat is constant. Thus, the number of an object of

interest (a license plate) is defined, basing on the value Zcoef. It has been found out

that aspect ratio value ZAspRat for single-line license plates is 0.2. If the value of

aspect ratio is higher or lower than 0.2, it means that the localized object is a two-

line license plate. In order a license plate could be converted into single-line type,

first, it is necessary to define what standard it belongs to, as they have

considerable differences in the structure and appearance. The Republic of

Kazakhstan flag image in the upper left corner of a number plate of 2012 serves as

the distinguishing feature.

,Sobj

Zcoef h w

bar bar

.hbarZ

AspRat wbar


– 198 –

In order to find the left edge of the flag, the algorithm of changing a number plate

colour from blue to white is used. In case flag image is found, a number plate is

considered to be a standard license plate of 2012. Further on the algorithm goes to

the next cutting stage. In this case, the cutting stage consists of two sub-stages:

horizontal cutting in half (see Figure 2(a)-I) and vertical cutting of the second cut

part in half (see Figure 2(a)-II). Thus cutting results in the dividing of a two-line

license plate into 3 parts (see Figure 2(b)). Figure 2 from (a) to (b) shows the

result of the described stage.

Figure 2

License plate split stages. (a) Split sub-stages: I - upper part and II – lower part (b) Result of split sub-

stages: 1, 2, 3

After the cutting stage, the cut parts of a license plate are joined in accordance

with the established procedure, and its result is provided in Figure 3.

Figure 3

Merge of license plate. (a) Merge of circumcised parts: 1, 2, 3. (b) Result of merge

If the flag image is not found, this number plate type is considered a standard

license plate of 2003, so only horizontal cutting in half algorithm is applied.

Further on the cut parts of a license plate are joined in accordance with the

established procedure.

Thus, all the types of two-line license plates are converted into single-line license

plate type. Some results are provided in Table1.

The conducted transformations resulted in conversion of the two-line license

plates into a single-line license plate type, their colours are as follows: black

characters on a white background, white characters on a red background, white

characters on a blue background, black characters on a yellow background, blue

characters on a white background.


– 199 –

Table1

Results of transformation two-line license plate to a single-line form

Standard Input two-

line license

plate

Transformed one-

line license plate

2012y

2003y

In order to convert a license plate image into the same grey colour, colours of a

license plate are converted according to the algorithm provided in Figure 4.

Figure 4

The scheme of the algorithm for background and alphanumeric characters colours conversion to

produce images in the same grey colour

Before all the required conversions are carried out, image pixels intensity is

normalized. The method suggested in the resource is used for normalization of all

image pixels intensity. The given method resulted in the same weight of red,

green, blue pixels. It can be explained by the following - while taking snapshots of

the plane field, the light going through the recording system was absolutely white

and the pixels sensitivity to different spectral regions was similar.

According to the suggested algorithm, the process of defining background colour

starts for change. If the background is red or blue, these images undergo inversion

that results in the value that is inverse to the original value. While an image is

inverted, the value of all pixels in channels are converted into opposites according

to the scale of 256 colours.

Then RGB system is converted into HSV. Representing colour model HSV (Hue,

Saturation, Value) is more sensitive to color distinction than RGB model, so it

describes colours nearly as well as a human does. The results of license plates

colours normalization and inversion are provided in Table 2.


– 200 –

Table 2

The results of license plates colours normalization and inversion

Transformed one-line license plate One-line license plate after inversion

The procedure includes image conversion from RGB model into HSV model and

breakdown into H, S, V channels, i.e. 4 images are created: one is for storing and

the other three for further segmenting of an image into separate channels H, S, V.

After that difference between colours is calculated and the current colour is

replaced with the required colour. Next, channels H, S, and V are combined and

color space is converted from HSV into RGB. Thus, all colourful license plates

are converted into number plates with a white background. The results of

background and alpha-numeric character’s colours conversion are provided in

Table 3.

Table 3

The results of background and alphanumeric characters colours conversion

Transformed single-

line license plate

Transformed single-line license plate

in the homogeneous palette of gray

Thus, the suggested conversion method enables converting of all types of license

plates into the single standard one-line type of the same grey colour.

3 Segmentation of a License Plate

In case this method is used for solving the problem of vehicle license plate

segmentation, the task is to find outlines on the input image and to evaluate them

according to the specified criteria. The area of the ruling box and aspect ratio are

these criteria. Then, the found outlines are sorted in the order license plate

reading. Character image is selected for every outline basing on the ruling box

(see Figure 5).


– 201 –

Figure 5

Example of segmented characters

The produced images of characters are scaled to common size. The obtained

character images are scaled to the total size of 34x44 and transferred to the output.

It should be noted that the disadvantage of this approach is that this algorithm

can’t retrieve the required outlines under conditions of noise pollution and when

low resolution images are processed.

4 Clustering Characters

As a rule, any object or image subjected to recognition and classification

possesses a range of distinctive qualities and features [21]. According to RK

license plate standards every character is characterized by the following feature

vector: height (h) and width (w).

Preprocessing procedure based on feature vector enabled to take aspect ratios as a

distinctive feature as they are invariable in relation to different conversions and

deformations.

The size of numerals and letters on license plates differ in their height and width.

If the height of numeric characters is 58 mm, their width is 30 and 33 mm, if their

height is 70 mm, the width is 35 mm and 40.5 mm, if the height of numeric

characters is 76 mm, their width is 38 mm and 44 mm. If the height of literal

characters is 58 mm, their width is 39 mm, 43 mm, 44 mm, if their height is 76

mm, the width is 51 mm, 57 mm, and 58 mm. Examples of numeric characters are

shown in Figure 6.

Figure 6

Dimensions of numeric characters «1», «5» and «0»


– 202 –

According to the evaluation of numeric and literal characters aspect ratio, clusters

of characters and their single-valued characteristics are clearly defined. This

statement validity is evaluated by an agglomerative hierarchical algorithm where

the distance between objects is calculated with Euclidean distance, the distance

between clusters is calculated using the nearest-neighbor principle [22]. A set of

numeric and literal symbols is divided into 4 clusters by calculating sequentially

all the required distances, by combining objects into clusters in accordance with

the suggested algorithms, and as a result of clusterization.

Thus, the following conclusions can be made on each group of clusters

1) Ratio values of the first cluster is ranged between 1.31 and 1.35. the objects of

the first cluster are the following literal characters: A, C, D, H, M, N, O, T, U, V,

W, X, Y, Z.

2) The ratio values of the second cluster is 1.49. The objects of the second cluster

are the other 8 literal characters: B, E, F, K, L, P, R, S. Besides the only ratio of

the letter “W” is 1.31, the ratio of other letters varies from 1.33 to 1.35. It should

be also noted that for letters of this cluster that are 76 mm high, the ratio is 1.33,

and for those ones 58 mm high the ratio is 1.35 mm. It means that the license plate

character’s height parameter is also a characteristic feature for classification. It is

significant to note that when literal character’s height is fixed (58 or 76 mm), the

height-to-width ratio is also fixed (1.33mm and 1.35mm respectively). As for the

letter “W”, in case it is 58 mm high, its height-to-width ratio is 1.32.

3) Ratio values of the third cluster vary from 1.93 to 2, its object is numeral 1.

4) Ratio values of the fourth cluster vary from 1.73 to 1.76. It is significant to note

that when numerical characters height is fixed (58, 70 or 76 mm), height-to-width

ratio is also fixed (1.73 mm – for the first and the third case and 1.93 mm for the

second case). The objects of this cluster are the following numbers: 0, 2, 3, 4, 5, 6,

7, 8, 9.

Alphanumeric characters ratio values rendering is represented in Figure 7.

Figure 7

Clustering of alphanumeric characters


– 203 –

The conducted analysis of literal and numerical characters applied on license

plates enabled to divide a set of alphanumeric characters into clusters which

further are used for development of an effective classifying program.

4.1 Character Recognition

Before recognition stage belonging of a current symbol to one of the clusters

Ai,i=1,4 is defined. It has been established that after cluster analysis alphanumeric

characters were divided into 4 clusters, where 2 clusters are clusters of literal

characters, and the other two are the clusters of numerical characters. Belonging

of a current symbol to one of the clusters Ai,i=1,4 is defined by calculating the

characteristic feature of this symbol and by comparing it with the average vector

Ai,i=1,4.

Thus, the classifying program calculates the aspect ratio of the current symbol,

which is invariant value and then compares it with the average value of each

cluster. The classifying program refers symbol x to cluster Ai if height-to-width

aspect ratio is 1.33; symbol x is referred to cluster A2, if height-to-width aspect

ratio is 1.49; symbol x belongs to cluster A3 if height-to-width aspect ratio is 2;

and finally symbol x is referred to cluster A4, if height-to-width aspect ratio is

1.73. Then, a symbol is recognized by the method of simple patterns, when

Hamming distance is used as image similarity measure.

On the basis of experimental research, it has been found out that the given

approach reduces dimensions of selected characters considerably due to pre-

determined clusters and enables to achieve iterates decrease thus providing

characters processing speeding up.

In order to test the suggested information technology, the software program of

characters recognition has been implemented. Computer vision library OpenCV

(Open Source Computer Vision Library) was used for prototype realization [23].

Both static images and video sequences were used as the initial data of the system.

Two bases of templates were formed for algorithm running. The base of numeric

characters templates is represented by 150 pattern images of characters. When

templates were developed, the font was used that complies with RK standards.

The base of literal characters templates consists of 330 images. Thus, 15 pattern

images with different inclination angles have been selected for every character.

This method has been tested on 7923 images of number plates, numeric characters

recognition probability is 96.8%, literal characters recognition probability is

95.1%. Average recognition probability is 96%.

Conclusion

Information technology for RK license plates recognition is described in the given

paper. HOG method is suggested for localization of license plates on an image.

x

x


– 204 –

This method is rather stable to photometric and geometric image distortions that

include brightness variations, shift, angling, scale conversion, view distortion.

Geometric and topological characteristics of RK license plates have been studied

in order to define service and symbolic-numeric information that is further used in

development of characters recognition algorithms.

Analysis of geometric and topological characteristics of vehicles license plates in

accordance with RK standards resulted in defining of basic characteristic features

of license plates characters that are invariant relating to any conversions.

Methods of vehicles license plates geometric and topological characteristics con-

version have been developed. They are based on conversion of license plates to

standard single-line view and generating colourful number plates in the same grey

colour.

The results of experimental research has proven that preliminary transformations

considerably influence the images classification results. Conversion of all license

plates to one type enabled reducing the time period for image processing.

Pilot information technology suggested in the given paper has provided rather

consistent results of recognition - 96%.

The suggested methodology can be used not only for license plates of the Repub-

lic of Kazakhstan, but also for those ones of other countries.

References

[1] Björklund T., Fiandrotti A., Annarumma M., Francini G., Magli E. Robust

license plate recognition using neural networks trained on synthetic images,

Pattern Recognition, 2019, 93, 134-146

[2] Puranic A., Deepak K. T., Umadevi V. Vehicle Number Plate Recognition

System: A Literature Review and Implementation using Template

Matching, International Journal of Computer Applications, 2016, 134(1),

12-16

[3] Bharath B. P., Mahalakshmi S. Automatic license plate detection using

deep learning techniques, International Journal of Scientific Research

Today, 2017, 5(1), 107-112

[4] Rizvi S. T. H., Patti D., Bjorklund T., Cabodi G, Francini G. Deep

classifiers-based license plate detection, localization and Recognition on

GPU-Powered Mobile Platform, Future Internet, 2017,

http://www.mdpi.com/1999-5903/9/4/66 (01.02.2018)

[5] Tlebaldinova A, Denissova N, Kassymkhanova D. Application of a

scenario approach in development of a recognition system of vehicle

identification numbers. In: IEEE 2015 6th

International Conference on

Modeling, Simulation and Applied optimization; Istanbul, Turkey; 2015,

pp. 1-4

http://www.mdpi.com/1999-5903/9/4/66


– 205 –

[6] Medjahed S. A. A comparative study of feature extraction methods in

images classification, I. J. Image, Graphics and Signal Processing, 2015, 3,

16-23

[7] Zhang G., Ma Z., Niu L., Zhang C. Modified Fourier descriptor for shape

feature extraction, Journal of Central South University, 2012, 19(2), 488-

495

[8] Mullen R. J., Monekosso D. N., Remagnino P. Ant algorithms for image

feature extraction, Expert Systems with Applications, 2013, 40(11), 4315-

4332

[9] Dwivedi U., Rajput P., Sharma M. K. License Plate Recognition System for

Moving Vehicles Using Laplacian Edge Detector and Feature Extraction,

2017, 4(3), 407-412

[10] He S., Yang C., Pan J-S. The Research of Chinese License Plates

Recognition Based on CNN and Length_Feature, Proceedings of

International Conference on Industrial, Engineering and Other Applications

of Applied Intelligent Systems (2-4 August 2016, Morioka, Japan), Trends

in Applied Knowledge-Based Systems and Data Science, 2016, 389-397

[11] Panchal P. M, Panchal S. R., Shah S. K. A Comparison of SIFT and SURF,

International Journal of Innovative Research in Computer and

Communication Engineering, 2013, 1(2), 323-327

[12] M. Guzel, A Hybrid Feature Extractor using Fast Hessian Detector and

SIFT, Technologies, 2015, 3(2), 103-110

[13] Hongbo Li, Ming Qi, Yu Wu A Real-Time Registration Method Of

Augmented Reality Based On Surf And Optical Flow, Journal Of

Theoretical And Applied Information Technology, 2012, 42(2), 281-286

[14] Dalal N., Triggs B. Histograms of Oriented Gradients for Human

Detection, In CVPR, 2005, 886-893

[15] Prates R. F., Cámara-Chávez G., Schwartz William R., Menotti D.

Brazilian License Plate Detection Using Histogram of Oriented Gradients

and Sliding Windows, International Journal of Computer Science &

Information Technology (IJCSIT), 2013, 5(6), 39-52

[16] Rahim Md. A., Hossain Md. N., Wahis T., Azam Md. Sh. Face Recognition

using Local Binary Patterns (LBP), Global Journal of Computer Science

and Technology Graphics & Vision, 2013, 13(4)

[17] J. Canny. A computational approach to edge detection. IEEE Transactions

on Pattern Analysis and Machine Intelligence, 8(6):679{698, November

1986.- ĐĐ. 679-698

[18] ST RK 986-2003 - Road transport. State registration number plates with a

retroreflective surface for motor vehicles and their trailers. Technical

specifications


– 206 –

[19] ST RK 1176-2010 - State registration plates with a light-reflecting coating

for separate types of vehicles and trailers. Specification

[20] ST RK 986-2012 - Road vehicles. State registration licence plates of

retroreflective surface for motor vehicles and their trailers, and blank plates.

Specification

[21] E. Rafajlowicz, “Data Structures for Pattern and Image Recognition to

Quality Control” Acta Polytechnica Hungarica, Vol. 15, No. 4, pp. 233-

262, 2018

[22] Ozaki R., Hamasuna Y., Endo Y. Agglomerative Hierarchical Clustering

Based on Local Optimization for Cluster Validity Measures, Proceedings of

IEEE International Conference on Systems, Man, and Cybernetics (SMC),

(5-8 October 2017, Banff, Canada), 2017, 1822-1827

[23] Z. Balogh, M. Magdin, G. Molnar “Motion Detection and Face Recognition

using Raspberry Pi, as a Part of the Internet of Things” Acta Polytechnica

Hungarica, Vol. 16, No. 3, pp. 167-185, 2019

Normalization of Vehicle License Plate Images Based on ...

Documents

Transcript of Normalization of Vehicle License Plate Images Based on ...