Robotic Sorting of Paper Items From a Random

7/28/2019 Robotic Sorting of Paper Items From a Random

1/12

Robotic sorting of paper items from a random

pile

C. Jones*, G. Ross, J.R. Hewit, A.P. Slade

Department of Applied Physics, Electronics & Mechanical Engineering, University of Dundee, Dundee,

DD1 4HN, UK

Received 2 April 1999; accepted 20 June 1999

Abstract

A random stack of documents is physically sorted into ordered stacks according to type

and condition. This is done by rst locating the topmost item using a machine visionsystem with algorithms based upon the Hough Transform. The edges of the items are

identied from the shadows created by illuminating the pile from dierent directions.

Having located the position and orientation of the topmost item, a robot arm is

programmed to pick the item from the pile using an end-eector with vacuum picking

mechanisms. The robot arm oers the picked item ``on-the-y'' to devices which ascertain

its physical tness and classication. Finally, the item is placed on the appropriate one of a

number of piles of similar category. 7 2000 Elsevier Science Ltd. All rights reserved.

1. Introduction

Bin-picking by robot is a generic task common to a wide range of industries.

Examples include the picking of piston rods from metal bins in a car engine plant,

the picking of apples from boxes of fruit, and the picking of socks from a random

pile at the output of a knitting machine.

As these three examples show, the bin-picking process can involve parts that are

rigid and geometrically determinate, or parts that are exible and deformable with

little geometrical denition. The former are easier to bin-pick than the latter for

0957-4158/00/$ - see front matter 7 2000 Elsevier Science Ltd. All rights reserved.

P I I : S 0 9 5 7 -4 1 5 8 (9 9 )0 0 0 4 5 -8

Mechatronics 10 (2000) 869880

* Corresponding author.


2/12

two major reasons their positions and orientations are easier to determine, and

stay xed after determination, and they are easier to pick up by a robot end-

eector.

In all cases, however, it is likely that the same basic processes will be used to

eect the bin-picking procedure. Computer vision will be used to locate the parts

to be picked and, in particular, to dierentiate between the part to be picked next

and the other parts in the bin. A suitable end-eector, possibly of a special design,

will be used to do the actual picking under the direction of the computer visionsystem.

As well as these basic processes there may be other processes taking place

during the bin-picking procedure. For example, it is sometimes possible to apply

tests to the picked part during the picking process; apples may be inspected for

blemishes or poor coloration; socks may be sized on the y; electronic

components may be tested for continuity or for resistance etc.

In the following a description is given of an industrial bin-picking procedure

applied to the picking, classifying and sorting of paper items presented as a

random pile. This application poses a number of dicult problems. These

problems are discussed and ways of overcoming them are presented which will

prove to be applicable in a wide range of applications.

2. Picking of paper items

There are many examples of industrial and commercial operations where the

picking of paper items must be done accurately, robustly and fast. Banks must

sort piles of banknotes of dierent denominations into other piles each containing

only a single denomination; the Post Oce must sort piles of letters and postcards

prior to delivery; the counting of votes during an election is done by sorting the

voting papers.

Traditionally, all of these tasks are done by humans who use their visual sense

to locate the appropriate piles of items and, in some cases, to detect the next itemto be picked. The human operator also makes use of tactile sensing and co-

ordination between the hand and the eye to eect an ecient pick. As mentioned

above, inspection operations are also undertaken the voting paper is inspected

to determine the voter's preference; the letter is inspected to locate the address.

Where robots are proposed to replace the human it is essential that these

sensory and co-ordinative capabilities are replicated as far as technology will

allow. For paper items, sensory and co-ordinative functions are used during the

following repeated sequence:

. nd the topmost item on the pile;

. determine its position and orientation;

. determine whether it is defective;

. control a suitable end-eector into place to pick the item;

C. Jones et al. / Mechatronics 10 (2000) 869880870


3/12

. apply tests on-the-y to the item;

. deposit the item in a suitable manner.

These individual functions are described in detail below.

3. Machine vision aspects

A random pile of media contained within a rectangular bin presents a bin-

picking scenario which is further complicated by the variable form and type of

such media. These items of media may have any or all of the following: printed

data on their surfaces; folded corners; torn edges or attachments such as staples or

tape. The contents of the bin are viewed from above with a monochrome CCD

camera and a typical pile as seen by the camera is shown in Fig. 1.

Items within the bin are to be removed singly from the top of the pile, hence

requiring the determination of the uppermost item of the pile. A wide range of

currency and cheques together with the possible inclusion of `foreign' objects such

as business cards, bus tickets etc. may be in the bin. This renders direct pattern

recognition unfeasible and item location is derived from the physical boundary of

items since the uppermost item will generally have its complete boundary visible.

3.1. Edge enhancement

Previously, thin and planar mechanical parts have been located in a bin-picking

situation [1]. These parts were located via edge tracking after grabbing a series of

images illuminated from a range of positions. Our application uses a similar

lighting arrangement, however, we create shadows at the physical edges of the

media, but need to remove any printed data (noise) from the image [2]. By

successively illuminating the bin from above each side we create the desired

Fig. 1. Typical pile of paper items.

C. Jones et al. / Mechatronics 10 (2000) 869880 871


4/12

shadows. A simple subtraction procedure would then, to a rst approximation,

remove noise. In reality the intensity of each image varies across each image;

theoretically this variation is non-linear, however in practice it is found to be

approximately linear. We therefore modify the images by taking on-line

measurements of the intensity at each side of the bin, and applying a correction

factor to each image. The on-line measurements enable accurate estimation of the

correction factor irrespective of the output of each source which may degrade over

time. Fig. 2 shows histograms of intensity through the image (Fig. 1) where sourceA is located parallel with the upper edge of the image and source B is located

Fig. 2. (a) Source A energised. (b) Intensity corrected. (c) Source B energised. (d) Intensity corrected.

(e) Eect of [(b)(d)+128].



5/12

parallel with the lower edge. The original variations of intensity are shown in

Figs. 2(a) and (c) and the respective corrected intensities are shown in Figs. 2(b)

and (d). Fig. 2(e) shows a histogram illustrating the result of a subtraction and a

128 grey level shift to indicate physical edges which step up or down when moving

across the bin from top to bottom in Fig. 1.

The histograms of Figs. 2(a)(e) indicate the procedure applied when light

sources parallel to the top and bottom of the image in Fig. 1 are applied. Further

identical processing takes place for sources applied at the left and right edges. Thenal result after edge detection and binarisation is shown in Fig. 3.

3.2. Feature detection and location

The items constituting the pile of media are essentially rectangular but no edge

orientation information is derived from the enhancement process and therefore the

use of a Generalised Hough transform [3] is not possible for object location. The

fragmentation of physical edge features renders other methods such as boundary

coding [4], for example, unusable. We locate individual features of the items via a

Hough transformation and have developed a simple, accurate and robust

procedure to locate these features, known as the COA algorithm [5]. The

fragmentation of the edge data and its variable width together with the lack ofedge orientation data produces a cluttered parameter space and non-ideal peak

formations. The COA algorithm searches for regions indicating peaks in

parameter space and is particularly eective when peaks are sharp [6] a

phenomenon which exists in this case. Having located these regions, the algorithm

takes a slice through the peak at a level below the summit of the peak and

computes the centre of area of this region. The eect of this is to ignore errors

from the distant parts of the distribution and also errors around the focal point of

Fig. 3. Binary edge-detected image.



6/12

the peak; this can be likened to the Least Median of Squares approach to mode

location [7]. We obtain sub-pixel accuracy of parameter estimation with little

computational eort and the method has proved to be robust.

3.3. Object location

Final location of the uppermost object of the pile is achieved by an analysis of

the detected features based on a simple search of these features and incorporatestolerances on the angular location of each feature. The nature of the items within

the bin and the preliminary image processing stages are such that very few items

indicated in the binary edge image have truly parallel and perpendicular features.

The complete image processing and analysis procedure incorporates many trade-

os, generally implemented via thresholding. Ultimately, the detection and

location of the uppermost item is not guaranteed. However, within the object

location procedure, safeguards are built in such that failure to locate the

uppermost item is indicated. Under these circumstances the contents of the bin are

disturbed mechanically and the procedure is repeated until successful identication

and location is achieved.

4. Media picking and handling

Individual items are to be picked from the bin and transported to sensing

stations and storage trays. A commercial industrial robot (Staubli RX90) is

utilised to provide six degrees of freedom. The automated handling of exible

materials [8] is a dicult task which in general requires a specially designed end

eector. The nature of the media to be handled in this application is such that it

may range from crisp new banknotes or cheques, to very worn, torn banknotes

and may be of a wide range of sizes. The end eector designed to pick and handle

this wide range of media is shown in Fig. 4.

4.1. Picking

Media can only be attached to the upper side and vacuum provides an ideal

method of attachment. The inner section of the end-eector houses two suction

cups which attach to the desired item. The movement of the robot in the z-

direction, when picking, is controlled by a vacuum sensor located in the line. The

variable form of the pile of media renders optical methods of detecting the height

of the pile ineective. Once attached to an item the end-eector is lifted away and

the inner section retracts. An ultrasonic transmitter is located within this

extendible section for condition sensing.

4.2. Secure attachment for transport and sensing

The physical condition of this media is unknown at the picking and initial



7/12

handling stages. Secure attachment is necessary so that accurate positioning over

sensing stations can be achieved. It is also required because high dynamic and

aerodynamic forces may exist when transporting the media at high speed. High

attachment forces are applied to each item irrespective of damage, such as torn

edges and porosity due to wear or damage, by application of vacuum to a porous

pad. A high mass ow rate fan is used to supply the vacuum force which, by

careful choice of the spacing and size of the holes in the pad, applies maximum

attachment force and attens the items to assist sensing.

5. Sensing

Having located the top item of the unordered stack of items and picked that

item, the document must be at least recognised if automatic sorting and placement

are to be carried out. It may also be necessary, depending on the document, to

validate it and screen it for condition. For example, a pile of valuable documents,

such as stamps or tickets, might contain forgeries or items too soiled to be

acceptable in some way. The establishment of these properties would seem simple

given that an imaging device is available within the system. This, however, is

optimised for a very particular operation. To extend its uses to recognition etc.would require a separate illumination system, a dierent image set and further

processing time. It is also unlikely that ecient validation and condition screening

could be performed without using colour imaging. Recognition itself could prove

time-consuming for the existing system due to registration being only as good as

Fig. 4. The end-eector.



8/12

the edge detection (22 mm laterally and 228 skew). Thus, OCR would be time-

consuming even for a small set of items.

The technique the system described actually uses the one which utilises the

robotic handling of the documents such that recognition, validation and condition

can be established while the document is in transit between the bin and its nal

destination, leaving the imaging system to establish the next top document ready

for picking when the end-eector returns.

5.1. Recognition

Recognition can be assisted by data on item size. The dimensions of items can

be determined to within 2 mm by the edge detect system. If the application is to

sort several dierent types of items into classes and reject all others as a separate

class, then it is possible rst to reject all items whose dimensions are incorrect.

However, even if each known class has unique dimensions this does not make its

members unique in the universal document set. Some further form of

dierentiation is necessary. In this case the colour and spatial pattern along a

xed stripe are utilised to recognise known document types. Whilst in transit, the

robot arm carries the item at a constant speed across a stationary colour sensor.

This sensor has a wide bandwidth photodiode as the sensing element, with variousLEDs positioned around the detector to provide known bandwidth illumination.

The LEDs range from blue to near IR. The position of the LEDs and the

aperture of the photodetector port are designed to exclude specular reections

from the item. The separate LEDs pulse sequentially and the reections from the

passing item provide a pulse train of colour information at a constant rate.

Sample start/stop is automatically controlled and is dependent on the position of

the end-eector and item dimensional information. The data is de-multiplexed

into individual colour signatures which are compared using Bayesian classication

(Eq. (1)) to a database from known items. Four stripes for each item are stored in

the database to cover the four possible ways the item might be presented to the

sensor. The closest t is determined. Those items within tolerance are deemed to

be recognised.

5.2. Validation

Depending on the tolerance of the recognition algorithm, most simple forgeries

would be rejected at the recognition stage. This would include opportunist

forgeries made by scanning originals and subsequently printing on normal printer

paper using a desk top printer. The more valuable items such as currency, stamps,

event tickets etc., are targeted by professional counterfeiters using more

sophisticated printing techniques. To counteract these forgeries the system

examines the signatures resulting from the colour dierences and by applying time

dependent discriminant analysis to the signatures. These discriminant functions

are generated by training the system to highlight the locations of dierences. This

is shown in Fig. 5.



9/12

5.3. Condition-screening

There are many aspects of an item covered by the term `condition-screening'. It

involves the aesthetic appearance of the item, its age and the degree of wear, and

the presence of folds, tears and attachments. All of these features can be

determined in various ways. Here age and degree of wear are considered and is

expressed subjectively by the limpness of the item. Appearance is included

implicitly, via the recognition and validation algorithms, but only for the specicpath examined.

To determine limpness, ultrasonic transmittance is employed. A transmitter is

located in the end-eector pad and at one point, during the passage of the item

from the pile, the transmitter passes directly above a receiver in a known position.

Processing of the limpness data must take into account the item recognition data.

This is because limpness is subjective, for example, a limp business card may not

be as limp as a new banknote.

5.4. Sorting

Sorting is very application dependent. In this case rejects at any stage are sent

to a specic reject bin for further mechanical or manual sorting. Items which arerecognised and pass the condition screening process are sorted into ordered stacks

within machine tolerance, or inserted into subsequent machines for binding or for

redistribution.

6. Results and discussion

The system was tested using ve known types of genuine paper document mixed

Fig. 5. Classication using colour.



10/12

randomly with unknown documents and colour copies of the known documents.

The results for the various aspects of the system are given below.

6.1. Recognition

Using the Bayesian function (Eq. (1)) the system correctly identied 100% of

the ve types of item it was trained to recognise. The system also had a 0% false

accept rate, i.e. all unknown documents were rejected.

Lj

i

x i mij2

s2ij

class j4Lj minLj 1

6.2. Validation

Tests for validation were carried out on genuine and copied items, where the

copies ranged from colour scans and those from desk top colour printers to those

that were printed using multicolour lithographic printing. In every case the copywas rejected as can be seen in Fig. 6.

6.3. Screening

Condition screening was also almost entirely successful. It was found that the

detected signal of a new item was substantially dierent to that of a used but

subjectively acceptable one. Both were dierent again to that of a used and

unacceptable item. These results have been obtained for all types of item

investigated, though the signal levels were dierent for each of the document types

examined.

Fig. 6. Validation results.



11/12

6.4. Vision

The machine vision sub-system is three-stage (edge enhanced and detection,

feature parameter estimation, feature analysis) with each stage itself being multi-

stage. For example, the COA algorithm rst located peaks in parameter space and

then estimates feature location. Many thresholds are required to be set to achieve

successful object detection and location and each stage is highly dependent on the

ecacy of previous stages. Sampling, using a wide range of items, resulted in

success of detection and location as indicated in Fig. 7. No complete failure to

detect and locate was indicated, however, success was not necessarily achieved at

the rst attempt.

Three modes of failure to detect and locate may occur:

1. there may be insucient edge data for any particular edge feature to be

detected;

2. there may be excessive noise in the image so that the search procedure registers

occlusion;

3. the detected `edge' may be too wide so that the COA algorithm fails to locate

any particular feature.

In the results obtained, where failure occurred, they occurred with the followingfrequencies; Mode 1: 10; Mode 2: 4; Mode 3: 7.

References

[1] Dessimoz JD, Kunt M, Zurcher JM. Recognition and handling of overlapping industrial parts. In:

Proceeding of the Ninth International Symposium on Industrial Robots, Washington DC, 1979. p.

35766.

[2] Jones C, Hewit JR. The extraction of physical edge features of deformed printed paper documents,

1998 (in preparation).

[3] Ballard DH. Generalising the Hough transform to detect arbitrary shapes. Pattern Recognition

1981;13:11122.

[4] Freeman H. On the encoding of arbitrary geometric congurations. IEEE Trans Electron Comput

1961;10:2608.

Fig. 7. Vision system success rate.



12/12

[5] Jones C, Hewit JR. The accurate and robust location of edge features of paper documents via the

COA algorithm. Pattern Recognition Letters 1998 (submitted for publication).

[6] Davies ER. Modelling peak shapes obtained by Hough transform. IEE Proceedings E

1992;139(1):912.

[7] Davies ER. Machine vision: theory, algorithms, practicalities. 2nd ed. Academic Press, 1997.

[8] Taylor P. Handling of exible materials in automation. International workshop on advanced

robotics and intelligent machines, University of Salford, 1994.


Robotic Sorting of Paper Items From a Random

Documents

Transcript of Robotic Sorting of Paper Items From a Random