PAVISYS: A computer vision system for theinspection of locomotive pantographsM. Sacchi∗‡, S. Cagnoni∗, D. Spagnoletti§, L. Ascari‡, G. Zunino∗¶, A. Piazzi∗

∗Dept. of Information Engineering, University of Parma, Italy‡Henesis srl, Parma, Italy

§Rete Ferroviaria Italiana S.p.a, Roma, Italy¶EFSA - European Food Safety Authority, Parma, Italy

Abstract—This paper describes PAVISYS, a vision-based pan-tograph inspection algorithm, which is the heart of the PANTO-BOT system (Henesis srl, Parma, Italy), a full-fledged monitoringsystem for train pantographs, which also adds remote analysisand management of images coming from the Inspection Pointslocated along the railway.

The automatic inspection algorithm consists of three mainsteps: a pantograph classifier, a modular quality assessmentsystem, and a report generator. The paper provides details aboutthis architecture, reporting the most significant experimentalresults obtained in the extensive set of lab tests that were run toassess its performances.

Presently, an installation of the system is monitoring the mainline of the Italian High Speed Train Network for a final field testin full operating conditions.

I. INTRODUCTION

Pantographs are the most frequently used means for con-ducting electricity from the contact line system (overheadcontact line) to the traction units. A pantograph consistsof one or more elements (collector strips), parallel to eachother and perpendicular to the direction of the velocity vectorof the traction units; these collector strips are made of aconducting material, typically copper or graphite. The contactsurface of pantographs is subject to wear due to frictionbetween the contact wire and the collector strips. Wear, incombination with other causes, such as an incorrect layout ofthe line (e.g. wrong wire height) or the presence of objectsnearby (e.g., tree branches) that may hit the pantograph, oftencause severe structural abnormalities, such as misalignmentof the collector strips or breakage of the extremities of thepantograph (horns). This is a critical factor in the maintenanceof the rail infrastructure due to the long time the maintenancecompany needs to repair and to fit the damaged contact line.

Moreover, structural damage of the pantograph can affectthe overhead wire and result, in the worst cases, in prolongedservice interruptions, if the line is pulled down and requiresrepair. The recovery operations following these events usuallyhas high direct and indirect costs and are a significant entryin budgets of railway managing companies [1]. Figure 1shows a pantograph which has been severely damaged afterhitting a tree branch. Rete Ferroviaria Italiana (RFI) spa(the Italian railway network managing company), as othercompanies in the world, has set up special units (VideoInspection Pantographs, VIPs) whose personnel inspects high-

Fig. 1: Example of a severely broken pantograph.

resolution images of pantographs, acquired at the so-calledRemote Shooting Points (RSPs), located along the network, todetect pantograph anomalies which may be potentially harmfulto the rail network. Typically, such a diagnosis is performedby a human operator who examines all images transmitted bythe RSPs to the VIP. Given the number of trains that dailytravel the network, it is clear that the cost of human visualinspection can be just as important, if not higher, than thecost deriving from damage to the line caused by pantographsin bad working conditions.

For these reasons, the development of automatic systemsaimed at diagnosing the state of maintenance of pantographsis a relevant research topic. The automatic diagnosis systemscan be installed directly on the locomotive, as are thosebased on the analysis of mechanical stress suffered by thepantograph [2] or of the noise generated by friction with thecatenary [3]. In other cases they can be external monitoringsystems located along the network. Among such systems,those based on visual inspection have recently attracted re-searchers’ attention.

Examples of such systems are the Pancam system, de-veloped by QR Limited, manager of a private railway inAustralia [4] or Apis, a software application for the visualinspection of pantographs, which has been developed usingthe DUOSTECH PRAESIDIUM commercial library produced

Fig. 2: Schematic representation of the architecture of Pavisys.

by DUOSTECH, Jacksonville, FL, USA [5].In this context, RFI has commissioned the University of

Parma a project to automate the analysis of pantographimages using computer vision techniques. This paper de-scribes PAVISYS (Pantograph Automatic Vision-based Inspec-tion SYStem), the software prototype developed within thisresearch.

At present Pavisys represents the core algorithms insidethe commercial software Pantobot owned and distribuited byHenesis srl1. Pantobot offers a plugin environment for easilyincluding new algorithms of analysis of the pantograph, a webservice interface (SOAP+HTTP based) for access to imageprocessing, and a remote management of the system.

II. DESCRIPTION AND QUALITY ASSESSMENT OFPAVISYS

The main goal of PAVISYS is to reduce the number ofimages of pantographs in touch with the wire on the networkthat must be inspected by human operators, who are stillthe ones who must express the final judgment on the stateof wear of pantographs and may request the activation ofthe alternative pantograph on the locomotive, or, in the mostcritical cases, even stop the train. Moreover PAVISYS isable to recognize the material on the collector strips: somematerials have higher friction than others (e.g. copper respectto graphite), accelerating in that way the wear of the contactwire. Thanks to this system the company in charge of themaintenance of the contact line system is able to track thequality of the material on the railway network.

PAVISYS is organized as a set of software modules whichevaluate the efficiency of the pantograph under several pointsof view: geometry of the system, wear of the contact areaof the collector strips, and integrity of the terminal elements

1Viale dei Mille 108, 43125 Parma, Italy. (http://www.henesis.eu)

(horns or bows). If any possibly significant defect is detected,the image is submitted to a human operator. Otherwise, theimages of pantographs in good conditions are directly storedon disk and remain available for possible future consideration.The architecture of PAVISYS (see Figure 2) is therefore highlymodular; this makes it easy to integrate new modules whichimplement new image analysis algorithms or to allow forthe inspection and diagnosis of new types of pantograph,in addition to those already taken into consideration by thesystem. Presently PAVISYS is designed to deal with the fourpantograph types which are most frequently used on the Italianrailway network (see Figure 3).

The first module of PAVISYS recognizes the model of thepantograph that is shown in the image under examination;This is necessary because the subsequent analysis relies onthe specific geometric parameters of the pantograph typeto identify and analyze the appropriate image regions onthe examined picture. The classification algorithm uses aninverse perspective transform (IPT) [6] which produces a birdview of the pantograph (see Figure 5 based on which thesimilarity degree between the geometric parameters of thepantograph under examination and those of the 4 modelsstored in the system is evaluated. A low correlation with allmodels automatically produces an alarm, because this situationtypically occurs if the pantograph model is different from thoseconsidered by the system, as well as when the pantographdeviates significantly from its correct geometry.

Table I reports, as a confusion matrix, the experimentalresults of the pantograph model detection module obtainedon a test set of 3737 pantographs. As it can be observed, theclassification is very accurate for three pantograph types out offour. In fact, the pantograph termed MINUETTO is much lessfrequently used than the other three models, as well as beingvery similar to the pantograph termed FS 52 AV. The similaritybetween the MINUETTO and the FS 52 AV pantographs, alsoin terms of geometry, is such that the misclassification of aMINUETTO pantograph as a FS 52 AV has a lower impact onthe results of the subsequent modules, as shown by the finalaccuracy obtained by PAVISYS in our tests. It should alsobe noticed that classifying a pantograph as ’Unknown’ hasno impact on accuracy (the image is selected for inspectionby a human operator anyway) but only on the percentage ofpantographs which are filtered out by the system (referred toas ’filtration index’ in the following).

Better performances were obtained using an algorithm thatcombines the classification of an MPF2-based algorithm withthe algorithm described here: results and a more detaileddescription of the new algorithm will be presented in futurework.

The pantograph classification module allows the systemto set the parameters of the modules which are cascadedwith it according to the pantograph geometry. The modulewhich immediately follows detects the type of material (copperor graphite) of which the pantograph collector strips are

2Memory Prediction Framework [7]

Fig. 3: Pantograph types; from left to right: MINUETTO, FS 52/92, ATR 90, FS 52 AV

TABLE I: Results of the pantograph classification module

FS 52 AV FS 52/92 ATR 90 MINUETTO UnknownFS 52 AV 97.9% 0.0% 0.06% 0.0% 2.0%FS 52/92 0.66% 98.2% 0.066% 0.0% 0.99 %ATR 90 0.2% 1.07% 98.4% 0.0% 0.2%

MINUETTO 5.16% 0.93% 0.47% 88.26% 5.16%Empty 0.0% 0.0% 0.0% 0.0% 100%

made. This module is also crucial for the system performancebecause it estimates wear on the basis of a statistical model,whose parameters depend on this information. This modulealso identifies the region in the image that is occupied bythe collector strips, for which statistical parameters of thedistribution of the differences between color planes are com-puted. These data are finally input into a feed-forward neuralnetwork [8] which produces the final classification of thematerial.

Table II reports the results of this module obtained on atest set of 1098 images for the two most common pantographmodels.

TABLE II: Results of the material classification module

Graphite CopperFS 52 AV Graphite 100% 0%FS 52 AV Copper 0.27% 99.73%

FS 52/92 Graphite 98.24% 1.76%FS 52/92 Copper 1.89% 98.11%

The module that analyzes the level of wear of the collectorstrips (see Figure 4) is the first of the three diagnosticmodules of PAVISYS. The analysis is based on several sta-tistical parameters derived from the color histograms of the

Fig. 4: Two types of defects: heavy wear on the strips (left)and damaged horn (right)

image region occupied by the strips, the longitudinal profileassociated with the intensity of the pixels, and the Gray-Level Co-occurrence matrices (GLCM) computed in such aregion [9]. All parameters are analyzed by a Bayesian filter thatcompares such parameters with three models corresponding tovarious classes of wear (low, medium, high), to produce an “aposteriori” estimate of the probability with which the regionunder consideration belongs to each class.

The second diagnostic module of the system assesses the or-thogonality between the collector strips of the pantograph andthe direction of the motion vector of the locomotive, becausepossible misalignments of the strips can cause an entanglementbetween the pantograph and the catenary, damaging it or eventearing it down.

The algorithm used in this module uses the IPT to obtain abird view of the pantograph, within which an analysis of thevertical edges is performed based on the Hough transform [10].This module uses a statistical filter to determine whether theimage that is analyzed shows a potentially abnormal panto-graph or a pantograph which is definitely in good conditions.An example of misaligned pantograph is shown in Figure 5.

The last diagnostic module of PAVISYS verifies the struc-

(a) Normal perspective (b) IPM perspective

Fig. 5: Misaligned pantograph: The defect is not clearly visiblein the original image (a) but it is in the IPT image (b) thatshows the left collector strip being not perfectly orthogonal tothe train direction.

TABLE III: Detection statistics: sensitivity of PAVISYS on the two test sets

Site A Site BPantS DIP PantS CEI PantS DIP PantS CEI

Defective pantographs 73 11 38 1Defective pantographs detected by PAVISYS 57 11 30 1

Sensitivity 0.78 1.00 0.79 1.00

TABLE IV: Detection statistics: specificity of PAVISYS on the two test sets

Site A Site BPantS DIP PantS CEI PantS DIP PantS CEI

Non-defective pantographs 9246 9308 5019 5056Non-defective pantographs detected by PAVISYS 7362 7378 3742 3750

Specificity 0.80 0.79 0.75 0.74

tural integrity of the extremes of the pantograph, more com-monly termed horns. The algorithm operates selectively onthe region occupied by the horns; the degree of integrity isestimated analyzing the symmetry of the edges of the collectorstrips in the Hough space, which makes it possible to check thesymmetry of the structure, in spite of the presence of stainingor oxidation phenomena that can alter the appearance of thehorns under analysis. An example of this defect is shown inFigure 4.

If any of three diagnostic modules has detected a significantanomaly, the image is submitted to a human operator for finalassessment. Therefore, the main specification that PAVISYShas to satisfy is to filter out as many pictures as possible,without submitting them to the human operator while, at thesame time, reaching the highest possible sensitivity in detecingpossible faults.

PAVISYS was developed and tested in laboratory on anextensive set of images acquired at the visual inspectionpoints of Milan Pioltello, on the line Milano-Venezia (SiteA) and Rome Casilina, on the line Rome-Naples (Site B). Inparticular, 22, 319 images acquired in January, February andMarch 2008 were analyzed to determine the parameters of thestatistical models used in the modules of PAVISYS and totraining the neural network. The system has then been testedon 14, 376 images acquired in May and June 2008.

To measure classification performance, we have computedspecificity and sensitivity with respect to the image setfor which the operators of RFI have actually generatedalarms (Pants RFI test set) and to a more stringent test set(Pants DIP) for which we required that an alarm signal begenerated also in cases of minor wear or in borderline cases.Since the main purpose of PAVISYS is to reduce the numberof images submitted to the operator, we also computed a“filtration index” F, defined as the ratio between the numberof images classified as non-defective by PAVISYS (whichtherefore would not have been submitted to human judgment)and the total number of images which have been analyzed.

The results obtained analyzing all images acquired in thetwo months under consideration, reported in Tables III and IV,show that the sensitivity achieved on the Pants RFI test set isexcellent (equal to 1.0) while on the more difficult PantS DIPtest set it is equal to 0.78 and 0.79 for sites A and B,

respectively. This very uniform performance on the image setscoming from the two different sites shows that the system isvirtually insensitive to changes in shooting angle, provided thatthe camera geometry meets the criteria for camera placementthat were identified during the research. These are already met,or can be satisfied with minor interventions virtually in allexisting RSPs in the RFI network. It should be noted that theRSPs on whose images tests were made, despite being selectedfor the good quality of the images they produce, do not fullymeet these specifications. Time constraints made it impossibleto modify the cameras installed at the test sites to adjust theirfeatures, which makes results even more significant.

The specificity of the system with respect to Plant A is equalto 0.8 for the two test sets, while being slightly lower for PlantB (0.75) probably due to the particular weather conditionsthat, at certain times of the day for a few days, resulted in thecamera being dazzled.

The filtration index shows a reduction of 74% and 79% ofthe images submitted to human operators with respect to thepresent situation, for the two plants, respectively.

III. CONCLUSIONS AND FUTURE DIRECTIONS

We have described a machine vision system for the in-spection of locomotive pantograph which has been shown toreduce the cases in which human inspection of the pantographimages acquired at PRRs is required by about 75%. Despitethis dramatic reduction, the sensitivity and specificity data arevery high when compared to results obtained on the same datasets in in-service conditions, in which all images are inspectedby a human operator, and still very good when comparedwith a more prudential reference set in which an alarm signalis expected to be generated also in borderline cases. The

Fig. 6: Filtering index statistics over two months.

large number of images on which the tests have been carriedout, along with the different environmental conditions (nightand day, winter and spring, different locations) in whichthe training and test images have been acquired show thatPAVISYS is robust and reliable. The system can be easilyinstalled in a centralized server, and can be remotely operatedby means of a web-service interface without hindering orbeing dependent on the specific hardware and software usedto acquire images (provided by third parties in the currentimplementation).

PANTOBOT, a patent pending product of Henesis srl,Parma, Italy, is a fully-fledged monitoring system for trainpantographs, which integrates a video analysis module directlyderived from PAVISYS, as well as functions for the remotemanagement and archival of images coming from the VideoInspection Points located along the railway network. PANTO-BOT can moreover be easily tailored on specific needs forother kinds of pantographs (for instance other models, tram orbus pantographs).

PANTOBOT has been recently installed and is working inone of the VIPs managed by RFI along the main line of theItalian Alta Velocita (High Speed) railway network, for a finalfield test in full operating conditions.

ACKNOWLEDGMENT

The authors would like to thank RFI for funding the initialresearch to University of Parma and the fruitful cooperation,and are grateful to the Emilia Romagna local government forco-funding the technology transfer process from Universityof Parma to Henesis s.r.l., in the framework of the SPIN-NER20133 project.

3Progetto Operativo Regionale Emilia Romagna, su Fondo Sociale Europeo2007-2013, Asse 4. www.spinner.it

REFERENCES

[1] A. Fumi, “On the catenary-pantograph anomalies,” La Tecnica Profes-sionale, vol. 1, 1998.

[2] C. O’Donnell, R. Palacin, and J. Rosinski, “Pantograph damage andwear monitoring system,” in Proc. of the Institution of Engineering andTechnology International Conference on Railway Condition Monitoring,2006, pp. 178–181.

[3] L. Palmucci, “Pantograph diagnosis by noise analysis (in Italian),”Master’s thesis, University of Rome ”La Sapienza”, 2002.

[4] L. Hamey, T. Watkins, and S. Yen, “Pancam: In-service inspection oflocomotive pantographs,” in Proc. of the Institution of Engineering andTechnology International Conference on Railway Condition Monitoring:9th Biennial Conference of the Australian Pattern Recognition Societyon Digital Image Computing Techniques and Applications, 2007, pp.493–499.

[5] Http://www.duostechnologies.com/DownloadCenter/APIS.pdf.[6] H. Mallot, H. Bulthoff, J. Little, and S.Bohrer, “Inverse perspective

mapping simplifies optical flow computation and obstacle detection,”Biological Cybernetics, vol. 64, pp. 177–185, 1991.

[7] J. Hawkins, On Intelligence. A Holt Paperback, 2004.[8] S. Haykin, Neural Networks: A Comprehensive Foundation. Prentice-

Hall, 1998.[9] L. Shapiro and G. Stockman, Computer Vision. Prentice-Hall, 2001.

[10] P. Hough, “Method and means for recognizing complex patterns,” USpatent 3069654, 1962.