Suitability of airborne video data for photogrammetry

Suitability of airborne video data for photogrammetry

Yoonjoo Lee

March, 2008

SUITABILITY OF AIRBORNE VIDEO DATA FOR PHOTOGRAMMETRY


i

Suitability of airborne video data for photogrammetry

by

Yoonjoo Lee Thesis submitted to the International Institute for Geo-information Science and Earth Observation in partial fulfillment of the requirements for the degree of Master of Science in Geo-information Science and Earth Observation.

Thesis Assessment Board

Chairman: Prof.Dr. M.G. Vosselman

External examiner: Dr. B.G.H. Gorte

Supervisor: Dr. M. Gerke

Second supervisor: Ms Y. Tian MSc

INTERNATIONAL INSTITUTE FOR GEO-INFORMATION SCIENCE AND EARTH OBSERVATION ENSCHEDE, THE NETHERLANDS


ii

Disclaimer This document describes work undertaken as part of a programme of study at the International Institute for Geo-information Science and Earth Observation. All views and opinions expressed therein remain the sole responsibility of the author, and do not necessarily represent those of the institute.


3

Abstract Simple and economic acquisition of video data results attractive for a wide range of applications. The acquisition of video data from airborne platforms producing sequences of highly overlapped images brings special interest for photogrammetric applications. The combination of this factors results in the possibility of implementing video data in low cost photogrammetric projects. However to exploit video data as an effective data source in the realm of photogrammetric, research on its advantages, limitations is needed. Likewise, exploration on the implementation of video data within photogrammetric application such as triangulation and 3D modeling is needed.

This research investigates on the suitability of airborne video data as a photogrammetric data source. Video data acquired from non metric digital camera at scale 1:45000 are used. The solution of unknown interior orientation parameters, achievable triangulation accuracy, and video based 3D building modeling are investigated in a frame of a series of experiments and practical set ups implemented by means of photogrammetric tools. As part of the findings of this study interior orientation parameters of a non metric camera are estimated and image distortion is eliminated using Jacobsen’s simple additional distortion parameter model. Different experiments on achievable triangulation accuracy produce acceptable RMSE of the order of 0.5 meter in horizontal and 0.6 to 0.8 meters in vertical measurement. In addition 3D building modeling of a residential area is successfully implemented producing acceptable RMSE given the low resolution of video data. The study shows that video data can be effectively considered as an alternative for low cost photogrammetric applications within a certain level of accuracy.

Keywords: airborne video; photogrammetry; self-calibration; bundle adjustment; 3D building model


4

Acknowledgements I take this opportunity to express my most sincere thanks to my two supervisors. Dr. Markus Gerke, my first supervisor, supports me and attentively guided me during thesis time to conduct my research and was always available and eager to give me constructive comments. He was always there to listen and oriented me showing me how to proceed when my mind did not know how. Furthermore he guided me on how to report for a scientific document. I would also like to extend my thanks and appreciations to my second supervisor Ms. Yixiang Tian. Without their encouragement and constant support I would not have been able to successfully finish this research.

The last 18 months in the Netherlands were a great experience. I gain theoretical and practical knowledge in the field of geoinformatics and I had the opportunity to meet excellent and valuable persons at ITC. Moreover I am very thankful to all those persons, colleagues and staff in GFM who always encourage me during my academic and personal life in Enschede. I would like to express my gratitude to my friends; my Chinese friend, Du Ye, Qujui and Shango who share all the time in ITC; “three musketeers” Arik, Punny, and Satish who show me another part of Asia with food and sweets; Irvin and Chin who always sweet for me and make me warm and happy with their jokes. Thanks also to my Korean colleagues Yoon, Jjang and Song for being friend and brother, sharing Korean food and culture. With you I could feel at home. I would like to extend all this thanks to the Colombian community as well, who make me feel like another one of them. They teach me Spanish, Salsa and give me an opportunity to join International Evening. Especially to Chippi who always there to help me and encouraging me, being travel guide and partner. For all of your time sharing with me I would like to say “Mucho gracias!!”

I am greatly indebted to my parents who make all of these things possible. They brought me to this world and ever since they have supported me and have put all their efforts in my whole life. I extend my sincere thanks to my sister So-Jung and her family for always sending me the greetings from Korea and encouraging me with tons of Korean food; my younger sister Seung-Min, for listening me and inspiring me to be strong; and to my brother Sung-yup, for supporting me even in his silence. Grandmother, thanks for always thinking of me and having me in your prays.

감사합니다. 그리고 사랑합니다…


5

Table of contents 1. Introduction ............................................................................................................................... 9

1.1. Motivation and problem statement ....................................................................................... 9 1.2. Research identification ......................................................................................................... 9

1.2.1. Research objectives ....................................................................................................... 9 1.2.2. Research questions ...................................................................................................... 10 1.2.3. Innovation aimed and motivation ................................................................................ 10

1.3. Structure of thesis .............................................................................................................. 11 2. Basic concept and related topics ............................................................................................... 13

2.1. Photogrammetry ................................................................................................................ 13 2.2. Basic concept of photogrammetry ...................................................................................... 14

2.2.1. Interior orientation ...................................................................................................... 14 2.2.2. Exterior orientation ..................................................................................................... 14 2.2.3. Collinearity condition.................................................................................................. 15 2.2.4. Bundle block adjustment ............................................................................................. 15

2.3. Feature extraction (feature tracking) ................................................................................... 18 2.4. Robust estimation .............................................................................................................. 19 2.5. 3D modelling ..................................................................................................................... 19

3. Methods ................................................................................................................................... 21 3.1. Part I : Aerial triangulation ................................................................................................ 21

3.1.1. Initial interior orientation parameters (IOP) estimation (Self-Calibration) .................... 22 3.1.2. Aerial triangulation by bundle adjustment ................................................................... 23 3.1.3. Triangulation result assessment ................................................................................... 24

3.2. Part II : 3D building modeling ........................................................................................... 24 3.2.1. Data input ................................................................................................................... 25 3.2.2. Camera calibration ...................................................................................................... 25 3.2.3. Building modelling ..................................................................................................... 26 3.2.4. Building model accuracy assessment ........................................................................... 26

4. Data processing and result ........................................................................................................ 27 4.1. Aerial triangulation ............................................................................................................ 27

4.1.1. Data preparation .......................................................................................................... 27 4.1.2. Self-calibration ........................................................................................................... 30 4.1.3. Applying estimated IOPs ............................................................................................. 35 4.1.4. Bundle adjustment for the whole strip ......................................................................... 39

4.2. 3D building modeling ........................................................................................................ 50 4.2.1. Input video image ....................................................................................................... 50 4.2.2. Image calibration ........................................................................................................ 51 4.2.3. Assessment of obtainable accuracy .............................................................................. 52 4.2.4. 3D building modelling ................................................................................................ 55

5. Discussion ................................................................................................................................ 61 5.1. Discussion on results .......................................................................................................... 61

5.1.1. Self-calibration ........................................................................................................... 61 5.1.2. Bundle adjustment with whole strip ............................................................................. 62


6

5.1.3. Feasibility of using tie points derived from KLT .......................................................... 64 5.1.4. 3D building modelling ................................................................................................. 65

5.2. Problems ............................................................................................................................ 66 5.3. Limitations ......................................................................................................................... 67

6. Conclusions .............................................................................................................................. 69 6.1. Answer to research questions .............................................................................................. 69 6.2. Recommendations .............................................................................................................. 70


7

List of figures Figure 1 Photogrammetric workflow ................................................................................................ 21 Figure 2 Self-calibration workflow ................................................................................................... 22 Figure 3 Workflow of 3D building modeling .................................................................................... 25 Figure 4 SONY XC 555P digital video camera ................................................................................. 27 Figure 5 Overlap area between two key frames ................................................................................ 28 Figure 6 Study area .......................................................................................................................... 28 Figure 7 Tracked points from KLT ................................................................................................... 29 Figure 8 Approximate focal length calculation ................................................................................. 31 Figure 9 Distribution of GCPs for self-calibration ............................................................................ 31 Figure 10 Distribution of calculated distortion value ........................................................................ 35 Figure 11 Applying distortion values as camera information ............................................................ 36 Figure 12 Compensate distortion by additional parameter in BLUH ................................................. 37 Figure 13 Check point RMSE with different number of control point use in bundle adjustment for whole strip ....................................................................................................................................... 43 Figure 14 GCP distributions on strips for GCP experiment on bundle adjustment. ............................ 44 Figure 15 Selection of GCP on ground for experiment on bundle adjustment .................................... 45 Figure 16 Check point RMSE with different GCP distribution in bundle adjustment ......................... 45 Figure 17 Example of blunder in tie point using KLT tracker. .......................................................... 47 Figure 18 RMSE on check point from the triangulation using with and without tie points ................. 49 Figure 19 Buildings in video imagery to be modeled ........................................................................ 50 Figure 20 Screen capture of calibration on IM using survey points ................................................... 51 igure 21 Applied definition of height of building .............................................................................. 52 Figure 22 Screen capture of distance measurement on ImageModeler and corresponding laser data .. 53 Figure 23 Example of disconnected roof edge due to laser point density ........................................... 53 Figure 24 Selected buildings for 3D modeling .................................................................................. 55 Figure 25 Modeled roof and applying extracted texture .................................................................... 56 Figure 26 Basic factors on building roof ........................................................................................... 56 Figure 27 Improved model from the roof model to whole building model by applying primitive cube model .............................................................................................................................................. 58 Figure 28 Example of the improved roof model after applying cube geometry .................................. 58 Figure 29 Overall result from GCP experiments on whole strip ........................................................ 63


8

List of tables Table 1 video data specification (provided by Fugro) ........................................................................ 29 Table 2 Coordinate system of different data set ................................................................................. 30 Table 3 Triangulation result with focal length 12mm ........................................................................ 30 Table 4 Triangulation with different distortion model ....................................................................... 32 Table 5 IOP estimation with different set-up ..................................................................................... 33 Table 6 Estimated IOP from selected set-up ...................................................................................... 34 Table 7 Calculated distortion value ................................................................................................... 34 Table 8 Triangulation result after distortion value implementation .................................................... 36 Table 9 Triangulation result implementing distortion value from BLUH ........................................... 37 Table 10 Triangulation result depend on use of initial GPS/IMU ....................................................... 39 Table 11 Fixed set up for GCP experiment ........................................................................................ 41 Table 12 Self-calibration result with different number of control points ............................................ 41 Table 13 RMSE with different number of control points ................................................................... 42 Table 14 Triangulation result with minimum number of control points and additional GCPs using in self-calibration .................................................................................................................................. 43 Table 15 Check point RMSE with different GCP distribution in bundle adjustment ........................... 45 Table 16 RMSE on check point and estimated EOP with the project with or without tie points .......... 46 Table 17 RMSE difference depend on blunder detection method ....................................................... 48 Table 18 Triangulation RMSE using different number of GCP with and without tie points ................ 49 Table 19 Planimetric distance difference between stereo model and laser point data .......................... 54 Table 20 Height difference between stereo model and laser point data .............................................. 54 Table 21 Difference between model measurement and laser point measurement ................................ 57 Table 22 Measurement difference between building model applying primitive cube model and laser data .................................................................................................................................................. 59


9

1. Introduction 1.1. Motivation and problem statement Increasing interest on photo realistic 3D modeling in numerous applications brought needs for simple and flexible data acquisition procedures. For those applications aerial video may be a reasonable data source due to several characteristics of video data. Advantages of video data are its economic and simple acquisition process compared to aerial photograph; also it does not require advanced operator skills or specific mounting on the aircraft. In addition, a high overlap area between successive video frames is one of the big advantages which provide strong block geometry and makes it possible to extract homologous points automatically. These considerations bring sufficient reasons to regard video data in the conventional photogrammetric workflow. With development of technology, resolution of video imagery is getting higher and the automation of photogrammetric process becomes possible. Therefore video imagery has been widely used for applications such as feature detection, monitoring and 3D object modeling. However, most of those applications are only related to computer vision or industrial applications that do not require high geometric accuracy.

There are several reasons that prevent from the use of video data in photogrammetric applications which require high geometric accuracy. Since the video camera is not a professional photogrammetric purpose instrument, quality of video imagery is still low. Therefore only large scale video images have been used for close range photogrammetry, since accurate 3D information is more difficult to obtain with smaller image scale. Only few known internal and external camera parameters are another problem of non metric camera which impedes to achieve high geometric accuracy. In order to compensate the uncertainty of exterior orientation parameters, direct sensor orientation can be used with assistance of GPS/IMU device. However the integration of GPS/IMU device would disturb the price advantage which is one of the strengths of video data. Also, the lens distortion of video camera causes larger systematic errors which should be eliminated for accurate photogrammetric use of video data. Even though these characteristics of video data are obstacles for accurate triangulation, highly overlapped area in video sequence is one of the advantages which make possible automatic image matching and extraction of homologous points between images. Automation of tie point extraction provides high redundant homologous point observations which support the triangulation process and increase the achievable accuracy.

The aforementioned advantages of video data and the wide range of potential applications impulse research on how to overcome the existing limitations with promising methods for photogrammetry fields. Establishing a clear methodology for video photogrammetry with an analysis on the achievable accuracy and correction of lens distortion effects remain as issues that should be investigated and solved before opening the door to the use of video data in the field of photogrammetry.

1.2. Research identification 1.2.1. Research objectives Simple and economic data acquisition is an advantage of video imagery. This attractive characteristic of video data brings interest for photogrammetric application. This research will analyze the suitability of using aerial video data in the photogrammetric workflow, thus it aims to:


10

− Analyze how interior orientation parameters for non metric camera can be estimated.

− Analyze the achievable accuracy from airborne video triangulation.

− Analyze the potential automation of airborne video triangulation.

− Analyze the feasibility of airborne video data for 3D building modeling.

1.2.2. Research questions To achieve the aimed objectives it is necessary to address the following questions:

− How can lens distortion be addressed in video data?

− What is the achievable accuracy of video triangulation?

− How many ground control points (GCP) are needed for successful triangulation?

− Can the state of the art Kanade-Lucas-Tomasi (KLT) tracking algorithms extract enough tie points for triangulation?

− Which accuracy can be achieved by using airborne video images for 3D building modeling?

1.2.3. Innovation aimed and motivation Video data has been used for spatial applications in many fields. Some researches have made use of tracking algorithms to find corresponding points in image sequences (Faugeras et al. 1998; Pollefeys et al. 2004). However most of those take place in laboratory models or fields of research in computer vision which do not require high geometric accuracy. There are similar works with video data which use commercial cameras to record objects along a road (Pollefeys et al. 2002), record around one object, or use the images which recorded oblique or side view. Those researches consider video data as potential source for large scale applications such as close range photogrammetry. Meanwhile small scale video imagery has not been researched in the photogrammetric field.

The low resolution and lack of reliable interior and exterior orientation information of airborne video data causes lower quality for small scale photogrammetric applications. Direct sensor orientation is possible using GPS/IMU, however accurate GPS/IMU data is not always available due to economic reason. Although, since most aircraft systems are GPS equipped, approximate exterior orientation parameters can be obtained from their position. On the other hand, the automation of the photogrammetric workflow should be considered since video data provide large number of image overlapping which must be handled. The potential of automation on tie point extraction within high overlapped video image is expected by combining automatic techniques such as feature tracking methods.

Besides the triangulation, 3D modeling using aerial video data is another active application. Due to the ability of simple data acquisition, video data has been used for many 3D modeling applications, although previous modeling applications have used close range video imagery to reconstruct specific objects or historical monuments. Therefore it is interesting to assess the usability of small scale airborne video data to model 3D objects.


11

Finally, this research aims to innovate by implementing new approaches in the photogrammetric automatic workflow by applying feature tracking methods.

1.3. Structure of thesis This thesis is organized as follows. The basic concept of photogrammetry and several studies which are related with this research are discussed in chapter 2, while the concept of basic photogrammetry is briefly introduced. In order to establish the automation of the video based photogrammetric workflow, KLT feature tracking and robust estimation is considered. Chapter 3 describes the methods applied for this research. It is divided into two parts which refers to aerial triangulation procedure and 3D building modeling procedure. Afterwards, experiments with real data following the methodology are described and analyzed in chapter 4. Chapter 5 discusses each process of this research with a brief summary. Also the problems and limitations faced during this research are described. Finally, answers for the corresponding research questions are presented by analyzing the results from the experiments and suggestions for further work are discussed in chapter 6.


12


13

2. Basic concept and related topics The main purpose of this research is the assessment of photogrammetric products derived from video imagery. As the fundamental principles of photogrammetric workflow using airborne video images are basically the same as those of conventional photogrammetry, necessary photogrammetric concepts in the context of this research are reviewed in this chapter. In addition, studies related to this thesis are introduced as a result of literature review.

2.1. Photogrammetry Photogrammetry is defined as an art, science and technology to obtain information about real world features through photographs. Traditionally the largest application of photogrammetry is the extraction of topographic information from aerial images in order to produce maps.

Photogrammetric techniques and theories have been growing with technology to get more accurate and broaden information. Development of technology has brought considerable extension to domains and applications of modern photogrammetry. In particular, the introduction of analytical photogrammetric systems and the advent of computer brought a new era of photogrammetry. Analytical photogrammetry can be traced back to Sebastian Finsterwalder in 1899, being used in practice after incorporation with computer assistance. Analytical photogrammetry uses mathematical models which explain relationships between measured ground, image coordinates, and camera parameters (Walstra 2006). Computer systems allow to calculate complicate mathematical algorithms rapidly and easily within analytical photogrammetric system. They solve collinearity condition by rigorous mathematical calculation. The advent of analytical photogrammetric systems eliminate many restrictions such as photo size, focal length, camera type and so forth by modeling the camera geometry mathematically. Thus the photogrammetric field is not anymore limited to the measurement of topographic information with film photographs anymore. Through the last half of the 20th century, computation and storing capacity of imagery in digital format has been growing. This has provided extension of analytical photogrammetry and automation on photogrammetric workflow as well (McClone et al. 2004). Today, photogrammetry can be used in many applications with many different data sources such as digital form of imagery from active sensors or non - photogrammetric cameras. Even though, photographs are still the main source of the photogrammetric process (Mikhail et al. 2001).

Videogrammetry is one of the branches of extended photogrammetry which uses video imagery. Normally it is also referred as close range photogrammetry due to the characteristics of imagery recorded by non-metric video devices in relatively low flying height (Ganci et al. 1998). Easy and fast data acquisition of video imagery without specific camera or platform configuration attracts many applications such as reconstruction of historical monuments (Pollefeys et al. 2000) or buildings, object tracking, accident analysis (Neale 2004), and many industrial applications. Meanwhile, this research uses airborne video data which has much smaller scale than close range photogrammetric application. Small scale airborne video data has not been used for conventional photogrammetric applications. However ability on fast and economic data acquisition and high overlapped images attempt to evaluate potential of the video data as a new resource for automatic photogrammetric workflow.


14

2.2. Basic concept of photogrammetry The main task of photogrammetry is the establishment of relationships between image and object space. The general problem of performing this task is the determination of camera orientation parameters and coordinates of object points (Karara 1989). The parameters which need to be determined are divided into interior and exterior orientation parameters.

2.2.1. Interior orientation The interior orientation process includes the geometric parameters of the camera itself such as focal length and principal point position as well as parameters that describe systematic errors related with camera properties. By determining the interior orientation parameters, image coordinates can be transformed to the camera coordinate system. Furthermore, transferred image coordinates are used to determine object coordinates by exterior orientation. This means that accurate object coordinates depends on accurate interior orientation solution. Therefore interior orientation with accurate camera calibration information without image distortion should be established. To obtain interior orientation parameters several solutions have been developed and investigated such as test-field camera calibration, self-calibration (Ackermann 1980), and calibration using line-constraint (Frank A.van den Heuvel 1999).

In case of metric camera, calibration information is determined in laboratory by using a test-field calibration method (Luhmann 2006). However if the used video camera is not designed for photogrammetric purposes it has only limited camera information or it might be even unknown. In case the calibration in laboratory is not reliable, it is common to include calibration process into bundle adjustment. This method is known as self-calibration. The introduction of self calibration method within bundle adjustment process eliminates the interior orientation problem. By applying this method it is possible to use non-metric device for photogrammetry and even to increase the accuracy of bundle adjustment (section 2.2.4.2).

2.2.2. Exterior orientation Exterior orientation determines the position of perspective center (X, Y, Z) and angular orientation (ω, Ф, κ) of camera when the image is captured (Bongbae Jang 2004). After geometry and physics of camera are modeled by interior orientation, camera model coordinate might be transferred into object coordinate using control points.

Exterior orientation requires sufficient number of well distributed control points. This is a time consuming process and difficult to achieve in many practical cases. However this problem can be reduced by using projection center coordinates and rotation parameters determined by the use of GPS/IMU (Inertial Measurement Unit). This does not only reduce the number of required control points but also increase the orientation quality. Therefore exterior orientation based only on GPS/IMU data during bundle adjustment process has being used and widely studied in a photogrammetric field known as direct sensor orientation (Cramer 2001; Jacobsen et al. 2001; Cramer et al. 2002). GPS/IMU based orientation method is being used in bundle adjustment as well. In theory, this method makes possible to adjust photographs without control points. Although, in practice, a minimum number of control points are still necessary for accurate photogrammetric solution (Jacobsen 1997; Wegmann et al. 2004).


15

2.2.3. Collinearity condition The basic geometry to solve the photogrammetric problem is known as the collinearity condition. Collinearity condition specifies that projection center, image point and the corresponding object point in real world must lie along one straight line (Geosystems 2005). Implementing the collinearity condition for adjustment of photographs is the most straightforward way to model the relationship between photographs and object space mathematically. The so call collinearity equation has been introduced and used in photogrammetric adjustment processes such as bundle block adjustment to estimate orientation parameters. When an object point measurement is given in the object coordinate system, the corresponding image space coordinate is composed based on exterior orientation parameters and the principal point distance as only interior orientation parameter (McClone et al. 2004).

The image point, projection center, and object point from different photos are intersected in each corresponding point. The collinearity equation, shown in 2-1 and 2-2 is formulated based on this concept. In this equation, the offset value of principal point (x0, y0) is considered as well.

x � x� � �c �m��X� � X� m��Y� � Y� m��Z� � Z�m��X� � X� m��Y� � Y� m��Z� � Z� � �cR�R� 2-1

y � y� � �c �m��X� � X� m��Y� � Y� m��Z� � Z�m��X� � X� m��Y� � Y� m��Z� � Z� � �cR�R� 2-2

It should be noticed that the bundle of light ray never travels in a perfect straight line from the object through the camera lens to the image due to atmospheric reflection or systematic lens distortion. Therefore a correction has to be made considering lens distortion for accurate airborne photogrammetric workflow (Walstra 2006). Meanwhile the effect of atmospheric reflection is neglectable for aerial photographs.

2.2.4. Bundle block adjustment The bundle block adjustment is a simultaneous solution of interior and exterior orientation which implements a mathematical model in a triangulation procedure. Traditionally, the relationship between image and object space is reconstructed using known ground object coordinates, known as control points. Images are connected by tie points and control points fixing the object coordinate system. The standard block configuration includes the strip with 60% forward overlap and 20-30% side overlap. Using bundle adjustment method, the required number of control points is reduced up to 3 within standard block configuration. Development of GPS/IMU device has led to the direct measurement of exterior orientation parameters and this method has been incorporated with bundle adjustment (Walstra 2006).

Where,

x, y = image measurement of point

x0,y0= principal point offset

X0, Y0, Z0= coordinates of perspective center

c = principal distance

m11-m33 / R1-R3= rotation matrix


16

Bundle block adjustment is the most accurate and flexible triangulation method (Mikhail et al. 2001). The flexibility of bundle adjustment enables to add additional unknown parameters. The additional self-calibration parameters eliminate systematic errors in the image and thus increase the accuracy of triangulation.

Since bundle adjustment method is necessary for orientation of photographs or camera calibration, various software have been developed based on the bundle adjustment method (Wiedemann et al. 2001). ERDAS Leica Photogrammetry Suite (LPS) is one of the well known software which allows to process photogrammetric workflow such as triangulation implementing bundle block adjustment, orthophoto generation and DEM extraction.

2.2.4.1. Ground control point (GCP) / tie points Ground control point (GCP) is an instrumental component of the photogrammetric process to establish the relationship between image space and real world. GCPs are known ground (XYZ) coordinates of identifiable objects in real world. Theoretically, aerial triangulation with bundle block adjustment can be performed with 2 full (XYZ) control point coordinates and 1 vertical (Z) control point for the whole block project (Mikhail et al. 2001). However, a larger number of GCPs are highly recommended for accurate triangulation and for the definition of checkpoints, which enables the verification of overall quality and accuracy of triangulation (Geosystems 2005).

Tie points are visually recognizable homologous points in the image pair which do not have known ground coordinates. Extracted points are used to connect two or more images. Tie points can be measured either manually or automatically. Since tie points measurement is normally based on visual information, orientation information is not used during tie points extraction (Luhmann 2006). The automatic generation of tie points may detect false tie points which should be detected and eliminated by gross error checking.

2.2.4.2. Self-calibration with additional parameters Traditionally, calibration was performed as a separate procedure from block adjustment by means of simplified geometric models and assuming that systematic effects were already corrected. However, systematic errors still remain in many situations due to the difference between calibration environment of laboratory and operational conditions. Even if non-metric camera is used, complete camera calibration is not feasible in laboratory (Mikhail et al. 2001). Especially systematic error parameters are unknown and even large in many applications which use non-metric camera. To solve this problem, several solutions have been sought and developed such as plumb-line calibration, a method which uses geometric constraints (Fryer et al. 1986; Frank A.van den Heuvel 1999), and self-calibration with additional parameters, a method which compensates systematic distortions (Ebner 1976).

Self-calibration was developed and accepted in conventional triangulation since it was introduced to the photogrammetric community and experimented in close-range photogrammetry (Fraser 1997). Self-calibration is a computational process where calibration parameters are included in bundle adjustment (Wolf et al. 2000). Self-calibration bundle adjustment can be performed either with or without ground control points (Remondino et al. 2006). Without any control points self-calibration can calculate the unknown parameters using mathematical equations only based on the corresponding points between images and scale information (Ha et al. 2005). However, since self-calibration within bundle adjustment estimates exterior orientation parameters and interior orientation parameters


17

simultaneously, there is a correlation between unknown parameters which reduces the reliability of self-calibration result itself. Therefore self calibration without control points is suitable when the target objects are recorded within at least three different orientations, and one of them are preferably rotated

by 180° to reduce the correlations between parameters (Maas 1999). Meanwhile, the images are

recorded only in nadir view, high redundancy of control points within rich geometry is recommended in order to minimize the correlation between unknown parameters. The rich geometry of control points refers to the large variation in the Z value of control points (Geosystems 2005).

The self-calibration uses collinearity equation extended with additional unknown parameters which estimate camera information and image errors. The extended collinearity equation can be written as (Cardenal et al. 2006):

x � x� ∆x � �c R�R� 2-3

y � y� ∆y � �cR�R� 2-4

With x, y, x0, y0, c, and R1-R3 as defined in 2.2.3

These equations contain the image coordinate correction function ∆x, ∆y which accounts for the departures from collinearity due to lens distortion. The image coordinate correction function ∆x, ∆y is composed of additional parameters associated with the principal source of systematic errors, such as symmetric radial distortion, tangential distortion, affinity and shear (Fraser 1997). Therefore image

distortion Δx and Δy is written as:

∆x � ∆x�� ∆x� ∆x�� 2-5

∆y � ∆y�� ∆y� ∆y�� 2-6

Each of the distortion factors occurs due to poor design of lens, decentring of individual lens elements and the sensor (Luhmann 2006).

Since quality of self-calibration depends on selection of additional parameters (AP), many researchers have been investigated (Ackermann 1980; Fraser 1997) on introducing sets of additional parameters (Ebner 1976; Fryer et al. 1986). LPS provides self-calibration function with five different AP models such as “lens distortion”, which focus on radial lens distortion; Bauer’s and Jacobsen’s (Jacobsen 2007) simple model which requires three and four additional parameters to compensate radial distortion and affine deformation; and the complex models from Ebner (Ebner 1976) and Brown (Brown 1971). Ebner’s model introduces 12 additional parameters to eliminate various systematic errors, including lens distortion, scanner error, affine deformation, and film deformation. Meanwhile, scanner error and film deformation is not any more the considering distortion factors for digital format images. Brown’s physical model proposes 14 additional parameters to compensate for a large variety of physical systematic errors. Ebner’s and Brown’s distortion model requires high number of control points and tie points within rich geometry to calculate accurate distortion parameters. Since more additional parameters compensate for more distortions in the image, it may seem better to add more AP than necessary. Nevertheless this cause an over-parameterization and decreases the final accuracy


18

of triangulation (Cardenal et al. 2006). In consequence, a selection of proper additional parameters is important to achieve accurate self-calibration results.

2.2.4.3. Integrated GPS/IMU (Direct georeferencing) With the development of GSP/IMU devices, integration of GPS/IMU into triangulation process has been proposed and extensive studies have been done known as direct georeferencing. Using the bundle adjustment method decreases the required number of control points. However, obtaining ground control points from field is still time consuming and difficult to acquired due to accessibility reasons. Meanwhile, by means of using perspective centres determined by kinematic GPS with the assistance of inertial measurement unit (IMU), the direct measurement of exterior orientation parameters without control points became possible (Jacobsen 2001). Also integration of GPS/IMU into triangulation strengthen the geometric stability of the model (Cramer et al. 2002). Due to these advantages of integrating GPS/IMU data, direct georeferencing method has been broadly used.

However, acquiring accurate GPS/IMU data for direct georeferencing is expensive for low budget project. This means that using accurate GPS/IMU data for airborne video data goes against the economic advantage of video data. Considering that most of the aircraft contains GPS device as a navigator, this positional information may provide approximate estimation of image orientation. Since camera device is mounted on the platform, using approximate exterior orientation value would assist on the estimation of accurate exterior orientation parameters.

2.3. Feature extraction (feature tracking) Feature extraction is a basic task in many technical applications. For instance, it has been widely used in photogrammetry and image interpretation. The term feature tracking has been widely used in computer vision, as regarded the most fundamental process to extract features in image sequences. Likewise, various feature tracking algorithms have been developed for different applications such as image matching, motion tracking, facial feature tracking, or motion understanding (Lucas et al. 1981; Bourel et al. 2000). Successful image matching using feature tracking algorithm in computer vision has recently had a large impact on digital photogrammetry (McClone et al. 2004). This has motivated the use of the feature tracking algorithm as feature extraction and image matching method in the photogrammetric field.

The feature tracking method to be used in this research is the Kanade-Lucas-Tomasi (KLT) feature tracking algorithm, which is one of the widely used feature tracking algorithms nowadays. It was proposed by Lucas and Kanade in 1981 as an image registration technique (Lucas et al. 1981). KLT tracker uses a window containing a group of pixels and tracks similar textures having small movements. The method calculates the displacement between previous and current frames by means of minimizing the sum of squared intensity difference over the windows (Tomasi et al. 1991).

Since the method compares the dissimilarity of components in two windows, the result of this method is strongly related with the windows size and the selected feature points. As an extension of this research, Shi proposed measuring dissimilarity using affine motion to find good features to track (Shi et al. 1994). The tracking method proposed by Lukas and Kanade has been used in computer vision and remote sensing applications such as human body motion tracking, moving object tracking, road feature extraction and traffic monitoring (Neale 2004; Sugaya et al. 2005).


19

2.4. Robust estimation Apart from systematic errors, gross errors (outliers) in input data affect all estimated unknowns leading to significant distortion of result. The gross errors are not normally distributed in overall observations, even several outliers can occur in the same image simultaneously. It is important to detect and eliminate only those erroneous observations (Luhmann 2006). Many approaches to detect and eliminate gross error have been developed in the photogrammetric community and those researches proposed several automatic gross error detection (Geosystems 2005).

Robust estimation is one of the methods to eliminate or compensate for outliers in the data. Since efficient elimination of outliers is difficult due to various potential error factors, large number of robust estimation methods have been developed as well. Several techniques have been introduced depending on strategies for finding blunders, such as complete search, random sample consensus (RANSAC), clustering, and maximum likelihood (ML) type estimation. Feasibility and efficiency of each technique depends on the characteristic of the input data, such as gross error size and percentage of gross error. For example, ML type estimation which is extension of weighted least square estimation can handle gross error up to 10-20% of the total measurements (McClone et al. 2004). Meanwhile the RANSAC method performed better with larger percentage of gross error (Ruzgiene et al. 2005).

During this research ML type robust estimation is used. This method defines the weights as function of the residuals in iterations based on redundancy of observations (Luhmann 2006). During iterations, the weight for the wrong observation decrease until zero and gross errors are detected. Since robust estimation method is based on the redundancy of observations, the effect of automatic gross error detection does not only depend on the mathematical model but also on the redundancy in the block. Therefore more observations in blocks bring better error detection results.

2.5. 3D modelling 3D modeling from image sequences is one of the important and ongoing research topics of photogrammetry and computer vision. The applications of 3D models are enormous such as object designing, planning and accurate reconstruction of virtual realistic world space. Since applications are diverse, flexibility of data acquisition is often preferred. The source for 3D models includes video data or photographs from amateur camera (Faugeras et al. 1998; Pollefeys et al. 2002), conventional aerial photograph or laser scanned point data. Traditionally aerial photograph was the major source of photogrammetric 3D modeling applications. Meanwhile video imagery or amateur photographs have been used only for industrial 3D modeling or close range applications. Nowadays, 3D building modeling from high density laser point data or integration of laser point data with aerial photograph is being widely used as an alternative option (Brenner 2005). If the application requires less geometrical accuracy, simple and economic 3D models produced by commercial digital camera are a good solution. Due to simple and easy means of data acquisition, close range video data has been used as source for modeling industrial objects, reconstruction of specific buildings or historical places (Karara 1989; Pollefeys et al. 2000; Pollefeys et al. 2002).

With increasing interest on photorealistic 3D modeling, various 3D modeling software applications such as AutoCAD, Google Sketchup, CyberCity, and ImageModeler (IM) have been developed and released. ImageModeler, used in this research, was released by Realviz as one of the end-user application 3D building modeling tool. Reconstruction and measurement of the object from multiple


20

photographs is done by calibrating images based on photogrammetric technique and derived 3D information.

In ImageModeler, several primitive models such as cube, cylinder, and plane are implemented. Each primitive model contains geometric constraints which are commonly addressed in characteristics of objects in the field. Therefore simple shape of building can be easily modeled in the image with few clicks by using parametric model. After the model of buildings is established, IM allows extracting texture from corresponding images in order to obtain photo realistic model.


21

3. Methods This chapter presents the methodology approach and the work flow adopted in this research. The study is divided into two parts. The first part examines the use of a practical photogrammetric workflow using video data as input for the process of triangulation (section 3.1). Phase two concentrates on the 3D modeling of building features using video data (section 3.2).

3.1. Part I : Aerial triangulation Aerial bundle block triangulation adjustment is performed to establish the relationship between images and object space estimating orientation parameters during part I. As an assessment method for feasibility of video data as photogrammetric resource, achievable triangulation accuracy is analyzed. Figure 1 resents the adopted methodology in this stage and the processing steps named by letters in alphabetical order.

Figure 1 Photogrammetric workflow


22

After the video images are inputted (a), approximate interior orientation parameter (IOP) and exterior orientation parameter (EOP) values are imported in order to perform bundle adjustment (b). Afterwards, GCP points are measured on the corresponding image points (c). The tie points are automatically extracted by means of KLT feature tracking method using all frames and later imported into the block (d). Considering the compression method applied for video images recording, only the key frames are selected in order to reduce the efforts and processing time (e). Once sufficient number of observations is measured, IOPs are estimated by self-calibration method (f). Considering the derived check point RMSE from self-calibration, estimated IOPs are accepted and applied to the whole block (g). At the end, triangulation is performed within whole block (h) and each triangulation result is used for accuracy assessment (i).

3.1.1. Initial interior orientation parameters (IOP) estimation (Self-Calibration) For accurate triangulation of video data, exact estimation of camera information and determination of proper distortion model are critical processes. As the camera recording video data used for this research is a non-metric camera, exact camera parameters are not provided. Moreover, as systematic distortion on video data is supposed to be larger than on aerial photography, the characteristics of the observing instrument cannot be neglected. Therefore proper interior orientation parameters and distortion parameters for video data are estimated by a self-calibration method. Figure 2 shows the workflow adopted for the self calibration process. In this figure, the sequence of steps is named by letters following alphabetical orders.

Figure 2 Self-calibration workflow


23

As explained in 2.2.4.2, self-calibration with additional parameters (AP) method requires redundant ground control points within rich geometry. Thus selection of the frames containing big height difference is the first step to estimate interior orientation parameters (a). Once selected images are imported, approximate focal length and principal point offset values are applied as initial values (b). Afterwards, well distributed GCPs are measured on the images (c). Before estimating the focal length and principal point offset, sufficient distortion model should be selected to eliminate image distortion (d). Several different distortion models are experimented and the one producing best fit of distortion model is selected considering derived RMSE on check points (d). Later, focal length, principal point offset values and corresponding distortion values are estimated within selected AP model (e). Estimated IOPs and distortion values are applied into whole frame as known values (f). This is carried out based on the assumption that interior orientation parameters and systematic errors are constant during the entire acquisition, as the movement of the platform is quite stable and imagery is recorded in short time.

The known or approximated interior orientation parameters such as focal length, principal point offset, and measured control points are inputted into block calibration project (b). Afterwards, a proper distortion model is selected in order to eliminate systematic image distortions (d). As shown in Cardenal’s research (Cardenal et al. 2006), selection of different distortion models affect the accuracy of observation coordinates. Different distortion AP models are used during self-calibration. Afterwards, respective self-calibration results are compared. The model producing the lowest root mean square error (RMSE) on control and check points is selected and used as known for further estimation.

After a best fit distortion model is selected, accurate focal length and principal offset coordinates are estimated. Experiments with several set ups, applying different IOP and exterior orientation parameters (EOP) weighting values, with and without initial GPS/IMU, data are performed to select the best estimation of interior orientation. Later the best interior orientation parameters within estimated IOP from different setups are selected by comparing derived RMSE on the check points and on the estimated exterior orientation parameters. The residual of estimated EOP is calculated from the difference with initial EOP. Since RMSE of estimated EOP is derived from initial value, it can not be considered as true residual. However, since the Z-value of estimated EOP is the most sensitive factor against calibration error and the video camera is mounted on the same platform as mapping camera, overall quality of estimated IOP can be assessed by comparing z-value for each set-up.

Finally, when the interior orientation parameters are estimated with selected frames by self-calibration, those parameters are applied into a whole block as known camera information for block triangulation. The aerial triangulation process is explained in the next section.

3.1.2. Aerial triangulation by bundle adjustment Once the block is defined with selected key frames, survey points or reference points are measured on corresponding images as control points or check points. Normally tie points are generated either manually or automatically to assist geometric connection between images. However, in case of video data which involves a large number of images with high overlapped area, manual tie point measurement is impossible. Therefore KLT feature tracking algorithm is used to extract feature points from the image sequence and later the extracted points are imported into LPS as a tie points. Although, as the KLT tracks features based on group of pixels, it may miss track the features due to occlusion,


24

different reflection of points within clusters of pixels and even moving objects. These miss tracked features or tracked points on moving objects result in gross errors during bundle adjustment process. To solve this problem robust estimation is performed to detect and eliminate false tie points.

After sufficient input data are imported and measured into triangulation block project, 3D information for point observations is derived from bundle adjustment process. Since the quality of triangulation depends not only on the number of control points but also on their distribution, different configurations of ground measurements are used as control points for triangulation assessment. In a first step, triangulation is done including all the measurements as control points, and later, those points showing big residuals are excluded from the process. Likewise, those points that do not lie on correct image points due to image quality are removed. Afterwards, GCP experiments on triangulation are done by selecting “good” control points while reducing the number of control points to the theoretically minimum required. As an additional experiment on GCPs, the triangulation is performed using different distribution of GCPs.

3.1.3. Triangulation result assessment In general, the quality of triangulation with bundle block model is evaluated by RMSE of check points. In this research, the triangulation results derived from different control point setups are analyzed by comparing them with theoretical expectation. The theoretical expectation calculates the precision of 3D measurement in stereo model based on scale and base length (Cramer 2006; Luhmann 2006). It is defined as:

S� � S � m� s� 3-1

S� � m �Hb � s� 3-2

Where,

SX, SY, SZ =theoretical accuracy of X,Y,Z sp = precision of image measurement m =scale b =base length H =object distance (flying height)

The theoretical accuracy calculated using these equations only considers the accuracy of a stereo model which uses two images. However, since bundle adjustment considers all the bundle of rays intersecting on same points, measurements from the bundle adjustment are supposed to be better. Also, the accuracy of bundle adjustment depends on more than image scale and base length. Therefore calculated theoretical accuracy is considered as minimum expectation for bundle adjustment.

3.2. Part II : 3D building modeling This section presents the general approach adopted for 3D building modeling in the second phase of the research. Digital video data has been used for reconstruction of single objects such as a historical monument or a building. However this research does not focus on reconstruction of a specific object or one building. Instead, several buildings in a residential area are modeled using images recorded within small scale (1:45000). By using ImageModeler, one of the end-user 3D modeling applications, several buildings are modeled. Figure 3 presents the general workflow of 3D building model with


25

ImageModeler (IM). In the figure, the 3D modeling procedures are named by letters following alphabetical orders.

Input photos

Measure tie points/input survey points

Camera calibration Is it well calibrated?

Define coordinate system

Measure distance of building model

Assess geometric model accuracy

Yes

a

b

c

Measure more points

No

d

Model 3D buildingse

f

g

(section 3.2.1)

(section 3.2.2)

(section 3.2.3)

(section 3.2.4)

Figure 3 Workflow of 3D building modeling

Once the buildings to be modeled are selected, corresponding images are selected and imported (a). Afterwards, tie points are measured on the images where clearly identifiable objects exist and survey point information is added on corresponding image points (b). Based on measured points, camera is calibrated (c) and coordinate system is defined by entered survey information (d). Once coordinate system is defined, simple buildings can be modeled using primitive models implemented in IM (e). The distance of modeled buildings is measured (f) and accuracy of model is assessed by comparing with the measurement from reference data (g).

3.2.1. Data input In the first step, initial camera information is required as well as several images of the study area. For a better three dimensional height measurement, forward view images should be imported. Since the characteristics of capturing camera are different from nadir view photograph, specific camera information should be also imported when forward view images are used.

3.2.2. Camera calibration Once adequate images are loaded, camera calibration can be performed. To calculate correct 3D information from model, calibration of camera is an essential procedure in modeling workflow. Calibration in IM involves retrieving the parameters of the camera used to shoot images, such as 3D


26

position and orientation of camera as well as camera dependent parameters such as focal length, radial distortion and principal point offset.

The identification of locators acting as tie points on the images is the first step to start the calibration process. These locators connect each frame and assist in the estimation of calibration parameters. When enough and well distributed locators are measured, calibration is done automatically. The quality of automatic calibration can be evaluated by checking the residuals of locators. If residuals are bigger than the expected tolerance, the calibration result should be improved. This can be done by adding more locators, adjusting the location of existing locators or applying geometric constraints. Since calibration quality has strong effect on the final accuracy of model, the calibration process is iterated until satisfactory results are obtained.

Once the calibration result is acceptable, relation between image and object space is established by defining coordinate system and scale. Coordinate system can be defined either manually or automatically when well distributed ground control points are available. When all the images are oriented, 3D models can be retrieved from the multiple images.

3.2.3. Building modelling Buildings can be modelled either creating surfaces separately and connecting them with each other or using primitive models implemented into IM. Creation of surface is based on measured points. However, since the resolution of the video images is low, some points can be poorly measured on the images and the created surface with those points might be poorly modelled. On the other hand, using primitive models such as cube, cylinder or plate, the simple shape of buildings can be easily modelled. Also, poorly measured points belonging to the building can be compensated by using a primitive model, since it contains geometric constraints. For example, the cube model is used as a primitive in order to improve the quality. By assigning cube to the building, a flat base is obtained which constraint that the four roof corners lay on the same plane, considering that cube allows only perpendicular plane. Another advantage of using cube is that only three points are required to define X, Y plane of cube. This enables obtaining accurate models even though one of the four roof corner points of the building has to be neglected.

IM does not provide parametric model for various roof shape. Therefore only the body of the buildings are modelled using parametric model while the roofs are modelled by connecting the modelled building body and created building roofs surfaces.

3.2.4. Building model accuracy assessment The accuracy of modeled buildings is assessed by comparing geometric information such as planimetric distance or height between the model and reference data. Comparing the measurement from modeled building and reference data, different experiments on 3D building modeling are performed. Horizontal accuracy assessment is done by comparing building roof edge length. Likewise, for vertical accuracy assessment, the height of several building roofs is compared. Estimation of achievable accuracy of 3D model is evaluated by comparing derived accuracy with theoretically expected accuracy.


27

4. Data processing and result This chapter presents the relevant details of data processing, the experiments done and the results obtained after following the proposed methodology approach introduced in chapter 3. The chapter presents the aerial triangulation processing workflow and results (chapter 4.1) and the later 3D building modelling and accuracy assessment by means of video data (chapter 4.2).

4.1. Aerial triangulation This section describes the triangulation process with aerial video data, from data preparation, estimation of IOPs, to the analysis of triangulation result. The process follows the conventional photogrammetric workflow as outlined in chapter 3.

4.1.1. Data preparation To proceed on the research and conduct the planned experiments, a necessary step is the preparation of the input available data sets. Here it is necessary to address the preparation of video data as well as the reference ground truth data and tie points.

4.1.1.1. Aerial video data Of relevant importance for this research is aerial video imagery data. The video image used was acquired over Enschede, in the Netherlands, by Fugro system FLIMAP on 14 March 2007 using SONY XC-555P non metric digital video camera shown in Figure 4.

Figure 4 SONY XC 555P digital video camera

The device was mounted on helicopter flying at 275 meters flight height and at 45kts (83.34 km/hour) velocity. During the helicopter flight over Enschede, video camera recorded 25 frames in every second within chosen flight height and velocity of helicopter. It provides more than 95% forward overlap between every adjacent image frame, which is very large as compared with conventional aerial photographs.


28

The video data was processed by MPEG compression technique in order to reduce the size of the storage information. MPEG reduce the volume of information by storing the information of key frames, while only the changes are stored between two key frames. After considering the applied MPEG technique, a frame is selected as a key frame every 30th frames. Since the only key frames contain full image information, every 30th frames within 1500 total frames are used for bundle adjustment. As Figure 5 shows, two adjacent considered key frames provide approximately 80% of overlapped area, represented by the red box in the figure. By using only the key frames for bundle adjustment, efforts and time for data processing can be reduced.

Figure 5 Overlap area between two key frames (30th frames)

On the other hand, the acquired video data does not provide side overlap. This decreases the stability of the block since there are not suitable constraints for the adjustment on the lateral edges. For this research only a part of the video data of one flight strip from GJ van Heek Park to train station is used. The study area covered by aerial video data is shown in Figure 6. It includes a residential area, big and tall buildings and a tunnel along the main street.

Figure 6 Study area


29

The following table shows the specification of video camera and the images used for this research.

Table 1 video data specification (provided by Fugro)

Pixel size (㎛) 8.6 x 8.6

Picture elements (H x V) 752 x 582 Focal length (mm) 12

Flying height (m) 275

Ground resolution (cm) Approximate 40

Image overlap (every 30th frame) Approximate 80%

Besides the video camera, a mapping camera, laser scanner and GPS/IMU device were mounted on the same platform. The laser scanner, which measures 20 points per square meter, is used as reference data. The GPS/IMU data provides the position and the orientation of the mapping camera. However, since the relative position of video camera with respect to the mapping camera is unknown, the provided GPS/IMU data can only be used as initial exterior orientation parameters.

4.1.1.2. GCPs / tie points A total of 98 GCPs (93 full / 4 horizontal / 1 vertical) are used as control points and check points for this research. 44 of them are collected on the field with a RTK-GPS device which provides less than 3cm standard deviations on point measurement and 53 of them are measured from high point density laser system (20 points per m²). Since ground resolution of video image is 40 cm, the laser point data providing 3 cm planimetric standard deviation and 2-3 cm vertical standard deviation (Vosselman 2008) is suitable to be used as reference data.

Tie points used in this research are extracted by using KLT (Kanade-Lucas-Tomasi) tracking method. Since automatic feature extraction performs more effectively with high overlap of images, all 1500 images are used. Doing so, more than 170 feature points are tracked in every image as shown in Figure 7. In Figure 7 green points are the imported tie points existing in both images and some of them are highlighted (red points) for better visualization. Total 3566 tracked points are imported as tie points for whole strip and 1208 points are involved in selected key frames.

Figure 7 Tracked points from KLT


30

4.1.1.3. Transformation of coordinates The data provided from Fugro have two sets of coordinates which were recorded in different coordinate system. One set is measured in the local RD/NAP system and the other set is in WGS84 coordinate system as shown in Table 2.

Table 2 Coordinate system of different data set

Data set Measured coordinate system

GPS/IMU data(25times per second) RD/NAP

Mapping camera position (every 3 seconds) WGS84/UTM (zone 32)

GCPs from laser RD/NAP

GCPs from field WGS84/UTM (zone 32)

To process and analyze these data, it is important to transform them into a common coordinate system. Therefore all the data sets are transformed into WGS84/UTM coordinate system. The coordinates of GCPs from laser point has been transformed into WGS84/UTM coordinate system using coordinate calculator software version 4.1 obtained from the Dutch Geometric Infrastructure website.

However, GPS/IMU data contains not only XYZ coordinate of perspective center but also rotation (ω, φ, κ) of image which makes impossible to simply transform by using a standard approach. As a solution to this, the difference between two GPS/IMU entries has been used to transform the data. To this end, the position of mapping camera at a given time is subtracted from the previous coordinate of GPS/IMU data. The found delta of distance is then added to the mapping camera position and thus all the information of GPS/IMU is transformed into WGS84/UTM system. Transformed GPS/IMU data will be imported into LPS as initial exterior orientation parameter (EOP).

4.1.2. Self-calibration After all input data are prepared, interior orientation parameters are estimated. Since the used video camera is non metric camera, not sufficient reliable interior orientation parameters (IOP) such as focal length, principal point offset, and image distortion parameters have been provided. Actually, a focal length of video camera has been provided as 12mm. However, when triangulation is performed with the given focal length, large residuals (around 200m) are observed on Z-value of the estimated exterior orientation parameters (EOP).

Table 3 Triangulation result with focal length 12mm

rX (m) rY(m) rZ(m) RMSE on control point 2.88 2.61 10.41

Residual of estimated EOP Average 40.21 31.97 253.04

Max. 101.84 98.55 284.72 Min. 4.41 0.021 233.52

From the triangulation results shown in Table 3, it is assumed that video camera is not really calibrated and given focal length does not fit to the video imagery. Therefore estimation of correct interior orientation parameters by self-calibration is the first task of this research for accurate triangulation of video data.


31

Approximate interior orientation values are required as an input in order to perform self-calibration. Since the given focal length is not a reliable value, initial focal length has been calculated manually using flying height, image width, and ground distance. Figure 8 explains the basic geometry of the image plane and the object plane. For principal point coordinate offset, an initial value of zero was set.

DE =image width =582 * 8.6µm

AB = ground width = 205~225m

H=flying height = 275m

f = focal length= H � �� 275 � ��~��

��.� � 6.25~6.7mm

Figure 8 Approximate focal length calculation

As explained in section 2.2.4.2, when self-calibration is performed to estimate focal length and principal point offset values, at least 6 well distributed full control points within rich geometry are required. Therefore four frames involving tall buildings and tunnel are selected. 14 GCPs are measured on corresponding image points. The distribution of GCPs used for self-calibration is shown in Figure 9.

Figure 9 Distribution of GCPs for self-calibration

A B

D E

Cf

H


32

Before the estimation of accurate focal length and principal offset, different distortion models are tested in order to select the best AP model which compensates image distortion. The selected best fitting model is used as known parameter for the later triangulation process. Five different distortion models: lens distortion, Bauer’s simple model, Jacobsen’s simple model, Ebner’s orthogonal model, and Brown’s physical model, which are available in LPS were used (Geosystems 2005). These methods calculate different number of distortion parameters as shown in Table 4 (#AP column). The focal length used for this experiment was taken as previously estimated from image geometry.

Table 4 Triangulation with different distortion model

Distortion model

#AP RMSE of control point (m) RMSE of check point (m) rX rY rZ rX rY rZ

without AP 0 1.62 1.10 0.55 3.06 1.45 7.35 Lens distortion 2 1.62 1.17 0.44 3.39 0.94 2.43 Bauer's 3 0.24 0.30 0.07 0.54 0.52 1.33 Jacobsen's 4 0.16 0.27 0.06 0.58 0.48 1.05 Ebner's 12 0.15 0.49 0.10 0.55 1.63 3.82 Brown's 14 0.16 0.28 0.04 0.62 0.75 1.12

The distortion model producing the best fit from the RMSE analysis is selected. As it can be seen on Table 4, the RMSE of control and check points varying with the distortion models applied. Considering the triangulation results, the AP model producing the lowest RMSE is considered as a best fit distortion model.

The results presented in Table 4 show that the selection of Jacobsen’s simple model is sufficient to compensate the distortions on the video images. Also it can be assume that radial distortion and affinity has a most significant effect on these video images since Jacobsen’s model calculate distortions mainly considering these factors. Therefore Jacobsen’s simple model is selected and applied for further work.

Jacobsen’s simple model implemented in LPS is written as following equations, based on Jacobsen’s research (Jacobsen 2007):

∆x � a�x�r� � r�� a�x a�y 4-1

∆y � a�y�r� � r�� a�x a�x a�x� 4-2 Where,

x, y : image coordinate (mm)

r: radial distance

r0: maximum distance from the center of image to the GCP which has largest radius

Δx, Δy: distortion on x, y direction in the image plane

a1-a4: to additional distortion parameter


33

The Jacobsen’s model contains four additional distortion parameters (a1~a4) which compensates different distortion factors; a1 compensates for radial symmetric distortion, a2 for affinity distortion, a3 for shearing and a4 for general deformation.

Using the Jacobsen’s distortion model, corresponding distortion parameters, focal length and principal point offset values are estimated. Different set ups such as use of initial GPS/IMU data, different weight for initial EOPs (shown as EOP weight column in Table 5), and fixed/weighted IOPs are applied in order to find the best fitting IOP estimation. Every set-up uses same standard deviation for image point and same weighting value for IOP as follows:

− Standard deviation for image point (pixel) = 0.33

− Standard deviation for GCPs(m) = 0.1

− Weight for IOP : Focal length weight (mm) = 0.08 / Principal point offset (mm) = 0.01

Table 5 shows the results of self-calibration with different set-ups. In this table, for the setup configurations used, the term “with” in brackets refers to the use of initial GPS/IMU data while the term “without” refers to a set up were initial GPS/IMU data was disregarded. When initial GPS/IMU data was used different weighting value has been applied.

Table 5 IOP estimation with different set-up

Set up( use of GPS/IMU) EOP weight RMSE of check point(m) RMSE of estimated EOP(m) XYZ(m)/ωφκ(°) rX rY rZ rX rY rZ

Fixed IOP(without) 0 / 0 0.64 0.45 0.98 2.08 4.21 10.55 Weighted IOP(without) 0 / 0 0.65 0.47 0.92 1.36 2.83 2.55 Fixed IOP(with) 2 / 0.2 0.69 0.47 0.63 3.72 0.81 9.84 Fixed IOP(with) 20 / 2 0.64 0.46 0.96 4.15 0.92 10.37 Weighted IOP(with) 2 / 0.2 0.67 0.54 0.43 2.44 0.51 0.33 Weighted IOP(with) 20 / 2 0.65 0.48 0.90 3.24 0.74 3.23

From the experiment of set up configurations, the best result is selected by comparing derived RMSE on check points and RMSE of estimated EOP. RMSE of estimated EOP is calculated from the difference between GPS/IMU data and the estimated EOPs. Since GPS/IMU data is an approximate value, it does not warrant absolute accuracy of estimation. However, RMSE of estimated EOP can be used as an additional indicator of the quality of estimated parameters. The best estimated IOP from the group of considered set ups is implemented for whole frame block adjustment as a known value.

Besides the estimation of best fitting focal length and distortion model, the quality of initial EOP can be assessed. As shown in Table 5, smaller RMSE on check points are observed when smaller standard deviation values are applied to initial EOP. This result refers that initial EOPs do not have big difference with real video camera EOPs; therefore it is acceptable to be used as an initial EOP values.

Considering the RMSE of each result, the set up using GPS/IMU and IOP weighting, highlighted in Table 5, is selected as best IOP estimation. The estimated IOP values from the selected set up are shown in Table 6.


34

Table 6 Estimated IOP from selected set-up

IOP estimated value

Estimated IOP

Focal length(mm) 6.0071 Principal offset on xo(mm) -0.013 Principal offset on yo(mm) 0.0176

Estimated AP value

a1 -6.50E-03 a2 -4.44E-02 a3 -2.24E-03 a4 2.97E-03

Focal length and principal point offset values are simply inputted into LPS as camera information. Estimated distortion parameters, which are a1-a4 in Table 6, are used to calculate distortion value along the radial distance using Jacobsen’s distortion equation (equation 4-1, 4-2). Table 7 shows the calculated distortion values when estimated distortion parameters are implemented in the Jacobsen’s model.

Table 7 Calculated distortion value

X (mm) Y (mm) Δx Δy Radial distance r (mm) radial distortion Δr (μm) -3.20 -2.50 0.4906 0.1945 4.0608 527.7392 -3.00 -2.34 0.4210 0.1501 3.8070 446.9694 -2.50 -1.95 0.2789 0.0652 3.1725 286.3995 -2.00 -1.56 0.1760 0.0124 2.5380 176.4451 -1.50 -1.17 0.1045 -0.0144 1.9035 105.5214 -1.00 -0.78 0.0566 -0.0213 1.2690 60.4832 -0.50 -0.39 0.0244 -0.0145 0.6345 28.3447 0.00 0.00 0.0000 0.0000 0.0000 0.0000 0.50 0.39 -0.0244 0.0159 0.6345 29.1299 1.00 0.78 -0.0566 0.0272 1.2690 62.8207 1.50 1.17 -0.1045 0.0278 1.9035 108.1568 2.00 1.56 -0.1760 0.0114 2.5380 176.3756 2.50 1.95 -0.2789 -0.0281 3.1725 280.2915 3.00 2.34 -0.4210 -0.0967 3.8070 431.9619 3.20 2.50 -0.4906 -0.1337 4.0608 508.4830 -3.20 2.50 0.4794 -0.1194 4.0608 494.0522 -3.00 2.34 0.4105 -0.0833 3.8070 418.8796 -2.50 1.95 0.2701 -0.0169 3.1725 270.6742 -2.00 1.56 0.1690 0.0203 2.5380 170.2365 -1.50 1.17 0.0993 0.0345 1.9035 105.1048 -1.00 0.78 0.0531 0.0317 1.2690 61.8606 -0.50 0.39 0.0226 0.0182 0.6345 29.0298 0.00 0.00 0.0000 0.0000 0.0000 0.0000 0.50 -0.39 -0.0226 -0.0167 0.6345 28.1240 1.00 -0.78 -0.0531 -0.0258 1.2690 59.0372 1.50 -1.17 -0.0993 -0.0211 1.9035 101.5117 2.00 -1.56 -0.1690 0.0034 2.5380 169.0565 2.50 -1.95 -0.2701 0.0540 3.1725 275.4923 3.00 -2.34 -0.4105 0.1367 3.8070 432.6855 3.20 -2.50 -0.4794 0.1802 4.0608 512.1531


35

Figure 10 Distribution of calculated distortion value

In the Figure 10, calculated distortion values are graphically displayed to analyze the pattern of image distortion. As it is seen, calculated distortion values are not symmetric along the axes and maximum

16μm difference which is about twice bigger than pixel size are observed at the edge of the image.

4.1.3. Applying estimated IOPs The calculated distortion values from Jacobsen’s model are imported into LPS as camera information. By doing this, additional parameters to compensate distortions do not need to be calculated for further work. Therefore, the number of unknown parameters is reduced and thus less control points, needed to estimate unknown parameters, are required.

In order to apply the calculated distortion values as camera information several experiments are done. These experiments involve:

− Input calculated distortion value: this approach simply implement calculated distortion value into LPS as known camera distortion value.

− Input distortion value derived from BLUH: implement independently calculated distortion value into LPS as same as previous approach. The distortion values are calculated using BLUH application which is developed based on Jacobsen’s research. Since Jacobsen’s model is applied for this research, more accurate distortion values are expected to be obtained from BLUH.

− Obtain undistorted images before triangulation: using calculated distortion values, the images can be distorted beforehand. Once the distortions in the images are eliminated, the undistorted images are used as an input for bundle adjustment.

x (mm)

y (mm

)

∆r= 0.528

∆r= 0.494 ∆r= 0.508

∆r= 0.512

x,y : image coordinate (mm) ∆r : radial distortion (mm)


36

− Calculate selected distortion model inside adjustment procedure: Instead of putting calculated values as camera information, the distortion parameters are calculated during bundle adjustment process. When direct implementation of calculated distortion value method is not sufficient to compensate distortions in the image, this approach can be used as an alternative.

The design and results of these experiments are presented in the following three sections.

4.1.3.1. Input calculated distortion value from LPS The calculated distortion values are entered into LPS along with focal length and principal point offset values as camera information in order to perform triangulation. However, as the distortion values are not symmetric as shown in Figure 10, the decrease of triangulation accuracy is expected. Once distortion values are added with corresponding radial distance, LPS computes distortion coefficients (K0, K1, and K2) as shown in Figure 11. Then, all the image measurements in the block are adjusted based on the calculated lens distortion (Geosystems 2005).

Figure 11 Applying distortion values as camera information

Triangulation is performed with same set-up as that used for self-calibration but without additional parameter calculation in order to check the derived accuracy after IOP implementation. It indicates whether implemented distortion values can compensate image distortion effectively or not. Table 8 presents the result of triangulation after IOP implementation with the result of self-calibration using same control points.

Table 8 Triangulation result after distortion value implementation

RMSE for check point RMSE on estimated EOP

rX rY rZ rX rY rZ Self-calibration 0.67 0.54 0.43 2.44 0.51 0.33 Implement calculated IOP 3.40 2.29 5.87 5.65 16.29 20.48

From Table 8 it is seen that the RMSE of check points are larger than two meters and the residual of estimated EOP is so big that it can not be acceptable. Two meters of RMSE in check point is about


37

five times the ground resolution of the image, which is around 40cm. This triangulation result shows that the approach used in this experiment is not sufficient for implementation of distortion value.

4.1.3.2. Input distortion value derived from BLUH Since the LPS is not a self-calibration purpose application, BLUH, the bundle block adjustment program system of Hannover University developed based on Jacobsen’s additional parameters is used to calculate distortions (Wiedemann et al. 2001). BLUH calculates distortion in image by self-calibration with 12 additional parameters as shown in Figure 12. Among these parameters BLUH selects the most sufficient parameters which compensate for image distortion (Jacobsen 2007).

Figure 12 Compensate distortion by additional parameter in BLUH (Jacobsen 2004)

As the selected distortion model is developed by Jacobsen, correct distortion values with corresponding radial distance are obtained from BLUH application software. Detailed result of BLUH process is shown in Appendix 1. Obtained distortion values are implemented into LPS as camera information as it was done in the previous experiment. Once distortion coefficients are computed, feasibility of implementation is checked by triangulation error assessment.

Like in the previous experiment, the triangulation result implementing calculated distortion value is compared with self calibration result. Table 9 presents the RMSE on check points and estimated EOP from each triangulation.

Table 9 Triangulation result implementing distortion value from BLUH

Set-up RMSE for check point (m) RMSE on estimated EOP(m) rX rY rZ rX rY rZ

Self-calibration 0.67 0.54 0.43 2.44 0.51 0.33 BLUH (fixed IOP) 3.12 1.31 5.32 3.71 12.53 19.95 BLUH (weight IOP) 3.14 1.30 5.17 2.18 12.31 9.78 BLUH (estimated IOP) 3.15 1.29 5.15 2.18 11.48 7.30

angular affinity affinity

tangent ial distort ion radial symmetric distort ion

radial sym.

1 2 3 4 5

6 7 8 9 10

11 12 1,2, 7~ 11 : Physical just if icat ion3~ 6, 12 : mathemat ical just if icat ion


38

The estimated focal length from BLUH (6.3mm) is applied as either fixed or initial interior parameter. However estimated EOP with initial focal length shows about 19.95m offset for z-value. Since z-value of EOPs is correlated with focal length, estimated focal length and principal point offset values from previous self-calibration are applied to check whether wrong EOP estimation is the problem of incorrect focal length. When the estimated focal length and principal point offset values from self-calibration are used instead of the focal length estimated from BLUH, about 12 meters smaller RMSE for z- value was observed as shown in Table 9. However, estimated EOP presents not only large residual on z-value but also the overall estimated EOPs are much different from initial GPS/IMU data. Even though GPS/IMU data does not refer to the exact position of video data, residuals larger than 10 meters are not expected, as both sensors were mounted in same helicopter.

Not only the RMSE on estimated EOP is big but also the check point RMSE is not acceptable compared to the self-calibration result using same focal length and principal point offset as well as same control points. Thus direct implementation of calculated distortion value into LPS is not sufficient to eliminate image distortion.

4.1.3.3. Obtain undistorted images beforehand Referring to the previous experiments, it was not feasible to eliminate the distortions by applying calculated distortion values into LPS as known camera information. Besides applying calculated distortion value into strip project, undistorting the images before triangulation is another possible solution to eliminate distortions. Using calculated distortion values, distortions are eliminated separately from bundle adjustment process. Afterwards, undistorted images are used as input data for triangulation.

However, this method has not been performed during this research. Since the calculated distortion values from LPS are not reliable, undistorting the images using incorrect distortion values is not a consistent approach. On the other hand, the distortion values derived from BLUH system should be most likely correct since they are calculated using an special application for self-calibration bundle adjustment. Therefore it is reasonable to undistort image using distortion values from BLUH. However, this approach was not done due to the time constraints as additional efforts are required to obtain undistorted image.

4.1.3.4. Calculate selected distortion model during adjustment procedure The implementation of distortion value by simply entering distortion values into LPS as known value is failed as it was shown in the previous experiment. From the result, it was assumed that LPS applying different algorithm to calculate distortion. Although it is unknown how implementation of distortion parameters is working inside LPS, the application only considers radial lens distortions and neglects other possible distortions such as tangential or affine distortion. On the other hand, it is possible that distortions different from radial component are also present in the captured data and thus the consideration of the distortion value is not enough to compensate for the total system distortion. Therefore, as an alternative approach to implement distortion parameter, additional parameters of selected Jacobsen’s distortion model should be calculated during every bundle block adjustment.

The estimated focal length and principal point offset values by self-calibration are derived from selected frames. However, as whole video frames are recorded in similar recording condition within one minute, interior orientation parameters can be assumed stable for the whole strip. In consequence,


39

estimated IOP from self-calibration with selected frames are applied into whole frames as fixed internal value for the bundle adjustment.

4.1.4. Bundle adjustment for the whole strip For the bundle adjustment process the estimated focal length and principal point offset values are imported in the whole strip as known values. Distortions are eliminated by calculating additional parameters of Jacobsen’s model during every bundle adjustment process. Within the process of bundle adjustment, different set-ups are applied to answer several research questions. These experiments allow to address issues such as the benefit of using initial GPS/IMU data (section 4.1.4.1), the effect of distribution and number of GCP (section 4.1.4.2) and the feasibility of using tie points extracted by KLT algorithm (section 4.1.4.3).

4.1.4.1. Use of initial GPS/IMU Integration of GPS/IMU data as position of camera into bundle block adjustment has been widely used in photogrammetric applications. With accurate calculation of GPS antenna and camera position, each camera exposure station assist direct sensor orientation. Thus, in theory, no additional control points are required for adjustment procedure.

Since correct GPS antenna and camera position is unknown in this research, given GPS/IMU data cannot be used as exact exposure station but may be used as initial approximated video camera position. Therefore GPS/IMU data transformed during data preparation is applied into LPS project as initial exterior orientation value with standard deviation of 2 meters for X,Y,Z and 0.2° for ω, φ, κ as same setup as applied for self-calibration. Effectiveness of initial GPS/IMU is assessed by comparing estimated check point RMSE from triangulation results. Each triangulation process uses the same control points, with the difference that one uses GPS/IMU data as initial EOP while the other neglects the use of initial GPS/IMU data and leave the EOP as unknown.

Table 10 shows the RMSE on control points and check points derived from bundle adjustment with and without initial GPS/IMU data.

Table 10 Triangulation result depend on use of initial GPS/IMU

Use of GPS/IMU # control point RMSE on check point (m) RMSE on estimated EOP (m) rX rY rZ rX rY rZ

With GPS/IMU

all GCP(85) 2.86 1.05 0.70 50 GCP 0.39 0.34 0.58 2.72 0.92 0.64 25 GCP 0.37 0.32 0.60 2.80 1.03 0.57 12 GCP 0.37 0.35 0.72 3.09 0.86 0.43

Without GPS/IMU

all GCP(85) 3.23 3.10 0.87 50 GCP Bundle adjustment failed 25 GCP Bundle adjustment failed 12 GCP 0.43 0.39 1.21 2.97 3.18 0.73

As shown in Table 10, bundle adjustment with 50 and 25 control points has been failed to be executed, with an error message complaining about the lack of control points. Meanwhile, bundle adjustment is successfully performed when 12 control points are used. Since not even initial EOP is available if GPS/IMU data is not used, LPS should figure out initial EOP from the measured GCPs. Estimation of initial EOP can be done by means such as direct linear transformation (DLT) method, iterative least


40

square while it is unknown which method is implemented in LPS. However, estimation of initial EOP is failed when 50 and 25 GCPs are used, while it succeeded with 12 GCPs. This result is still surprising since it was expected that triangulation can not be done with 12 GCPs since it failed with larger number of GCPs.

Since the result is not consistent, derived results are not reliable to evaluate the effect of using initial GPS/IMU data. Meanwhile decrease of triangulation quality still can be seen within derived result by comparing triangulation result using the same number of control points.

4.1.4.2. GCP configuration The quality of bundle adjustment and self calibration is highly related to the configuration of control points. In order to quantify the effect of control point configuration, different number and distribution of control points are applied to the same project. The effect of GCP configuration on the photogrammetric process is assessed within two photogrammetric processes, one in self-calibration with selected frame and the other in bundle adjustment with whole frames. Different configurations of control points are used in both projects. For bundle adjustment the attention is put on the location of control points selected. Meanwhile, for the self calibration process height of control points is considered during control point selection.

The effect of GCP configuration is evaluated by comparing check point RMSE. During this process the minimum number of required control points and achievable accuracy of 3D information derived from video data is checked as well. The check points RMSE, which refers empirical accuracy, are compared with theoretical expectation, calculated as explained in methodology (chapter 3.1.3):

S� � S � 2756 � 10�� 8.6 � 10�� 0.39�m

S��base length 30m � 2756 � 10�� 27530 � 8.6 � 10�� 3.61�m

S��base length 180m � 2756 � 10�� 275180 � 8.6 � 10�� 0.60�m

The precision of image measurement is low due to the resolution; therefore one pixel deviation is applied as point measurement precision. This equation calculates theoretical expectation when the point is measured in a stereo model while bundle block adjustment, used in this research, estimates the position of the observation with the bundle of rays from several images. Considering that several bundle of rays from different images are intersecting on same point, vertical accuracy is estimated within two base lengths; 30 meters considering based length between only two key frames, 180 meters with the consideration of several images within the bundle. Therefore 0.39 meters for horizontal accuracy and 0.60-3.61 meters for vertical accuracy is expected. However, since the accuracy of bundle adjustment depends on various factors, estimated accuracy is used as minimum expectation.

- Self calibration GCP experiment

As a first experiment with GCP configuration, the effect on the result of self-calibration depends on the use of control points. The GCP experiment carried out during self-calibration process is similar to


41

previous experiment done in section 4.1.2. However, self-calibration results strongly depend on the geometry of control points due to the correlation between unknown parameters. Therefore, only the frames which have rich geometry in image, as they involve a good variation in height difference, are used. In addition, the selection of control points for GCP distribution experiment considers only the height difference.

A total of 14 GCPs measured by GPS device are used for self-calibration. In a reiterative process the number of control points is decreased until reaching a minimum of six points. When decreasing the number of control points, a careful selection of them is done for self-calibration process by considering the height of each control point. As a contrasting experiment, only the points which have elevation between 80 and 83 meters are selected as control points in poor geometry for self-calibration. After the self calibration process is carried out with the two experiments the result is analyzed to assess how geometry of control points effect the self-calibration results. Since there are many factors that influence on self-calibration results, the values shown in Table 11 are used for the initialization of the process:

Table 11 Fixed set up for GCP experiment

Parameter Input value

Image point standard deviation 1 pixel

GCP standard deviation 0.1m

IOP weight focal length :0.08 / principal point offset : 0.01

EOP weight X, Y, Z : 2m / ω, φ, κ : 0.2°

Distortion model Jacobsen’s simple model

A standard deviation of 1 pixel is applied for image points, as it was assumed during theoretical accuracy estimation. The GCP standard deviation used in the model implies that a small fluctuation, equivalent to 0.1 meter, is allowed. This parameter addresses the difficulty of measuring control points on the image due to image quality. The weighting values applied to IOP and EOP are used to estimate the correct parameters from the initial input values.

After running the self calibration process with the considerations described before regarding distribution of GCP, the following results shown in Table 12 are obtained.

Table 12 Self-calibration result with different number of control points

RMSE on check point RMSE on estimate EOP GCP set up rX rY rZ rX rY rZ 10 GCP 0.59 0.60 0.39 3.41 0.28 0.65 6 GCP(min.) 0.72 0.58 1.49 3.33 0.37 0.70 poor geometry (9 GCP) 0.71 0.66 1.30 3.43 0.25 1.20

Referring to Geosystems, at least 6 control points are recommended in every overlapped image to calculate accurate unknown parameters (section 2.2.4.2). However, as Table 12 shows, it is clear that the minimum control points do not guarantee good self calibration results. Either small number of control points or poor geometry has big influence on z direction of check point RMSE. The result


42

using GCPs within poor geometry shows the influence of GCP geometry on the quality of self-calibration. Using GCPs within poor geometry for the self-calibration derives a decrease of overall accuracy of self-calibration, even though a large number of control points are used.

Besides considering the RMSE of check points, RMSE on estimated EOP is compared. Since the position of video camera and GPS/IMU device is unknown residuals on X and Y direction are not used for evaluation. The Z-value on estimated EOP which is most sensitive against self-calibration result, shows that RMSE increase when GCPs within poor geometry are used, although it does not show dramatic changes when the minimum number of control points are used.

- GCP experiment on the bundle adjustment for whole strip After the experiments carried out on the GCP configuration effect within self-calibration process, similar experiments are done within bundle adjustment for whole strip. There are 98 GCPs (93 full / 4 horizontal/ 1 vertical) used in this research. Among them 56 control points are measured from laser point data and 42 points are measured by GPS device during field survey. As a first step of bundle adjustment GCP experiment, it is necessary to eliminate bad control points from the total of 98 GCPs considered. Due to low image resolution, several reference points are measured at not easily identifiable image pixels, thus the GCP quality decrease. This might affect the whole triangulation process. Therefore, those points which are considered as inaccurate should be eliminated to proceed on further GCP experiments.

a. Experiment on the number of control point

Control points to remove are selected after a triangulation process which uses all GCP as control point. Those points for which a residual larger than 1.5m is found are disregarded. Thus, 85 control points are selected and accepted as control points. Afterwards bundle adjustment is performed with the selected control points. To measure the sensitivity of the process, the number of control points is decreased until a minimum of four. Four control points are the theoretically required number of control points to calculate the six exterior orientation parameters of the block (X, Y, Z and ω, Ф, κ) and four additional distortion parameters. Out of the 85control points available, 50, 25, 12, 8, 6, and 4 control points are selected manually by considering their distribution in the whole strip. As already mentioned, this is done in order to analyze the effect of number of control points on accuracy of triangulation. The Table 13 shows the results obtained from varying the number of GCPs considered in the triangulation process.

Table 13 RMSE with different number of control points

RMSE on check point RMSE on estimated EOP GCP set up rX rY rZ rX rY rZ 85 GCP 2.86 1.05 0.70 50 GCP 0.39 0.34 0.58 2.72 0.92 0.64 25 GCP 0.37 0.32 0.60 2.80 1.03 0.57 12 GCP 0.37 0.35 0.72 3.09 0.86 0.43 8 GCP 0.39 0.49 0.81 3.19 0.77 0.41 6 GCP 0.70 0.64 0.96 3.04 0.80 0.68 4 GCP 1.00 0.67 0.98 2.81 0.81 0.55


43

Figure 13 Check point RMSE with different number of control point use in bundle adjustment for whole strip

From this result it can be seen that the quality of triangulation is affected by the number of control points (Figure 13). Especially, when the number of control points reaches 6, RMSE on X and Y becomes larger than ground resolution. Considering the theoretical expectation, derived RMSE on Z value is within expectation range. Also, RMSE on X and Y direction shows similar value with expectation.

Four control points are used as a minimum number for this experiment. However it is only possible when focal length and principal point offset values are known. Since IOP for non metric video camera are usually unknown, self-calibration should be performed with bundle adjustment process.

At this point, it is interesting to compare which accuracy can be achieved when dense control points are obtained within selected area for self-calibration while minimum number of control points is available for whole strip. Since the control points are concentrated on one area of whole strip, the remaining frames which do not have control points are poorly adjusted. This affects the calculation of overall self-calibration results. Therefore the following experiment analyzes the triangulation result using the minimum control points for bundle adjustment with additional control points which is distributed only in the frames used for self-calibration. As a minimum control points, 3 GCPs are selected manually considering the distribution of points. Then triangulation is performed without calculating additional parameters while focal length and principal point offset values are fixed. Afterwards, 13 additional control points used in self-calibration are added to the triangulation process and additional distortion parameters are calculated. The result of this experiment is shown in Table 14.

Table 14 Triangulation result with minimum number of control points and additional GCPs using in self-calibration

RMSE on check point RMSE on estimated EOP GCP use rX rY rZ rX rY rZ 3 GCP without AP 2.71 3.30 4.32 2.78 2.59 5.18 3 GCP+13 SC-GCP 0.41 0.49 1.05 2.52 0.80 0.65

0.00

0.30

0.60

0.90

1.20

50 GCP 25 GCP 12 GCP 8 GCP 6 GCP 4 GCP

RMSE

(m)

rX rY rZ


44

The triangulation results using only the minimum number of control points shows big RMSE on both check points and estimated EOP since this does not calculate distortion parameters. Meanwhile, after the control points used for self-calibration are added, the triangulation quality increases to a similar level as in the triangulation results using well distributed eight GCPs. Although, overall triangulation quality decreases compared to the triangulation result using similar number of well distributed control points. The results show that around 0.5 meter horizontal RMSE and 1 meter vertical RMSE is achievable when three control points are available in the whole strip and 13 additional points are densely concentrated in one specific part of the strip. Likewise, the results do not show significant influence of remaining frames (frames where points are not available) on the self-calibration results. This may suggests that since the frames have large overlapping area in a small distance, the weak adjustment of remaining frames is compensated by adjacent existing control points.

According to the previous GCP experiments, a minimum of three well distributed GCPs with more than six additional GCPs only on limited area can be considered as the required minimum number of control points for self-calibration bundle adjustment process in whole strip. On the other hand, the results show still lower quality of overall triangulation results compared to when using control points well distributed over whole strip.

b. Experiment on distribution of control points

Besides evaluating the effect of number of control points, the distribution of control points is one of the important factors that affects the quality of bundle adjustment and thus should be considered. In order to check this effect, the whole strip is divided into 3 strips: right, left and center as shown in Figure 14. The GCP which are included in one strip are used as control points, while those falling in the other two strips are used as check points during the bundle adjustment.

Figure 14 GCP distributions on strips for GCP experiment on bundle adjustment. (GCP on right left and center strip)

As an additional experiment on GCP’s distribution, only those points which are located on the ground are selected and used as control points. This is done to check whether the geometry of control points affects the bundle adjustment results. Finally the results of each triangulation process using the mentioned GCP configurations are analyzed


45

.

Figure 15 Selection of GCP on ground for experiment on bundle adjustment

The effect of GCP distribution is evaluated by comparing the RMSE on check points which are derived from the bundle adjustment results using each GCP configuration. The results obtained are shown in Table 15. For the sake of comparison, the table also presents the residuals for the scenario considering 25 GCPs points during the number of control points experiment. It should be noticed that those 25 GCP’s follow a regular distribution on the strip.

Table 15 Check point RMSE with different GCP distribution in bundle adjustment

GCP distribution (number of control points)

RMSE on check point (m) RMSE on control point (m) rX rY rZ rX rY rZ

25 GCP 0.37 0.32 0.60 0.16 0.23 0.34 GCP on ground (27) 0.40 0.38 0.72 0.20 0.19 0.28 Center strip(29) 0.43 0.32 0.88 0.20 0.22 0.35 Right strip(22) 0.84 1.53 0.95 0.31 0.28 0.47 Left strip(11) 0.74 1.59 0.75 0.11 0.05 0.17

Figure 16 Check point RMSE with different GCP distribution in bundle adjustment

In Figure 16 it can be seen that when GCPs located on the right or left strip are selected for bundle adjustment, RMSE on check points is larger than the RMSE when the points in the central strip are used as control points. This implies that if the control points are laid on one side of strip, the stability

0.00

0.40

0.80

1.20

1.60

2.00

25 GCP GCP on ground (27) Center(29) Right(22) Left(11)

RMSE

(m)

rX rY rZ


46

of strip decrease and eventually overall RMSE of triangulation increase. Meanwhile, the influence of poor geometry of control points on the bundle adjustment is not significant. Only the small increase of RMSE is observed on the check points. Nevertheless, bundle adjustment quality is still worse than the achieved bundle adjustment quality using well distributed control points, even when the used number of control points is similar.

4.1.4.3. Feasibility of using tie points extracted by KLT algorithm Previous experiments used only the GCPs measured from the field or from the laser point data. Meanwhile, for this experiment, automatically extracted tie points are added for bundle adjustment; therefore an improvement of the bundle adjustment accuracy is expected.

Within highly overlapped area of video data, the features in the images are tracked automatically using KLT algorithm. Afterwards, tracked feature points are imported as tie points into LPS. Usually tie points assist bundle adjustment process by tying images based on common points and even increase adjustment results especially when only few known GCPs are available. Feasibility of using extracted points from KLT as a tie points is checked by triangulation results. This is done to verify if adding automatically extracted tie points increases the bundle adjustment quality when only few control points are available. Therefore, only 12 well distributed control points with 1208 tie points are used for tie point experiment. Other 73 GCPs are used as check points in order to check the quality of each triangulation result. However, as shown in Table 16, triangulation with automatically extracted tie points derive large RMSE on check points and estimated exterior orientation parameters. This evidences that the imported tie points contain miss tracked features which act as blunders during the triangulation process.

Table 16 RMSE on check point and estimated EOP with the project with or without tie points

GCP set up

RMSE on check points (m) RMSE on estimated EOPs (m) rX rY rZ rX rY rZ

12 GCP without tie 0.37 0.35 0.72 3.09 0.86 0.14 with tie (without robust estimation) 3.32 2.95 8.02 13.51 7.89 6.97 with tie (robust estimation) 1.17 0.60 1.74 0.43 3.81 0.89

In consequence, robust estimation is performed to detect wrong tie points and exclude them from the triangulation process. However, as shown in Table 16, triangulation result was still inconsistent even after blunder detection. It means that robust estimation did not work properly on blunder checking. Problems of tie points and robust estimation can be checked by analyzing the triangulation report. It can be seen that robust estimation excluded 6 control points as blunders from the input of 12 control points. Even residuals on excluded control points are estimated to be larger than 10 pixels. Since every control point is measured manually and around 1 pixel image point residuals are estimated (when triangulation was performed using same control points but without tie points) control points should not be recognized as blunders.


47

Figure 17 Example of blunder in tie point using KLT tracker. (Up: moving object; middle: small point movement; bottom: miss tracked point)

Figure 17 illustrates the example of detected wrong tie points such as moving objects or miss tracked points.

Robust estimation detected 179 blunder points. However some of those points are blunders and some of them are excluded but they are not actually blunders. Besides, there remain some blunders which are not detected by robust estimation. These problems are due to wrong robust estimation performance resulting in poor triangulation results. Since robust estimation is a mathematical process which only considers redundancy of input observations, the reason for poor results can be related to input data such as image observations of control points and tie points.

Since the GCP observations are not expected to influence the poor estimation, tie points should be verified as another possible source of error for poor robust estimation. It is obvious that blunders exist in automatically extracted tie points. However, when a large number of blunders exist in observations, it causes wrong calculation of weight on observations during robust estimation iteration procedure.


48

Robust estimation using modified least square weighting procedure performs reliable blunder detection with a total of blunders up to 10-20% of the observations. Since the study area is laid along the main street, many tracked points on moving objects are observed. 179 blunders, about 15% of total tie points, are detected by robust estimation. Nevertheless all those 179 points cannot be consider as blunders since robust estimation result is not reliable. It might be expected that at least 10% of the points are actual blunders, considering that some of the blunders were not excluded by robust estimation.

It is a time consuming work checking how many blunders are included in the total observation since robust estimation does not perform for automatic blunder detection. Thus, the residual of each tie point on the image is used as alternative way to detect blunders in observations instead of robust estimation. However this method can only be used effectively when residuals on the image points are calculated with correct exterior orientation parameters. Since the exterior orientation parameters in this research are approximated values, the actual parameters are estimated based on input observations. It means that exterior orientation values are always changing depending on input observations. Moreover, wrong EOPs can be estimated when the observations contain errors. Since image residuals are calculated based on EOP, the EOPs should be fixed with accurate values in order to check correct image residual.

Referring to the previous results in Table 13, 25 GCPs are applied as fixed EOP value. Since the triangulation result from previous experiment which only use GCPs yield acceptable accuracy, it can be assumed that the estimated EOP are correct. From the triangulation result using fixed EOP, the tie points which have more than 1 pixel RMSE on image measurement are selected and excluded as blunders. 601 tie points are excluded from 1208 imported tie points using this method. To check the quality of this method as alternative blunder detection method, triangulation is performed after exclusion of detected blunders.

Table 17 RMSE difference depend on blunder detection method

GCP set up Blunder checking RMSE on check point (m) rX rY rZ

12 GCP without tie 0.37 0.35 0.72 12GCPs+tie image residual 0.36 0.32 0.67 12 GCPs+ tie robust estimation 1.17 0.60 1.74

Table 17 compares the RMSE of check points derived from triangulation results with different blunder detection method. Compared to the triangulation result using only GCP points, RMSE values are large when blunders are detected by robust estimation since it does not perform efficiently. Meanwhile, when the blunders in tie points are excluded based on image point residuals, slight decrease of RMSE on check points is observed. As this result shows, this alternative blunder detection method is successfully performed. In contrast, eliminating all the points, which have more than one pixel residual, causes lost of many good tie points. Thus, this method is not applicable when only a limited number of tie points are available. Considering the practicability, effective blunder elimination should be further studied with different blunder detecting algorithms.


49

After elimination of blunder points in the previous experiment, the remaining points are used to evaluate triangulation results with different number of control points. The derived results are compared with the triangulation results using same number of control points but without tie points as shown in Table 18.

Table 18 Triangulation RMSE using different number of GCP with and without tie points

RMSE on check point (m)

(without tie points) RMSE on check point (m)

(after extract bad tie points) GCP set up rX rY rZ rX rY rZ) 50 GCP 0.39 0.34 0.58 0.39 0.33 0.60 25 GCP 0.37 0.32 0.60 0.37 0.31 0.60 12 GCP 0.37 0.35 0.72 0.36 0.32 0.67 8 GCP 0.39 0.49 0.81 0.40 0.41 0.83 6 GCP 0.70 0.64 0.96 0.69 0.41 0.92 4 GCP 1.00 0.67 0.98 1.09 0.47 1.06

This result shows that elimination of blunders using image point residuals performs efficiently as an alternative blunder detection method. Meanwhile, the increase of X and Y RMSE is observed from the triangulation using four GCP. Most of triangulation accuracy is improved after selected tie points are used. Thus the applied blunder detection strategies can be considered as sufficient methods.

Figure 18 RMSE on check point from the triangulation using with and without tie points (up-left: RMSE on X; up-right: RMSE on Y; down: RMSE on Z)

Figure 18 shows that the differences of triangulation results depend on the use of tie points. Referring to the RMSE on Y, improvement of triangulation result is shown especially when smaller number of

0.00

0.30

0.60

0.90

1.20


RM

SE

(m

)

rX (w ith tie) rX(w ithout tie)

0.00

0.30

0.60

0.90

1.20


RM

SE

(m

)

rY(w ith tie) rY(w ithout tie)

0.00

0.30

0.60

0.90

1.20


RM

SE

(m

)

rZ(w ith tie) rZ(without tie)


50

control points is used. On the other hand, RMSE on X and Z does not show large improvement and even larger RMSE is observed when four control points are used. Since only four control points are used, low quality of one or two GCPs or distribution of GCP affects the final result. Therefore improvement of the result can be expected by adjusting the used control points.

4.2. 3D building modeling Besides triangulation issue, 3D modeling is another photogrammetric product which has a wide range of applications. Therefore suitability of video data for 3D building modeling is assessed here as well. This chapter discusses the process of 3D building modeling with video data using ImageModeler (IM) software. As study area, a residential area which involves many houses is selected. The selected area is shown in Figure 19. Afterwards, several buildings are modeled in IM and the quality of models is assessed by comparing the geometric accuracy with laser point data.

Figure 19 Buildings in video imagery to be modeled (red box)

4.2.1. Input video image As a first step, the frames involved in the study area are selected. Each frame is selected considering 50 meters distance between two frames to obtain different angle of view. Forward and downward imagery is selected and used for modeling. Once sufficient frames are selected, they are imported into ImageModeler with initial camera information. Focal length 6.0 mm is applied as an initial camera parameter.

Forward view and downward view imageries are recorded using same video camera type. However camera parameters for each view should be defined separately since their camera configuration (such as focal length, principal points) is different. Therefore two camera systems are defined in IM. Each camera containing constant internal parameters is assigned to the downward and to the forward images respectively.


51

4.2.2. Image calibration

Figure 20 Screen capture of calibration on IM using survey points (blue box)

From the previous experiment within LPS, accurate calibration information is obtained such as focal length, principal point offset values, and distortion value. Using these values, more accurate calibration result can be obtained. However this was not done due to some limitations in IM. The software accepts only the focal length as camera information and considers that principal point is fixed. Therefore, the camera calibration is done within IM using input focal length and measured points.

Once the images are loaded into IM they are calibrated using measuring points common in those images and six GCPs measured as survey point. This can be seen in blue boxes as shown in Figure 20. When enough locators (10 locators foe this research) are measured, IM internally orients the image by calculating camera information such as focal length and distortions. Afterwards, eleven additional points are measured in order to improve calibration results. During the calibration process, the coordinate system is automatically defined based on the survey points. When the calibration result is acceptable and the coordinate system is defined, all the input images are oriented. At this point 3D information of the objects can be measured and modeled.

All the calibration process is done using the measured points on the image. However there are several problems when measuring points on exact corner of buildings. Among the problems faced for the positioning of those points we have:

− One flying direction without side overlap resulting in lack of different view angle

− Low resolution of video image

− Houses too close to each other


52

− Bottom of houses hidden by trees or other houses

These factors decrease the precision of point measurement. Also the accuracy of the stereo model decreases since the calibration process is based on image measurement. The expected accuracy from video data is calculated using the equation explained in chapter 3.1.3.

S� � S � 2756 � 10�� 12.9 � 10�� 0.59�m

S� � 2756 � 10�� 27550 � 12.9 � 10�� 3.25�m

Since the points are measured imprecisely, 12.9µm (equivalent to 1.5 pixels) is used as precision of point measurement. Therefore 0.59 / 3.25 meters are expected as horizontal and vertical accuracy. As described in section 4.1.4.2, theoretical accuracy is calculated only considering the stereo model, while IM establish the model using several images. Therefore expected accuracy is used as minimum expectation.

4.2.3. Assessment of obtainable accuracy Once scale and coordinate system is established, distance and height between two locators in the stereo model can be measured. Since the quality of the final building model depends on the accuracy of established stereo model, the quality of stereo model is checked before modeling the building.

Measurements between locators are used to assess the accuracy of the established stereo model. Usually, aerial imagery is used to determine accurate building outlines and length while laser points are the best for determining height information. However, the aerial video data used in this research has lower resolution (around 40 cm ground resolution) than laser point cloud, which has higher resolution (20 points per m2). Thus, laser point data provides higher accuracy for both building length and height determination in this research. In conclusion quality of stereo model derived from video data is assessed by comparing measured distance or height from model with the respective distance from laser point data.

Where,

A : height of the building roof

B: height of the building body

A+B : absolute height of the building

igure 21 Applied definition of height of building

The distance between two or more buildings or the length of building roof is used to assess horizontal accuracy of the stereo model. Besides, only the height of roof (“A” in igure 21) is measured in order to


53

assess the vertical accuracy of model since absolute height of building (A+B in igure 21) is difficult to measure due to occlusion of the ground corner of the building.

4.2.3.1. Planimetric distance / height measurement Distance and heights of the model are measured based on locators which are located on corresponding image points. In order to measure the distance, 44 additional locators are measured in the image while these points are independent from the points for calibration. 14 horizontal distances between two roof edges or the distance between two buildings and 10 heights of different building roofs are measured in the stereo model. Reference distance and height of buildings are measured by means of corresponding laser point coordinates.

Figure 22 Screen capture of distance measurement on ImageModeler and corresponding laser data

It is usually difficult to find correct edge point in laser point clouds. The provided laser data was already processed on building edge extraction. The edge of building roofs is extracted based on the intersection of segmented laser points. Since the edge of building roof is clearly defined by the edge extraction process, corresponding building edge coordinates can be easily selected and measured.

However processed laser data contains disconnected edges on top of the roofs as shown in Figure 23. It causes confusion on measuring correct height of building roof.

Figure 23 Example of disconnected roof edge due to laser point density


54

In cases such as the one presented in Figure 23, the point with the highest z value is selected, based on the assumption that the top of the roof is the highest point of roof. Since the height difference between disconnected points are within 0.1~0.2m, this assumption do not cause big problem on the model accuracy assessment.

4.2.3.2. Achieved accuracy When the model and reference measurements are obtained, measured distance and height is compared as shown in Table 19 and Table 20. The difference between two measurements derived from model and laser point data is used for stereo model accuracy assessment. From the obtained average of absolute difference, achievable accuracy of aerial video based 3D building model can be expected.

Table 19 Planimetric distance difference between stereo model and laser point data

Line IM distance (m) Laser distance (m) difference (m) 1 16.60 16.66 -0.06 2 6.19 5.88 0.31 3 7.74 7.69 0.05 4 10.27 10.77 -0.50 5 6.52 6.19 0.33 6 9.68 10.21 -0.53 7 23.95 23.52 0.42 8 12.03 11.85 0.18 9 16.02 15.23 0.79 10 10.25 10.18 0.06 11 16.04 15.21 0.82 12 11.23 10.73 0.50 13 11.53 11.28 0.25 14 14.07 13.38 0.69

RMSE on distance measurement (m) 0.39

Table 20 Height difference between stereo model and laser point data

Height height from IM (m) height from laser (m) difference (m) 1 4.56 4.92 0.36 2 4.44 4.20 -0.24 3 4.33 4.85 0.52 4 4.90 4.99 0.09 5 4.19 4.73 0.54 6 4.66 4.41 -0.25 7 4.59 5.01 0.42 8 4.22 4.60 0.38 9 4.10 4.37 0.27 10 4.68 5.02 0.33

RMSE on height measurement (m) 0.34

From the presented results it can be seen that 0.39 meter RMSE on horizontal measurement and 0.34 meter RMSE on vertical measurement has been achieved from video based stereo model. Horizontal accuracy achieved is similar with expected theoretical accuracy, being 0.2m smaller. Meanwhile


55

derived vertical accuracy is much higher than expectation. Considering theoretical expectation, derived RMSE is acceptable to reconstruct accurate 3D building model.

4.2.4. 3D building modelling Five rectangular shape buildings (red box in Figure 24) are selected to be modeled.

Figure 24 Selected buildings for 3D modeling

To obtain a 3D building model, the height of the building should be defined. However, the building edges on the ground are difficult to be measured due to occlusion of objects. Therefore, the roof of each building is modeled considering only the height of the roof, shown as “A” in igure 21.

4.2.4.1. Roof modelling Since the roof planes are reconstructed by connecting the points which were measured on each corner of building roofs, 29 points are measured on selected buildings in order to model roofs. Afterwards, roof planes are modeled based on the measured points on the image. Once the roof plane is modeled, texture of roof can be extracted using their corresponding image. Extracted texture is applied on corresponding roof plane as shown in Figure 25.


56

Figure 25 Modeled roof and applying extracted texture

Once the entire building roofs are modeled, simple outline of buildings are identifiable. However some complex details on roofs cannot be modeled due to the resolution of video data.

4.2.4.2. Roof model accuracy check The assessment on the accuracy of the modelled roof is done in a similar way as previous stereo model assessment. As shown in Figure 26, the modelled building roof contains four building edges, four corner points, and four heights.

Where,

L1-L4 : length of the roof edge

H1-H4 : height between top and corner of the roof

p1-p4 : roof corner point

Figure 26 Basic factors on building roof

Each element on modelled roofs are measured and compared with the measurement from laser points. Differences between two measurements, shown in Table 21, can be used to evaluate the absolute accuracy of modelled roof.

L1

L2

L3

L4

p1 p2

p4 p3

H1 H2

H3H4


57

Table 21 Difference between model measurement and laser point measurement

Distance difference (m) Height difference (m)

Building 1 L1 0.52 H1 0.09 L2 0.82 H2 0.59 L3 0.40 H3 0.78 L4 0.59 H4 0.39





Average RMSE 0.39 0.44

The derived horizontal and vertical accuracy of modelled roof is 0.39 meters and 0.44 meters, which refers to the average difference between the two measurements. Both values are acceptable within theoretical expectation limits. Although several measurements have more than 0.5 meters residual and even more than 1 meters difference (highlighted on table).

The bigger residuals are observed on points which were not located in the image precisely due to occlusion of object points such as hidden edges by trees or other buildings. In order to improve the quality of the model a simple cube is used as a primitive. By applying this method, the whole building model can be obtained as shown in Figure 27, while the absolute height of building is an approximation.


58

Figure 27 Improved model from the roof model to whole building model by applying primitive cube model

Afterwards, the texture of the roof is extracted from the image and applied on corresponding roof plane. Artificial colours are applied for texturing the building body since the building façade is occluded by other buildings.

As an example of the difference between two models, the corner of building 4 containing a poor point measurement on the occluded corner is shown Figure 28. The red line refers to the roof model based only on the measured image points and the light gray cube is the model applying cube geometry on three points to reconstruct the building.

Figure 28 Example of the improved roof model after applying cube geometry (red line: roof model based on measured image point; grey cube: building model applying cube model (example of building 4))

As shown in Figure 28, if the building roof is modelled only based on the image points containing a poorly measured point, one corner of building roof model is distorted. On the other hand, the roof modelled by connecting the roof top point and the primitive cube model is evenly modelled along the building body. In order to check whether this approach is sufficient to improve the quality of model,


59

the same building edges and the heights of modelled roofs are measured again. The measurement difference between modelled buildings and laser data is shown in Table 22.

Table 22 Measurement difference between building model applying primitive cube model and laser data

Distance difference (m) Height difference (m)






Average RMSE 0.34 0.36

From the presented results, an improvement on the overall accuracy is observed. Especially, the measurement H4 for building 4 (highlighted on the table) was previously showing more than 1 meter residual when the roof is modelled considering all the points. It explains that poor image measurement can be compensated by applying the constraint of parametric model such as cube. Therefore the quality of roof model is improved.


60


61

5. Discussion

This research demonstrated the potential of airborne video imagery for conventional photogrammetric application. Airborne video based triangulation and 3D building modeling has been successfully implemented within the ordinary photogrammetric workflow although additional efforts were required such as the estimation of camera parameters. As it will be indicated later, the results are still subject to improvement considering that some problems and limitations have not been solved yet. This chapter discusses on the main findings of this research and the limitations and the problems faced during its execution.

5.1. Discussion on results This section discusses the execution of this research in the light of the main findings. The discussion follow the course of the main procedures reported in this document: self-calibration, bundle adjustment for whole strip, and 3D modeling.

5.1.1. Self-calibration

Since a non metric camera does not provide reliable camera information, accurate focal length and principal point offset values are estimated by self-calibration. Self-calibration requires a large number of control points within rich geometry as explained in 2.2.4.2. However, it is difficult to obtain large redundancy of control points for whole strip as the study site is a residential area with public access restricted and some objects identifiable in the image are not clearly defined in the ground or do not even exist anymore due to the changes after data acquisition. Therefore only four frames which contain dense control points with rich geometry are selected for self-calibration.

As elimination of systematic distortion is critical to obtain accurate triangulation results, a distortion model was selected in order to remove systematic distortion before the estimation of correct focal length and principal point offset. As Table 3 shows, considering the check point RMSE derived from experiments using different AP models, Jacobsen’s AP model is the most effective model to remove the image distortion. Meanwhile, the other AP models considering different distortion factors derived worse results in all cases. The experiment carried out in section 4.1.2 demonstrates that the used video data presents mainly radial and affine distortions. If more than these distortion parameters are considered for image adjustment, worse bundle adjustment results are obtained due to over parameterization.

After the selection of the AP model, correct focal length and principal point offset are estimated along with distortion parameters using with different set-ups (section 4.1.2). The focal length and principal point offset values are estimated from the best fitting set up. This is done by means of evaluation of RMSE obtained for each tested set up. According to the RMSE, the selected IOP was considered as a reliable estimation to be used as known values. Besides estimating IOP values, quality of GPS/IMU data was checked from the result of same experiment. This was done by analyzing the triangulation results applying different standard deviation values for initial EOP. From this experiment it was observed that when smaller standard deviation values are allowed for initial EOP, triangulation


62

provided better accuracy. This result suggests that the used GPS/IMU data does not have significant difference with real EOP of video data; therefore it can be used as a reliable initial EOP value.

Once focal length and principal point offset values are estimated, they are simply imported into whole strip as known camera information. By applying estimated IOP as known value three unknown parameters are reduced from bundle adjustment process, thus less control points are required. On the other hand, estimated additional distortion parameters cannot be imported as known values. Referring to the experiments on applying calculated distortion value as known camera parameters (section 4.1.3), simply inputting the calculated distortion value is not sufficient. Since it is unknown how distortion parameters are internally used in LPS, the distortion parameters should be calculated during every triangulation process.

Another possibility to compensate image distortion is to obtain undistorted images by adjusting the images separately and using them as an input data for triangulation. By doing this, distortion parameters do not need to be calculated anymore during triangulation although it requires additional efforts on image processing. This method was not implemented during this research due to the allotted time.

Since the selected distortion model obtains only four unknown additional parameters, calculating distortion parameters during bundle adjustment does not cause the increasing of the required number of additional control points. However if the distortion model involves large number of unknown parameters and only limited number of GCPs are available, obtaining undistorted images is more appropriate than calculating distortion values during triangulation.

5.1.2. Bundle adjustment with whole strip In order to assess the achievable accuracy of the triangulation using airborne video data, several experiments were executed during this research (section 4.1.4). Achieved triangulation accuracy depends on input data such as control points, tie points, and initial GPS/IMU. Therefore the triangulation results using different inputs are compared each other and evaluated considering the theoretical expectation. From this experiment, achievable accuracy and minimum number of required control points were determined.

As a first experiment, efficiency of using approximate GPS/IMU data as an initial EOP is checked. For this research the GPS/IMU data for mapping camera mounted on same video recording platform is provided, although relative position of mapping camera and video camera is unknown. Since both camera systems are mounted on the same platform, provided GSP/IMU data can be used as initial EOPs to estimate correct EOPs. In the case any approximate EOP value is not available; LPS should calculate initial EOP by alternative means such as DLT. As shown in Table 10, triangulation with initial GPS/IMU was executed successfully. In contrast bundle adjustment process derived inconsistent results during execution without initial EOPs. The triangulation using 85 and 12 GCPs successfully run without GPS/IMU data while it complains for the lack of GCPs when 50 and 25 GCPs are used. The failure of execution of initial EOP estimation in LPS can be considered as a reason for this. Since the triangulation process succeeds with 12 GCPs while it failed when larger number of GCPs are used, the result from this experiment is not reliable. From this result, usability of approximate GPS/IMU data as initial EOP is tested; however, the effects of approximate GPS/IMU on the triangulation quality could not be verified due to technical problem of LPS.


63

Afterwards, triangulation is performed with different configuration of control points and the derived accuracy is compared with theoretical expectation. Since experiment without GPS/IMU did not yield reliable results, all the experiments are done only with initial GPS/IMU data (section 4.1.4.1). All the GCP experiments used only measured GCPs and not additional tie points are added.

The first GCP experiment is performed within four frames, using the same points as those used for self-calibration. This is done in order to measure the effect of GCP distribution on self-calibration results. According to Geosystem (Geosystems 2005), at least six control points within rich geometry are recommended for reliable self-calibration. Nevertheless as shown in Table 12, six control points are not sufficient to obtain accurate estimation even when they are located within rich geometry having large height variation. On the other hand, when GCPs located only on the ground are selected, self-calibration derives worse results although larger number of control points (nine control points) were used. This experiment shows that more than six control points within rich geometry is a critical set up to obtain reliable self-calibration results.

The overall achieved result (RMSE on check points) is shown in Figure 29. The red line in the graph refers the theoretical expectation.

Figure 29 Overall result from GCP experiments on whole strip (up: RMSE on X, Y; down: RMSE on Z)

After the experiment for self-calibration, GCP experiments are performed within whole strip in order to evaluate the required number of control points needed and the achievable triangulation accuracy

0

0.4

0.8

1.2

1.6

50 GCP 25 GCP 12 GCP 8 GCP 6 GCP 4 GCP Ground (27)

Center(29) Right(22) Left (11)

rX

rY

X,Y expect.

0.2

0.7

1.2

1.7

2.2

2.7

3.2

3.7

50 GCP 25 GCP 12 GCP 8 GCP 6 GCP 4 GCP Ground (27)

Center(29) Right(22) Left(11)

rZ

Z expec t.


64

from airborne video images. After executing several experiments with different number of GCP, a significant effect of the number of used control points was observed as shown in Figure 29. Theoretically, only four control points are required when focal length and principal point offset values are known. However, the results show that when less than eight control points are used, overall triangulation accuracy decreases. This result comes as no surprise since using more control points will produce higher accuracy in the triangulation result. Meanwhile it can be also concluded that at least eight control points are recommended to achieve reliable triangulation accuracy according to the theoretical expectation.

The previous two experiments focused on the required number of control points for self-calibration and bundle adjustment separately. However required control points for self-calibration within selected frames and bundle adjustment for whole strip should be considered together since non-metric video data requires both processes in one project. Therefore only a minimum number of three well distributed GCPs were selected for whole strip and 14 GCPs located in selected frames for self-calibration are added in order to perform bundle adjustment. The results reported in Table 14 show that when only three control points are available on the whole strip and a large number of control points for self-calibration are concentrated within selected frames, triangulation within whole strip can achieve similar accuracy as triangulation using eight GCPs.

Besides the experiments on number of control points, the effect of the distribution of control points is assessed as well (section 4.1.4.2). The Whole strip is divided into three parts (right, left, and center) and the GCPs included in each part are used as control points for each experiment. As an additional experiment, the effect of GCP geometry is considered as well by using GCPs located only on the ground. The results presented in Figure 29 show that the effect of GCP distribution is critical for triangulation results. If control points are located on only the one side of the strip, the other side of strip loses stability and this results in low quality triangulation. It was also found that the geometry of GCP does not affect significantly the triangulation results. This suggests that in order to obtain better triangulation results GCPs should be selected with careful consideration on the planimetric distribution of GCPs, while geometry of GCPs can receive less attention. However, since the use of rich geometry on GCPs is critical for self-calibration, as it was demonstrated by the previous experiment, geometry of GCP should be considered for proper GCP selection.

As a conclusion of the GCP experiments conducted, it can be observed that the selection of the GCPs is critical to obtain consistent triangulation results. From these experiments, using more than six control points for self-calibration and at least eight control points for whole strip bundle adjustment can be recommended. For the whole project, using three well distributed control points with additional GCPs for self-calibration is possible to obtain sufficient accuracy according to the theoretical expectation.

5.1.3. Feasibility of using tie points derived from KLT In order to achieve automation in airborne video based triangulation, the usability of tracked points from KLT tracker as tie points is investigated. KLT tracker automatically extracted features from the image sequence and provided more than 170 points for every frame. Extracted points are simply imported into LPS and used as tie points. By adding tie points into the block, an improvement of triangulation accuracy is expected. However, due to the presence of blunders in the tracked points, triangulation with tie points derived worse results than triangulation without tie points (section 4.1.4.3).


65

Since KLT tracker algorithm automatically extracts features based on a pixels window, several features are miss tracked due to similarity between windows, different behavior of pixels, occlusion problems, etc. Besides, many extracted points are observed on moving objects which produce blunders during adjustment process. In order to eliminate these points, robust estimation was selected as blunder detection method. However, robust estimation did not perform efficiently as a blunder detection method. The implemented robust estimation method in LPS detects blunders using weighting function based on observation redundancy. This method yields effective blunder detection when up to 10~20% of blunders exist within total observations (McClone et al. 2004). However, imported points contain many points on the moving object and several miss tracked points as well. Besides percentage of blunders, small movements of tracked points might cause poor estimation as well. Since images are selected within short time period (one second) with short base length, small difference between two video frames is difficult to be detected.

As the robust estimation cannot provide reliable blunder extraction, blunders were alternatively eliminated based on image point residuals. To estimate correct image residual, EOPs are fixed using estimated EOP derived from triangulation using 25 GCPs. Since the applied EOP values are estimated by triangulation they are not true values. However, considering the derived RMSE from triangulation, it was assumed as a reliable value to be used as fixed value. Using the estimated EOP, triangulation is performed with all the tie points. From this triangulation result, the points showing more than 1 pixel RMSE are considered as blunder and excluded for further triangulation process. The convenience of this method can be confirmed by the obtained check point RMSE. Afterwards, triangulation was performed using selected tie points and the GCPs used for GCP configuration experiments in order to check the effect of “good” tie points in triangulation result. Compared to the result of triangulation using only GCPs, an increase of overall triangulation quality is observed. Meanwhile the triangulation result with four GCP shows worse results. Since the good tie points are selected after blunder elimination, the reason of this result might be the quality of used control points or technical problem of LPS during the bundle adjustment process.

From the experiments in section 4.1.4.3, the improvement on the triangulation quality is observed after removing blunders based on image point residuals. However, since this method removes all the points which have more than 1 pixel RMSE, many good tie points are eliminated as well. Therefore this method is not sufficient to be used for practical applications. This suggests that more additional blunder detection methods should be tested in order to detect blunders efficiently.

5.1.4. 3D building modelling Building roofs are modeled from airborne video image using ImageModeler (section 4.2). The images are selected considering the distance between frames. For this research 50 meters distance between two images are considered in order to obtain different views. However, since only one flight direction is available, various angles of views are difficult to be acquired. Once the selected images are imported, corresponding cameras are calibrated based on measured tie points. Afterwards, the coordinate system is defined based on survey points and then all the images are oriented.

With the oriented images, 3D information of the objects in the images can be obtained. Before modeling the buildings, it is interesting to check the accuracy of the stereo model. Distance and height of stereo model was measured and compared with laser point measurements in order to evaluate it. Theoretically, the accuracy of 3D points of stereo model established in IM is comparable to the


66

derived accuracy from triangulation in LPS (Part I). Since IM is not a professional photogrammetric application, there are some limitations which restrict achievable accuracy such as fixed principal point. Therefore high accuracy of the model reconstructed by IM was not expected. However, similar horizontal RMSE and even smaller RMSE for vertical measurements were achieved compared with previous triangulation results. The reason for the higher accuracy on the vertical measurements can be most probably explained by the use of oblique images.

For modeling assessment, five buildings having similar shape and size are selected. Since the building corners on the ground are difficult to be measured and laser point data does not provide corresponding height of ground, only the roofs of the buildings are modeled. As Table 21 shows, overall horizontal and vertical RMSE is within 0.5 meters. However, from several measurements residuals between 0.5 and 1.13 meters are observed. These large measurement residuals are derived from poorly measured image points due to occlusion of objects and blurry edges. As a solution, those poor measurements are compensated by adding cube geometry model. The use of the cube geometry improved the overall accuracy as it is seen in Table 22. Although, caution should be taken as this method might decrease the accuracy of several measurements accuracy if the applied constraint does not fit with the actual shape of the buildings on the ground. By using primitive cube model, not only higher accuracy is obtained but also it is possible to reconstruct the whole building. Since accurate height of building is difficult to be obtained from the images, it is approximated by using adjacent ground points. The absolute height accuracy of obtained building model could not be assessed since reference building heights are not available. Therefore it should be answered in a further study.

5.2. Problems Due to technical problems with LPS software, experiments on effect of using initial GPS/IMU data and implementation of calculated distortion values could not be achieved.

− Effect of using initial GPS/IMU data

As accurate GPS/IMU data is not available in many practical situations due to their cost, this prevents from the implementations within low budget project (Yuan et al. 2004). However nowadays most of aircraft contains GPS system and this can be used for the approximate position of camera. This research used GPS/IMU data of mapping camera position as an initial EOP value of video camera. However comparison between the triangulation using and not using initial GPS/IMU position data has not been evaluated due to technical problems. The incidence of initial GPS/IMU data on increasing final triangulation accuracy should be evaluated in order to check efficiency of approximate GPS/IMU data used as initial value for further applications.

− Implementation of calculated distortion value

The use of non metric video camera for photogrammetric workflow requires calculation of additional unknown parameters such as interior orientation parameters and distortion parameters. Therefore high redundancy of ground control points is needed. This is rarely available in practical applications. Six EOPs (X0, Y0, Z0, ω, φ, κ), three IOPs (focal length, principal point offset x0, y0) and four additional distortion parameters (a1-a4) should have been estimated during self calibration bundle adjustment process. In order to reduce the number of control points needed for bundle block adjustment, IOPs and distortion parameters are estimated first by self-calibration and estimated values are planned to be used


67

as known value. Estimated IOPs are implemented successfully, however, distortion values calculated from Jacobsen’s simple equation could not be implemented as known camera value. Since the operations behind the software are unknown it would have been ideal to count with expert support from the LPS provider.

Due to technical problems with LPS software, experiments on usability of initial GPS/IMU data and implementation of calculated distortion values could not be achieved.

5.3. Limitations − Automation of blunder detection: feasibility of tie points from KLT tracking algorithm is one

of the objective of this research. KLT tracking algorithm provides sufficient number of tie points for whole frames. However, robust estimation did not perform efficiently for automatic blunder detection since distribution and number of blunders on imported tie points is unpredictable. Therefore blunders in tie points are eliminated manually considering image point residuals. By removing all the tie points with residuals larger than 1 pixel, the overall triangulation accuracy increased. However, good tie points are eliminated as well by this method. Only one robust estimation method was used for this research while there are more techniques for robust estimation which could have been further explored (McClone et al. 2004).

− Experiment on limited photogrammetric application: This research focused on the airborne video based triangulation which is one of the most basic and important photogrammetric applications. Although advanced photogrammetric applications such as DEM extraction, orthophoto generation has not been investigated. The triangulation of airborne video data has provided acceptable results. However, the successful implementation into conventional triangulation workflow is not enough to qualify the feasibility of airborne video data as photogrammetric source. Thus it is still subject to be tested on advanced photogrammetric applications.

− Limited size and shape of building model: the 3D modeling experiment has been done only with selected images which involve residential area. The accuracy of building model is fairly achieved considering theoretical expectation while theoretical accuracy has been estimated low. Since the test site is a dedicated residential area, most of buildings are somewhat simple, having rectangular and similar shape. Therefore, accuracy assessment of 3D building model could be only implemented in a case with rather simple geometry.

− Height assessment of building model: the height of building is approximately estimated due to difficulty of ground building edge measurement both in the images and in the laser reference data. The point on the ground near the modeled building is used to define height of building based on assumption that the buildings are located on same ground height. Applying this method a model of the whole building is produced. However, accuracy of whole building model could not be assessed due to lack of reference data. Since aerial laser point data does not provide absolute height of corresponding building ground, additional reference data is required to evaluate the height accuracy of modeled building.


69

6. Conclusions

This research aimed at evaluating the feasibility of airborne video data to be used as a photogrammetric source. Four research objectives are proposed related with video based photogrammetric procedures. In order to reach them, several experiments were done by evaluating research questions. This chapter answers the proposed research question based on the achieved results (section 6.1) and states final recommendations for further work (section 6.2).

6.1. Answer to research questions − How can lens distortion be addressed in video data?

Referring to the experiments (section 4.1.2), the distortions in the video images are handled by means of the self-calibration with additional parameters. Among the five implemented distortion model in LPS, Jacobsen’s Simple model is selected considering the derived RMSE obtained from triangulation experiments. The distortion vales are calculated both by manually and by BLUH based on Jacobsen’s distortion model. However, an effective method to apply camera distortion values as known parameters was not found. Therefore image distortions are compensated by calculating distortion parameters during bundle adjustment.

− What is the achievable accuracy of video triangulation? / How many ground control points (GCP) are needed for successful triangulation?

These two questions can be answered together since the number of GCP affects the accuracy of triangulation. According to the result from several experiments (section 4.1.4), around 0.5 meter horizontal RMSE and 0.6 - 0.8 meters vertical RMSE can be achieved. With the theoretically required of four control points, accuracies of 0.6-1.0 meter horizontal RMSE and 0.98 meter vertical RMSE are achieved. From this result, more than eight control points are recommended in order to obtain reliable accuracy. Meanwhile, considering the number of control points required to perform self-calibration, minimum three well distributed control points on whole strip with more than 10 dense control points on the selected frames derive 0.41-0.49 meter RMSE on X,Y direction and 1.05 meter RMSE on vertical direction.

− Can the state-of-art Kanade-Lucas-Tomasi (KLT) tracking algorithms extract enough tie points for triangulation?

KLT tracking algorithm extracts more than 170 tie points for every frame. These points provide high redundancy of input observations. However, the outliers existing in tracked points should be eliminated by means of additional blunder detection methods. Since robust estimation implemented in LPS did not detect the blunders effectively, blunders in tie points are manually removed considering the image residuals.

− Which accuracy can be achieved by using airborne video images for 3D building modeling?

Five building roofs are modeled and each measurement on the model is compared with the measurement from laser point data. The roof models reconstructed by using only the measured image


70

points derived 0.39 meter horizontal RMSE and 0.44m vertical RMSE. In order to improve the quality of roof models, primitive cube model is used. By applying the primitive cube model, higher overall accuracy of roof models is obtained, 0.34 meter horizontal RMSE, and 0.36 meter vertical RMSE is achieved.

6.2. Recommendations Considering the problems and limitations (section 5.2 and section 5.3) of this research, some recommendations for further work are

− Further studies should focus on evaluating the efficiency of use of approximate GPS/IMU position data as initial exterior orientation parameters for triangulation.

− There is a need for improvement automatic blunder detection method in the case an unpredictable number of blunders exists in the observations.

− Further studies should focus on feasibility of airborne video data for advanced photogrammetric applications.

− As this model investigates 3D modeling on simple geometry buildings, forthcoming studies should address more complex building structures.


71

Appendix 1. Self-calibration result from BLUH PROGRAM DISTOP UNIVERSITY OF HANNOVER JAN 2005 =================================================== PLOT OF DISTORTION FUNCTION DETERMINED BY BLUH INSTITUTE FOR PHOTOGRAMMETRY AND ENGINEERING SURVEYS DATE: 21.11.2007 15:59:28 ================================== DISTORTION INFORMATION FROM FILE bluh.lst FOCAL LENGTH IN BLUH 6.300 .000 .000 .000 .000 .000 .000 .000 RADIAL SYMMETRIC DISTORTION FROM BLUH RADIUS .25 .50 .75 1.00 1.25 1.50 1.75 2.00 MM DISTORTION - 8.8 - 17.6 - 24.9 - 29.9 - 33.0 - 33.9 - 31.5 - 25.1 MICRONS RADIUS 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00 MM DISTORTION - 16.3 - 7.8 .5 12.3 32.9 64.0 103.5 148.5 MICRONS RADIUS 4.25 4.50 4.75 5.00 5.25 5.50 5.75 6.00 MM DISTORTION 199.5 259.3 328.3 401.9 475.0 548.7 631.3 731.6 MICRONS RADIUS 6.25 6.50 6.75 7.00 7.25 7.50 7.75 8.00 MM DISTORTION 851.4 985.2 1128.0 1280.8 1448.1 1629.5 1817.3 2004.3 MICRONS COMPENSATION BY K1 - K3 +/ - 1.31 K1 = - .932522E+04 K2 = .796205E+09 K3 = - .333575E+14 LINEAR .39086E- 01 CORRESPONDS TO CHANGE OF FOCAL LENGTH FROM 6.300 TO 6.546 PLOT INFORMATION STORED IN blplot.dat END OF PROGRAM DISTOP


72

Systematic distortion in video image calculated by BLUH


73

Reference Ackermann, F. (1980). Blockadjustment with additional parameters(photogrammetric image correction). International archives of photogrammetry: 1-10.

Bongbae Jang, K. G., WungGi Min, Jungkwan Choi, Il Kim,Sangu Kang, Sungkyu Park (2004). 포인트포인트포인트포인트 지적기술사지적기술사지적기술사지적기술사, 예문사예문사예문사예문사.

Bourel, F., C. C. Chibelushi and A. A. Low (2000). Robust facial feature tracking. Proceedings of the 11th British Machine Vision Conference 1: 232-241.

Brenner, C. (2005). Building reconstruction from images and laser scanning. International Journal of Applied Earth Observations and Geoinformation 6(3-4): 187-198.

Brown, D. C. (1971). Close-range camera calibration. Photogrammetric Engineering 37(8): 855-866.

Cardenal, J., J. Delgado, E. Mata, A. Gonzalez and I. Olague (2006). Use of historical flight for landslide monitoring. 7th International Symposium on Spatial Accuracy Assessment in Natural Resources and Environmental Science, Lisbon, Portugal, Instituto Geogrfico Portugues.

Cramer, M. (2001). On the use of direct georeferencing in airborne photogrammetry. Proceedings of III rdInternational Symposium on Mobile Mapping Technology, Cairo.

Cramer, M. (2006). The ADS40 Vaihingen/Enz geometric performance test. ISPRS Journal of Photogrammetry and Remote Sensing 60(6): 363-374.

Cramer, M. and D. Stallmann (2002). System calibration for direct georeferencing. Photogrammetric Computer Vision, ISPRS Commission III Symposium, Graz, Austria.

Ebner, H. (1976). Self calibrating block adjustment. Bildmessung und Luftbildwesen 44: 128-139.

Faugeras, O., L. Robert, S. Laveau, G. Csurka, C. Zeller, C. Gauclin and I. Zoghlami (1998). 3-D reconstruction of urban scenes from image sequences. COMPUT VISION IMAGE UNDERSANDING 69(3): 292-309.

Frank A.van den Heuvel (1999). Estimation of interior orientation parameters from constraints on line measurements in a single image. International Archives of Photogrammetry and Remote Sensing 32: 81-88.

Fraser, C. S. (1997). Digital camera self-calibration. ISPRS Journal of Photogrammetry and Remote Sensing 52(4): 149-159.

Fryer, J. G. and D. C. Brown (1986). Lens distortion for close-range photogrammetry. Photogrammetric engineering and remote sensing 52: 51-58.

Ganci, G. and H. Handley (1998). Automation in videogrammetry. International Archives of Photogrammetry and Remote Sensing 32(5): 53-58.

Geosystems, L. (2005). Leica photogrammetry suite project manager user guide, Leica Geosystems Geospatial Imaging, LLC, Norcross, USA.


74

Ha, J. E. and D. J. Kang (2005). Initialization method for self-calibration using 2-views. Pattern Recognition 38(1): 143-150.

Jacobsen, K. (1997). Operational block adjustment without control points. ASPRS Annual Convention, Seattle, USA.

Jacobsen, K. (2001). Exterior orientation parameters. Photogrammetric engineering and remote sensing 67: 12-47.

Jacobsen, K. (2004). Issues and methods for in-flight and on-orbit calibration. ISPRS Book Series. ISPRS, Balkema Publishers. 2: 83-92.

Jacobsen, K. (2007). Geometric handling of large size digital airborne frame camera images. Optical 3D Measurement Techniques, Zurich.

Jacobsen, K. and H. Wegmann (2001). Dependencies and problems of direct sensor orientation. OEEPE Workshop, Integrated Sensor Orientation, Hannover, Germany, Sept: 17-18.

Karara, H. M. e. (1989). Non - topographic photogrammetry. Falls Church, American Society for Photogrammetry and Remote Sensing.

Lucas, B. D. and T. Kanade (1981). An iterative image registration technique with an application to stereo vision. Proc. DARPA Image Understanding Workshop: 121-130.

Luhmann, T. (2006). Close range photogrammetry : principles, techniques and applications. Caithness, Whittles.

Maas, H.-G. (1999). Image sequence based automatic multi-camera system calibration techniques. ISPRS Journal of Photogrammetry and Remote Sensing 54(5-6): 352-359.

McClone, J. C. e., E. M. e. Mikhail and J. S. e. Bethel (2004). Manual of photogrammetry. Bethesda, American Society for Photogrammetry and Remote Sensing (ASPRS).

Mikhail, E. M., J. S. Bethel and J. C. McGlone (2001). Introduction to modern photogrammetry. New York etc., Wiley & Sons.

Neale, W. T. C. (2004). A video tracking photogrammetry technique to survey roadways for accident reconstruction. SAE SP: 223-234.

Pollefeys, M., L. V. Gool, M. Vergauwen, F. Verbiest, K. Cornelis and J. Tops (2002). Video-to-3D. Proceedings of the Symposium on Photogrammetric Computer Vision, vol. A: 252-257.

Pollefeys, M., R. Koch, M. Vergauwen and L. Van Gool (2000). Automated reconstruction of 3D scenes from sequences of images. ISPRS Journal of Photogrammetry and Remote Sensing 55(4): 251-267.

Pollefeys, M., L. Van Gool, M. Vergauwen, F. Verbiest, K. Cornelis, J. Tops and R. Koch (2004). Visual modeling with a hand-held camera. International Journal of Computer Vision 59(3): 207-232.

Remondino, F. and C. Fraser (2006). Digital camera calibration methods: considerations and comparisons. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 36(5): 266-272.

Ruzgiene, B. and W. F?stner (2005). RANSAC for outlier detection. Geodesy and Cartography 31(3): 528.73.


75

Shi, J. and C. Tomasi (1994). Good features to track. Computer Vision and Pattern Recognition, 1994. Proceedings CVPR'94., 1994 IEEE Computer Society Conference on: 593-600.

Sugaya, Y. and K. Kanatani (2005). Extracting moving objects from a moving camera video sequence. Mem. Fac. Eng. Okayama Univ 39: 56?2.

Tomasi, C. and T. Kanade (1991). Detection and tracking of point features. School Comput. Sci., Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-CS-91?32.

Vosselman, G. (2008). Analysis of planimetric accuracy of airborne laser scanning surveys. XXI. ISPRS congress,Beijing.

Walstra, J. (2006). Historical Aerial Photographs and Digital Photogrammetry for Landslide Assessment, Loughborough University. Doctor: 247.

Wegmann, H., C. Heipke, K. Jacobsen and I. Wegmann (2004). Direct sensor orientation based on GPS network solutions. International Archives of Photogrammetry and Remote Sensing 35(B1): 153-158.

Wiedemann, A., J. Moré, T. Suthau, I. Theodoropoulou, U. Weferling and B. Ergün (2001). Comparison of bundle block adjustments for close range applications. Fourth Turkish-German Joint Geodetic Days. Paper presented to the conference organized at Berlin, Germany, April: 2-6.

Wolf, P. R. and B. A. Dewitt (2000). Elements of photogrammetry : with applications in GIS. Boston etc., McGraw-Hill.

Yuan, X. X., C. Xie and S. G. Wang (2004). Image Orientation by Combined Bundle Adjustment with Fixed Imageries. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 35(B2): 34-38.

Suitability of airborne video data for photogrammetry

Documents

Transcript of Suitability of airborne video data for photogrammetry