Plant and Canopy Reconstruction

User Documentation The University of Nottingham

2014

1

Table of Contents

Overview ............................................................................................................................. 2

Introduction ..................................................................................................................... 2

Program Input ..................................................................................................................... 3

Point Clouds ..................................................................................................................... 3

Image Sets and Camera Geometries ........................................................................... 5

Directory Structure .......................................................................................................... 7

Executing the Tool .............................................................................................................. 8

References ........................................................................................................................ 11

Contact ............................................................................................................................. 11

2

Overview

Introduction

The Centre for Plant Integrative Biology’s 3D plant reconstruction tool is a command-

line based windows program that can be used to reconstruct surface information in

plant canopies.

The input of the tool is a 3D point cloud, combined with an input image set and

camera geometries. The output is a completed plant mesh in the standard PLY

format that can be exported into modelling or graphical applications.

The tool is written in C# .NET, and makes extensive use of the .NET libraries. For this

reason a modern Windows machine is essential. A version of this software for other

operating systems is under consideration, but is unlikely for the foreseeable future.

Due to the complexity of the data structures used within our software, it is likely that

the computer running the reconstruction will require at least 4Gb of available RAM.

We would recommend at least 8Gb of available RAM for most reconstructions.

3

Program Input

Out tool reconstructs plant mesh data from an existing point cloud. Point clouds can

be obtained through a variety of means, such as with laser scanning, however in our

experiments we have made use of the algorithms presented in [1] and the

associated tool PMVS. Along with the point cloud, our software requires a set of

images of the plant, or plants, to be reconstructed. The image set must be

calibrated, in other words the 3D world position of each camera must be known, or

have been calculated before reconstruction can begin. Details on this process can

be found below.

With all data available, files must be placed in a single location, using a consistent

and specific directory structure, also described below. This approach avoids the

requirement of manually specifying the location of each file before reconstruction

can begin.

Point Clouds

A point cloud is simply a list of 3-dimensional points that comprise our input data.

Various file formats exist to store point clouds, and three formats can be read by the

reconstruction tool. Point cloud data can be read in either straight text format, the

Stanford PLY format or the PMVS Patch format. The file format used will often

depend on the means by which the point cloud was produced, however the

choice of format will not affect the operation of our reconstruction tool. Figure 1

provides a sample of each of the three file formats that can be input into the

reconstruction software. PLY and Patch files are identified by their header

information, the text format should contain no header information. The text format

should contain only the X, Y and Z co-ordinates of each point, with spaces used to

separate each value. Each point should be output on a single line. The PLY and

Patch formats are more verbose, with options to include additional information, such

as the normal direction of a point, and colour. This information can be included, but

will be ignored by our software as it is not used during reconstruction.

During our experiments we have used the PMVS software to construct the initial point

clouds. This software outputs clouds as both a PLY and Patch file, they are usually

located in the models directory of the PMVS reconstruction. Either is suitable as a

starting point for reconstruction using our tool.

An optional file named clip.txt can be included inside the patches directory, and will

be read if the --plane-filter option is set when the program is run. This file simply

contains six numbers, X Y Z U V W, where X Y Z is a point on the clip plane, and U V W

is the normal orientation. If the plane filter is applied, any points below the plane will

be removed.

4

Text Format

0.031054 -0.055910 0.525988

0.033435 -0.057221 0.529514

0.028908 -0.054901 0.519939

0.027287 -0.053935 0.518550

0.027287 -0.053935 0.518550

…

The recognised text format contains

no header. Each line contains a

single point, given as X Y Z

separated by spaces

PLY Format

ply

format ascii 1.0

element vertex 116736

property float x

property float y

property float z

end_header

0.031054 -0.055910 0.525988

0.033435 -0.057221 0.529514

0.028908 -0.054901 0.519939

…

A PLY file contains a header, found

between the ply and end_header

lines. After this a number of points

are listed, one on each line. The

number of points must equal the

value given by element vertex for

the file to be read successfully.

Patch Format

PATCHES

112472

PATCHS

0.289144 -0.369772 1.77477 1

0.442019 -0.818687 -0.366567 0

…

The patch file is output by the PMVS

point cloud reconstruction software.

The file begins with a header

PATCHES, and a value representing

how many points are included in this

file. The first line after each PATCHS

sub-heading gives the position of

each point, other information is

included but is ignored.

clip.txt

0.28 -0.36 1.77 0 1 0

The optional clip file gives the

position and orientation of a plane

that is used to filter points. This is

useful for removing large areas of

background, such as points that

have been reconstructed on the

floor below the plant. These points

will likely be removed by the colour

filter, but the plane filter is faster and

more precise in some cases.

Figure 1: Examples showing the three available file formats for point cloud

input.

5

Image Sets and Camera Geometries

Our tool requires images of the captured scene, taken from a variety of angles. The

number of images required will depend on the scene, but in general the greater the

number of images, the better the reconstruction is likely to be. This is because

reconstruction of each small section of a leaf surface is best when at least one

image has a good “view” of that leaf. To be specific, we define a good view as

taken perpendicular to the leaf surface, and not obscured by other leaves or

objects. In our experiments we use more than twenty images per set, however we

have experimented with as many as 70. We would encourage users to test their

image capture setup and ascertain the optimum number of images for their

experiment. Note that the greater the number of images, the greater the memory

and computational requirements of the tool. The majority of allocated memory is

given to image and related data, thus doubling the number of input images is likely

to double the memory requirements of the tool.

In order to use the 2D images within the reconstruction process, our software must

have access to the geometric information for each camera that was used to

capture each image. This can be obtained through camera calibration, when used

with a static image capture system, or through so-called structure from motion

algorithms that will operate on arbitrarily placed images.

If automated calibration is required, we recommend the use of the VisualSFM [2]

system for automated camera calibration. This software also utilises PMVS for point

cloud reconstruction, so can produce all necessary input files for the reconstruction

tool. Users are encouraged to read the documentation supplied with these external

tools for more information.

Output of VisualSFM is provided in numerous files, our tool reads the detailed data

file named cameras_v2.txt, which can be found in the root directory of each

VisualSFM reconstruction. This file can be renamed as required, however the

contents mut remain unchanged. Where VisualSFM is not the calibration tool used,

our tool can read standard text files containing the camera projection matrix and

the camera normal (direction of the view). Figure 2 shows examples of these file

formats.

Please note that our software requires camera data associated with each image,

thus the number and order of cameras listed in cameras_v2.txt should match the

number and order of images found in the image directory. Alternatively, the number

of individual files in text format should match the number of images provided.

6

Text Format

3783.965087 1718.976806 33.28966 4787.862304

2.817852 1224.604003 -3757.105468 7840.200195

0.002300 0.999811 0.019264 2.815705

The directory should contain multiple numerically ordered files, one for each

camera. Each file contains three lines for the 3x4 projection matrix for this camera.

Details of the derivation of this matrix can be found in our paper, published

alongside this tool.

VisualSFM Format

40

00000000.jpg

C:\Directory\images\0004.jpg

3118.22875977

1728 1152

0.0832426307487 -0.815510288973 -0.746382757736

0.772396707068 0.793051025891 0.0595472036664

0.0467306567339 -1.49557456378 0.107119690748

0.731643253292 0.021235248586 -0.679615906568

…

The VisualSFM camera format contains a number identifying the camera count,

followed by a list of parameters for each camera. The parameters included

contain all the information necessary to reconstruct the projection matrix. All

camera geometry is contained in a single text file, the name is unimportant.

Figure 2: Examples showing the two possible file formats for inputting camera

geometry into our tool.

7

Directory Structure

Our tool requires a specific directory structure in which input files are stored, with

each folder stored in a working directory that is specified when the program

executes. Figure 3 provides an overview of this directory structure.

Figure 3: Example of the file structure expected by the tool, containing a

working directory and three sub-directories.

The working directory can have any name, and is specified when execution begins.

Below this are three mandetory directories, named patches, images and cameras.

The patches folder contains the point cloud data associated with this reconstruction.

Usually this will be a single file, but multiple files can be read if the point cloud is

stored in this way. The images folder contains each of the captured images. Specific

names for each image are not required, but images are loaded in alphabetical or

numerical order. Finally, the cameras folder contains the calibration data for each

camera, either in a series of files, or a single VisualSFM file as discussed above. As

with the image folder, each camera is loaded in order. This means that the ordering

of both images and cameras must be consistent to match each image to the

geometry.

8

Executing the Tool

The tool can be executed from the command line, with additional parameters

providing the information required to direct the reconstruction. Users may find it

easier to store the entire command in a Windows batch file, if it is to be used or

adjusted multiple times.

To run the tool with all default settings, the program can be executed as follows:

reconstructor.exe path-to-working-directory

Most parameters used in the tool, such as the radius to use for point segmentation,

provide default values that will work on many datasets. However, it is recommended

that these values be customised for a given image capture setup to ensure

optimum reconstruction. Options are given after the working directory, when the

program is run. For example:

reconstructor.exe C:\Reconstruction\ --min-cluster-size 30

Brief details of each optional parameter are given with the command:

reconstructor.exe --help

Each option can be given as a full command, preceded by a double dash, --. Some

options that don’t require values can be shortened to a single letter preceded by a

dash. A detailed description of each option is provided in Table 1.

The optional flags, also shown in table 1, do not require values, it is assumed that a

specific flag is true if passed as a parameter. For example:

reconstructor.exe C:\Reconstruction\ -fs

This command includes both -f -s, and will instruct the software to output both a

filtered cloud and a segmented cloud before resuming the reconstruction process.

Where such options are chosen, output files will be saved in the working directory,

under the sub-directory output. The final output of the reconstruction software will be

saved as working-directory/output/triangulation.ply in the PLY file

format.

9

Option Expected values Default Description

--camera-type Either of:

VSFM

PR

VSFM Indicates which of the two accepted

formats will be used for the camera

calibration data

--segmentation-radius A positive real number 0.01 The distance between points below

which they will be considered for the

same cluster. This value will depend on

the scale of the point cloud

--min-cluster-size Any integer greater

than zero

10 The minimum number of points

allowed in a single segmented cluster.

Points in clusters smaller than this will

be discarded

--max-cluster size An integer greater than

the minimum cluster size

60 The maximum number of points

allowed in a single cluster. Points

above this value will be split into other

clusters

--alpha radius A positive real number 0.01 The alpha value used when creating

alpha-shape surface estimates. This

number should usually be similar to the

segmentation radius, as both are

representative of the expected

distance between points on the same

surface

--level-set-iterations Any positive integer 200 The number of level set iterations to

run

--halting-percentage Any real number zero or

greater

0.0 A value indicating when level sets

should halt due to inactivity. If a level

set changes size by less than the

indicated percentage, it will stop

iterating.

--zbuffer-resync-frequency Any integer greater

than zero

20 How many level set iterations to run

between resynchronising the z buffer

data structures

--terminate-after One of:

filtering segmentation

surface-estimation

all

all Indicates that processing should stop

before the full reconstruction is

complete. This is useful for testing

earlier stages of the reconstruction,

such as segmentation. This should be

used with the -f, -s, and -a flags to view

output at the appropriate stage

-p, --plane-filter - - Indicates whether to apply a planar

clipping line to the points before other

processing. The position and

orientation of this plane must be

supplied in clip.txt within the patches

folder

-c, --colour-filter - - Indicates whether to apply a green-

based colour filter to remove non

plant points before surface

reconstruction

-f, --output-filtered-points - - Indicates whether to output the plane

and colour filtered points. The points

are saved in ply file format in the

software output folder, as filtered.points.ply

10

-s, --output-segmented-points - - Indicates whether to output the

segmented point cloud, with vertices

coloured based on the cluster they

are in. The points are saved in ply file

format in the software output folder,

as segmented.points.ply

-a, --output-alpha-triangulation - - Indicates whether to output the initial

surface reconstruction based on

alpha shapes. The mesh is saved in ply

file format in the software output

folder, as alpha.triangulation.ply

Table 1: Details of all optional parameters and flags that can be passed as

command line arguments to reconstructor.exe.

The following example shows the full command line execution of the software, using

a number of the optional parameters. Those parameters that are left at the default

values are not required.

reconstructor.exe C:\Reconstructions\Test\ -pcfs --camera-type PR --

max-cluster-size 140 --segmentation-radius 0.05 --level-set-

iterations 100

These options indicate that both the plane and colour filters should be applied, and

that the filtered and segmented point clouds should be output. The camera type is

set to NP, that is, the normal and projection matrices stored in separate files for each

camera. The maximum cluster size is increased to 140, and the segmentation radius

is increased to 0.05. Finally, the number of level set iterations is decreased from the

default value of 200, to 100.

As the working directory has been set as C:\Reconstructions\Test\, the program will

look for the necessary files within:

C:\Reconstructions\Test\patches\

C:\Reconstructions\Test\images\

C:\Reconstructions\Test\cameras\

All output files will be saved in:

C:\Reconstructions\Test\output\

11

Once execution begins, the console window will provide information on the progress of the

reconstruction. Depending on the stage of reconstruction being processed, it will appear much like

Figure 4.

Reading camera parameters: Done

Loading images: Done 40

Reading patches: Done

Colour Filter: Done

Constructing Search Tree: Done

Clustering patches: Done

Flattening Clusters: Done

Triangulating Clusters: Done

Calculating Cluster Visibility: Done

Calculating Distance Maps: Done

Converting RGB images to NG: Done

Syncing Z Buffers: Done

Analysing Cluster Histograms: Done

Running Level Sets...

Iteration 2

Figure 4: Expected output of the reconstruction tool

References

[1] Furukawa, Yasutaka and Ponce, Jean. Accurate, Dense, and Robust Multi-View

Stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32,

Issue 8, Pages 1362-1376, 2010.

PMVS software available at http://www.di.ens.fr/pmvs/

[2] Wu, Changchang. VisualSFM: A visual structure from motion system. 2011.

VisualSFM software available at http://ccwu.me/vsfm/

Contact

Details on the tool and its development can be found on the Centre for Plant

Integrative Biology website at www.cpib.ac.uk.