SIGNet: Semantic Instance Aided Unsupervised 3D Geometry Perception
Yue Meng1 Yongxi Lu1 Aman Raj1 Samuel Sunarjo1 Rui Guo2
Tara Javidi1 Gaurav Bansal2 Dinesh Bharadia1
1UC San Diego 2Toyota InfoTechnology Center
{yum107, yol070, amraj, ssunarjo, tjavidi, dineshb}@ucsd.edu
[email protected] [email protected]
Abstract
Unsupervised learning for geometric perception (depth,
optical flow, etc.) is of great interest to autonomous sys-
tems. Recent works on unsupervised learning have made
considerable progress on perceiving geometry; however,
they usually ignore the coherence of objects and perform
poorly under scenarios with dark and noisy environments.
In contrast, supervised learning algorithms, which are ro-
bust, require large labeled geometric dataset. This paper
introduces SIGNet, a novel framework that provides ro-
bust geometry perception without requiring geometrically
informative labels. Specifically, SIGNet integrates seman-
tic information to make depth and flow predictions con-
sistent with objects and robust to low lighting conditions.
SIGNet is shown to improve upon the state-of-the-art unsu-
pervised learning for depth prediction by 30% (in squared
relative error). In particular, SIGNet improves the dynamic
object class performance by 39% in depth prediction and
29% in flow prediction. Our code will be made available at
https://github.com/mengyuest/SIGNet
1. Introduction
Visual perception of 3D scene geometry using a monoc-
ular camera is a fundamental problem with numerous appli-
cations, like autonomous driving and space exploration. We
focus on the ability to infer accurate geometry (depth and
flow) of static and moving objects in a 3D scene. Supervised
deep learning models have been proposed for geometry pre-
dictions, yielding “robust” and favorable results against the
traditional approaches (SfM) [38, 39, 10, 2, 1, 26]. How-
ever, supervised models require a dataset labeled with ge-
ometrically informative annotations, which is extremely
challenging as the collection of geometrically annotated
ground truth (e.g. depth, flow) requires expensive equip-
ment (e.g. LIDAR) and careful calibration procedures.
Ours State-of-the-art unsupervised
Xs
Xu
Xt
Ys
Yt
Yu
ft → u
Fram
e s
Fram
e t
Fram
e u
ft → s
ft → u
ft → s
Figure 1: On the right, state-of-the-art unsupervised learn-
ing approach relies on pixel-wise information only, while
SIGNet on the left utilizes the semantic information to en-
code the spatial constraints hence further enhances the ge-
ometry prediction.
Recent works combine the geometric-based SfM meth-
ods with end-to-end unsupervised trainable deep models
to utilize abundantly available unlabeled monocular cam-
era data. In [54, 41, 51, 9] deep models predict depth
and flow per pixel simultaneously from a short sequence of
images and typically use photo-metric reconstruction loss
of a target scene from neighboring scenes as the surro-
gate task. However, these solutions often fail when dealing
with dynamic objects1. Furthermore, the prediction quality
is negatively affected by the imperfections like Lambertian
reflectance and varying intensity which occur in the real
world. In short, no robust solution is known.
In Fig 1, we highlight the innovation of our system (on
the left) comparing to the existing unsupervised frameworks
(on the right) for geometry perception. Traditional unsu-
pervised models learn from the pixel-level feedback (i.e.
1Section 5 presents empirical results that explicitly illustrate this short-
coming of state-of-the-art unsupervised approaches.
9810
photo-metric reconstruction loss), whereas SIGNet relies on
the key observation that inherent spatial constraints exist in
the visual perception problem as shown in Fig 1. Specifi-
cally, we exploit the fact that pixels belonging to the same
object have additional constraints for the depth and flow
prediction.
How can those spatial constraints of the pixels be en-
coded? We leverage the semantic information as seen in
Fig 1 for unsupervised frameworks. Intuitively, seman-
tic information can be interpreted as defining boundaries
around a group of pixels whose geometry is closely related.
The knowledge of semantic information between different
segments of a scene could allow us to easily learn which
pixels are correlated, while the object edges could imply
sharp depth transition. Furthermore, note that this learn-
ing paradigm is practical 2 as annotations for semantic pre-
diction tasks such as semantic segmentation are relatively
cheaper and easier to acquire. To the best of our knowl-
edge, our work is the first to utilize semantic information in
the context of unsupervised learning for geometry percep-
tion.
A natural question is how do we combine semantic in-
formation with an unsupervised geometric prediction? Our
approach to combine the semantic information with RGB
input is two-fold: First, we propose a novel way to augment
RGB images with semantic information. Second, we pro-
pose new loss functions, architecture, and training method.
The two-fold approach precisely accounts for spatial con-
straints in making geometric predictions:
Feature Augmentation: We concatenate the RGB input
data with both per-pixel class predictions and instance-level
predictions. We use per pixel class predictions to define se-
mantic mask which serves as a guidance signal that eases
unsupervised geometric predictions. Moreover, we use the
instance-level prediction and split them into two inputs, in-
stance edges and object masks. Instance edges and object
masks enable the network to learn the object edges and
sharp depth transitions.
Loss Function Augmentation: Second, we augment the
loss function to include various semantic losses, which re-
duces the reliance on semantic features in the evaluation
phase. This is crucial when the environment contains less
common contextual elements (like in dessert navigation or
mining exploitation). We design and experiment with var-
ious semantic losses, such as semantic warp loss, masked
reconstruction loss, and semantic-aware edge smoothness
loss. However, manually designing a loss term which
can improve the performance over the feature augmenta-
tion technique turns out to be very difficult. The chal-
2Semantic labels can be easily curated on demand on unlabeled data.
On the contrary, geometrically informative labels such as flow and depth
require additional sensors and careful annotation at the data collection
stage.
lenge comes from the lack of understanding of error dis-
tributions because we are generally biased towards simple,
interpretable loss functions that can be sub-optimal in un-
supervised learning. Hence, we propose an alternative ap-
proach of incorporating a transfer network that learns how
to predict semantic mask via a semantic reconstruction loss
and provides feedback to improve the depth and pose esti-
mations, which shows considerable improvements in depth
and flow prediction.
We empirically evaluate the feature and loss func-
tion augmentations on KITTI dataset [14] and compare
them with the state-of-the-art unsupervised learning frame-
work [51]. In our experiments we use class-level predic-
tions from DeepLabv3+ [4] trained on Cityscapes [6] and
Mask R-CNN [18] trained on MSCOCO [27]. Our key find-
ings:
• By using semantic segmentation for both feature and
loss augmentation, our proposed algorithms improves
squared relative error in depth estimation by 28% com-
pared to the strong baseline set by state-of-the-art un-
supervised GeoNet [51].
• Feature augmentation alone, combining semantic with
instance-level information, leads to larger gains.
With both class-level and instance-level features, the
squared relative error of the depth predictions im-
proves by 30% compared to the baseline.
• Finally, as for common dynamic object classes
(e.g. vehicles) SIGNet shows 39% improvement (in
squared relative error) for depth predictions and 29%
improvement in the flow prediction, thereby showing
that semantic information is very useful for improving
the performance in the dynamic categories of objects.
Furthermore, SIGNet is robust to noise in image
intensity compared to the baseline.
2. Related Work
Deep Models for Understanding Geometry: Deep mod-
els have been widely used in supervised depth estimation
[8, 29, 36, 53, 5, 49, 50, 11, 46], tracking, and pose es-
timation [43, 47, 2, 17] , as well as optical flow predic-
tions [7, 20, 25, 40]. These models have demonstrated su-
perior accuracy and typically faster speed in modern hard-
ware platforms (especially in the case of optical flow esti-
mation) compared to traditional methods. However, achiev-
ing good performance with supervised learning requires a
large amount of geometry-related labels. The current work
addresses this challenge by adopting an unsupervised learn-
ing framework for depth, pose, and optical flow estimations.
Deep Models for Semantic Predictions: Deep models are
widely applied in semantic prediction tasks, such as image
classification [24], semantic segmentation [4], and instance
9811
segmentation [18]. In this work, we utilize the effectiveness
of the semantic predictions provided by DeepLab v3+ [4]
and Mask R-CNN [18] in encoding spatial constraints to ac-
curately predict geometric attributes such as depth and flow.
While we particularly choose [4] and [18] for our SIGNet,
similar gains can be obtained by using other state-of-the-art
semantic prediction methods.
Unsupervised Deep Models for Understanding Geome-
try: Several recent methods propose to use unsupervised
learning for geometry understanding. In particular, Garg
et al. [13] uses a warping method based on Taylor expan-
sion. In the context of unsupervised flow prediction, Yu et
al. [21] and Ren et al. [37] introduce image reconstruction
loss with spatial smoothness constraints. Similar methods
are used in Zhou et al. [54] for learning depth and camera
ego-motions by ignoring object motions. This is partially
addressed by Vijayanarasimhan et al. [41], despite the fact,
we note, that the modeling of motion is difficult without
introducing semantic information. This framework is fur-
ther improved with better modeling of the geometry. Ge-
ometric consistency loss is introduced to handle occluded
regions, in binocular depth learning [16], flow prediction
[32] and joint depth, ego-motion and optical flow learning
[51]. Mahjourian et al. [31] focuses on improved geometric
constraints, Godard et al. [15] proposes several architectural
and loss innovations, while Zhan et al. [52] uses reconstruc-
tion in the feature space rather than the image space. In con-
trast, the current work explores using semantic information
to resolve ambiguities that are difficult for pure geometric
modeling. Methods proposed in the current work are com-
plementary to these recent methods, but we choose to vali-
date our approach on a state-of-the-art framework known as
GeoNet [51].
Multi-Task Learning for Semantic and Depth: Multi-
task learning [3] achieves better generalization by allowing
the system to learn features that are robust across different
tasks. Recent methods focus on designing efficient archi-
tectures that can predict related tasks using shared features
while avoiding negative transfers [35, 19, 30, 34, 23, 12].
In this context, several prior works report promising results
combining scene geometry with semantics. For instance,
similar to our method Liu et al. [28] uses semantic predic-
tions to provide depth. However, this work is fully super-
vised and only uses sub-optimal traditional methods. Wang
et al. [44], Cross-Stitching [35], UberNet [23] and NDDR-
CNN [12] all report improved performance over single-task
baselines. But they have not addressed outdoor scenes and
unsupervised geometry understanding. Our work is also re-
lated to PAD-Net [48]. PAD-Net reports improvements by
combining intermediate tasks as inputs to final depth and
segmentation tasks. Our method of using semantic input
similarly introduces an intermediate prediction task as input
to the depth and pose predictions, but we tackle the problem
setting where depth labels are not provided.
3. State-of-the-art Unsupervised Geometry
Prediction
Prior to presenting our technical approach, we provide
a brief overview of state-of-the-art unsupervised depth and
motion estimation framework, which is based on image re-
construction from geometric predictions [54, 51]. It trains
the geometric prediction models through the reconstruc-
tions of a target image from source images. The target and
source images are neighboring frames in a video sequence.
Note that such a reconstruction is possible only when cer-
tain elements of the 3D geometry of the scene are under-
stood: (1) The relative 3D location (and thus the distance)
between the camera and each pixel. (2) The camera ego-
motion. (3) The motion of pixels. Thus this framework can
be used to train a depth estimator and an ego-motion esti-
mator, as well as a optical flow predictor.
Technically, each training sample I = {Ii}ni=1 consists
of n contiguous video frames Ii ∈ RH×W×3 where the cen-
ter frame It is the “target frame” and the other frames serve
as the “source frame”. In training, a differentiable warping
function ft→s is constructed from the geometry predictions.
The warping function is used to reconstruct the target frame
Is ∈ RH×W×3 from source frame Is via bilinear sampling.
The level of success in this reconstruction provides training
signals through backpropagation to the various ConvNets in
the system. A standard loss function to measure reconstruc-
tion success is as follows:
Lrw = α1− SSIM(It, Is)
2+ (1− α)||It − Is||1 (1)
where SSIM denotes the structural similarity index [45] and
α is set to 0.85 in [51].
To filter out erroneous predictions while preserving
sharp details, the standard practice is to include an edge-
aware depth smoothness loss Lds weighted by image gradi-
ents
Lds =∑
pt
|∇D(pt)| · (e−|∇I(pt)|)T (2)
where | · | denotes element-wise absolute operation, ∇ is the
vector differential operator, and T denotes transpose of gra-
dients. These losses are usually computed from a pyramid
of multi-scale predictions. The sum is used as the training
target.
While the reconstruction of RGB images is an effective
surrogate task for unsupervised learning, it is limited by the
lack of semantic information as supervision signals. For ex-
ample, the system cannot learn the difference between the
car and the road if they have similar colors or two neighbor-
ing cars with similar colors. When object motion is consid-
ered in the models, the learning can mistakenly assign mo-
tion to non-moving objects as the geometric constraints are
9812
RGB Frames
DepthNet
PoseNet
Camera Motion
Depth Map
Semantic FramesInstance EdgesInstance Frames
ResFlowNet Optical Flow
⊕
Concat Concat Concat
Figure 2: Our unsupervised architecture contains DepthNet, PoseNet and ResFlowNet to predict depth, poses and motion
using semantic-level and instance-level segmentation concatenated along the input channel dimension.
ill-posed. We augment and improve this system by leverag-
ing semantic information.
4. Methods
In this section, we present solutions to enhance geome-
try predictions with semantic information. Semantic labels
can provide rich information on 3D scene geometry. Impor-
tant details such as 3D location of pixels and their move-
ments can be inferred from a dense representation of the
scene semantics. The proposed methods are applicable to
a wide variety of recently proposed unsupervised geometry
learning frameworks based on photometric reconstruction
[54, 16, 51] represented by our baseline framework intro-
duced in Section 3. Our complemented pipeline in test time
is illustrated in Fig 2.
4.1. Semantic Input Augmentation
Semantic predictions can improve geometry prediction
models when serving as input features. Unlike RGB im-
ages, semantic predictions mark objects and contiguous
structures with consistent blobs, which provide important
information for the learning problem. However, it is un-
certain that using semantic labels as input could indeed im-
prove depth and flow predictions since training labels are
not available. Semantic information could be lost or dis-
torted, which would end up being a noisy training signal.
An important finding of our work is that using semantic
predictions as inputs significantly improves the accuracy in
geometry predictions, despite the presence of noisy training
signal. Input representation and the type of semantic la-
bels have a large impact on the performance of the system.
We further illustrate this by Fig 3, where we show various
semantic labels (semantic segmentation, instance segmen-
tation, and instance edge) that we use to augment the input.
This imposes additional constraints such as depth of the pix-
els belonging to a particular object (e.g. a vehicle) which
helps the learning process. Furthermore, sudden changes
RG
B I
ma
ge
Se
ma
nti
c
seg
me
nta
tio
n
Inst
an
ce c
lass
seg
me
nta
tio
n
Inst
an
ce
ed
ge
ma
p
Figure 3: Top to bottom: RGB image, semantic segmen-
tation, instance class segmentation and instance edge map.
They are used for the full prediction architecture. The se-
mantic segmentation provides accurate segments grouped
by classes, but it fails to differentiate neighboring cars.
in the depth predictions can be inferred from the boundary
of vehicles. The semantic labels of the pixels can provide
important information to associate pixels across frames.
Encoding Pixel-wise Class Labels: We explored two in-
put encoding techniques for class labels: dense encoding
and one-hot encoding. In dense encoding, dense class la-
bels are concatenated along the input channel dimension.
The added semantic features are centralized to the range of
[−1, 1] to be consistent with RGB inputs. In the case of
9813
one-hot encoding, the class-level semantic predictions are
first expanded to one-hot encoding and then concatenated
along the input channel dimension. The labels are repre-
sented as one-hot sparse vectors. In this variant, semantic
features are not normalized since they have similar value
range as the RGB inputs,
Encoding Instance-level Semantic Information: Both
dense and one-hot encoding are natural to class-level se-
mantic prediction, where each pixel is only assigned a class
label rather than an instance label. Our conjecture is that
instance-level semantic information is particular well-suited
to improve unsupervised geometric predictions, as it pro-
vides accurate information on the boundary between indi-
vidual objects of the same type. Unlike class-level label, the
instance label itself does not have a well-defined meaning.
Across different frames, the same label could refer to differ-
ent object instances. To efficiently represent the instance-
level information, we compute the gradient map of a dense
instance map and use it as an additional feature channel con-
catenating to the class label input (dense/one-hot encoding).
Direct Input versus Residual Correction: Complemen-
tary to the choice of encoding, we also experiment with dif-
ferent architectures to feed semantic information to the ge-
ometry prediction model. In particular, we make a residual
prediction using a separate branch that takes in only seman-
tic inputs. Notably, using residual depth prediction leads
to further improvement on top of the gains from the direct
input methods.
4.2. Semantic Guided Loss Functions
The information from semantic predictions could be di-
minished due to noisy semantic labels and very deep archi-
tectures. Hence, we design training loss functions that are
guided by semantic information. In such design, the se-
mantic predictions provide additional loss constraints to the
network. In this subsection, we introduce a set of seman-
tic guided loss functions to improve depth and flow predic-
tions.
Semantic Warp Loss: Semantic predictions can help learn
scenarios where reconstruction of the RGB image is cor-
rect in terms of pixel values but violates obvious semantic
correspondences, e.g. matching pixels to incorrect seman-
tic classes and/or instances. In light of this, we propose to
reconstruct the semantic predictions in addition of doing so
for RGB images. We call this “semantic warping loss” as it
is based on warping of the semantic predictions from source
frames to the target frame. Let Ss be the source frame se-
mantic prediction and Srigs be the warped semantic image,
we define semantic warp loss as:
Lsem = ||Srigs − St||2 (3)
The warped loss is added to the baseline framework using a
hyper-tuned value of the weight w.
Masking of Reconstruction Loss via Semantics: As de-
scribed in Section 3, the ambiguity in object motion can
lead to sub-optimal learning. Semantic labels can par-
tially resolve this by separating each class of region. Moti-
vated by this observation, we mask the foreground region
out to form a set of new images Jkt,c = It,c ⊙ St,k for
c = 0, 1, 2 and k = 0, ...,K − 1 where c represents the
RGB-channel index, ⊙ is the element-wise multiplication
operator and Ss,k is the k-th channel of the binary semantic
segmentation (K classes in total). Similarly we can obtain
Jrig,ks,c = Irigs,c ⊙ St,k for c = 0, 1, 2 and k = 0, ...,K − 1.
Finally, the image similarity loss is defined as:
L′rw =
K−1∑
k=0
α1− SSIM(Jk
t , Jrig,ks )
2+(1−α)||Jk
t −Jrig,ks ||1
(4)
DepthNet
PoseNet
VGG16
Image Loss
depth
poseSemantic
Maps
*Optional
RGB
Total Loss⊕
Semantic
Loss
predicted semantic
maps
Concat
Figure 4: Infer semantic labels from depth predictions. The
transfer function uses RGB and predicted depth as input.
We experimented the variants with and without semantic
input.
Semantic-Aware Edge Smoothness Loss: Equation 2 uses
RGB to infer edge locations when enforcing smooth re-
gions of depth. This could be improved by including an
edge map computed from semantic predictions. Given a se-
mantic segmentation result St, we define a weight matrix
Mt ∈ [0, 1]H×W where the weight is low (close to zero)
on class boundary regions and high (close to one) on other
regions. We propose a new image similarity loss as:
L′′rw =
K−1∑
k=0
α1− SSIM(It ⊙Mt, I
rigs ⊙Mt)
2
+ (1− α)||It ⊙Mt − Irigs ⊙Mt||1
(5)
Semantic Loss by Transfer Network: Motivated by the
observation that high-quality depth maps usually depict ob-
ject classes and background region, we designed a novel
transfer network architecture. As shown in Fig 4 the trans-
fer network block receives predicted depth maps along
with the original RGB images and outputs semantic labels.
The transfer network introduces a semantic reconstruction
loss term to the objective function to force the predicted
depth maps to be richer in contextual sense, hence refines
9814
the depth estimation. For implementation, we choose the
ResNet-50 as the backbone and alter the dimensions for the
input and output convolutional layers to be consistent with
the segmentation task. The network generates one-hot en-
coded heatmaps and use cross-entropy as the semantic sim-
ilarity measure.
5. Experiments
To quantify the benefits that semantic information brings
to geometry-based learning, we designed experiments sim-
ilar to [51]. First, we showed our model’s depth predic-
tion performance on KITTI dataset [14], which outper-
formed state-of-the-art unsupervised and supervised mod-
els. Then we designed ablation studies to analyze each in-
dividual component’s contribution. Finally, we presented
improvements in flow predictions and revisited the perfor-
mance gains using a category-specific evaluation.
5.1. Implementation Details
To make a fair comparison with state-of-the-art models
[8, 54, 51], we divided KITTI 2015 dataset into train set
(40238 images) and test set (697 images) according to the
rules from Eigen et al [8]. We used DeepLabv3+ [4] (pre-
trained on [6]) for semantic segmentation and Mask-RCNN
[18] (pretrained on [27]) for instance segmentation. Similar
to the hyper-parameter settings in [51], we used Adam opti-
mizer [22] with initial learning rate as 2e-4, set batch size to
4 per GPU and trained our modified DepthNet and PoseNet
modules for 250000 iterations with random shuffling and
data augmentation (random scaling, cropping and RGB per-
turbation). The training took 10 hours on two GTX1080Ti.
5.2. Monocular Depth Evaluation on KITTI
We augmented the image sequences with corresponding
semantic and instance segmentation sequences and adopted
the scale normalization suggested in [42]. In the evalua-
tion stage, the ground truth depth maps were generated by
projecting 3D Velodyne LiDAR points to the image plane.
Followed by [51], we clipped our depth predictions within
0.001m to 80m and calibrated the scale by the medium num-
ber of the ground truth. The evaluation results are shown in
Table 1, where all the metrics are introduced in [8]. Our
model benefits significantly from feature augmentation and
surpasses the state-of-the-art methods substantially in both
supervised and unsupervised fields.
Moreover, we found a correlation between the improve-
ment region and object classes. We visualized the absolute
relative error (AbsRel) among image plane from our model
and from the baseline. As shown in Fig 5, most of the im-
provements come from regions containing objects. This in-
dicates that the network is able to learn the concept of ob-
jects to improve the depth prediction by rendering extra se-
mantic information.
Figure 5: Comparisons of depth evaluations on KITTI. Top
to bottom: Input RGB image, AbsRel error map of [51],
AbsRel error map of ours, and improvements of ours on Ab-
sRel map compared to [51]. The ground truth is interpolated
to enhance visualization. Lighter color in those heatmaps
corresponds to larger errors or improvements.
5.3. Ablation Studies
Here we took a deeper look of our model, testified its ro-
bustness under noise from observations, and presented vari-
ations of our framework to show promising explorations for
future researchers. In the following experiments, we kept all
the other parameters the same in [51] and applied the same
training/evaluation strategies mentioned in Section 5.2
How much gain from various feature augmentation?
We tried out different combinations and forms of
semantic/instance-level inputs based on “Yin et al” [51]
with scale normalization. From Table 2, our first conclusion
is that any meaningful form of extra input can ameliorate the
model, which is straightforward. Secondly, when we use
“Semantic” and “Instance class” for feature augmentation,
one-hot encoding tends to outperform the dense map form.
Conceivably one-hot encoding stores richer information in
its structural formation, whereas dense map only contains
discrete labels which may be more difficult for learning.
Moreover, using both “Semantic” and “Instance class” can
provide further gain, possibly due to the different label dis-
tributions of the two datasets. Labels from Cityscape [6]
cover both background and foreground concepts, while the
COCO dataset [27] focuses more on objects. At last, when
we combined one-hot encoded “Semantic” and “Instance
class” along with “Instance id” edge features, the network
exploited the most from scene understanding, hence greatly
enhanced the performance.
Can our model survive under low lighting conditions?
To testify our model’s robustness for varied lighting condi-
tions, we multiplied a scalar between 0 and 1 to RGB inputs
9815
Method SupervisedError-related metrics Accuracy-related metrics
Abs Rel Sq Rel RSME RSME log δ < 1.25 δ < 1.252 δ < 1.253
Eigen et al. [8] Coarse Depth 0.214 1.605 6.653 0.292 0.673 0.884 0.957
Eigen et al. [8] Fine Depth 0.203 1.548 6.307 0.282 0.702 0.890 0.957
Liu et al. [29] Depth 0.202 1.614 6.523 0.275 0.678 0.895 0.965
Godard et al. [16] Pose 0.148 1.344 5.927 0.247 0.803 0.922 0.964
Zhou et al. [54] updated No 0.183 1.595 6.709 0.270 0.734 0.902 0.959
Yin et al. [51] No 0.155 1.296 5.857 0.233 0.793 0.931 0.973
OursNo
0.133 0.905 5.181 0.208 0.825 0.947 0.981
(improved by) 14.04% 30.19% 11.55% 10.85% 3.14% 1.53% 0.80%
Table 1: Monocular depth results on KITTI 2015 [33] by the split of Eigen et al. [8] (Our model used scale normalization.)
SemanticInstance Instance Error-related metrics Accuracy-related metrics
class id Abs Rel Sq Rel RSME RSME log δ < 1.25 δ < 1.252 δ < 1.253
0.149 1.060 5.567 0.226 0.796 0.935 0.975
Dense 0.142 0.991 5.309 0.216 0.814 0.943 0.980
One-hot 0.139 0.949 5.227 0.214 0.818 0.945 0.980
Dense 0.142 0.986 5.325 0.218 0.812 0.943 0.978
One-hot 0.141 0.976 5.272 0.215 0.811 0.942 0.979
Edge 0.145 1.037 5.314 0.217 0.807 0.943 0.978
Dense Edge 0.142 0.969 5.447 0.219 0.808 0.941 0.978
One-hot One-hot Edge 0.133 0.905 5.181 0.208 0.825 0.947 0.981
Table 2: Depth prediction performance gains due to different semantic sources and forms. (Scale normalization was used.)
in the evaluation. Fig 6 showed that our model still holds
equal performance to [51] when the intensity drops to 30%.
(a) Observations under decreased light condition (left to right)
0.0 0.2 0.4 0.6 0.8 1.0Darkness = 1 - Intensity
1
2
3
Squ
are
Rela
tive
Erro
r Test under Varied Light ConditionsYin et alOurs
(b) Robustness under decreased light condition
Figure 6: The abs errs change as lighting condition drops.
Our model can still be better than baseline even if the light-
ing intensity drops to 0.30 of the original ones.
Which module needs extra information the most?
We fed semantics to only DepthNet or PoseNet to see the
difference in their performance gain. From Table 3 we can
see that compared to DepthNet, PoseNet learns little from
the semantics to help depth prediction. Therefore we tried
to feed the semantics to a new PoseNet with the same struc-
ture as the original one and compute the predicted poses by
taking the sum from two different PoseNets, which led to
performance gain; however, performance gain was not ob-
served from applying the same method to DepthNet.
How to be “semantic-free” in evaluation?
Though semantic helps depth prediction, this idea relies on
semantic features during the evaluation phase. If semantic
is only utilized in the loss, it would not be needed in evalua-
tion. We attempted to introduce a handcrafted semantic loss
term as a weight guidance among image plane but it didn’t
work well. Also we designed a transfer network which uses
the predicted depth to predict semantic maps along with a
reconstruction error to help in the training stage. The result
in Table 4 shows a better result can be obtained by training
from pretrained models.
5.4. Optical Flow Estimation on KITTI
Using our best model for DepthNet and PoseNet in Sec-
tion 5.2, we conducted rigid flow and full flow evaluation
on KITTI [14]. We generated the rigid flow from estimated
depth and pose, and compared with [51]. Our model per-
formed better in all the metrics shown in Table 5.
9816
DepthNet PoseNetError-related metrics Accuracy-related metrics
Abs Rel Sq Rel RSME RSME log δ < 1.25 δ < 1.252 δ < 1.253
0.149 1.060 5.567 0.226 0.796 0.935 0.975
Channel 0.145 0.957 5.291 0.216 0.805 0.943 0.980
Channel 0.147 1.076 5.385 0.223 0.808 0.938 0.975
Channel Channel 0.139 0.949 5.227 0.214 0.818 0.945 0.980
Extra Net Channel 0.147 1.036 5.593 0.226 0.803 0.937 0.975
Channel Extra Net 0.135 0.932 5.241 0.211 0.821 0.945 0.980
Table 3: Each module’s contribution toward performance gain from semantics. (Scale normalization was used.)
CheckpointTransfer Error-related metrics Accuracy-related metrics
Network Abs Rel Sq Rel RSME RSME log δ < 1.25 δ < 1.252 δ < 1.253
Yin et al. [51] 0.155 1.296 5.857 0.233 0.793 0.931 0.973
Yin et al. [51] Yes 0.150 1.141 5.709 0.231 0.792 0.934 0.974
Yin et al. [51] +sn 0.149 1.060 5.567 0.226 0.796 0.935 0.975
Yin et al. [51] +sn Yes 0.145 0.994 5.422 0.222 0.806 0.939 0.976
Table 4: Gains in depth prediction using our proposed Transfer Network. (+sn: “using scale normalization”.)
MethodEnd Point Error Accuracy
Noc All Noc All
Yin et al. [51] 23.5683 29.2295 0.2345 0.2237
Ours 22.3819 26.8465 0.2519 0.2376
Table 5: Rigid flow prediction from first stage on KITTI on
non-occluded regions(Noc) and overall regions(All).
MethodEnd Point Error
Noc All
DirFlowNetS 6.77 12.21
Yin et al. [51] 8.05 10.81
Ours 7.66 13.91
Table 6: Full flow prediction on KITTI 2015 on non-
occluded regions(Noc) and overall regions(All). Results
from DirFlowNetS are shown in [51]
We further appended the semantic warping loss intro-
duced in Section 4.2 to ResFlowNet in [51] and trained our
model on KITTI stereo for 1600000 iterations. As demon-
strated in Table 6, flow prediction got improved in non-
occluded region compared to [51] and our model produced
comparable results in overall regions.
5.5. CategorySpecific Metrics Evaluation
This section will present the improvements by seman-
tic categories. As shown in the bar-chart in Fig 7, most
improvements were shown in “Vehicle” and “Dynamic”
classes3, where errors are generally large. Our network did
not improve much for other less frequent categories, such
3For “Dynamic” classes, we choose “person”, “rider”, “car”, “truck”,
“bus”, “train”, “motorcycle” and “bicycle” classes as defined in [6]
Car Motorcycle DynamicClasses
0
2
4
6
Squa
re R
elat
ive
Erro
r 5.42
0.16
5.59
3.72
0.21
3.39
Depth Prediction ComparisonYin et alOurs
Car Motorcycle DynamicClasses
0
5
10
15
20
Endp
oint
Erro
r
19.62
5.45
19.50
13.60
6.40
13.86
Flow Prediction ComparisonYin et alOurs
Figure 7: Performance gains in depth (left) and flow (right)
among different classes of dynamic objects.
as “Motorcycle”, which are generally more difficult to seg-
ment in images.
6. Conclusion
In SIGNet, we strive to achieve robust performance
for depth and flow perception without using geometric
labels. To achieve this goal, SIGNet utilizes semantic
and instance segmentation to create spatial constraints on
the geometric attributes of the pixels. We present novel
methods of feature augmentation and loss augmentation to
include semantic labels in the geometry predictions. This
work presents a first of a kind approach which moves away
from pixel-level to object-level depth and flow predictions.
Most notably, our method significantly surpasses the
state-of-the-art solution for monocular depth estimation. In
the future, we would like to extend our SIGNet to various
sensor modalities (IMU, LiDAR or thermal).
Acknowledgement: This work was supported by UCSD
faculty startup (Prof. Bharadia), Toyota InfoTechnology
Center and Center for Wireless communication at UCSD.
9817
References
[1] Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan
Leutenegger, and Andrew J Davison. Codeslam-learning a
compact, optimisable representation for dense visual slam.
arXiv preprint arXiv:1804.00874, 2018. 1
[2] Arunkumar Byravan and Dieter Fox. Se3-nets: Learning
rigid body motion using deep neural networks. In Robotics
and Automation (ICRA), 2017 IEEE International Confer-
ence on, pages 173–180. IEEE, 2017. 1, 2
[3] Rich Caruana. Multitask learning. Mach. Learn., 28(1):41–
75, July 1997. 3
[4] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Flo-
rian Schroff, and Hartwig Adam. Encoder-decoder with
atrous separable convolution for semantic image segmenta-
tion. arXiv preprint arXiv:1802.02611, 2018. 2, 3, 6
[5] Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-
image depth perception in the wild. In Proceedings of the
30th International Conference on Neural Information Pro-
cessing Systems, NIPS’16, pages 730–738, USA, 2016. Cur-
ran Associates Inc. 2
[6] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo
Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe
Franke, Stefan Roth, and Bernt Schiele. The cityscapes
dataset for semantic urban scene understanding. In Proceed-
ings of the IEEE conference on computer vision and pattern
recognition, pages 3213–3223, 2016. 2, 6, 8
[7] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip
Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der
Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learn-
ing optical flow with convolutional networks. In The IEEE
International Conference on Computer Vision (ICCV), De-
cember 2015. 2
[8] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map
prediction from a single image using a multi-scale deep net-
work. In Advances in neural information processing systems,
pages 2366–2374, 2014. 2, 6, 7
[9] Xiaohan Fei, Alex Wang, and Stefano Soatto. Geo-
supervised visual depth prediction. arXiv preprint
arXiv:1807.11130, 2018. 1
[10] Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip
Hausser, Caner Hazırbas, Vladimir Golkov, Patrick Van der
Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learn-
ing optical flow with convolutional networks. arXiv preprint
arXiv:1504.06852, 2015. 1
[11] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Bat-
manghelich, and Dacheng Tao. Deep ordinal regression net-
work for monocular depth estimation. In The IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR),
June 2018. 2
[12] Yuan Gao, Qi She, Jiayi Ma, Mingbo Zhao, Wei Liu, and
Alan L. Yuille. Nddr-cnn: Layer-wise feature fusing in
multi-task cnn by neural discriminative dimensionality re-
duction. CoRR, abs/1801.08297, 2018. 3
[13] Ravi Garg, G VijayKumarB., and Ian D. Reid. Unsupervised
cnn for single view depth estimation: Geometry to the res-
cue. In ECCV, 2016. 3, 12, 13, 14, 15, 16, 17, 18, 19, 20,
21, 22
[14] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we
ready for autonomous driving? the kitti vision benchmark
suite. In Computer Vision and Pattern Recognition (CVPR),
2012 IEEE Conference on, pages 3354–3361. IEEE, 2012.
2, 6, 7
[15] Clement Godard, Oisin Mac Aodha, and Gabriel J. Bros-
tow. Digging into self-supervised monocular depth estima-
tion. CoRR, abs/1806.01260, 2018. 3
[16] Clement Godard, Oisin Mac Aodha, and Gabriel J Bros-
tow. Unsupervised monocular depth estimation with left-
right consistency. In CVPR, volume 2, page 7, 2017. 3,
4, 7
[17] Daniel Gordon, Ali Farhadi, and Dieter Fox. Re 3: Real-time
recurrent regression networks for visual tracking of generic
objects. IEEE Robotics and Automation Letters, 3(2):788–
795, 2018. 2
[18] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Gir-
shick. Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE
International Conference on, pages 2980–2988. IEEE, 2017.
2, 3, 6
[19] Keke He, Zhanxiong Wang, Yanwei Fu, Rui Feng, Yu-Gang
Jiang, and Xiangyang Xue. Adaptively weighted multi-task
deep network for person attribute classification. In Proceed-
ings of the 25th ACM International Conference on Multime-
dia, MM ’17, pages 1636–1644, New York, NY, USA, 2017.
ACM. 3
[20] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper,
Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evo-
lution of optical flow estimation with deep networks. In The
IEEE Conference on Computer Vision and Pattern Recogni-
tion (CVPR), July 2017. 2
[21] J Yu Jason, Adam W Harley, and Konstantinos G Derpanis.
Back to basics: Unsupervised learning of optical flow via
brightness constancy and motion smoothness. In European
Conference on Computer Vision, pages 3–10. Springer, 2016.
3
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for
stochastic optimization. arXiv preprint arXiv:1412.6980,
2014. 6
[23] I. Kokkinos. Ubernet: Training a universal convolutional
neural network for low-, mid-, and high-level vision using
diverse datasets and limited memory. In 2017 IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR),
pages 5454–5463, July 2017. 3
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.
Imagenet classification with deep convolutional neural net-
works. Commun. ACM, 60(6):84–90, May 2017. 2
[25] Wei-Sheng Lai, Jia-Bin Huang, and Ming-Hsuan Yang.
Semi-supervised learning for optical flow with generative ad-
versarial networks. In NIPS, 2017. 2
[26] Konstantinos-Nektarios Lianos, Johannes L Schonberger,
Marc Pollefeys, and Torsten Sattler. Vso: Visual semantic
odometry. In Proceedings of the European Conference on
Computer Vision (ECCV), pages 234–250, 2018. 1
[27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence
Zitnick. Microsoft coco: Common objects in context. In
9818
European conference on computer vision, pages 740–755.
Springer, 2014. 2, 6
[28] Beyang Liu, Stephen Gould, and Daphne Koller. Single im-
age depth estimation from predicted semantic labels. 2010
IEEE Computer Society Conference on Computer Vision and
Pattern Recognition, pages 1253–1260, 2010. 3
[29] Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian D Reid.
Learning depth from single monocular images using deep
convolutional neural fields. IEEE Trans. Pattern Anal. Mach.
Intell., 38(10):2024–2039, 2016. 2, 7
[30] Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng,
Tara Javidi, and Rogerio Feris. Fully-adaptive feature shar-
ing in multi-task networks with applications in person at-
tribute classification. In The IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), July 2017. 3
[31] Reza Mahjourian, Martin Wicke, and Anelia Angelova. Un-
supervised learning of depth and ego-motion from monoc-
ular video using 3d geometric constraints. In The IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), June 2018. 3
[32] Simon Meister, Junhwa Hur, and Stefan Roth. UnFlow: Un-
supervised learning of optical flow with a bidirectional cen-
sus loss. In AAAI, New Orleans, Louisiana, Feb. 2018. 3
[33] Moritz Menze and Andreas Geiger. Object scene flow for au-
tonomous vehicles. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages 3061–
3070, 2015. 7
[34] Elliot Meyerson and Risto Miikkulainen. Beyond shared hi-
erarchies: Deep multitask learning through soft layer order-
ing. CoRR, abs/1711.00108, 2017. 3
[35] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Mar-
tial Hebert. Cross-stitch networks for multi-task learning.
2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pages 3994–4003, 2016. 3
[36] N.Mayer, E.Ilg, P.Hausser, P.Fischer, D.Cremers,
A.Dosovitskiy, and T.Brox. A large dataset to train
convolutional networks for disparity, optical flow, and scene
flow estimation. In IEEE International Conference on
Computer Vision and Pattern Recognition (CVPR), 2016.
arXiv:1512.02134. 2
[37] Zhe Ren, Junchi Yan, Bingbing Ni, Bin Liu, Xiaokang Yang,
and Hongyuan Zha. Unsupervised deep learning for optical
flow estimation. In AAAI, volume 3, page 7, 2017. 3
[38] Ashutosh Saxena, Sung H Chung, and Andrew Y Ng. Learn-
ing depth from single monocular images. In Advances in
neural information processing systems, pages 1161–1168,
2006. 1
[39] Ashutosh Saxena, Min Sun, and Andrew Y Ng. Make3d:
Learning 3d scene structure from a single still image. IEEE
transactions on pattern analysis and machine intelligence,
31(5):824–840, 2009. 1
[40] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.
Pwc-net: Cnns for optical flow using pyramid, warping, and
cost volume. In The IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), June 2018. 2
[41] Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia
Schmid, Rahul Sukthankar, and Katerina Fragkiadaki. Sfm-
net: Learning of structure and motion from video. arXiv
preprint arXiv:1704.07804, 2017. 1, 3
[42] Chaoyang Wang, Jose Miguel Buenaposada, Rui Zhu, and
Simon Lucey. Learning depth from monocular videos us-
ing direct methods. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages 2022–
2030, 2018. 6
[43] Lijun Wang, Wanli Ouyang, Xiaogang Wang, and Huchuan
Lu. Visual tracking with fully convolutional networks. In
Proceedings of the IEEE international conference on com-
puter vision, pages 3119–3127, 2015. 2
[44] Peng Wang, Xiaohui Shen, Zhe Lin, Scott Cohen, Brian
Price, and Alan L. Yuille. Towards unified depth and seman-
tic prediction from a single image. In The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), June
2015. 3
[45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si-
moncelli. Image quality assessment: from error visibility to
structural similarity. IEEE transactions on image processing,
13(4):600–612, 2004. 3
[46] Ke Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, Yang Xiao,
Ruibo Li, and Zhenbo Luo. Monocular relative depth percep-
tion with web stereo data supervision. In The IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR),
June 2018. 2
[47] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and
Dieter Fox. Posecnn: A convolutional neural network for
6d object pose estimation in cluttered scenes. arXiv preprint
arXiv:1711.00199, 2017. 2
[48] Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe.
Pad-net: Multi-tasks guided prediction-and-distillation net-
work for simultaneous depth estimation and scene parsing.
CoRR, abs/1805.04409, 2018. 3
[49] Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, and
Nicu Sebe. Multi-scale continuous crfs as sequential deep
networks for monocular depth estimation. In The IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR), July 2017. 2
[50] Dan Xu, Wei Wang, Hao Tang, Hong Liu, Nicu Sebe, and
Elisa Ricci. Structured attention guided convolutional neural
fields for monocular depth estimation. In The IEEE Confer-
ence on Computer Vision and Pattern Recognition (CVPR),
June 2018. 2
[51] Zhichao Yin and Jianping Shi. Geonet: Unsupervised learn-
ing of dense depth, optical flow and camera pose. In Pro-
ceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), volume 2, 2018. 1, 2, 3, 4, 6,
7, 8, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22
[52] Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera,
Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised
learning of monocular depth estimation and visual odometry
with deep feature reconstruction. In The IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), June
2018. 3
[53] Ziyu Zhang, Alexander G. Schwing, Sanja Fidler, and
Raquel Urtasun. Monocular object instance segmentation
and depth ordering with cnns. 2015 IEEE International
9819
Conference on Computer Vision (ICCV), pages 2614–2622,
2015. 2
[54] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G
Lowe. Unsupervised learning of depth and ego-motion from
video. In CVPR, volume 2, page 7, 2017. 1, 3, 4, 6, 7
9820
Top Related