You can also find my articles on my Google Scholar Profile.
Show all by date / Show all by topic / Show selected


CLOUD: Contrastive Learning of Unsupervised Dynamics
Jianren Wang*, Yujie Lu*, Hang Zhao (* indicates equal contribution)
2020 Conference on Robot Learning
[Project Page] [Code] [Abstract] [Bibtex]

Developing agents that can perform complex control tasks from high dimensional observations such as pixels is challenging due to difficulties in learning dynamics efficiently. In this work, we propose to learn forward and inverse dynamics in a fully unsupervised manner via contrastive estimation. Specifically, we train a forward dynamics model and an inverse dynamics model in the feature space of states and actions with data collected from random exploration. Unlike most existing deterministic models, our energy-based model takes into account the stochastic nature of agent-environment interactions. We demonstrate the efficacy of our approach across a variety of tasks including goal-directed planning and imitation from observations.

    Author = {Wang, Jianren and Lu, Yujie and Zhao, Hang},
    Title = {CLOUD: Contrastive Learning of Unsupervised Dynamics},
    Booktitle = {CORL},
    Year = {2020}
Inverting the Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting
Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, Nick Rhinehart
2020 Conference on Robot Learning
[Project Page] [Code] [Abstract] [Bibtex]

Many autonomous systems forecast aspects of the future in order to aid decision-making. For example, self-driving vehicles and robotic manipulation systems often forecast future object poses by first detecting and tracking objects. However, this detect-then-forecast pipeline is expensive to scale, as pose forecasting algorithms typically require labeled sequences of object poses, which are costly to obtain in 3D space. Can we scale performance without requiring additional labels? We hypothesize yes, and propose inverting the detect-then-forecast pipeline. Instead of detecting, tracking and then forecasting the objects, we propose to first forecast 3D sensor data (e.g., point clouds with $100$k points) and then detect/track objects on the predicted point cloud sequences to obtain future poses, i.e., a forecast-then-detect pipeline. This inversion makes it less expensive to scale pose forecasting, as the sensor data forecasting task requires no labels. Part of this work's focus is on the challenging first step -- Sequential Pointcloud Forecasting (SPF), for which we also propose an effective approach, SPFNet. To compare our forecast-then-detect pipeline relative to the detect-then-forecast pipeline, we propose an evaluation procedure and two metrics. Through experiments on a robotic manipulation dataset and two driving datasets, we show that SPFNet is effective for the SPF task, our forecast-then-detect pipeline outperforms the detect-then-forecast approaches to which we compared, and that pose forecasting performance improves with the addition of unlabeled data.

    author = {Weng, Xinshuo and Wang, Jianren and Levine, Sergey 
    and Kitani, Kris and Rhinehart, Nick}, 
    journal = {CoRL}, 
    title = {Inverting the Pose Forecasting Pipeline with SPF2: 
    Sequential Pointcloud Forecasting for Sequential Pose Forecasting}, 
    year = {2020} 
PanoNet3D: Combining Semantic and Geometric Understanding for LiDARPoint Cloud Detection
Xia Chen, Jianren Wang, David Held, Martial Hebert
2020 International Virtual Conference on 3D Vision
[Project Page] [Code] [Abstract] [Bibtex]

Visual data in autonomous driving perception, such as camera image and LiDAR point cloud, can be interpreted as a mixture of two aspects: semantic feature and geometric structure. Semantics come from the appearance and context of objects to the sensor, while geometric structure is the actual 3D shape of point clouds. Most detectors on LiDAR point clouds focus only on analyzing the geometric structure of objects in real 3D space. Unlike previous works, we propose to learn both semantic feature and geometric structure via a unified multi-view framework. Our method exploits the nature of LiDAR scans -- 2D range images, and applies well-studied 2D convolutions to extract semantic features. By fusing semantic and geometric features, our method outperforms state-of-the-art approaches in all categories by a large margin. The methodology of combining semantic and geometric features provides a unique perspective of looking at the problems in real-world 3D point cloud detection.

    Author = {Chen, Xia and Wang, Jianren and Held, David and Hebert, Martial},
    Title = {PanoNet3D: Combining Semantic and Geometric Understanding
    for LiDARPoint Cloud Detection},
    Booktitle = {3DV},
    Year = {2020}
GSIR: Generalizable 3D Shape Interpretation and Reconstruction
Jianren Wang, Zhaoyuan Fang
2020 The European Conference on Computer Vision
[Project Page] [Code] [Abstract] [Bibtex]

3D shape interpretation and reconstruction are closely related to each other but have long been studied separately and often end up with priors that are highly biased towards the training classes. In this paper, we present an algorithm, Generalizable 3D Shape Interpretation and Reconstruction (GSIR), designed to jointly learn these two tasks to capture generic, class-agnostic shape priors for a better understanding of 3D geometry. We propose to recover 3D shape structures as cuboids from partial reconstruction and use the predicted structures to further guide full 3D reconstruction. The unified framework is trained simultaneously offline to learn a generic notion and can be fine-tuned online for specific objects without any annotations. Extensive experiments on both synthetic and real data demonstrate that introducing 3D shape interpretation improves the performance of single image 3D reconstruction and vice versa, achieving the state-of-the-art performance on both tasks for objects in both seen and unseen categories.

    title={GSIR: Generalizable 3D Shape Interpretation and Reconstruction},
    author={Wang, Jianren and Fang, Zhaoyuan},
    booktitle={European Conference on Computer Vision},
Uncertainty-aware Self-supervised 3D Data Association
Jianren Wang, Siddharth Ancha, Yi-Ting Chen, David Held
2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
[Project Page] [Code] [Abstract] [Bibtex]

3D object trackers usually require training on large amounts of annotated data that is expensive and time-consuming to collect. Instead, we propose leveraging vast unlabeled datasets by self-supervised metric learning of 3D object trackers, with a focus on data association. Large scale annotations for unlabeled data are cheaply obtained by automatic object detection and association across frames. We show how these self-supervised annotations can be used in a principled manner to learn point-cloud embeddings that are effective for 3D tracking. We estimate and incorporate uncertainty in self-supervised tracking to learn more robust embeddings, without needing any labeled data. We design embeddings to differentiate objects across frames, and learn them using uncertainty-aware self-supervised training. Finally, we demonstrate their ability to perform accurate data association across frames, towards effective and accurate 3D tracking.

    Author = {Wang, Jianren and Ancha, Siddharth and Chen, Yi-Ting and Held, David},
    Title = {Uncertainty-aware Self-supervised 3D Data Association},
    Booktitle = {IROS},
    Year = {2020}
Motion Prediction in Visual Object Tracking
Jianren Wang*, Yihui He*(* indicates equal contribution)
2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
Deep Mixture Density Network for Object Detection under Occlusion
Yihui He*, Jianren Wang*(* indicates equal contribution)
2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
3D Multi-Object Tracking: A Baseline and New Evaluation Metrics
Xinshuo Weng, Jianren Wang, David Held, Kris M. Kitani
2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
[Project Page] [Code] [Abstract] [Bibtex]

3D multi-object tracking (MOT) is an essential component for many applications such as autonomous driving and assistive robotics. Recent work on 3D MOT focuses on developing accurate systems giving less attention to practical considerations such as computational cost and system complexity. In contrast, this work proposes a simple real-time 3D MOT system. Our system first obtains 3D detections from a LiDAR point cloud. Then, a straightforward combination of a 3D Kalman filter and the Hungarian algorithm is used for state estimation and data association. Additionally, 3D MOT datasets such as KITTI evaluate MOT methods in the 2D space and standardized 3D MOT evaluation tools are missing for a fair comparison of 3D MOT methods. Therefore, we propose a new 3D MOT evaluation tool along with three new metrics to comprehensively evaluate 3D MOT methods. We show that, although our system employs a combination of classical MOT modules, we achieve state-of-the-art 3D MOT performance on two 3D MOT benchmarks (KITTI and nuScenes). Surprisingly, although our system does not use any 2D data as inputs, we achieve competitive performance on the KITTI 2D MOT leaderboard. Our proposed system runs at a rate of 207.4 FPS on the KITTI dataset, achieving the fastest speed among all modern MOT systems.

    author = {Weng, Xinshuo and Wang, Jianren and Held, David and Kitani, Kris}, 
    journal = {IROS}, 
    title = {3D Multi-Object Tracking: A Baseline and New Evaluation Metrics}, 
    year = {2020} 
PanoNet: Real-time Panoptic Segmentation through Position-Sensitive Feature Embedding
Xia Chen, Jianren Wang, Martial Hebert
AlignNet: A Unifying Approach to Audio-Visual Alignment
Jianren Wang*, Zhaoyuan Fang*, Hang Zhao (* indicates equal contribution)
2020 Winter Conference on Applications of Computer Vision
[Project Page] [Code] [Data] [Abstract] [Bibtex]

We present AlignNet, a model designed to synchronize a video with a reference audio under non-uniform and irregular misalignment. AlignNet learns the end-to-end dense correspondence between each frame of a video and an audio. Our method is designed according to simple and well-established principles: attention, pyramidal processing, warping, and affinity function. Together with the model, we release a dancing dataset Dance50 for training and evaluation. Qualitative, quantitative and subjective evaluation results on dance-music alignment and speech-lip alignment demonstrate that our method far outperforms the state-of-the-art methods.

    Author = {Wang, Jianren and Fang, Zhaoyuan
            and Zhao, Hang},
    Title = {AlignNet: A Unifying Approach to Audio-Visual Alignment},
    Booktitle = {WACV},
    Year = {2020}


Physics-Aware 3D Mesh Synthesis
Jianren Wang, Yihui He
2019 International Conference on 3D Vision
Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks
Yihui He*, Jianing Qian*, Jianren Wang (* indicates equal contribution)
2019 IEEE Conference on Computer Vision and Pattern Recognition Workshop
Bounding Box Regression with Uncertainty for Accurate Object Detection
Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, Xiangyu Zhang
2019 IEEE Conference on Computer Vision and Pattern Recognition


Vertical Jump Height Estimation Algorithm Based on Takeoff and Landing Identification Via Foot-Worn Inertial Sensing
Jianren Wang, Junkai Xu, Peter B Shull
2018 Journal of biomechanical engineering
Integration of a Low-Cost Three-Axis Sensor for Robot Force Control
Shuyang Chen, Jianren Wang, Peter Kazanzides
2018 Second IEEE International Conference on Robotic Computing


Prioritization and static error compensation for multi-camera collaborative tracking in augmented reality
Jianren Wang, Long Qian, Ehsan Azimi, Peter Kazanzides
2017 IEEE Virtual Reality
Validation of a smart shoe for estimating foot progression angle during walking gait
Haisheng Xia, Junkai Xu, Jianren Wang, Michael A Hunt, Peter B Shull
2017 Journal of biomechanics