BLOG Inside Playment

Playment’s Sensor-Fusion Data Labeling Tools For Better CV/ML Models

Merlin Peter
October 21, 2020

Perception is the foundational base for all autonomous systems to function safely. Robotics, autonomous vehicles, AR technologies like the all-new iPad pro lidar scanners, and VR technologies are increasingly using data from multiple sensors to counter the limitations of relying on a single sensor for context awareness. Sensor fusion primarily increases perception accuracy for computer vision models by identifying and classifying objects, understanding relationships, and interactions between objects, and helping autonomous systems predict behaviours more accurately.

For example, the most commonly combined sensors for autonomous vehicles are the Camera and LiDAR sensors that provide both 2D and 3D data for building improved perception systems. While the 2D labeled data may seem accurate on the surface, they account for many different inaccuracies that cause models to misbehave while predicting its environment. Combining 3D data from LiDAR sensors has considerably improved depth perception in such safety-critical systems. 

Accurate sensor fusion data labeling is the cornerstone of high-performing computer vision models. Playment has been at the forefront of sensor fusion labeling innovations by consistently developing efficient tools for overcoming technical challenges and meeting the evolving needs of ML teams working with multi-sensor data. 

Efficient 2D-3D Linking With Playment

We offer cutting-edge labeling tools for sensor fusion and particularly linking 2D-3D data for autonomous vehicles, robotics, and AR/VR use cases. Here’s our sensor-fusion feature stack you can leverage to create high-quality ground truth and training datasets for your computer vision models. 

Automated Projections

By utilising the spatial relationship between multiple sensors, the labeling process can be simplified by just creating labels in one sensor modality and automatically projecting it on to other sensors. These projections are possible because these tools can read the calibrated extrinsic parameters like sensor poses, camera intrinsics, distortion models, and synchronising data between multiple sensors across time. 

For example, where well-calibrated sensor parameters are available, the annotator manually labels an object in the 3D scene, and the annotation automatically projects on the 2D image. For improved precision, the objects can also be adjusted manually in the same task interface. The feature drastically reduces the annotator time spent on labeling. 

Manual Linking

When sensor parameters are not well-calibrated, our tool allows easy navigation between 3D and 2D scenes so that annotators can manually add objects in both sensors and link them with each other. Annotators can also use multiple keyboard shortcuts to create, adjust, and edit annotations on point clouds and images on the same task interface.  

Sensor Details Linked To Objects 

The annotator can select an object on any sensor to access all corresponding information about the object detected on multiple sensors. Secondly, the object, if present on several cameras, can be easily identified via camera thumbnails. 

Multi-Camera Support 

For sensor setups with multiple cameras, displaying all the images together help annotators identify the object class and shape faster by enabling a complete view of the surroundings in the scenario. This feature also makes it easier to annotate objects quickly in the 3D scenes.  

Easy Sensor and Auto-Image Switching

Switch between primary and secondary sensors using keyboard shortcuts for faster navigation between 2D and 3D scenes. You can also hide the secondary sensor for improved focus on the primary sensor to precisely annotate objects in respective sensors.

Additionally, you can enable/disable the auto-switching of camera images. You can choose to disable auto-image switching while creating annotations and enable the feature while reviewing annotated images. This auto-switching feature helps you better relate the objects between the sensors while editing/reviewing annotations.

Tracking IDs

The sensor-fusion tool automatically assigns the same tracking identifiers for annotations of the same real-world object created in 2D images and 3D point clouds, making it easier to identify and track objects across multiple frames. 

One-Click Cuboids 

The one-click cuboid feature lets you create 3D cuboids for sensor fusion scenes 5X faster. The tool auto-detects objects in the 3D scenario; the annotators can use one click to complete the annotation accurately. The orthogonal view allows further manual adjustments to improve the annotation precision if required. 

Default Dimensions

The tool also allows you to pre-configure dimensions of particular objects while setting up annotation projects and makes it easier to label objects in 3D scenes. This feature also helps you annotate faraway objects that have fewer points visible, and hence you can’t estimate the length, width, and height with proper estimations.

Auto-Ground Segmentation 

All ground points on a 3D scene are programmatically segmented. The annotators can get started on annotating everything above the ground with ease. This feature creates better visibility of objects above the ground, making it easier for annotators to annotate objects in 3D point clouds.

Ground and Ceiling Mover

The ground and ceiling mover enables annotators to hide the ground or ceiling points in 3D point clouds. This feature hides large backdrops like vegetation, buildings, sky, and other stationary objects to improve the focus on moving objects like vehicles, pedestrians, etc.

Ego-Motion Compensation 

Stationary objects are labeled automatically across frames in 3D scenarios by compensating for the ego motion of the vehicle. This feature makes it easier for annotators to label objects faster in 3D point clouds. 

Points Resizer

The annotators can effortlessly increase/decrease the sizes of points on 3D scenes using the points resizer. This feature helps them identify farther objects and unlabelled points to improve the recall and precision of the annotated images.

Quality Control Mechanisms For Accurate Sensor Fusion Labeling

We ensure the highest labeling accuracies by following a robust QC process for all our annotation projects. After having worked on more than a billion annotations, we have devised useful metrics to measure the quality of annotations created on both 2D and 3D scenes. 

We check all 2D and 3D annotations in every frame for the following errors: 

  1. Geometric Accuracy
  2. Class Accuracy
  3. Attribute Accuracy
  4. Tracker ID Error
  5. Linking Error
  6. Missing Annotations

All the errors get recorded in our unique customer dashboards, along with the overall precision and recall rates. All inconsistencies in each frame qualify for re-work. We have maintained the highest precision and recall rates for all our annotation projects, and provide up to 99% recall and precision for all our customers.

Sensor-fusion annotations have quickly gained popularity among ML teams building autonomous systems because these annotations provide autonomous systems with a better understanding of surroundings and objects. 

Are you working on a sensor fusion project? We can help you navigate through complexities. Contact us at and we’ll set up a pilot for you.