lidar_transfusion#
Purpose#
The lidar_transfusion package is used for 3D object detection based on lidar data (x, y, z, intensity).
Inner-workings / Algorithms#
The implementation bases on TransFusion [1] work. It uses TensorRT library for data process and network inference.
We trained the models using https://github.com/open-mmlab/mmdetection3d.
Inputs / Outputs#
Input#
| Name | Type | Description |
|---|---|---|
~/input/pointcloud |
sensor_msgs::msg::PointCloud2 |
Input pointcloud. |
Output#
| Name | Type | Description |
|---|---|---|
~/output/objects |
autoware_perception_msgs::msg::DetectedObjects |
Detected objects. |
debug/cyclic_time_ms |
tier4_debug_msgs::msg::Float64Stamped |
Cyclic time (ms). |
debug/pipeline_latency_ms |
tier4_debug_msgs::msg::Float64Stamped |
Pipeline latency time (ms). |
debug/processing_time/preprocess_ms |
tier4_debug_msgs::msg::Float64Stamped |
Preprocess (ms). |
debug/processing_time/inference_ms |
tier4_debug_msgs::msg::Float64Stamped |
Inference time (ms). |
debug/processing_time/postprocess_ms |
tier4_debug_msgs::msg::Float64Stamped |
Postprocess time (ms). |
debug/processing_time/total_ms |
tier4_debug_msgs::msg::Float64Stamped |
Total processing time (ms). |
Parameters#
TransFusion#
| Name | Type | Description | Default | Range |
|---|---|---|---|---|
| class_names | array | An array of class names will be predicted. | ['CAR', 'TRUCK', 'BUS', 'BICYCLE', 'PEDESTRIAN'] | N/A |
| trt_precision | string | A precision of TensorRT engine. | fp16 | ['fp16', 'fp32'] |
| voxels_num | array | A maximum number of voxels [min, opt, max]. | [5000, 30000, 60000] | N/A |
| point_cloud_range | array | An array of distance ranges of each class. | [-76.8, -76.8, -3.0, 76.8, 76.8, 5.0] | N/A |
| voxel_size | array | Voxels size [x, y, z]. | [0.3, 0.3, 8.0] | N/A |
| onnx_path | string | A path to ONNX model file. | $(var model_path)/transfusion.onnx | N/A |
| engine_path | string | A path to TensorRT engine file. | $(var model_path)/transfusion.engine | N/A |
| down_sample_factor | integer | A scale factor of downsampling points | 1 | ≥1 |
| densification_num_past_frames | integer | A number of past frames to be considered as same input frame. | 1 | ≥0 |
| densification_world_frame_id | string | A name of frame id where world coordinates system is defined with respect to. | map | N/A |
| circle_nms_dist_threshold | float | A distance threshold between detections in NMS. | 0.5 | ≥0.0 |
| iou_nms_target_class_names | array | An array of class names to be target in NMS. | ['CAR'] | N/A |
| iou_nms_search_distance_2d | float | A maximum distance value to search the nearest objects. | 10.0 | ≥0.0 |
| iou_nms_threshold | float | A threshold value of NMS using IoU score. | 0.1 | ≥0.0 ≤1.0 |
| yaw_norm_thresholds | array | A thresholds array of direction vectors norm, all of objects with vector norm less than this threshold are ignored. | [0.3, 0.3, 0.3, 0.3, 0.0] | N/A |
| score_threshold | float | A threshold value of confidence score, all of objects with score less than this threshold are ignored. | 0.2 | ≥0.0 |
Detection class remapper#
| Name | Type | Description | Default | Range |
|---|---|---|---|---|
| allow_remapping_by_area_matrix | array | Whether to allow remapping of classes. The order of 8x8 matrix classes comes from ObjectClassification msg. | [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] | N/A |
| min_area_matrix | array | Minimum area for specific class to consider class remapping. | [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 12.1, 0.0, 36.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] | N/A |
| max_area_matrix | array | Maximum area for specific class to consider class remapping. | [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.0, 0.0, 999.999, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 999.999, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 999.999, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] | N/A |
The build_only option#
The lidar_transfusion node has build_only option to build the TensorRT engine file from the ONNX file.
Although it is preferred to move all the ROS parameters in .param.yaml file in Autoware Universe, the build_only option is not moved to the .param.yaml file for now, because it may be used as a flag to execute the build as a pre-task. You can execute with the following command:
ros2 launch lidar_transfusion lidar_transfusion.launch.xml build_only:=true
The log_level option#
The default logging severity level for lidar_transfusion is info. For debugging purposes, the developer may decrease severity level using log_level parameter:
ros2 launch lidar_transfusion lidar_transfusion.launch.xml log_level:=debug
Assumptions / Known limits#
Trained Models#
You can download the onnx format of trained models by clicking on the links below.
- TransFusion: transfusion.onnx
The model was trained in TIER IV's internal database (~11k lidar frames) for 20 epochs.
Changelog#
(Optional) Error detection and handling#
(Optional) Performance characterization#
References/External links#
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu and Chiew-Lan Tai. "TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers." arXiv preprint arXiv:2203.11496 (2022).
[2] https://github.com/wep21/CUDA-TransFusion
[3] https://github.com/open-mmlab/mmdetection3d
[4] https://github.com/open-mmlab/OpenPCDet
[5] https://www.nuscenes.org/nuscenes