Scenario 9
License
If you want to use the dataset or scripts on this page, please use the link below to generate the final list of papers that need to be cited.
Overview
Scenario 8 emulates a Vehicle-to-Infrastructure (V2I) mmWave communication setup. The adopted testbed comprises of three units. Unit 1 primarily consists of a stationary base station equipped with an RGB camera, a LiDAR sensor, an FMCW radar, and a mmWave phased array. The stationary unit adopts a 16-element 60GHz-band phased array and it receives the transmitted signal using an over-sampled codebook of 64 pre-defined beams. The second unit (Unit 2) is a mobile vehicle unit equipped with a mmWave transmitter and GPS receiver. The transmitter consists of a quasi-omni antenna constantly transmitting (omnidirectional) at the 60 GHz band. The scanning range of the LiDAR is 16 meters and the motor spin frequency is 10Hz. The radar consists of 3 Tx and 4 Rx antennas and captures data at 10 FPS. The operational frequency range is 76 to 81 GHz with 4 GHz bandwidth. It has a maximum range of ~100 meters with a range resolution of 60 cm making it highly suitable for this vehicular-communication application. Each data sample has multiple modalities including an RGB image, a LiDAR 360-degree point cloud, and FMCW radar data both collected by Unit 1. Please refer to the detailed description of the testbed presented here.
McAllister Ave: The testbed is deployed during the daytime at McAllister avenue, the same street as that of Scenarios 1, 2, and 5, but a little south to other locations. It is a two-way street with 2 lanes, a width of 10.6 meters, and a vehicle speed limit of 25mph (40.6 km per hour. The stationary unit is placed close to the entrance of a vehicle parking lot. This location has vehicle traffic and also footfall and cycling traffic. Vehicles can be seen driving through the street or driving into or out of the parking structure. All of these features make this scenario diverse from visual and wireless perspectives alike.
Collected Data
Overview
Number of Data Collection Units: 2 (using DeepSense Testbed #1)
Number of Data Samples: 5964
Data Modalities: RGB images, 64-dimensional received power vector, GPS locations
Average Data Capture Rate: 7.67 FPS
Sensors at Unit 1: (Stationary Receiver)
- Wireless Sensor [Phased Array]: A 16-element antenna array operating in the 60 GHz frequency band and receives the transmitted signal using an over-sampled codebook of 64 pre-defined beams
- Visual Sensor [Camera]: The main visual perception element in the testbed is an RGB-D camera. The camera is used to capture RGB images of 960×540 resolution at a base frame rate of 30 frames per second (fps)
- Position Sensor [GPS Receiver]: A GPS-RTK receiver for capturing accurate real-time locations for the stationary unit.
- LiDAR Sensor [2D laser scanner]: This system provides the range and angle data of objects within its range of view. This system covers a maximum range of 16 meters around the master component and it operates at a maximum scan rate of 10 Hz.
- MmWave radar sensor [Ranging sensor]: For the Frequency Modulated Continuous Wave (FMCW) radar each sample contains a 3D complex radar frame samples of (# of RX antennas) x (# of samples per chirp) x (# of chirps per frame).
- Active TX antennas: 1
- Active RX antennas: 4
- # of samples per chirp: 256
- # of chirps per frame: 128
- ADC sampling rate: 5000 Ksps
- Chirp slope: 15.015 MHz/us
- Chirp Start Frequency: 77 GHz
- Ramp end time: 60 us
- ADC start time: 6 us
- Idle time: 5 us
- Receiver gain: 30dB
- Radar frames per second: 10
Sensors at Unit 2: (Mobile Transmitter)
- Position Sensor [GPS Receiver]: A GPS-RTK receiver installed on the top of the mobile unit and is used to capture accurate real-time locations at 10 frames per seconds (fps). The collected data comprises the Latitude and Longitude information in addition to other important data. also
| Testbed | 1 |
|---|---|
| Instances | 5964 |
| Number of Units | 2 |
| Data Modalities | RGB images, 64-dimensional received power vector, GPS locations, LiDAR point cloud, Radar data |
| Unit1 | |
| Type | Stationary |
| Hardware Elements | RGB camera, mmWave phased array receiver, GPS receiver, LiDAR, mmWave Radar System |
| Data Modalities | RGB images, 64-dimensional received power vector, GPS locations, LiDAR point cloud, Radar data |
| Unit2 | |
| Type | Mobile |
| Hardware Elements | mmWave omni-directional transmitter, GPS receiver |
| Data Modalities | GPS locations |
Data Visualization
Download
Please login to download the DeepSense datasets
How to Access Scenario 9 Data?
Step 1. Download Scenario 9 Data
Step 2. Extract the scenario9.zip file
Scenario 9 folder consists of three sub-folders:
- unit1: Includes the data captured by unit 1
- unit2: Includes the data captured by unit 2
- resources: Includes the scenario-specific annotated dataset, data labels and other additional information. For more details, refer the resources section below.
Scenario 9 folder also includes the “scenario 9.csv” file with the paths to all the collected data. For each coherent time, we provide the corresponding visual, wireless and GPS data.
Resources
What are the Additional Resources?
Resources consist of the following information:
- visual data annotations: For the visual data, we provide the coordinates of the 2D bounding box and attributes for each frame
- data labels: The labels comprises of the ground-truth beam indices computed from the mmWave received power vectors, the direction of travel (unit2), and the sequence index
- additional information: Includes the scenario-specific additional data. Details of the information is provided below
Visual Data Annotations
After performing the post-processing steps presented here, we generate the annotations for the visual data. Using state-of-the-art machine learning algorithms and multiple validation steps, we achieve highly accurate annotations. In this particular scenario, we provide the coordinates of the 2D bounding box and attributes for each frame. We, also, provide the ground-truth labels for 2 object classes, “Tx”, and “Distractor”. The “Tx” refers to the transmitting vehicle in the scene and “Distractor” for any other objects, such as human, other vehicles, etc. We follow the YOLO format for the bounding-box information. In the YOLO format, each bounding box is described by the center coordinates of the box and its width and height. Each number is scaled by the dimensions of the image; therefore, they all range between 0 and 1. Instead of category names, we provide the corresponding integer categories. We follow the following assignment: (i) “Tx” as “0” , and (ii) “Distractor” as “1”.
Data Labels
The labels comprises of the ground-truth beam indices computed from the mmWave received power vectors, the direction of travel (unit2), and the sequence index.
- Ground-Truth Beam: The phased array of unit 1 utilizes an over-sampled beamforming codebook of N = 64 vectors, which are designed to cover the field of view. It captures the received power by applying the beamforming codebook elements as a combiner. For each received power vector of dimension [64 x 1], the index with the maximum received power value is selected as the optimal beam index. This data is provided in the column 11 [‘unit1_beam_index’] of the scenario9.csv
- Sequence Index: During the data collection process, the mobile transmitter (unit2) travelled multiple times in front of the base station (unit1). For each run, the testbed collects multiple data samples. All the data samples with the same sequence index belongs to the same run
- Direction of Travel: For this scenario, during the data collection process, the mobile unit generally moves either from left-to-right or right-to-left of the base station. Here “0” represents the left-to-right movement of the transmitter and “1” represents the “right-to-left” movement. The movement is measured is from the point of view of the base station. This is provided in the ‘unit2_direction’ column of the csv
Additional Information
We, further, provide additional information for each sample present in the scenario dataset. The details are provided in the columns 12 – 20 of the scenario9.csv. The contents of the additional data is listed below:
- index: It represents the sample number
- time_stamp[UTC]: This represents the time of data capture in “hr-mins-secs-ms” format
- unit2_num_sat: For each data sample, it is an integer value representing the number of connected satellites at that time instant
- unit2_sat_used: At each time instant, these were the satellites that the receiver was connected
- unit2_fix_type: This shows whether or not there was a 3D fix. A 3D (three dimensional) position fix includes horizontal coordinates plus altitude. It requires a minimum of four visible satellites
- unit2_DGPS: Binary indicator representing whether or not there was Differential GPS was used
- unit2_PDOP: PDOP (position dilution of precision) describes the error caused by the relative position of the GPS satellites
- unit2_HDOP: HDOP represents the horizontal dilution of precision
An example table comprising of the data labels and the additional information is shown below.
| index | unit1_beam_index | seq_index | time_stamp[UTC] | unit2_direction | unit2_num_sat | unit2_sat_used | unit2_fix_type | unit2_DGPS | unit2_PDOP | unit2_HDOP |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 1 | ['16-45-05-799'] | 0 | 13 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 | 3D | Yes | 3.3 | 1.2 |
| 2 | 2 | 1 | ['16-45-05-910'] | 0 | 13 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 | 3D | Yes | 3.3 | 1.2 |
| 3 | 2 | 1 | ['16-45-06-022'] | 0 | 17 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 S123 E4 E9 E11 | 3D | Yes | 2.8 | 1 |
| 4 | 2 | 1 | ['16-45-06-132'] | 0 | 17 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 S123 E4 E9 E11 | 3D | Yes | 2.8 | 1.2 |
| 5 | 3 | 1 | ['16-45-06-243'] | 0 | 17 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 S123 E4 E9 E11 | 3D | Yes | 1.8 | 0.9 |
| 6 | 3 | 1 | ['16-45-06-354'] | 0 | 17 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 S123 E4 E9 E11 | 3D | Yes | 1.8 | 0.9 |
| 7 | 6 | 1 | ['16-45-06-465'] | 0 | 17 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 S123 E4 E9 E11 | 3D | Yes | 1.8 | 0.9 |
| 8 | 6 | 1 | ['16-45-06-576'] | 0 | 17 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 S123 E4 E9 E11 | 3D | Yes | 1.9 | 0.9 |
| 9 | 9 | 1 | ['16-45-06-687'] | 0 | 17 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 S123 E4 E9 E11 | 3D | Yes | 1.9 | 0.9 |
| 10 | 10 | 1 | ['16-45-06-798'] | 0 | 17 | G8 G10 G18 G23 G24 G27 G32 R1 R7 R8 R21 R22 R23 S123 E4 E9 E11 | 3D | Yes | 1.9 | 0.9 |
