/NEAR-VI-Dataset

NEAR: The NetEase AR Oriented Visual Inertial Dataset

Abstract

The existing datasets for evaluating Visual Inertial Odometry (VIO) have boosted the research of autonomous agents, but they don’t meet the prosperous research of Augmented Reality (AR) or Mixed Reality (MR) given that they are not collected at real AR scenes and do not account for affecting factors of mobile devices. This paper presents the NEAR dataset, an AR oriented visual-inertial dataset collected with commodity handheld phones with ground truthrts. The dataset has a total of 113 sequences in 49 elaborately designed collection cases at two typical indoor scenes, i.e. the living area and the table area. It also covers plenty of setting adjustments for comparison, including the comparisons of different level textures, illuminations, motion patterns, camera settings and the difference between the rolling shutter and the global shutter.

If you wanna acknowledge our work, please cite

  1. Cheng Wang, Yu Zhao, Jiabin Guo, Ling Pei, Yue Wang, Haiwei Liu. NEAR: The NetEase AR Oriented Visual Inertial Dataset. [Accepted by ISMAR 2019].

The dataset link is here.

Case design

case environment scene lighting motion pattern special device setting devices and link preview
1 Easy[1] Rich-textured desktop uniform & stable circle (slow+normal) default[5] iPhone 7, Huawei P20 001.gif
2 Easy Rich-textured desktop uniform & stable hybrid default iPhone 7, Huawei P20 002.gif
2_control[4] Easy Rich-textured desktop uniform & stable hybrid default iPhone 7, Huawei P20, MYNT 002.gif
3 Easy Rich-textured desktop uniform & stable circle (alwayse points to center) default iPhone 7, Huawei P20 003.gif
3_control Easy Rich-textured desktop uniform & stable circle (alwayse points to center) default iPhone 7, Huawei P20, MYNT 003.gif
4 Easy Medium-textured desktop uniform & stable circle (slow+normal) default iPhone 7, Huawei P20 004.gif
5 Easy Medium-textured desktop uniform & stable hybrid default iPhone 7, Huawei P20 005.gif
6 Easy Medium-textured desktop with reflection uniform & stable circle (slow+normal) default iPhone 7, Huawei P20 006.gif
7 Easy Medium-textured desktop with reflection uniform & stable hybrid default iPhone 7, Huawei P20 007.gif
8 Easy Rich-textured livingroom uniform & stable translation (slow+normal) default iPhone 7, Huawei P20 008.gif
9 Easy Rich-textured livingroom uniform & stable circle (slow+normal) default iPhoneXR_OnePlus5T_part1
iPhoneXR_OnePlus5T_part2		 009.gif
9_control Easy Rich-textured livingroom uniform & stable circle (slow+normal) default iPhoneXR_OnePlus5T_MYNT_part1
iPhoneXR_OnePlus5T_MYNT_part2		 009.gif
10 Easy Rich-textured livingroom uniform & stable hybrid default iPhone XR, OnePlus 5T 010.gif
10_control Easy Rich-textured livingroom uniform & stable hybrid default iPhone XR, OnePlus 5T, MYNT 010.gif
11 Easy Rich-textured livingroom uniform & stable user scan simulation default iPhoneXR_OnePlus5T_part1
iPhoneXR_OnePlus5T_part2		 011.gif
11_control Easy Rich-textured livingroom uniform & stable user scan simulation default iPhoneXR_OnePlus5T_MYNT_part1
iPhoneXR_OnePlus5T_MYNT_part2		 011.gif
12 Normal[2] Rich-textured desktop uniform & stable circle (slow+normal+fast) default iPhone XR, OnePlus 5T 012.gif
13 Normal Rich-textured desktop uniform & stable hybrid default iPhone XR, OnePlus 5T 013.gif
14 Normal Rich-textured desktop uniform & stable free default iPhone XR, OnePlus 5T 014.gif
15 Normal Rich-textured desktop uniform & stable game simulation (YuMe) default iPhone XR, OnePlus 5T 015.gif
16 Normal Medium-textured desktop uniform & stable circle (slow+normal+fast) default iPhone XR, OnePlus 5T 016.gif
17 Normal Medium-textured desktop uniform & stable hybrid default iPhone XR, OnePlus 5T 017.gif
18 Normal Medium-textured desktop uniform & stable free default iPhone XR, OnePlus 5T 018.gif
19 Normal Medium-textured desktop uniform & stable game simulation (YuMe) default iPhone XR, OnePlus 5T 019.gif
20 Normal Medium-textured desktop dynamic circle (slow+normal+fast) default iPhone XR, OnePlus 5T 020.gif
21 Normal Medium-textured desktop dynamic hybrid default iPhone 7, Huawei P20 021.gif
22 Normal Medium-textured desktop dynamic free default iPhone 7, Huawei P20 022.gif
23 Normal Medium-textured desktop with reflection uniform & stable circle (slow+normal+fast) default iPhone 7, Huawei P20 023.gif
24 Normal Medium-textured desktop with reflection uniform & stable hybrid default iPhone 7, Huawei P20 024.gif
25 Normal Medium-textured desktop with reflection uniform & stable free default iPhone 7, Huawei P20 025.gif
26 Normal Median-textured livingroom uniform & stable translation (slow+normal+fast) default iPhone 7, Huawei P20 026.gif
27 Normal Median-textured livingroom uniform & stable translation (slow+normal+fast) 30fps iPhone 7, Huawei P20 027.gif
28 Normal Median-textured livingroom uniform & stable translation (slow+normal+fast) autofocus iPhone 7, Huawei P20 028.gif
29 Normal Median-textured livingroom uniform & stable circle (slow) default iPhone XR, OnePlus 5T 029.gif
30 Normal Median-textured livingroom uniform & stable circle (normal) default iPhone XR, OnePlus 5T 030.gif
31 Normal Median-textured livingroom uniform & stable circle (fast) default iPhone XR, OnePlus 5T 031.gif
32 Normal Median-textured livingroom uniform & stable circle (normal) 30fps iPhone XR, OnePlus 5T 032.gif
33 Normal Median-textured livingroom uniform & stable circle (normal) autofocus iPhone XR, OnePlus 5T 033.gif
34 Normal Median-textured livingroom uniform & stable hybrid default iPhone XR, OnePlus 5T 034.gif
35 Normal Median-textured livingroom uniform & stable hybrid 30fps iPhone XR, OnePlus 5T 035.gif
36 Normal Median-textured livingroom uniform & stable hybrid autofocus iPhone XR, OnePlus 5T 036.gif
37 Normal Median-textured livingroom uniform & stable free default iPhone XR, OnePlus 5T 037.gif
38 Normal Median-textured livingroom uniform & stable user scan simulation default iPhone XR, OnePlus 5T 038.gif
39 Normal Median-textured livingroom uniform & stable game simulation (Shot) default iPhone 7, Huawei P20 039.gif
40 Normal textureless floor and furnitures uniform & stable translation (slow+normal+fast) default iPhone 7, Huawei P20 040.gif
41 Normal textureless floor and furnitures uniform & stable circle (slow) default iPhone 7, Huawei P20 041.gif
42 Normal textureless floor and furnitures uniform & stable circle (normal) default iPhone 7, Huawei P20 042.gif
43 Normal textureless floor and furnitures uniform & stable circle (fast) default iPhone 7, Huawei P20 043.gif
44 Normal textureless floor and furnitures uniform & stable hybrid default iPhone 7, Huawei P20 044.gif
45 Normal textureless floor and furnitures uniform & stable free default iPhone 7, Huawei P20 045.gif
46 Hard[3] Rich-textured desktop non-uniform & stable hybrid default iPhone 7, Huawei P20 046.gif
48 Hard Medium-textured desktop non-uniform & stable hybrid default iPhone 7, Huawei P20 048.gif
50 Hard Medium-textured desktop with reflection non-uniform & stable hybrid default iPhone 7, Huawei P20 050.gif
52 Hard textureless floor and furnitures non-uniform & stable hybrid default iPhone 7, Huawei P20 052.gif

notes:

  1. "Easy" means no repetitive texture, no reflection, no opened windows. Rich-textured carpets, sofas and walls.
  2. "Normal" means a common livingroom settings. Median-textured carpets, sofas and walls. Some opened windows and a mirror.
  3. "Hard" means Opened windows, mirrors, textureless floor, textureless furnitures, textureless walls, lamps.
  4. "_control" means we have an additinal MYNT camera attached to record data besides of the two mobile phones.
  5. default device setting means as below. default_config

The full list of the case design can be downloaded with CaseTableEva.xlsx and the better resolution preview video can be found at NEAR-dataset-preview-video.

For full understand our design, please refer to our paper[1].

Tentative Grouping

We tested some open source VIOs including ICE-BA, VINS-Mono, OKVIS, ROVIO, MSCKF-Mono on our NEAR dataset tentatively. Note that the results are not for benchmarking.

Easy

Medium

Based on the results, we devide our dataset into several groups as below for reference. group

Dataset structure

The sequences are organized following the structure of the EuRoC, so the usage is fair easy.

./Case ID
│
└───ip7                                         A folder with phone name.
     └───P_lp2lv.txt                            The extrinsic between phone camera and rigid body.
     └───rts_compensate_extrinsic_and_time.txt  The ground truth trajectory (TUM format) of camera frame.
     └───timeoffset.txt                         Timestamp offset between phone camera and RTS system.
     │
     └───mav0
          │
          └───cam0
          │    └───data                         The folder contains image sequence.
          │    └───cam.yaml                     Camera settings when record this sequence.
          │    └───data.csv                     Timestamps of camera frames.
          │    └───sensor.csv                   Camera intrinsics and camera-IMU extrinsic.
          └───imu0
                └───data.csv                    Timestamps and IMU measurements.
                └───sensor.yaml                 IMU intrinsics and IMU-rig extrinsic which is identity.