prasenjit52282/OpenCamera-Sensors

Android app for synchronized recording of video and IMU data on one or multiple smartphones with advanced camera options, useful for 3D reconstruction, SLAM, AR, video stabilization. Supports remote control over network.

Note: this is the main repository of OpenCamera Sensors. The old repository is now archived and will not receive any updates.

OpenCamera Sensors is an Android application for synchronized recording of video and IMU data on one or multiple smartphones. It records sensor data (accelerometer, gyroscope, magnetometer) and video with frame timestamps synced to the same clock.

This project is based on Open Camera — a popular open-source camera application with flexibility in camera parameters settings, actively supported by the community. By regular merging of Open Camera updates our app will adapt to new smartphones and APIs — this is an advantage over the other video + IMU recording applications built from scratch for Camera2API.

Related papers

If you use OpenCamera Sensors for your research consider citing these papers:

• A. Akhmetyanov, A. Kornilova, M. Faizullin, D. Pozo and G. Ferrer, "Sub-millisecond Video Synchronization of Multiple Android Smartphones," 2021 IEEE Sensors, 2021, pp. 1-4
• Faizullin, M.; Kornilova, A.; Akhmetyanov, A.; Ferrer, G. Twist-n-Sync: Software Clock Synchronization with Microseconds Accuracy Using MEMS-Gyroscopes. Sensors 2021, 21, 68

Install

Get OpenCamera Sensors on F-Droid or install the latest APK from GitHub Releases.

Usage

Our app requires full Camera2 API support. Additional restrictions are described in the sections below as needed.

IMU recording

Important: synchronized timestamping for camera and IMU data isn’t available on all the devices with Camera2 API support. You can check whether your device supports this feature in preferences.

• Go to preferences, enable Camera2 API and press the “Enable sync video IMU recording” switch in "IMU settings..."
• (Optional) Disable video stabilization in video preferences of OpenCamera Sensors to minimize preprocessing effects
• (Optional) Enable save frames option if you want to verify recorded data correctness
• (Optional) Enable flash strobe and specify its frequency in additional sensor settings
• Switch to video, setup ISO and exposure time
• Record video
• Get data from DCIM/OpenCamera:
  • Video file
  • Sensor data and frame timestamps in the directory {VIDEO_DATE}:
    • {VIDEO_NAME}_gyro.csv, data format: X-data, Y-data, Z-data, timestamp (ns)
    • {VIDEO_NAME}_accel.csv, data format: X-data, Y-data, Z-data, timestamp (ns)
    • {VIDEO_NAME}_magnetic.csv, data format: X-data, Y-data, Z-data, timestamp (ns)
    • {VIDEO_NAME}_timestamps.csv, data format: timestamp (ns)
    • {VIDEO_NAME}_flash.csv, data format: timestamp (ns) (timestamps of when the flash fired)

Remote recording

• Connect smartphone to the same network as PC
• Use scripts provided in ./api_client/ directory to send requests for the application.
  • Note: phase, which is returned by start_recording method, can be used to perform synchronization with external devices

Synchronized recording on multiple smartphones (RecSync)

Important: smartphones are required to support real-time timestamping to be correctly synchronized. This can be checked on the preview message when RecSync is enabled ("Timestamp source" should be "realtime").

Leader smartphone setup:

• Start a Wi-Fi hotspot
• Open OpenCamera Sensors, go to preferences - "RecSync settings..." and enable the "Use RecSync" switch
• (Optional) Enable phase alignment option if synchronization precision better than half of a frame duration is required
• (Optional) Choose which camera settings will be broadcasted to client smartphones in the "Sync settings" section
• Switch to video, adjust the camera settings as needed and press the settings synchronization button
• Wait for client smartphones to connect if needed
• (Optional) If phase alignment was enabled, press the phase alignment button to start the alignment and wait for it to finish ("Phase error" on the preview indicates how much the current phase differs from the targeted one -- when it becomes green, the phase is considered aligned)
• Start a video recording

Client smartphones setup:

• Connect to the leader's Wi-Fi hotspot
• Open OpenCamera Sensors, go to preferences - "RecSync settings..." and enable the "Use RecSync" switch
• Adjust the camera settings as needed (the ones that will not be broadcast by the leader) and wait for the leader to start the recording

Note: the phase needs to be re-aligned before every recording.

Good practices for data recording

• When recording video with audio recording enabled, MediaRecorder adds extra frames to the video to match the sound. Due to this problem, the audio recording feature is disabled in our app by default.

• To minimize the amount of preprocessing done by the smartphone, we also disable video stabilization and OIS options.

Contribution

The project follows AOSP Java Code Style, main principles, which include the following:

• Non-public fields should start with m, constants are ALL_CAPS_UNDERSCORES
• Standard brace style:

if () {
    // ...
} else {
    // ...
}

• Limit line length