MiliPoint (A Point Cloud Dataset for mmWave Radar)

Magnemite (Japanese: コイル Coil) is a dual-type Electric/Steel Pokémon introduced in Generation I. Prior to Generation II, it was a pure Electric-type Pokémon. This Pokemon specialises in generating waves.

For more, you can watch this video.

Dataset Download and Preparation

Clone the project.

git clone https://github.com/yizzfz/MiliPoint.git
cd MiliPoint

Download MiliPoint_data.zip from Google drive, unzip all contents to data/raw/:

MiliPoint
└─data
  └─raw
    ├─1.pkl
    ├─2.pkl
    ├─...

The actions involved in this study are shown in this video.

Simple Usage: Using only the Dataset

We provide a direct pip installation by running the following

${CUDA} should be replaced by either cpu, cu116, or cu117 depending on your PyTorch installation.

python -m pip install torch-cluster -f https://data.pyg.org/whl/torch-2.0.0+${CUDA}.html
python -m pip install .

Make a directory mkdir data/processed/mmr_kp, the following code shows how to run a dataset using the pip-installed package.

from mmrnet.dataset import get_dataset
from mmrnet.models import model_map

import torch

dataset_config = {
		'seed': 20,
		'train_split': 0.8,
		'val_split': 0.1,
		'test_split': 0.1,
		'stacks': 3,
		'zero_padding': 'per_data_point'
}
loaders = get_dataset(
		'mmr_kp',
		batch_size=32,
		workers=0,
		mmr_dataset_config=dataset_config)
train_loader, val_loader, test_loader, info = loaders
# each dataset element in the loader is a tuple of (data, label)

print(train_loader.dataset[0][0].shape)
# Input point cloud size: torch.Size([66, 3])

print(train_loader.dataset[0][1].shape)
# Output keypoint size torch.Size([9, 3])

Using the Datasets, Models and our Training and Evaluation Frameworks

Installation by ./install_mmrnet.sh, then you can follow the commands recorded below to use different functionalities.

Commands

Format: ./mm <train/test> <task> <model> <-args>

Replace ./mm with python mm if using Windows.

Train

Example Training run

 ./mm train mmr_kp dgcnn
 	# cli argument for the saving directory
 	\ --save session_name 
 	# A config file that defines how to construct a dataset
 	\ -config ./configs/keypoints/mmr_keypoints_stack_3_point.toml
 	# to use GPU
 	\ -a gpu

The config file defines the dataset building, for example ./configs/keypoints/mmr_keypoints_stack_5_point.toml is:

  seed = 20
  processed_data = 'data/processed/mmr_keypoints/stacks_3_padd_point_task_keypoints.pkl'
  train_split = 0.8
  val_split = 0.1
  test_split = 0.1
  stacks = 3
  zero_padding = 'per_data_point'

These parameters will change the parameters in the dataset class (e.g. MMRKeypointData).

Command line arguments have a higher priority
```
 ./mm train mmr mlp --save session_name -config ./configs/keypoints/mmr_keypoints_stack_3_stack.toml --dataset_stacks 5
```
- In this case, stacks would be set to be 5 instead of 3 (defined in .toml), because command line arguments have a higher priority.

Test

./mm eval mmr_kp dgcnn
	\ --load ckpt_name 
	\ -config ./configs/keypoints/mmr_keypoints_stack_3_stack.toml
	# to visualize test result as a video, only applicable for keypoint
	\ -v

Dataset hyperparameters

--dataset_stacks (in cli) or stacks (in toml): this defines how many previous frames we stack together with the current frame.
--dataset_zero_padding (cli) or zero_padding (toml): there are two styles of padding: 1) pad per frame (--dataset_zero_padding data_point) or 2) pad per stack (--dataset_zero_padding stack)
--dataset_num_keypoints (cli) or num_keypoints (toml): the number of keypoints used to represent the human skeleton. low uses 9 points (left and right shoulders, elbows, waists, knees, and a head), high uses 18 points (the COCO18 format).

processed_data: this declares the data path for storing the processed data file, if the file exists, the processing step will be skipped. However, you can use -d_forced_rewrite or --dataset_forced_rewrite to force to rewrite the cached file.

The rest hyperparameters are fairly straightforward and are in mmr_keypoints.toml

Models

MLP: simple network with dense layers only.
DGCNN
PointNet++
PointTransformer
PointMLP

The performance of these models are included in the paper.

Tasks

mmr_kp - Keypoint estimation

X: Point cloud (n, 3), Y: The 18 keypoints of the subject.

Train command example:

 python mm train mmr_kp dgcnn --save mmr_dgcnn -config ./configs/keypoints/mmr_keypoints_stack_5_point.toml -w 0 -a gpu -m 300 -n -1

Test command example:

 python mm test mmr_kp dgcnn --load mmr_dgcnn -config ./configs/keypoints/mmr_keypoints_stack_5_point.toml -w 0 -a gpu

mmr_iden - Identification

X: Point cloud (n, 3), Y: The subject id (1 - 18).

Train command example:

 python mm train mmr_iden dgcnn --save mmr_iden_dgcnn -config ./configs/iden/mmr_iden_stack_5_point.toml -w 0 -a gpu -m 300 -n -1

Test command example:

 python mm test mmr_iden dgcnn --load mmr_iden_dgcnn -config ./configs/iden/mmr_iden_stack_5_point.toml -w 0 -a gpu

mmr_act - Action classification

X: Point cloud (n, 3), Y: The action id.

Train command example:

 python mm train mmr_act dgcnn --save mmr_act_dgcnn -config ./configs/act/mmr_act_stack_5_point.toml -w 0 -a gpu -m 300 -n -1

Test command example:

 python mm test mmr_act dgcnn --load mmr_act_dgcnn -config ./configs/act/mmr_act_stack_5_point.toml -w 0 -a gpu

Ciation

If you find this work useful, please consider citing our paper:

@inproceedings{cui2023milipoint,
 author = {Cui, Han and Zhong, Shu and Wu, Jiacheng and Shen, Zichao and Dahnoun, Naim and Zhao, Yiren},
 booktitle = {Advances in Neural Information Processing Systems},
 pages = {62713--62726},
 title = {MiliPoint: A Point Cloud Dataset for mmWave Radar},
 volume = {36},
 year = {2023}
}