Siplab wristband

Bachelor Thesis

Benchmarking Deep Learning-Based Pose Estimation from Sparse IMUs

Source description

These are the interesting subdirectories:

/data

Insert needed datasets here. It could be, that this folder needs to be created first.

data
├── smplx_models
│   └── smpl
│       └── SMPL_FEMALE.npz
│       └── ...
├── AMASS
│   └── ACCAD
│       └── ...
├── ...

/src/experimental

Contains all evaluated methods, and under /pip also the evaluate.py and preprocess.py used for the benchmark.

/src/platform

Contains the code for the platform, seperated in /frontend and /backend.

The server is located in /src/platform/backend/src/index.ts, and the default submissions directory in /src/platform/backend/submissions.

/src/tracking

Each subdirectory has its own README's.

• /uwb_wearables/recording/processing. uwb_wearables in general contains all the code for hardware and firmware of the commodity sensors. The recording/processing directory in particular contains the code to process the raw uwb.csv and imu.csv, including the orientation estimator.
• /processing contains all the post-processing. It also contains info.csv and shape.csv if you need the cut intervals, beta shape parameters or similar.
• VirtualCaliper is the program used to generate the SMPL shape parameters.

Execution Environments

Docker for backend evaluation

Do one of these things:

1. Setup or rebuild: docker-compose build

2. To create a non-persistent session to execute the evaluation: docker-compose run --rm --remove-orphans pip-env

3. Optional: If you want to be sure, that it builds correctly after changes, do: docker-compose build --no-cache

Docker for Visualization with aitviewer

Same as for the backend, but use vis-env. e.g. docker-compose run --rm --remove-orphans vis-env. Note that the visualizer.py files need to be run with xvfb-run -s "-screen 0 1280x720x24" python path/to/visualizer.py, as the docker does not has access to the display.

Conda/Micromamba

We have the following environments: uip, pip-env, mocap. You can find them under

• /pip_env.yml. Please refer to /pip_env_README.md and worst case /docker/pip-env/Dockerfile in the comments for an installation guide. The setup for rbdl was not easy. Works with python 3.7.
• /mocap_env.yml
• /src/experimental/UIP/environment.yml. As this python version differs, rbdl might has to be re-build. Works with python 3.8.17.

Execution of Benchmark and Methods

Running evaluation of Benchmark

Use pip-env. Be in project root. These scripts automatically evaluate all .pt files named as the datasets that are in a folder. You always have to provide the path of the folder and not the files for this evaluation.

Preparation:

• Have the predictions ready, and files named uip_db.pt, totalcapture.pt, amass.pt, gip_person1.pt, gip_perspn2.pt, hidden.pt in folder/ - no substructure.
• Have the files config.json (optional but recommended), submission_info.json (really optional, used for the display of information on the website).

Example config.json:

{
    "eval_tran": true
}

Run the evaluation of the benchmark:

python ./src/experimental/pip/evaluate.py eval --path <path_to_folder>

If you also run GIP (as its predicted as two single person), or want to re-evaluate the overall scores, additionally do this to calculate the overall scores.

python ./src/experimental/pip/evaluate.py overall --path <path_to_folder>

The result will be in folder/evaluation.json. The first part shows the results per dataset per sequence, and the last rows ("overall") have the final results. More details can be found in the Additional Information section.

For clearity: An example file structure

/.../submissions/1_PIP
    ├── amass.py
    ├── totalcapture.pt
    ├── ...
    ├── config.json
    ├── submission_info.json

Tip: If you only want to test things and run different versions, create an archive folder inside your actual evaluation folder. As the backend only searches for the exact files in the top level, they are not evaluated. E.g. folder/totalcapture.pt will be evauated, but folder/archive/totalcapture.pt will not.

Running PIP method

Uses environment "pip-env".

Go into project root. Preprocess:

python src/experimental/pip/preprocess.py

Evaluating PIP on a method:

python src/experimental/pip/evaluate.py predict --dataset <dataset>

with dataset in [hidden, totalcapture, uip_db, amass, gip_person1, gip_person2]

Get the output in a unified format. It creates an output file in the project root that has to be moved where desired.

python src/experimental/utils/change_pt_file.py correct_pip_output <path-to-results-file.pt>

Running UIP method

Use environment "uip" from src/experimental/UIP/environment.yml file, I used python 3.8.17. Go into /src/experimental/UIP

For preprocessing:

python modules/dataset/preprocess.py

For evaluating UIP on a dataset:

python modules/evaluate/evaluator.py --network UIP\
                    --ckpt_path data/processed/UIP/pretrained.pt\
                    --data_dir data/processed/UIP/ground_truth_datasets/HiddenDataset\
                    --eval_trans\
                    --normalize_uwb\
                    --add_guassian_noise\
                    --model_args_file config/model_args.json\
                    --eval_save_dir data/processed/UIP/evaluation_res/HiddenDataset

To change the UIP output from the per-sequence based file to one file for benchmark evaluation:

python src/experimental/utils/change_pt_file.py correct_uip_output <path/to/uip-output-folder> <path/to/gt-file-for-betas-gender>

# Or specifically (notice that UIP creates folders with spaces, do not forget the "")
python src/experimental/utils/change_pt_file.py correct_uip_output "/local/home/**/data/processed/UIP/result/HiddenDataset/Ultra Inertial Poser" /local/home/**/data/processed/ground_truth_datasets/hidden_test_gt.pt

This produces an output, that has to be renamed and put into one submission folder for evaluation.

Check out the official UIP GitHub README for open questions.

Running PNP method

Use pip-env environment, same as in PIP method.

For a pleasant experience, start in project root. Comment the datasets you want to use in the process.py before execution. Preprocessing

python src/experimental/PNP/process.py

Prediction: Comment in/out the datasets you want to run in test.py before execution.

cd src/experimental/PNP
python test.py

Get the output in a unified format. It creates an output file in the project root that has to be moved where desired.

python src/experimental/utils/change_pt_file.py correct_pnp_output <path-to-results-file.pt>

Utility Functions

Visualize

To visualize a sequence, and/or ground truth to it, use this:

python ./tracking/processing/vis.py --input <gt-file.pt> --output <prediction.pt> --seq 1 --second-input <gip-person2.pt> --origin

• All args are optional
• --input expects a ground truth file, specifically a file in which there is pose of shape [N, T, 24, 3].
• --output expects a prediction file, specifically a file which has keys 0, ..., N-1 and where each i has a pose of shape [T, 24, 3].
• --seq The sequence ID to show
• --second-input same as --input, but especially useful for visualization of a second person like in GIP-DB.
• --origin to reset origin in x and z axis (no height).

Platform

Can be found in /src/platform, seperated into frontend and backend.

To test it:

cd /src/platform/frontend
ng serve

cd /src/platform/backend
npm run dev

Additional information

Evaluation Metrics Schema

This is the template for the evaluation.json file, generated for each method. All of the following values can be null/nan.

{ sequence_metrics: { [dataset: string]: { [sequenceId: number]: { frames: number, tran_win_count: number, angle_mean: number, angle_std: number, sip_mean: number, sip_std: number, joint_pos_mean: number, joint_pos_std: number, jitter_mean: number, jitter_std: number, abs_jitter_mean: number, abs_jitter_std: number, drift_mean: number, drift_std: number, foot_pen_mean: number, foot_pen_std: number, foot_sliding_mean: number, mpjve_mean: number, mpjve_std: number, trans_2m_mean: number, trans_5m_mean: number, }, ... } }, overall: { [dataset: string]: { frames: number, angle: number, sip: number, joint_pos: number, jitter: number, abs_jitter: number, drift: number, foot_pen: number, foot_sliding: number, mpjve: number, trans_2m: number, trans_5m: number, }, ... } }

Submission Info (saved on disk)

This is generaetd for each method from the server and optional, it defines the public information for each submission.

{ shortName: string, version: string, fullName: string, pubTitle: string, pubAuthors: string, pubVenue: string, pubUrl: string, description: string, hardware: string, parameters: string, availability: string, availabilityUrl: string, status: string, contactEmail: string }

License and Acknowledgement

This project is released under the MIT license. Our code is partially based on PIP.

Hidden Dataset

Recorded at SIP Lab, Computer Science Department, ETH Zurich, Switzerland.

General Information

The sequences are numbered AA01, AA02, AA03 where AA is the subject identifier. Not all subjects have all three sequences.

File Structure

Main directory

In the main directory are the test.pt and train.pt, which contain the dataset with the synthetic magnetometer variant. This is the default version of this dataset. pose is stored in axis angle format. Note that tran does not start at origin to ensure compatibility for the camera feed.

test.pt has the following keys:

acc, ori, betas, gender, vuwb, offset (= None)

train.pt has the following keys:

acc, ori, betas, gender, vuwb, offset (= None), pose, tran

Videos

This directory contains all the videos for each subject. Note that not the subject (TM**) does not have videos, and for some sequences only 3 of 4 cameras worked. The subfolders are front, back, left, right indicating from where you usually see the person from. As most participants looked at the instructor, front is the default viewing direction for most people, the other directions are derived from that. The videos are pre-cut, but you have to do the exact frame-synchronization.

Extras

In the Extras folder, one can find additional versions of the dataset, with more data access for advanced processing. It contains test_no-mag.pt and train_no-mag with the same test and train data and keys, but where the orientation estimation is done without magnetometer readings. This leads to a significantly higher orientation error and more difficulty to estimate pose from.

It also contains the sequences with the following raw data. We recommend checking out the code and how it was generated, before using them. Wo = World frame of OptiTrack, so in which it was recorded. Wx= World frame of Xsens, where the pose originates. vis = Visualization/export frame (Y-up), this is the system in which everything like pose is exported by default if nothing is mentioned otherwise. Check out the paper "Benchmarking Deep Learning-Based Pose Estimation" for a more detailed definition of the coordinate systems etc.

"pose": to_numpy(pose_vis),  # (N, 72)
"pose_wo": to_numpy(pose_wo),  # (N, 72) in wo frame
"pose_unoptimized_vis": to_numpy(archive_pose_vis),  # (N, 72) in vis frame, but no optimization"
"pose_raw_xsensframe": to_numpy(data1["poses"][begin_idx:end_idx+1]),  # (N, 72) in Xsens frame

"tran": to_numpy(tran_vis),  # (N, 3), Optitrack Translation in vis frame
"tran_wo": to_numpy(tran_wo),  # (N, 3), Optitrack Translation in wo frame
"tran_unoptimized_vis": to_numpy(archive_tran_vis),  # (N, 3) OptiTrack translation in vis frame, but no optimization
"tran_raw_xsensframe": to_numpy(data1["trans"][begin_idx:end_idx+1]),  # (N, 3) in Xsens frame

# sensor readings
"rot":  to_numpy(rot_vis),            # (N, S, 3, 3) rotmat
"rot_wo":  to_numpy(rot_wo),            # (N, S, 3, 3) rotmat in wo frame
"acc":  to_numpy(acc_vis),            # (N, S, 3)
"acc_wo":  to_numpy(acc_wo),            # (N, S, 3) in wo frame
"acc_sensor":  to_numpy(acc_sensor),            # (N, S, 3) sensor frame acc
"acc_gtrotated": to_numpy(acc_vis_opt),  # (N, S, 3) # acc rotated with optitrack rot to remove sensor drift
"uwb":  to_numpy(uwb),               # (N, S, S), sensor UWB, frame independant

# OptiTrack ground truth
'opt_rot': to_numpy(opt_rot_vis),     # (N, S, 3, 3) rotmat
'opt_rot_wo': to_numpy(opt_rot_wo),     # (N, S, 3, 3) rotmat in wo frame
'opt_rot_wo_raw': to_numpy(archive_opt_rot_wo),     # (N, S, 3, 3) rotmat in wo frame, but no filtering
'opt_acc': to_numpy(opt_acc),        # (N, S, 3)