UPose3D
Multi-view 3D human pose estimation that carries an uncertainty for every keypoint from 2D to 3D, scales to any number of cameras, and learns without 3D-labelled video. ECCV 2024.
European Conference on Computer Vision (ECCV) 2024 · Queen’s University and Ubisoft La Forge
Markerless multi-view motion capture sounds like a solved problem. Put a few calibrated cameras around a person, find each joint in every image, and triangulate. In a studio with clean images it works well. The trouble starts when some of the 2D guesses are wrong: a hand hidden behind the body, a left foot labelled as the right one, a camera looking from an unusual angle. Plain triangulation trusts every camera equally, so one bad view drags the whole joint with it.
The usual alternative is to train a network end to end on multi-view video with 3D labels. That works in the setting it was trained on, but 3D-labelled video is scarce outside the lab, and such models tend to break in new rooms, with new people or new camera layouts.
UPose3D[1] is our answer to both problems. It refines each camera's 2D guesses using what the other cameras and the neighbouring frames see, it keeps an uncertainty for every keypoint all the way to the 3D estimate, and it learns all of this from synthetic data made from motion capture, without any 3D-labelled video. The work was led by Vandad Davoodnia at Queen's University and Ubisoft La Forge, and was published at ECCV 2024.
This page walks through the method one step at a time, with interactive figures that simulate each idea using real camera geometry, then shows what the experiments found. The code snippets are simplified Python sketches rather than the actual implementation.
- Overview
- Keypoints With Uncertainty
- One Joint, Every Camera
- The Pose Compiler
- From 2D to 3D
- Training Without 3D Labels
- Results
- Limitations
- Citation
Overview
UPose3D takes synchronised video from any number of calibrated cameras and outputs the 3D position of every joint. Between the two are four steps, and one idea runs through all of them: never throw away how sure you are.
How UPose3D works
Every step keeps an uncertainty next to each keypoint, all the way to the 3D answer
A 2D pose estimator first finds each joint in each image, along with an uncertainty. Every camera's guesses are then projected onto every other camera's view, giving each joint a small cloud of points. The pose compiler reads those clouds across neighbouring frames and outputs refined keypoints with their own uncertainties. Finally, both sets of keypoints and uncertainties go into an optimisation that finds the most likely 3D position of each joint.
Keypoints With Uncertainty
Most 2D pose estimators predict a heatmap per joint and take its peak. UPose3D instead puts a Residual Log-likelihood Estimation (RLE) head[2] on a standard backbone (CPN on Human3.6M, HRNet-W48 for the out-of-distribution experiments). RLE regresses the joint's position directly and, through a small normalising flow, also learns the shape of the error around it.
So for every joint in every image we get a position, an uncertainty, and a probability density we can evaluate anywhere in the image. That last part matters later: it lets the 3D step ask "how likely is it that the joint projects here?" for any candidate 3D point, rather than only comparing against a single best guess.
One Joint, Every Camera
The next step brings the other cameras in. With calibrated cameras, a 2D point in one image doesn't pin down a point in another image, but it does pin down a line: the epipolar line, where that point's ray would appear from the other camera's viewpoint. The true joint must lie on it.
For each joint and each reference camera, UPose3D takes every other camera's guess, draws its epipolar line in the reference image, and keeps the point on that line closest to the reference camera's own guess. Those points, together with the reference guess, form a small 2D point cloud for that joint, normalised by the person's bounding box. Try it below, and then make one camera unreliable:
One joint, seen from every camera
Click a camera on the left to make it the reference view. The image on the right is what it sees.
Capture volume, from above
What camera 1 sees
When the cameras agree, the cloud is a tight cluster around the joint. When one camera is wrong, its point lands visibly far from the rest, which is exactly the kind of pattern a network can learn to discount. The step itself is only a few lines:
def point_cloud(ref, keypoints, cameras):
"""2D point cloud for one joint in reference camera `ref` (Sec. 3.2)."""
cloud = [keypoints[ref]]
for j, cam in enumerate(cameras):
if j == ref:
continue
F = fundamental_matrix(cameras[ref], cam) # from the known calibration
line = F.T @ homogeneous(keypoints[j]) # epipolar line in the reference view
cloud.append(closest_point(line, keypoints[ref]))
return normalise_to_bbox(cloud, ref) Because every other camera adds one more point to the same cloud, the input has the same form whether there are two cameras or ten. That is what lets UPose3D handle any number of cameras without retraining, and it is why we call this a scalable cross-view fusion.
The Pose Compiler
The pose compiler turns these clouds into better keypoints. It has two parts. A point-cloud encoder, inspired by the Point Cloud Transformer[3], turns each joint's cloud into a feature vector. We modified it to keep the coordinates, adding a residual connection from the reference view's features to its output.
A spatiotemporal encoder then looks at those features across all joints and a window of frames (27 in the main model) and predicts each joint's refined position and uncertainty, again with an RLE head. Full attention over every joint in every frame gets expensive fast, so it uses criss-cross attention[4]: each token attends only along its own frame and its own joint, and stacking a few layers lets information reach everywhere.
Criss-cross attention, one token at a time
Each square is one joint in one frame. Hover, tap or use the arrow keys to pick a token and see what it attends to.
Two things make the compiler useful. It sees time, so a joint that flickers in one frame is pulled back by its neighbours. And it sees the clouds, so it can learn which kinds of disagreement mean "this camera is wrong" rather than "the joint moved".
From 2D to 3D
Now we have two sets of 2D keypoints for every joint in every camera: the image branch's and the pose compiler's, each with its full predicted distribution. The 3D step looks for the 3D point whose projections are most likely under all of them. It starts from a plain triangulation (DLT) and refines it with L-BFGS, maximising the combined likelihood:
def estimate_3d(cameras, image_kps, compiler_kps):
X = dlt(cameras, image_kps) # plain triangulation to start
def nll(X):
loss = 0
for cam, a, b in zip(cameras, image_kps, compiler_kps):
u = cam.project(X)
loss -= a.log_prob(u) + b.log_prob(u) # both predicted 2D distributions
return loss
return lbfgs(nll, X) # no 3D labels involved The effect is easiest to see with one bad camera. Plain triangulation weighs every view the same, so the bad one pulls the answer away. When each view comes with its uncertainty, a confident view counts for more and an unsure one for less:
One bad camera, two ways to triangulate
Six cameras look at the left wrist. Push camera 3's guess off and see where the 3D estimate lands.
From above: each camera's ray through its guess
Zoomed in: 30 cm around the true wrist
Plain triangulation off
Weighted by uncertainty off
This is not just a toy effect. In the paper's ablation on Human3.6M, removing uncertainty from both branches raises the error from 26.4 mm to 48.1 mm, and the full method's lead on unseen data, shown below, comes largely from discarding unreliable views.
Training Without 3D Labels
The pose compiler only ever sees 2D point clouds, so it doesn't need images or 3D-labelled video to train. Instead, we make training data from motion capture. Starting from AMASS[5], each sample is built like this:
- Vary the person: add noise to the body shape, mirror the motion half the time, and rotate it randomly.
- Place virtual cameras on a cylinder around them, at realistic heights, each aimed near the middle of the capture volume. Up to 8 cameras per sample.
- Project the 3D joints into each camera to get perfect 2D keypoints, the targets the compiler learns to recover.
- Corrupt them the way real detectors fail, then build the point clouds from the corrupted keypoints.
Making training data without 3D labels
A random virtual camera, the true 2D keypoints (grey), and a corrupted copy for the pose compiler to learn from
Samples are generated on the fly, so every sample has a fresh random camera layout, and training takes about six hours on a single RTX 2080 GPU. Nothing is tied to one dataset's cameras, and the skeleton is just whichever joint regressor is applied to the motion capture: 17 joints for the Human3.6M experiments, 133 whole-body joints for the out-of-distribution ones.
Results
In the lab: Human3.6M
Human3.6M[6] is the standard multi-view benchmark, with four cameras in one studio. Among methods trained with 2D labels only, UPose3D is the most accurate, at 26.4 mm against 30.5 mm for plain triangulation of the same 2D estimator. Methods trained on Human3.6M's own 3D labels, or with its camera parameters during training, are still ahead here (Epipolar Transformers[7] reaches 19.0 mm), which is expected when the test set looks exactly like the training set.
Somewhere new: RICH
The more telling test is a setting none of the models trained on. RICH[8] was recorded in three new environments (a construction site, a gym and a lecture hall) with different people and cameras. Here UPose3D roughly halves the error of the next best method, 34.7 mm against 64.4 mm. AdaFuse[9], one of the strongest methods on Human3.6M, falls apart at 524 mm: it does well only when every view's prediction is reasonable, and occasional noisy views cause large triangulation errors. UPose3D's uncertainty lets it set those views aside.
What matters
The ablations show which parts do the work. The two branches need each other: without the pose compiler the error rises to 37.1 mm, and without the image branch it rises to 69.9 mm. Uncertainty is what makes combining them work: remove it everywhere and the error jumps to 48.1 mm, and with no uncertainty and no compiler we are back to plain triangulation (30.5 mm).
Cost, and more cameras
Because the heavy lifting happens on small point clouds rather than image features, UPose3D is also the cheapest of the multi-view methods compared, at roughly half the compute of the next one with both 4 and 10 cameras. In the paper's breakdown, the pose compiler needs only 0.385 GFLOPs against 508.5 for the 2D backbone. On CMU Panoptic and HUMBI, adding cameras keeps lowering UPose3D's error while plain triangulation and RANSAC level off, and the run time per sample stays roughly constant.
Compute as cameras are added
Billions of operations (GFLOPs) to estimate one frame, with 4 and with 10 cameras; lower is better
Show as a table
| Method | Parameters (M) | 4 cameras (GFLOPs) | 10 cameras (GFLOPs) |
|---|---|---|---|
| UPose3D | 65.4 | 208.7 | 517.7 |
| Epipolar Transformers | 68.1 | 406.5 | 1016.2 |
| MTF-Transformers | 78.6 | 407 | 1017.8 |
| AdaFuse | 69.7 | 595 | 1487.6 |
| Learnable Triangulation | 80.6 | 716.1 | 1326.9 |
Limitations
- Calibrated cameras. The cross-view projection needs each camera's intrinsic and extrinsic parameters.
- Not real time yet. The paper's focus was accuracy; it suggests learned second-order optimisers for the likelihood maximisation as a way to make it fast enough for real-time use.
- Longer windows aren't better. A 243-frame model did worse than the 27-frame one, probably because the synthetic corruptions were tuned for short windows.
- In the lab, 3D supervision still wins. When training and test data match, methods trained on that dataset's 3D labels remain more accurate.
Citation
Vandad Davoodnia, Saeed Ghorbani, Marc-André Carbonneau, Alexandre Messier, Ali Etemad. UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues. ECCV 2024.
@misc{davoodnia2024upose3d,
title = {UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues},
author = {Vandad Davoodnia and Saeed Ghorbani and Marc-André Carbonneau and Alexandre Messier and Ali Etemad},
year = {2024},
eprint = {2404.14634},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2404.14634}
} References
- V. Davoodnia, S. Ghorbani, M.-A. Carbonneau, A. Messier, A. Etemad. UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues. ECCV 2024.
- J. Li, S. Bian, A. Zeng, C. Wang, B. Pang, W. Liu, C. Lu. Human Pose Regression with Residual Log-likelihood Estimation. ICCV 2021.
- M.-H. Guo, J.-X. Cai, Z.-N. Liu, T.-J. Mu, R. R. Martin, S.-M. Hu. PCT: Point Cloud Transformer. Computational Visual Media, 2021.
- Z. Huang, X. Wang, Y. Wei, L. Huang, H. Shi, W. Liu, T. S. Huang. CCNet: Criss-Cross Attention for Semantic Segmentation. ICCV 2019.
- N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, M. J. Black. AMASS: Archive of Motion Capture as Surface Shapes. ICCV 2019.
- C. Ionescu, D. Papava, V. Olaru, C. Sminchisescu. Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. IEEE TPAMI, 2014.
- Y. He, R. Yan, K. Fragkiadaki, S.-I. Yu. Epipolar Transformers. CVPR 2020.
- C.-H. P. Huang, H. Yi, M. Höschle, M. Safroshkin, T. Alexiadis, S. Polikovsky, D. Scharstein, M. J. Black. Capturing and Inferring Dense Full-Body Human-Scene Contact. CVPR 2022.
- Z. Zhang, C. Wang, W. Qiu, W. Qin, W. Zeng. AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild. IJCV 2021.