← Projects

UPose3D

Multi-view 3D human pose estimation that carries an uncertainty for every keypoint from 2D to 3D, scales to any number of cameras, and learns without 3D-labelled video. ECCV 2024.

Vandad Davoodnia, Saeed Ghorbani, Marc-André Carbonneau, Alexandre Messier, Ali Etemad

European Conference on Computer Vision (ECCV) 2024 · Queen’s University and Ubisoft La Forge

Read the paper → Official project page Cite

Markerless multi-view motion capture sounds like a solved problem. Put a few calibrated cameras around a person, find each joint in every image, and triangulate. In a studio with clean images it works well. The trouble starts when some of the 2D guesses are wrong: a hand hidden behind the body, a left foot labelled as the right one, a camera looking from an unusual angle. Plain triangulation trusts every camera equally, so one bad view drags the whole joint with it.

The usual alternative is to train a network end to end on multi-view video with 3D labels. That works in the setting it was trained on, but 3D-labelled video is scarce outside the lab, and such models tend to break in new rooms, with new people or new camera layouts.

UPose3D[1] is our answer to both problems. It refines each camera's 2D guesses using what the other cameras and the neighbouring frames see, it keeps an uncertainty for every keypoint all the way to the 3D estimate, and it learns all of this from synthetic data made from motion capture, without any 3D-labelled video. The work was led by Vandad Davoodnia at Queen's University and Ubisoft La Forge, and was published at ECCV 2024.

This page walks through the method one step at a time, with interactive figures that simulate each idea using real camera geometry, then shows what the experiments found. The code snippets are simplified Python sketches rather than the actual implementation.

Overview

UPose3D takes synchronised video from any number of calibrated cameras and outputs the 3D position of every joint. Between the two are four steps, and one idea runs through all of them: never throw away how sure you are.

How UPose3D works

Every step keeps an uncertainty next to each keypoint, all the way to the 3D answer

Multi-view videoAny number of calibrated cameras
2D pose estimatorPer image: each joint's position and uncertainty (an RLE head on CPN or HRNet)
Cross-view projectionEvery other camera's guess, moved onto this view along its epipolar line: a 2D point cloud per joint
Pose compilerPoint-cloud encoder, then a spatiotemporal transformer: refined positions and uncertainties
3D estimationStart from triangulation, then find the 3D point most likely under both sets of uncertainties
3D poseEvery joint, every frame, in metres
Training the pose compiler Motion capture (AMASS)Random virtual camerasCorrupted 2D keypointsPoint clouds No 3D-labelled video needed
Redrawn from Figure 1 of the paper, with the training path from its supplementary material.

A 2D pose estimator first finds each joint in each image, along with an uncertainty. Every camera's guesses are then projected onto every other camera's view, giving each joint a small cloud of points. The pose compiler reads those clouds across neighbouring frames and outputs refined keypoints with their own uncertainties. Finally, both sets of keypoints and uncertainties go into an optimisation that finds the most likely 3D position of each joint.

Keypoints With Uncertainty

Most 2D pose estimators predict a heatmap per joint and take its peak. UPose3D instead puts a Residual Log-likelihood Estimation (RLE) head[2] on a standard backbone (CPN on Human3.6M, HRNet-W48 for the out-of-distribution experiments). RLE regresses the joint's position directly and, through a small normalising flow, also learns the shape of the error around it.

So for every joint in every image we get a position, an uncertainty, and a probability density we can evaluate anywhere in the image. That last part matters later: it lets the 3D step ask "how likely is it that the joint projects here?" for any candidate 3D point, rather than only comparing against a single best guess.

One Joint, Every Camera

The next step brings the other cameras in. With calibrated cameras, a 2D point in one image doesn't pin down a point in another image, but it does pin down a line: the epipolar line, where that point's ray would appear from the other camera's viewpoint. The true joint must lie on it.

For each joint and each reference camera, UPose3D takes every other camera's guess, draws its epipolar line in the reference image, and keeps the point on that line closest to the reference camera's own guess. Those points, together with the reference guess, form a small 2D point cloud for that joint, normalised by the person's bounding box. Try it below, and then make one camera unreliable:

One joint, seen from every camera

Click a camera on the left to make it the reference view. The image on the right is what it sees.

Capture volume, from above

What camera 1 sees

A simulation with a 17-joint skeleton and pinhole cameras on a 3.2 m ring; the 2D "detections" are the true projections plus noise. The geometry is the paper's: the fundamental matrix comes from the known camera parameters, and each projected point is the point on the epipolar line nearest the reference keypoint.

When the cameras agree, the cloud is a tight cluster around the joint. When one camera is wrong, its point lands visibly far from the rest, which is exactly the kind of pattern a network can learn to discount. The step itself is only a few lines:

def point_cloud(ref, keypoints, cameras):
    """2D point cloud for one joint in reference camera `ref` (Sec. 3.2)."""
    cloud = [keypoints[ref]]
    for j, cam in enumerate(cameras):
        if j == ref:
            continue
        F = fundamental_matrix(cameras[ref], cam)     # from the known calibration
        line = F.T @ homogeneous(keypoints[j])        # epipolar line in the reference view
        cloud.append(closest_point(line, keypoints[ref]))
    return normalise_to_bbox(cloud, ref)

Because every other camera adds one more point to the same cloud, the input has the same form whether there are two cameras or ten. That is what lets UPose3D handle any number of cameras without retraining, and it is why we call this a scalable cross-view fusion.

The Pose Compiler

The pose compiler turns these clouds into better keypoints. It has two parts. A point-cloud encoder, inspired by the Point Cloud Transformer[3], turns each joint's cloud into a feature vector. We modified it to keep the coordinates, adding a residual connection from the reference view's features to its output.

A spatiotemporal encoder then looks at those features across all joints and a window of frames (27 in the main model) and predicts each joint's refined position and uncertainty, again with an RLE head. Full attention over every joint in every frame gets expensive fast, so it uses criss-cross attention[4]: each token attends only along its own frame and its own joint, and stacking a few layers lets information reach everywhere.

Criss-cross attention, one token at a time

Each square is one joint in one frame. Hover, tap or use the arrow keys to pick a token and see what it attends to.

Shown for 9 frames; the paper's main model uses 27. Stacking several of these layers lets information reach every token, at a fraction of the memory of full attention.

Two things make the compiler useful. It sees time, so a joint that flickers in one frame is pulled back by its neighbours. And it sees the clouds, so it can learn which kinds of disagreement mean "this camera is wrong" rather than "the joint moved".

From 2D to 3D

Now we have two sets of 2D keypoints for every joint in every camera: the image branch's and the pose compiler's, each with its full predicted distribution. The 3D step looks for the 3D point whose projections are most likely under all of them. It starts from a plain triangulation (DLT) and refines it with L-BFGS, maximising the combined likelihood:

def estimate_3d(cameras, image_kps, compiler_kps):
    X = dlt(cameras, image_kps)                       # plain triangulation to start
    def nll(X):
        loss = 0
        for cam, a, b in zip(cameras, image_kps, compiler_kps):
            u = cam.project(X)
            loss -= a.log_prob(u) + b.log_prob(u)     # both predicted 2D distributions
        return loss
    return lbfgs(nll, X)                              # no 3D labels involved

The effect is easiest to see with one bad camera. Plain triangulation weighs every view the same, so the bad one pulls the answer away. When each view comes with its uncertainty, a confident view counts for more and an unsure one for less:

One bad camera, two ways to triangulate

Six cameras look at the left wrist. Push camera 3's guess off and see where the 3D estimate lands.

From above: each camera's ray through its guess

Zoomed in: 30 cm around the true wrist

Plain triangulation off

Weighted by uncertainty off

A simulation: the other cameras' guesses have 2 px of noise and say so; camera 3 reports an uncertainty as large as its error when the button is on. Weighting each view by one over its uncertainty is a simplified version of what UPose3D does, which starts from triangulation and then finds the 3D point most likely under every keypoint's full predicted distribution.

This is not just a toy effect. In the paper's ablation on Human3.6M, removing uncertainty from both branches raises the error from 26.4 mm to 48.1 mm, and the full method's lead on unseen data, shown below, comes largely from discarding unreliable views.

Training Without 3D Labels

The pose compiler only ever sees 2D point clouds, so it doesn't need images or 3D-labelled video to train. Instead, we make training data from motion capture. Starting from AMASS[5], each sample is built like this:

Making training data without 3D labels

A random virtual camera, the true 2D keypoints (grey), and a corrupted copy for the pose compiler to learn from

The corruptions are the ones listed in the paper's supplementary material: Gaussian noise of varying strength, occlusions of varying size and probability, left/right (mediolateral) flips, and occasional truncation. Strengths here are illustrative.

Samples are generated on the fly, so every sample has a fresh random camera layout, and training takes about six hours on a single RTX 2080 GPU. Nothing is tied to one dataset's cameras, and the skeleton is just whichever joint regressor is applied to the motion capture: 17 joints for the Human3.6M experiments, 133 whole-body joints for the out-of-distribution ones.

Results

In the lab: Human3.6M

Human3.6M[6] is the standard multi-view benchmark, with four cameras in one studio. Among methods trained with 2D labels only, UPose3D is the most accurate, at 26.4 mm against 30.5 mm for plain triangulation of the same 2D estimator. Methods trained on Human3.6M's own 3D labels, or with its camera parameters during training, are still ahead here (Epipolar Transformers[7] reaches 19.0 mm), which is expected when the test set looks exactly like the training set.

Human3.6M: error by training supervision

Mean per-joint position error (MPJPE) on the full test set; lower is better

From Table 1 of the paper. Rows the paper reports without an MPJPE are left out. The two 'from scratch' rows were trained with no additional data.
Show as a table
MethodError (mm)Notes
TesseTrack (Trained with 3D labels)18.7HRNet, 5 frames
AdaFuse (Trained with 3D labels)19.5ResNet152, 1 frame
Learnable Triangulation (Trained with 3D labels)20.7ResNet152, 1 frame
Canonical Fusion (Trained with 3D labels)21.0ResNet152, 1 frame
MTF-Transformer+ (27 frames) (Trained with 3D labels)25.8CPN, 27 frames
MTF-Transformer+ (1 frame) (Trained with 3D labels)26.5CPN, 1 frame
Jiang et al. (Trained with 3D labels)27.8ResNet152, 1 frame, uncalibrated cameras
Flex (Trained with 3D labels)30.2ResNet152, all frames, uncalibrated cameras
UPose3D (CPN, 27 frames) (Trained with 2D labels only)26.4CPN, 27 frames
UPose3D (CPN, 1 frame) (Trained with 2D labels only)26.9CPN, 1 frame
UPose3D (ResNet152, 27 frames) (Trained with 2D labels only)29.9ResNet152, 27 frames
DLT (CPN) (Trained with 2D labels only)30.5CPN, 1 frame
UPose3D (ResNet152, 1 frame) (Trained with 2D labels only)31.0ResNet152, 1 frame
DLT (ResNet152) (Trained with 2D labels only)36.3ResNet152, 1 frame
EpipolarPose (Trained with 2D labels only)55.08ResNet50, 1 frame
Epipolar Transformers (2D labels + camera parameters in training)19.0ResNet152, 1 frame
TransFusion (2D labels + camera parameters in training)25.8ResNet50, 1 frame
TR Loss (2D labels + camera parameters in training)25.8HRNet, 1 frame; uses 1% of unlabelled test data in training
UPose3D (CPN, 27 frames) (2D labels + camera parameters in training)26.2CPN, 27 frames
Cross-view Fusion (2D labels + camera parameters in training)26.2ResNet152, 1 frame
UPose3D (ResNet50, from scratch) (2D labels + camera parameters in training)57.3ResNet50, 27 frames, trained from scratch with no extra data
DLT (ResNet50, from scratch) (2D labels + camera parameters in training)71.2ResNet50, 1 frame, trained from scratch with no extra data

Somewhere new: RICH

The more telling test is a setting none of the models trained on. RICH[8] was recorded in three new environments (a construction site, a gym and a lecture hall) with different people and cameras. Here UPose3D roughly halves the error of the next best method, 34.7 mm against 64.4 mm. AdaFuse[9], one of the strongest methods on Human3.6M, falls apart at 524 mm: it does well only when every view's prediction is reasonable, and occasional noisy views cause large triangulation errors. UPose3D's uncertainty lets it set those views aside.

RICH: error in unseen scenes

MPJPE on the RICH test set; none of the methods trained on RICH. Lower is better.

From Table 2 of the paper. Baselines marked as the authors' own implementations in the paper: AdaFuse and both HRNet-W48 baselines.
Show as a table
MethodError (mm)Notes
UPose3D (27 frames)34.7Trained on COCO (2D) and AMASS (pose compiler)
UPose3D (1 frame)36.2Trained on COCO (2D) and AMASS (pose compiler)
UPose3D (1 frame, Human3.6M 2D)49.9Trained on Human3.6M + MPII (2D) and AMASS (pose compiler)
UPose3D (27 frames, Human3.6M only)51.8Trained on Human3.6M + MPII (2D) and Human3.6M (pose compiler)
HRNet-W48 + grid search64.4Trained on COCO; triangulation by grid search
HRNet-W48 + DLT66.0Trained on COCO; plain triangulation
AdaFuse524.0Trained on Human3.6M + MPII and Human3.6M

What matters

The ablations show which parts do the work. The two branches need each other: without the pose compiler the error rises to 37.1 mm, and without the image branch it rises to 69.9 mm. Uncertainty is what makes combining them work: remove it everywhere and the error jumps to 48.1 mm, and with no uncertainty and no compiler we are back to plain triangulation (30.5 mm).

Ablations on Human3.6M

MPJPE with a 27-frame window when parts of UPose3D are removed; lower is better

From Table 3 of the paper.
Show as a table
MethodError (mm)Notes
UPose3D26.42The full method
Without the image uncertainty27.61Uncertainty from the 2D estimator removed
Without the compiler uncertainty26.42Same MPJPE; PA-MPJPE rises from 23.42 to 23.58
Without the pose compiler37.14Only the 2D estimator’s keypoints
Without any uncertainty48.11Both branches, plain triangulation
Without the image branch69.9Only the pose compiler’s keypoints
No uncertainty, no compiler30.48Image keypoints only, plain triangulation: the same as DLT
No uncertainty, no image branch77.25Compiler keypoints only, plain triangulation

Cost, and more cameras

Because the heavy lifting happens on small point clouds rather than image features, UPose3D is also the cheapest of the multi-view methods compared, at roughly half the compute of the next one with both 4 and 10 cameras. In the paper's breakdown, the pose compiler needs only 0.385 GFLOPs against 508.5 for the 2D backbone. On CMU Panoptic and HUMBI, adding cameras keeps lowering UPose3D's error while plain triangulation and RANSAC level off, and the run time per sample stays roughly constant.

Compute as cameras are added

Billions of operations (GFLOPs) to estimate one frame, with 4 and with 10 cameras; lower is better

From Table 4 of the paper (single-frame models, including each method's 2D backbone).
Show as a table
MethodParameters (M)4 cameras (GFLOPs)10 cameras (GFLOPs)
UPose3D65.4208.7517.7
Epipolar Transformers68.1406.51016.2
MTF-Transformers78.64071017.8
AdaFuse69.75951487.6
Learnable Triangulation80.6716.11326.9

Limitations

Citation

Vandad Davoodnia, Saeed Ghorbani, Marc-André Carbonneau, Alexandre Messier, Ali Etemad. UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues. ECCV 2024.

BibTeX
@misc{davoodnia2024upose3d,
  title         = {UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues},
  author        = {Vandad Davoodnia and Saeed Ghorbani and Marc-André Carbonneau and Alexandre Messier and Ali Etemad},
  year          = {2024},
  eprint        = {2404.14634},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2404.14634}
}

References

  1. V. Davoodnia, S. Ghorbani, M.-A. Carbonneau, A. Messier, A. Etemad. UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues. ECCV 2024.
  2. J. Li, S. Bian, A. Zeng, C. Wang, B. Pang, W. Liu, C. Lu. Human Pose Regression with Residual Log-likelihood Estimation. ICCV 2021.
  3. M.-H. Guo, J.-X. Cai, Z.-N. Liu, T.-J. Mu, R. R. Martin, S.-M. Hu. PCT: Point Cloud Transformer. Computational Visual Media, 2021.
  4. Z. Huang, X. Wang, Y. Wei, L. Huang, H. Shi, W. Liu, T. S. Huang. CCNet: Criss-Cross Attention for Semantic Segmentation. ICCV 2019.
  5. N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, M. J. Black. AMASS: Archive of Motion Capture as Surface Shapes. ICCV 2019.
  6. C. Ionescu, D. Papava, V. Olaru, C. Sminchisescu. Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. IEEE TPAMI, 2014.
  7. Y. He, R. Yan, K. Fragkiadaki, S.-I. Yu. Epipolar Transformers. CVPR 2020.
  8. C.-H. P. Huang, H. Yi, M. Höschle, M. Safroshkin, T. Alexiadis, S. Polikovsky, D. Scharstein, M. J. Black. Capturing and Inferring Dense Full-Body Human-Scene Contact. CVPR 2022.
  9. Z. Zhang, C. Wang, W. Qiu, W. Qin, W. Zeng. AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild. IJCV 2021.