Saeed Ghorbani

Selected publications

1. ECCV

   UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues

   Vandad Davoodnia, Saeed Ghorbani, Marc-André Carbonneau, Alexandre Messier, and Ali Etemad

   In European Conference on Computer Vision (ECCV) 2024

   Abs Bib PDF Website

   We introduce UPose3D, a novel approach for multi-view 3D human pose estimation, addressing challenges in accuracy and scalability. Our method advances existing pose estimation frameworks by improving robustness and flexibility without requiring direct 3D annotations. At the core of our method, a pose compiler module refines predictions from a 2D keypoints estimator by leveraging temporal and cross-view information. Our novel cross-view fusion strategy is scalable to any number of cameras, while our synthetic data generation strategy ensures generalization across diverse actors, scenes, and viewpoints. UPose3D leverages prediction uncertainty of both the 2D keypoint estimator and the pose compiler module, providing robustness to outliers and noisy data, resulting in state-of-the-art performance in out-of-distribution settings.

   @inproceedings{davoodnia2024upose3d,
     title = {UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues},
     author = {Davoodnia, Vandad and Ghorbani, Saeed and Carbonneau, Marc-Andr{\'e} and Messier, Alexandre and Etemad, Ali},
     booktitle = {European Conference on Computer Vision (ECCV)},
     year = {2024},
   }

2. ECCVW

   SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers

   Vandad Davoodnia, Saeed Ghorbani, Alexandre Messier, and Ali Etemad

   In ECCV Workshop on CV for Metaverse 2024

   Abs Bib PDF Website

   We present a markerless motion capture pipeline for 3D human pose and shape estimation. Our two-stage approach first leverages existing 2D keypoint detectors to obtain 3D joint positions, then applies a regression-based inverse-kinematic skeletal transformer that maps the joint positions to pose and shape representations from heavily noisy observations. By integrating pose space priors and separating 3D keypoint detection from inverse kinematics, our model achieves strong generalization to unseen noisy data. We evaluate our approach across three public datasets in both in-distribution and out-of-distribution settings, demonstrating robust performance against noise and occlusions.

   @inproceedings{davoodnia2024skelformer,
     title = {SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers},
     author = {Davoodnia, Vandad and Ghorbani, Saeed and Messier, Alexandre and Etemad, Ali},
     booktitle = {ECCV Workshop on CV for Metaverse},
     year = {2024},
   }

3. CGF

   ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech

   Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F. Troje, and Marc-André Carbonneau

   Computer Graphics Forum 2023

   Abs Bib PDF Blog Code Website

   We present ZeroEGGS, a neural network framework for speech-driven gesture generation with zero-shot style control by example. Style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the input, addressing the stochastic nature of gesture motion. In a user study, we show that our model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal. We also release a high-quality dataset of full-body gesture motion including fingers, with speech, spanning across 19 different styles.

   @article{ghorbani2023zeroeggs,
     author = {Ghorbani, Saeed and Ferstl, Ylva and Holden, Daniel and Troje, Nikolaus F. and Carbonneau, Marc-Andr{\'e}},
     title = {ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech},
     journal = {Computer Graphics Forum},
     year = {2023},
     keywords = {animation, gestures, character control, motion capture},
     doi = {10.1111/cgf.14734},
   }

4. PLOS ONE

   MoVi: A Large Multi-Purpose Human Motion and Video Dataset

   Saeed Ghorbani, Kimia Mahdaviani, Anne Thaler, Konrad Kording, Douglas James Cook, Gunnar Blohm, and Nikolaus F. Troje

   PLOS ONE 2021

   Abs Bib PDF Code Website

   We introduce MoVi, a large human Motion and Video dataset made publicly available. It contains 60 female and 30 male actors performing a collection of 20 predefined everyday actions and sports movements, and one self-chosen movement. In five capture rounds, the same actors and movements were recorded using different hardware systems, including an optical motion capture system, video cameras, and inertial measurement units (IMU). In total, the dataset contains 9 hours of motion capture data, 17 hours of video data from 4 viewpoints (including one hand-held camera), and 6.6 hours of IMU data. We present state-of-the-art estimates of skeletal motions and full-body shape deformations associated with skeletal motion.

   @article{ghorbani2021movi,
     title = {MoVi: A Large Multi-Purpose Human Motion and Video Dataset},
     author = {Ghorbani, Saeed and Mahdaviani, Kimia and Thaler, Anne and Kording, Konrad and Cook, Douglas James and Blohm, Gunnar and Troje, Nikolaus F.},
     year = {2021},
     journal = {PLOS ONE},
   }

5. CGF

   Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model

   Saeed Ghorbani, Calden Wloka, Ali Etemad, Marcus A. Brubaker, and Nikolaus F. Troje

   Computer Graphics Forum (Symposium on Computer Animation) 2020

   Abs Bib PDF Website

   We present a probabilistic framework to generate character animations based on weak control signals, such that the synthesized motions are realistic while retaining the stochastic nature of human movement. The proposed architecture, a hierarchical recurrent model, maps each sub-sequence of motions into a stochastic latent code using a variational autoencoder extended over the temporal domain. We also propose an objective function which respects the impact of each joint on the pose and compares joint angles based on angular distance. We demonstrate the ability of our model to generate convincing and diverse periodic and non-periodic motion sequences without the need for strong control signals.

   @article{ghorbani2020motion,
     journal = {Computer Graphics Forum (Symposium on Computer Animation)},
     title = {Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model},
     author = {Ghorbani, Saeed and Wloka, Calden and Etemad, Ali and Brubaker, Marcus A. and Troje, Nikolaus F.},
     year = {2020},
     publisher = {The Eurographics Association and John Wiley \& Sons Ltd.},
     issn = {1467-8659},
     doi = {10.1111/cgf.14116},
     talk = {https://www.youtube.com/watch?v=r9F74LcGC0A}
   }

6. CGI

   Auto-labelling of Markers in Optical Motion Capture by Permutation Learning

   Saeed Ghorbani, Ali Etemad, and Nikolaus F. Troje

   In Computer Graphics International (CGI) 2019

   Abs Bib PDF

   Optical marker-based motion capture is a vital tool in applications such as motion and behavioural analysis, animation, and biomechanics. Labelling, i.e., assigning optical markers to pre-defined positions on the body, is a time-consuming and labour-intensive postprocessing step. We present a framework for automatic marker labelling which estimates a permutation matrix for each individual frame using a differentiable permutation learning model, then utilizes temporal consistency to identify and correct remaining labelling errors.

   @inproceedings{ghorbani2019auto,
     title = {Auto-labelling of Markers in Optical Motion Capture by Permutation Learning},
     author = {Ghorbani, Saeed and Etemad, Ali and Troje, Nikolaus F.},
     booktitle = {Computer Graphics International (CGI)},
     pages = {167--178},
     year = {2019},
     organization = {Springer},
     award = {Best Paper Award}
   }