Saeed Ghorbani

Selected publications

1. ECCV

   UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues

   Vandad Davoodnia, Saeed Ghorbani, Marc-André Carbonneau, Alexandre Messier, and Ali Etemad

   European Conference on Computer Vision (ECCV) 2024

   Abs Bib PDF Website

   Abstract. We introduce UPose3D, a novel approach for multi-view 3D human pose estimation, addressing challenges in accuracy and scalability. Our method advances existing pose estimation frameworks by improving robustness and flexibility without requiring direct 3D annotations. At the core of our method, a pose compiler module refines predictions from a 2D keypoints estimator that operates on a single image by leveraging temporal and cross-view information. Our novel cross-view fusion strategy is scalable to any number of cameras, while our synthetic data generation strategy ensures generalization across diverse actors, scenes, and viewpoints. Finally, UPose3D leverages the prediction uncertainty of both the 2D keypoint estimator and the pose compiler module. This provides robustness to outliers and noisy data, resulting in state-of-the-art performance in out-of-distribution settings. In addition, for in-distribution settings, UPose3D yields performance rivalling methods that rely on 3D annotated data while being the state-of-the-art among methods relying only on 2D supervision.

   @article{davoodnia2024upose3d,
     title = {UPose3D: Uncertainty-Aware 3D Human Pose Estimation with Cross-View and Temporal Cues},
     author = {Davoodnia, Vandad and Ghorbani, Saeed and Carbonneau, Marc-Andr{\'e} and Messier, Alexandre and Etemad, Ali},
     journal = {European Conference on Computer Vision (ECCV)},
     year = {2024},
   }

2. ECCVW

   SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers

   Vandad Davoodnia, Saeed Ghorbani, Alexandre Messier, and Ali Etemad

   ECCVW Workshop on Video Games 2024

   Abs Bib PDF Website

   Abstract. We introduce UPose3D, a novel approach for multi-view 3D human pose estimation, addressing challenges in accuracy and scalability. Our method advances existing pose estimation frameworks by improving robustness and flexibility without requiring direct 3D annotations. At the core of our method, a pose compiler module refines predictions from a 2D keypoints estimator that operates on a single image by leveraging temporal and cross-view information. Our novel cross-view fusion strategy is scalable to any number of cameras, while our synthetic data generation strategy ensures generalization across diverse actors, scenes, and viewpoints. Finally, UPose3D leverages the prediction uncertainty of both the 2D keypoint estimator and the pose compiler module. This provides robustness to outliers and noisy data, resulting in state-of-the-art performance in out-of-distribution settings. In addition, for in-distribution settings, UPose3D yields performance rivalling methods that rely on 3D annotated data while being the state-of-the-art among methods relying only on 2D supervision.

   @article{davoodnia2024skelformer,
     title = {SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers},
     author = {Davoodnia, Vandad and Ghorbani, Saeed and Messier, Alexandre and Etemad, Ali},
     journal = {ECCVW Workshop on Video Games},
     year = {2024},
   }

3. CGF

   ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech

   Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F. Troje, and Marc-André Carbonneau

   Computer Graphics Forum 2023

   Abs Bib PDF Blog Code Website

   Abstract We present ZeroEGGS, a neural network framework for speech-driven gesture generation with zero-shot style control by example. This means style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the input, addressing the stochastic nature of gesture motion. In a series of experiments, we first demonstrate the flexibility and generalizability of our model to new speakers and styles. In a user study, we then show that our model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal. Finally, we release a high-quality dataset of full-body gesture motion including fingers, with speech, spanning across 19 different styles. Our code and data are publicly available at https://github.com/ubisoft/ubisoft-laforge-ZeroEGGS.

   @article{ghorbani2023zeroeggs,
     author = {Ghorbani, Saeed and Ferstl, Ylva and Holden, Daniel and Troje, Nikolaus F. and Carbonneau, Marc-Andr{\'e}},
     title = {ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech},
     journal = {Computer Graphics Forum},
     year = {2023},
     keywords = {animation, gestures, character control, motion capture},
     doi = {https://doi.org/10.1111/cgf.14734},
     url = {https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14734},
   }

4. CGF

   Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model

   Saeed Ghorbani, Calden Wloka, Ali Etemad, Marcus A. Brubaker, and Nikolaus F. Troje

   Computer Graphics Forum (Symposium on Computer Aanimation) 2020

   Abs Bib PDF Website

   We present a probabilistic framework to generate character animations based on weak control signals, such that the synthesized motions are realistic while retaining the stochastic nature of human movement. The proposed architecture, which is designed as a hierarchical recurrent model, maps each sub-sequence of motions into a stochastic latent code using a variational autoencoder extended over the temporal domain. We also propose an objective function which respects the impact of each joint on the pose and compares the joint angles based on angular distance. We use two novel quantitative protocols and human qualitative assessment to demonstrate the ability of our model to generate convincing and diverse periodic and non-periodic motion sequences without the need for strong control signals.

   @article{ghorbani2020b,
     journal = {Computer Graphics Forum (Symposium on Computer Aanimation)},
     title = {{Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model}},
     author = {Ghorbani, Saeed and Wloka, Calden and Etemad, Ali and Brubaker, Marcus A. and Troje, Nikolaus F.},
     year = {2020},
     publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
     issn = {1467-8659},
     doi = {10.1111/cgf.14116},
     talk = {https://www.youtube.com/watch?v=r9F74LcGC0A}
   }

5. PLOS ONE

   MoVi: A large multi-purpose human motion and video dataset

   Saeed Ghorbani, Kimia Mahdaviani, Anne Thaler, Konrad Kording, Douglas James Cook, Gunnar Blohm, and Nikolaus F. Troje

   Plos one 2021

   Abs Bib PDF Code Website

   Human movements are both an area of intense study and the basis of many applications such as character animation. For many applications, it is crucial to identify movements from videos or analyze datasets of movements. Here we introduce a new human Motion and Video dataset MoVi, which we make available publicly. It contains 60 female and 30 male actors performing a collection of 20 predefined everyday actions and sports movements, and one self-chosen movement. In five capture rounds, the same actors and movements were recorded using different hardware systems, including an optical motion capture system, video cameras, and inertial measurement units (IMU). For some of the capture rounds, the actors were recorded when wearing natural clothing, for the other rounds they wore minimal clothing. In total, our dataset contains 9 hours of motion capture data, 17 hours of video data from 4 different points of view (including one hand-held camera), and 6.6 hours of IMU data. In this paper, we describe how the dataset was collected and post-processed; We present state-of-the-art estimates of skeletal motions and full-body shape deformations associated with skeletal motion. We discuss examples for potential studies this dataset could enable.

   @article{ghorbani2020movi,
     title = {MoVi: A large multi-purpose human motion and video dataset},
     author = {Ghorbani, Saeed and Mahdaviani, Kimia and Thaler, Anne and Kording, Konrad and Cook, Douglas James and Blohm, Gunnar and Troje, Nikolaus F.},
     year = {2021},
     journal = {Plos one},
     primaryclass = {cs.CV},
   }

6. CGI

   Auto-labelling of markers in optical motion capture by permutation learning

   Saeed Ghorbani, Ali Etemad, and Nikolaus F. Troje

   In Computer Graphics International 2019

   Abs Bib PDF

   Optical marker-based motion capture is a vital tool in applications such as motion and behavioural analysis, animation, and biomechanics. Labelling, that is, assigning optical markers to the pre-defined positions on the body is a time consuming and labour intensive postprocessing part of current motion capture pipelines. The problem can be considered as a ranking process in which markers shuffled by an unknown permutation matrix are sorted to recover the correct order. In this paper, we present a framework for automatic marker labelling which first estimates a permutation matrix for each individual frame using a differentiable permutation learning model and then utilizes temporal consistency to identify and correct remaining labelling errors. Experiments conducted on the test data show the effectiveness of our framework.

   @inproceedings{ghorbani2019auto,
     title = {Auto-labelling of markers in optical motion capture by permutation learning},
     author = {Ghorbani, Saeed and Etemad, Ali and Troje, Nikolaus F.},
     booktitle = {Computer Graphics International},
     pages = {167--178},
     year = {2019},
     organization = {Springer},
     award = {Best Paper Award}
   }