Saeed Ghorbani

Selected publications

1. CGF

   ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech

   Saeed Ghorbani, Ylva Ferstl, Daniel Holden, Nikolaus F. Troje, and Marc-André Carbonneau

   Computer Graphics Forum 2023

   Abs Bib PDF Blog Code Website

   Abstract We present ZeroEGGS, a neural network framework for speech-driven gesture generation with zero-shot style control by example. This means style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the input, addressing the stochastic nature of gesture motion. In a series of experiments, we first demonstrate the flexibility and generalizability of our model to new speakers and styles. In a user study, we then show that our model outperforms previous state-of-the-art techniques in naturalness of motion, appropriateness for speech, and style portrayal. Finally, we release a high-quality dataset of full-body gesture motion including fingers, with speech, spanning across 19 different styles. Our code and data are publicly available at https://github.com/ubisoft/ubisoft-laforge-ZeroEGGS.

   @article{ghorbani2023zeroeggs,
     author = {Ghorbani, Saeed and Ferstl, Ylva and Holden, Daniel and Troje, Nikolaus F. and Carbonneau, Marc-Andr{\'e}},
     title = {ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech},
     journal = {Computer Graphics Forum},
     year = {2023},
     keywords = {animation, gestures, character control, motion capture},
     doi = {https://doi.org/10.1111/cgf.14734},
     url = {https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14734},
   }

2. CGF

   Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model

   Saeed Ghorbani, Calden Wloka, Ali Etemad, Marcus A. Brubaker, and Nikolaus F. Troje

   Computer Graphics Forum (Symposium on Computer Aanimation) 2020

   Abs Bib PDF Website

   We present a probabilistic framework to generate character animations based on weak control signals, such that the synthesized motions are realistic while retaining the stochastic nature of human movement. The proposed architecture, which is designed as a hierarchical recurrent model, maps each sub-sequence of motions into a stochastic latent code using a variational autoencoder extended over the temporal domain. We also propose an objective function which respects the impact of each joint on the pose and compares the joint angles based on angular distance. We use two novel quantitative protocols and human qualitative assessment to demonstrate the ability of our model to generate convincing and diverse periodic and non-periodic motion sequences without the need for strong control signals.

   @article{ghorbani2020b,
     journal = {Computer Graphics Forum (Symposium on Computer Aanimation)},
     title = {{Probabilistic Character Motion Synthesis using a Hierarchical Deep Latent Variable Model}},
     author = {Ghorbani, Saeed and Wloka, Calden and Etemad, Ali and Brubaker, Marcus A. and Troje, Nikolaus F.},
     year = {2020},
     publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
     issn = {1467-8659},
     doi = {10.1111/cgf.14116},
     talk = {https://www.youtube.com/watch?v=r9F74LcGC0A}
   }

3. PLOS ONE

   MoVi: A large multi-purpose human motion and video dataset

   Saeed Ghorbani, Kimia Mahdaviani, Anne Thaler, Konrad Kording, Douglas James Cook, Gunnar Blohm, and Nikolaus F. Troje

   Plos one 2021

   Abs Bib PDF Code Website

   Human movements are both an area of intense study and the basis of many applications such as character animation. For many applications, it is crucial to identify movements from videos or analyze datasets of movements. Here we introduce a new human Motion and Video dataset MoVi, which we make available publicly. It contains 60 female and 30 male actors performing a collection of 20 predefined everyday actions and sports movements, and one self-chosen movement. In five capture rounds, the same actors and movements were recorded using different hardware systems, including an optical motion capture system, video cameras, and inertial measurement units (IMU). For some of the capture rounds, the actors were recorded when wearing natural clothing, for the other rounds they wore minimal clothing. In total, our dataset contains 9 hours of motion capture data, 17 hours of video data from 4 different points of view (including one hand-held camera), and 6.6 hours of IMU data. In this paper, we describe how the dataset was collected and post-processed; We present state-of-the-art estimates of skeletal motions and full-body shape deformations associated with skeletal motion. We discuss examples for potential studies this dataset could enable.

   @article{ghorbani2020movi,
     title = {MoVi: A large multi-purpose human motion and video dataset},
     author = {Ghorbani, Saeed and Mahdaviani, Kimia and Thaler, Anne and Kording, Konrad and Cook, Douglas James and Blohm, Gunnar and Troje, Nikolaus F.},
     year = {2021},
     journal = {Plos one},
     primaryclass = {cs.CV},
   }

4. CGI

   Auto-labelling of markers in optical motion capture by permutation learning

   Saeed Ghorbani, Ali Etemad, and Nikolaus F. Troje

   In Computer Graphics International 2019

   Abs Bib PDF

   Optical marker-based motion capture is a vital tool in applications such as motion and behavioural analysis, animation, and biomechanics. Labelling, that is, assigning optical markers to the pre-defined positions on the body is a time consuming and labour intensive postprocessing part of current motion capture pipelines. The problem can be considered as a ranking process in which markers shuffled by an unknown permutation matrix are sorted to recover the correct order. In this paper, we present a framework for automatic marker labelling which first estimates a permutation matrix for each individual frame using a differentiable permutation learning model and then utilizes temporal consistency to identify and correct remaining labelling errors. Experiments conducted on the test data show the effectiveness of our framework.

   @inproceedings{ghorbani2019auto,
     title = {Auto-labelling of markers in optical motion capture by permutation learning},
     author = {Ghorbani, Saeed and Etemad, Ali and Troje, Nikolaus F.},
     booktitle = {Computer Graphics International},
     pages = {167--178},
     year = {2019},
     organization = {Springer},
     award = {Best Paper Award}
   }