Is there a fingerprint in AI-generated motion? I tested my own model
A frequency-domain look at real and generated gestures: what energy and coordination reveal, what fooled me, and where ZeroEGGS differs from a real actor.
· 17 min read
Generated motion has got good. Watch a modern speech-to-gesture or text-to-motion model and, at a glance, it is often hard to say exactly what is off. AI-generated images, on the other hand, have a well-known weakness: look at them in the frequency domain and many generators leave regular traces that a simple classifier can pick up[1].
So a question I've been curious about for a while is whether generated motion has a fingerprint of its own, and if it does, whether that fingerprint survives the cleanup filtering every motion-capture pipeline applies anyway. To find out I picked the model I know best: ZeroEGGS[2], a speech-to-gesture model I developed with colleagues at Ubisoft La Forge.
I want to be upfront that this is a case study rather than a definitive answer (one actor, one model), but I learned a few things along the way which I think are worth sharing. Including the fact that my hypothesis was half wrong.
The article is split into a few parts. First I'll introduce the two fingerprints I tried and show how to compute them. Second, I'll test them on a toy walker where I know the right answer. Third, I'll run the real test: an actor's motion capture against ZeroEGGS's output for the same speech. And finally I'll close with the lessons I took away. The important pieces of code (Python, NumPy and SciPy) are included throughout.
- Two Fingerprints
- A Toy Walker
- ZeroEGGS in a Nutshell
- Real Gestures
- What's Different About ZeroEGGS
- Lessons
Two Fingerprints
Overview
Here is the set-up. We have a window of motion, and we want a set of numbers, a fingerprint, which tells us whether it came from a person or from a model.
Since gestures are mostly about the upper body, I use eleven joints from the chest up: the chest, neck and head, plus the collarbones, shoulders, elbows and wrists on each side. Before anything else I remove the root's translation and rotation, so we're looking at the body's own movement rather than where the actor happens to be standing:
def body_positions(path):
d = bvh.load(path)
lrot = quat.from_euler(np.radians(d['rotations']), order=d['order'])
lpos = d['positions'].copy()
# Remove the root's translation and rotation so we only
# see the body's own movement, not where it is standing
lpos[:, 0] = 0.0
lrot[:, 0] = np.array([1.0, 0.0, 0.0, 0.0])
_, gpos = quat.fk(lrot, lpos, d['parents'])
x = gpos[:, joint_indices] # frames x joints x 3
return x.transpose(1, 2, 0).reshape(-1, len(x)) # signals x frames This gives us 33 signals (11 joints × 3 axes) per window, which we then split up by tempo: from slow sway (0.25–0.5 Hz), through the 1–3 Hz rhythm of gesture strokes, up to quick beats and twitches (4–8 Hz).
1. Energy
The first fingerprint is the obvious one, and the motion equivalent of the image trick: how much each signal moves at each tempo. We estimate the power spectrum using Welch's method and average it within each band. I also compute a variant I'll call energy shape, which divides out each signal's total power, so it describes how movement is spread across tempos rather than how much movement there is:
bands = [(0.25, 0.5), (0.5, 1), (1, 2), (2, 4), (4, 8)]
f, P = signal.welch(x, fs=60, nperseg=256, axis=-1)
band_power = np.stack([
np.log10(P[:, (f >= lo) & (f < hi)].mean(1)) for lo, hi in bands], axis=1)
energy = band_power.ravel()
# Energy shape: divide out each signal's total power
shape = (band_power - np.log10((10 ** band_power).sum(1, keepdims=True))).ravel() 2. Coordination
The second fingerprint looks at pairs of joints rather than single ones. For each pair we compute the coherence, which measures how consistently the two keep time with each other at each tempo, plus their timing offset, which is just the angle of the cross-spectrum:
f, Pxy = signal.csd(x[:, None, :], x[None, :, :], fs=60, nperseg=256, axis=-1)
Pxx = np.real(np.einsum('iif->if', Pxy))
# How consistently each pair keeps time, per frequency (0 to 1)
coherence = np.abs(Pxy) ** 2 / (Pxx[:, None, :] * Pxx[None, :, :])
# And their timing offset, per frequency
offset = np.angle(Pxy) This was the fingerprint I was betting on, for two reasons. First, coherence ignores how big the movement is. Second, if you filter every joint with the same filter (which is exactly what mocap cleanup does), the filter cancels out of the ratio above. So cleanup doesn't change coherence, as long as some signal survives. The little demo below shows both properties:
What coherence measures
Two joints moving to the same 1 Hz rhythm. Change how steady their timing is, or how big joint B moves.
–coherence at the 1 Hz rhythm
(0 = no steady timing, 1 = locked)
It also hints at the one thing coherence does not measure: what the offset actually is. Two joints can keep perfectly steady time with entirely the wrong offset, and we'll see shortly why that matters.
3. The Detector
For the detector I wanted the realistic setting, where you don't have any examples of the fake you're up against. So it only ever sees real motion. It learns the mean and spread of every fingerprint number, and scores a new window by how far it falls outside of that:
def one_class_score(train, test):
mu, sd = train.mean(0), train.std(0, ddof=1)
return (((test - mu) / sd) ** 2).mean(1)
def auc(fake_scores, real_scores):
u = mannwhitneyu(fake_scores, real_scores).statistic
return u / (len(fake_scores) * len(real_scores)) The scores then get turned into an AUC: the chance that a fake window scores as less real than a real one. An AUC of 0.5 means the detector can't tell them apart, and 1 means it always catches the fake.
I should say that the question itself isn't new. In 2005, Ren, Patrick, Efros, Hodgins and Rehg trained statistical models to tell natural motion from keyframed, edited and noisy motion[3], spectra are used to score predicted motion[4], and there are detectors for AI-generated videos of people[5]. What I couldn't find was a frequency fingerprint applied to skeleton motion from a modern generator, and tested against cleanup.
A Toy Walker
Before touching real data I wanted a case where I know the right answer, so I built a toy walker: ten joint angles driven by one shared stride rhythm, plus some postural sway, a little tremor and sensor noise. Then I built a set of fakes, each one breaking something different:
- Cleaned: a real walk, low-pass filtered at 6 Hz. This one is a control. It is real motion, so a good detector should not flag it.
- Frankenstein: every joint is real motion, but each one is taken from a different walk.
- Learned-style: the right stride, but the small motions are generated for each joint independently.
- Wrong arm swing: the arms swing with the same-side leg instead of opposite it.
- Perfect loop: a flawless repeating cycle, like a keyframed walk.
- Jitter: per-frame noise added to a real walk.
The cleaned control and the Frankenstein fake are only a few lines each:
# Cleaned: real motion, low-pass filtered at 6 Hz like a mocap cleanup
b, a = signal.butter(4, 6.0, fs=120) # the toy walker runs at 120 frames per second
cleaned = signal.filtfilt(b, a, x, axis=-1)
# Frankenstein: every joint is real, but from a different recording
def frankenstein(rng, recordings):
picks = rng.choice(len(recordings), size=num_joints, replace=False)
return np.stack([recordings[p][j] for j, p in enumerate(picks)]) The easiest way to see what these fakes do to coordination is to plot one joint against another. Here is the left hip angle against the left shoulder angle for three of them:
One joint plotted against another
Left hip angle (across) against left shoulder angle (up), over 20 seconds of a simplified toy walker simulated in your browser
And here is what the detector makes of all of them:
Toy walker: how well each kind of motion is told apart from real
Detector trained only on real walks. Dot: average score over 20 runs; line: ± one standard deviation.
Show as a table
The two fingerprints fail in opposite places. Energy flags every cleaned walk as fake, yet misses the Frankenstein, learned-style and wrong-arm walkers almost entirely. Coordination ignores the cleanup (0.52) and catches all three (1.00). Jitter is the one fake where energy does better (0.87 vs 0.69), which makes sense: it adds movement without changing much about timing.
Getting here took a few attempts, and each thing that went wrong taught me something:
- The cleanup control failed at first. Above about twice the 6 Hz cutoff the filtered motion is essentially gone, and what remains is leakage from the big stride motion. That alone made coordination flag every cleaned walk. So now the analysis stops below the point where signal runs out.
- The wrong-arm walker passed at first. Coherence only checks that two joints keep steady time, and these arms keep perfectly steady time with the wrong leg. Adding each pair's timing offset to the fingerprint fixed it.
- My first Frankenstein was caught for the wrong reason. I originally scrambled each joint's timing with a standard phase-randomization trick, which also injected broadband noise from the clip's edges, so energy "caught" it through an artifact. Stitching real joints from different walks together avoids that.
Of course, the toy has an obvious limit: I built the "real" motion myself, so it shows how the method works, not that real generators actually fail this way. For that we need real data.
ZeroEGGS in a Nutshell
Before running the real test it's worth a quick recap of what ZeroEGGS actually is, since its design matters when we come to interpret the results.
ZeroEGGS takes two inputs: a recording of speech, and a short clip of motion showing the style you want. From those it generates full-body gesture motion that moves in time with the speech, in the style of the example. The "zero-shot" part is that the example clip can be in a style the model never saw during training: give it a few seconds of someone gesturing nervously, and it will try to gesture nervously to any new speech you give it.
Under the hood it has three parts:
How ZeroEGGS works
Two encoders feed a decoder that writes the motion one frame at a time
- A speech encoder: a stack of 1-D convolutions which turns the audio (a mel spectrogram plus its energy) into a compact speech code for every frame.
- A style encoder: uses attention to summarize the example clip into a style embedding of 64 numbers. It is variational, which means that instead of a single embedding it predicts a distribution and samples from it. That's what lets the same inputs produce a variety of outputs, and it also means styles can be blended, or exaggerated, by moving around in the embedding space.
- A recurrent decoder: a two-layer GRU which writes the motion one frame at a time, each new pose conditioned on the previous one, the speech code and the style embedding. The output is the full skeleton, 75 joints including the fingers, at 60 frames per second.
We published it in Computer Graphics Forum in 2023[2], and I presented it at SIGGRAPH Asia 2022 in Daegu. In the paper's user study it outperformed the previous state of the art on naturalness of motion, appropriateness for the speech, and style portrayal. Alongside the code and pretrained models we released the ZEGGS dataset, which I'll use below: about 2¼ hours of one actor's monologues in 19 styles, from neutral to sarcastic to laughing, captured with full-body and finger motion.
One property is worth keeping in mind for later. Like most models that predict motion frame by frame, the decoder is trained to reconstruct the recorded motion, here with L1 losses on joint positions, rotations and velocities. When the next frame is uncertain, the safest prediction lies somewhere in the middle of the possibilities, and that tends to smooth away small, fast detail.
Real Gestures
Set-up
ZeroEGGS was trained on 46 of the dataset's 67 takes and never saw the other 21, which gives us a clean test. For each of those 21 takes I fed its audio to the pretrained model:
for take in held_out_takes:
# A *training* take of the same style provides the style example,
# so the only thing taken from the unseen take is its speech
example = training_take_with_style(take.style)
generate_gesture(
audio_file=f'{take}.wav',
styles=[(f'{example}.bvh', None)],
first_pose=f'{example}.bvh',
temperature=1.0,
seed=1234,
network_path=models_dir, # pretrained v1 model
data_path=processed_dir,
results_path=out_dir) Everything is cut into 30-second windows: 165 from the training takes, which the detector learns from, and 69 each from the held-out takes and from ZeroEGGS's output. I kept both controls, built from the held-out takes. If cleaned real motion gets flagged, or Frankenstein motion doesn't, nothing else can be trusted.
One practical detail: the real data fades into a sensor noise floor above about 15 Hz, so I only analyze up to 8 Hz. That leaves the 6 Hz cleanup control something to measure, which was the lesson from the toy.
Results
Here are the detection results:
Real gestures: how well each kind of motion is told apart from real
Dot: score; line: 95% interval from resampling the 21 held-out takes.
Show as a table
The controls behave: coordination ignores the cleanup and catches every Frankenstein window. But the ZeroEGGS row did not go the way I expected. The detector which only knows real motion catches ZeroEGGS best with energy shape (0.82). Coordination only manages 0.62, and its interval includes chance.
Looking at the spectrum makes it clear why:
ZeroEGGS is smoother than the actor
How much the right wrist moves up and down at each tempo (median over the 21 held-out takes, log scale)
Show as a table
ZeroEGGS is too smooth. Across all joints it has about 40× less movement at 4–8 Hz than the actor: the quick, small motions of a real body are mostly not there. That's the averaging effect from the primer above, and it is by far the cheapest giveaway.
What's Different About ZeroEGGS
So is ZeroEGGS's coordination fine? Not quite. The detector above averages about 2,500 coordination numbers per window, and a difference concentrated in some joint pairs easily gets lost in that average. The fairer question is whether a difference exists at all, and for that we can train a classifier with labelled examples, always testing on takes it has never seen:
prob = np.zeros(len(y))
for take in held_out_takes:
test = (groups == take) # this take's real AND generated windows
model.fit(X[~test], y[~test])
prob[test] = model.predict_proba(X[test])[:, 1]
auc = roc_auc_score(y, prob) With examples, coordination separates ZeroEGGS from the actor best of all, with an AUC of 0.99, ahead of energy (0.95) and energy shape (0.94). It also confirms the toy's warning: from energy, the classifier flags cleaned real motion as fake (0.77). From coordination, it barely notices (0.56). You can flip between the two detectors in the chart above.
The difference also has a very clear shape. Here it is averaged over all 55 joint pairs:
Too locked when slow, too loose when fast
Coherence of ZeroEGGS minus the actor's, averaged over all 55 joint pairs, from slow to fast tempos
Show as a table
At slow tempos ZeroEGGS keeps the upper body more in lockstep than the actor does, and at fast tempos it keeps it less. Drawing every joint pair on the body makes the flip easy to see. Try dragging the tempo slider:
Which joints keep time differently
Every line is a joint pair. Warm: ZeroEGGS keeps them more in lockstep than the actor. Cool: less. Faint: no clear difference.
–
Show the biggest differences in this band as a table
In the slowest band the elbows are the story: in ZeroEGGS they move with the chest and the opposite shoulder about twice as steadily as in the actor (right elbow and chest go from 0.19 for the actor to 0.39 for ZeroEGGS). In the fastest band it is the head: its quick movements are less tied to the shoulders and arms (head and left shoulder go from 0.67 to 0.55). And the flip is almost total. In the slowest band 38 of the 55 pairs are clearly more in lockstep than the actor, and only 4 less. In the fastest band 37 are clearly more independent, and only 5 more locked.
My reading of this (and it is a reading rather than a result) is that the model moves the arms and torso more as one block on slow time scales, where a person's slow drifts are more independent. On fast time scales a person's quick movements, like a beat or a laugh, ripple through the head and shoulders together, while the model's fast movements are weaker and less shared. Put simply, real people are loose when slow and tightly linked when fast. ZeroEGGS is the other way around.
Before believing any of this I checked the two things most likely to fake it. The two skeletons have identical bone lengths, so the smaller ZeroEGGS motion isn't a units mix-up. And ZeroEGGS's weak fast motion still sits far above any noise floor, so noise can't explain its lower fast-tempo coherence.
Lessons
To finish, here are the things I took away from all this.
Smoothness is the easy tell
If you want to spot generated motion without having examples of it, look at how much fast movement there is first. It is cheap to compute, it needs no labels, and for ZeroEGGS it was the strongest signal. I suspect this will be true of many learned models for a while, though it is also the easiest thing for a generator to fix.
Coherence is about timing, not size
That's what makes coordination robust. Cleanup filters and changes in scale both change energy, but leave coherence alone. If you compare motion from different pipelines, energy will happily mistake a cleaned take for a fake one. Coordination won't.
Only measure where the signal survives
Coherence is only invariant to filtering in theory. In practice, once a band has been filtered away there is nothing left to measure but leakage. Check where the spectrum hits its noise floor, and stop before it.
Steady isn't the same as right
Coherence tells you that two joints keep time, not that they keep the right time. Arms swinging with the wrong leg are perfectly steady. Always look at the timing offset as well.
Coordination is where the interesting differences are
Smoothness tells you that motion is generated. Coordination across tempos tells you how it differs, and for ZeroEGGS it points at something specific to fix: loosen the slow coupling between the arms and torso, and let fast movements travel through the upper body.
And the caveats, which are important:
- This is one actor, one model and one random seed. It is a case study of ZeroEGGS, not a claim about generated motion in general.
- The cleanup control is weak on this data, since the mocap was already smooth above 6 Hz, so a 6 Hz filter changes little.
- The Speech style has no training takes, so its three held-out takes used another unseen Speech take as the style example.
References
- J. Frank, T. Eisenhofer, L. Schönherr, A. Fischer, D. Kolossa, T. Holz. Leveraging Frequency Analysis for Deep Fake Image Recognition. ICML 2020.
- S. Ghorbani, Y. Ferstl, D. Holden, N. F. Troje, M.-A. Carbonneau. ZeroEGGS: Zero-shot Example-based Gesture Generation from Speech. Computer Graphics Forum, 2023. Dataset and model: github.com/ubisoft/ubisoft-laforge-ZeroEGGS.
- L. Ren, A. Patrick, A. A. Efros, J. K. Hodgins, J. M. Rehg. A Data-Driven Approach to Quantifying Natural Human Motion. SIGGRAPH 2005.
- A. Gopalakrishnan, A. Mali, D. Kifer, C. L. Giles, A. G. Ororbia. A Neural Temporal Model for Human Motion Prediction. CVPR 2019.
- M. Boháček, H. Farid. Human Action CLIPs: Detecting AI-generated Human Motion. 2024.
The ZEGGS dataset and the ZeroEGGS model are © Ubisoft and licensed CC BY-NC-ND 4.0. The figures here show statistics computed from them; no motion from the dataset is reproduced. The toy walker is a simulation.