All work

CVPR 2024 · 3D humans · Sign language

Neural Sign Actors

A diffusion model for 3D sign language production from text

Vasileios Baltatzis1, Rolandos Alexandros Potamias1, Evangelos Ververas1, Guanxiong Sun2, Jiankang Deng1, Stefanos Zafeiriou1

1Imperial College London · 2Queen’s University Belfast

Fig. 1From the paper

“You want to just kinda take it easy and then use more paint as you start filling in your details.”

Seven video frames of a How2Sign signer on a green screen signing the sentence above, each with the matching 3D SMPL-X avatar pose below it.
All 7 frames: reference How2Sign signer (top) and 3D SMPL-X avatar (bottom). … Source: paper Fig. 1, official project page.

In one paragraph

Sign language production needs expressive hands, bodies and faces, not 2D keypoints or glosses. Neural Sign Actors builds what the authors describe as the first large-scale 3D dataset of American Sign Language, by reconstructing How2Sign with SMPL-X through a fitting pipeline constrained by pose priors. It then trains a text-conditioned diffusion model whose denoiser is an anatomically informed graph network on the SMPL-X skeleton. It generates 3D signing avatars directly from text, and fluent ASL signers judged the results in a user study.

The problem

Sign languages are the main way of communicating for more than 70 million Deaf and Hard of Hearing people. Deep learning has moved sign language recognition and translation forward quickly. Production, turning text into signing, is harder: the output has to look natural and carry exact meaning, through the shape, orientation, position and motion of the hands, and through the arms and the face.

Most production methods generate 2D skeletons, and their output is limited in how expressive the hands and body can be. The obvious fix, generating a full 3D body with articulated fingers and a face, had no data to learn from: there was no large-scale dataset pairing text with 3D signing avatars.1

Figures and challenges as stated on the CVPR 2024 poster.

Data: a 3D version of How2Sign

So the first step was data. How2Sign (Duarte et al., CVPR 2021) is a large corpus of continuous American Sign Language video with English transcripts. The team fitted the SMPL-X model, body, hands and face together, to its frames, turning the corpus into a dataset of 4D signing avatars.

ASL video (69.62 / 3.91 / 5.59 h train / val / test)
~79hours
sentences, each an English transcript with its signing (31,128 / 1,741 / 2,322)
35,191pairs
SMPL-X body, hands and face, fitted frame by frame
4Davatars

Source: CVPR 2024 poster, How2Sign dataset table (totals are the sums of the three splits).

OSX initialises SMPL-X; the fit is then refined against MediaPipe joints with temporal-consistency and pose-prior losses, 𝓛 = 𝓛rec + λ1𝓛temp + λ2𝓛prior, where the prior is a PCA hand-pose prior that keeps the poses feasible. Fig. 2 compares the resulting fits with off-the-shelf whole-body estimators.

Fig. 2From the paper
3D reconstruction error of the fitting pipeline against whole-body estimators
MethodBodyLeft handRight hand
FrankMoCap78.0720.4719.62
PIXIE60.1125.0222.42
PyMAF-X68.6121.4619.19
SMPLify-X56.0722.2318.83
SGNify55.6319.2217.50
OSX47.3218.3418.12
Proposed fit46.4216.1715.23

The largest margin over the best estimator is on the hands. Unit and benchmark to confirm.

Source: CVPR 2024 poster, Dataset Evaluation table (“Proposed” in the original).

Text in, signing out

With 3D data in hand, the model learns to generate signing straight from a sentence, with no glosses in between. Fig. 3 puts the paper’s qualitative comparison under your hands: four sentences, each with the How2Sign signer for reference and two avatars generated from the transcript, one by an earlier method (Stoll et al.) and one by Neural Sign Actors.

Step through the frames and watch the hands. Then switch to Average: it lays the four frames over each other, so a pose that barely changes stays sharp and a hand that moves leaves a trail of ghosts. The frames are not aligned to each other, so some of the blur is the body swaying.

Fig. 3 Real output

Sentence explorer

“Another technique is you can braid the hair on first and then start wrapping cause some people have really short hair and we can create the look with adding the hair end.”

Look at frame 2. The signer lifts a hand to the side of the head, and so does the Neural Sign Actors avatar. The Stoll et al. avatar keeps its hands in front of its body in all four frames.

Ground truth (video)Reference signer, How2Sign

Four video frames of the reference signer on a green screen.

Stoll et al.Generated from the text

Four frames of the 3D avatar generated by Stoll et al.

Neural Sign ActorsGenerated from the text

Four frames of the 3D avatar generated by Neural Sign Actors.
2
View

Frame 2 of 4: reference signer, Stoll et al. and Neural Sign Actors.

Showing sentence 1 of 4. With JavaScript on, you can switch sentences, step through the frames and average them. All four sentences are in the paper’s qualitative figure on the official project page.

Source: paper qualitative comparison (official project page), cropped into one strip per row.

How it works

Neural Sign Actors is a diffusion model over whole signing sequences. A sequence p1:F holds, for every frame, the SMPL-X joint rotations θ and the facial expression parameters ψ. Training adds noise to it step by step; the network learns to predict that noise, so at test time it can start from pure noise and denoise its way to a new sequence, conditioned on the sentence.

Fig. 4From the paper
Enlarge figure: Method diagram: forward diffusion adds noise to the ground-truth sign; reverse diffusion uses the denoiser, which combines a CLIP text encoding with a graph-network pose encoder and an expression encoder, then an auto-regressive RNN and a regression head that output the generated pose and expression.
  1. Forward diffusion. Noise is added to the ground-truth pose sequence p1:F.
  2. Reverse diffusion εθ. The denoiser predicts the noise at each step.
  3. CLIP text encoder. The transcript conditions every denoising step.
  4. Pose embedding + GNN pose encoder. Per-joint embeddings, then a graph network on the SMPL-X skeleton (anatomically informed).
  5. Expression embedding + MLP encoder. Facial expression parameters ψ.
  6. Auto-regressive RNN. Runs over frames 1…F.
  7. Regression head. Outputs pose θ and expression ψ.
The denoiser εθ(·) inside the reverse diffusion. Green blocks are learned; grey blocks (the forward process and CLIP) are not. Select the diagram to enlarge it. Source: paper method figure, official project page.

Inside the denoiser (Fig. 4), the noisy joint rotations get a learned per-joint pose embedding and go through a graph neural network defined on the SMPL-X skeleton, so messages pass along the skeleton and a finger joint is encoded together with the wrist and arm it hangs from. The facial expressions take a parallel path through their own embedding and an MLP. The transcript is encoded by CLIP, and the CLIP encoding conditions the model.

The per-frame features x1…xF then pass through an auto-regressive RNN, which carries information across frames, and an MLP regression head turns them back into pose θ1:F and expression ψ1:F. Training minimises the usual noise-prediction loss, 𝓛t = ‖εt − εθ(pt1:F, t, w1:F)‖2. At inference the model starts from zT ∼ 𝒩(0, I) and runs the reverse process to produce a new signing sequence for any sentence.

Results

On How2Sign, Neural Sign Actors is compared with four earlier sign-language production baselines on four errors: vertex and joint position (MPVPE, MPJPE), a distribution distance (FID) and a sequence-alignment distance (DTW).2 It has the lowest error on all four.

best baseline 55.02
31.47MPVPE ↓
best baseline 3.98
1.56FID ↓
best baseline 13.78
7.83DTW ↓

Source: CVPR 2024 poster, SLP Evaluation table. ↓ lower is better.

Fig. 5From the paper
Sign language production from text, How2Sign
MethodMPVPEMPJPEFIDDTW
Saunders et al. (i)67.2170.064.7114.15
Saunders et al. (ii)63.1965.253.9813.78
Hwang et al.62.7463.254.4513.94
Stoll et al.55.0260.324.9613.99
Neural Sign Actors31.4735.871.567.83

Saunders et al. (i) and (ii) are two baselines with the same first author, in the poster’s order. …

Source: CVPR 2024 poster, SLP Evaluation table (“Proposed” in the original).

Signers in the loop

Numbers on poses do not say whether the signing is understandable. In a user study, 15 ASL-fluent participants rated how well each clip aligns with its transcript: the original video, its 3D fit, and the generations of Stoll et al. and Neural Sign Actors.

Fig. 6From the paper
User study panel: 15 ASL-fluent subjects rated the alignment between generations and text transcripts. Horizontal bars with error bars on a 0 to 10 alignment score for Stoll et al., Proposed, 3D Fitting and Video.
Alignment score between each clip (video, 3D fit or generation) and its transcript, rated by 15 ASL-fluent participants. … Source: CVPR 2024 poster, User Study panel (not re-plotted).

Talk and poster

Fig. 7From the paper
Watch the CVPR 2024 presentation (YouTube)
CVPR 2024 presentation. Nothing loads from YouTube until you press play. Open on YouTube Source: official project page. Thumbnail: crop of paper Fig. 1.
Fig. 8From the paper
Enlarge figure: The CVPR 2024 poster: introduction, fitting optimisation, dataset evaluation, the SLP pipeline, SLP evaluation and the user study.
The CVPR 2024 poster. Select it to enlarge. Source: official project page.

Where it led

  1. 2024Neural Sign Actorstext to 3D signing motion
  2. 2025Limited-data diffusiondiffusion when data is scarce, led by Zhaoyu Zhang
  3. In progressPhysical AIhuman motion as a prior for robot motion

Signing avatars are a demanding case of a general problem: generating human motion from language, with the hands and timing right. Robots that learn from people face the same problem from the other side, which is where I want to take it next.

Cite

@InProceedings{Baltatzis_2024_CVPR,
  author    = {Baltatzis, Vasileios and Potamias, Rolandos Alexandros and Ververas, Evangelos and Sun, Guanxiong and Deng, Jiankang and Zafeiriou, Stefanos},
  title     = {Neural Sign Actors: A diffusion model for 3D sign language production from text},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month     = {June},
  year      = {2024}
}

Vasileios Baltatzis, Rolandos Alexandros Potamias, Evangelos Ververas, Guanxiong Sun, Jiankang Deng, Stefanos Zafeiriou. Neural Sign Actors: A Diffusion Model for 3D Sign Language Production from Text. CVPR 2024.

BibTeX from the official project page (its empty pages field removed).