Home

Vision

Next: memory for machines that act.

Where the thread leads: from memory banks for video to memory for agents that act in the physical world, and the work in progress.

My perception work has circled one question: what should a machine keep from what it has seen? In video the answer was a memory bank that stores features from the entire video and reads them cheaply (MAMBA), and prompts that carry context from neighbouring frames into the backbone (STPN). In generation the question was structure: learning human motion well enough to produce signing avatars from text (Neural Sign Actors, with colleagues at Imperial College London).

Physical AI raises the stakes. A robot must remember over minutes rather than frames, on a fixed compute budget. Memory is arriving in vision-language-action policies, but mostly as a record of what the robot itself did. Robots that share space with people must also remember what others change.

Perception with memory, generation with structure, action with both.

Direction · 2026

Physical AI: perception and generation, grounded in interaction

Schematic · work in progress
Schematic: a loop from perceive, to predict with a world model, to act, to the environment and back to perceive, with memory at the centre linked to every step. memory perceive predict (world model) act environment

Grounding perception and generative models in physical interaction, for robots that share space with people.

Open questions

  • What should a robot remember about the people it works with?
  • What belongs in weights, and what in context?
  • What transfers from human-motion generation to robot motion?

See the open problems