ICCV 2023 · Video understanding
STPN
Spatio-temporal Prompting Network for Robust Video Feature Extraction
Affiliations to be added from the paper.
In one paragraph
Deteriorated frames (motion blur, defocus, occlusion) hurt every video task. Prior work compensates with heavy integration modules, a different one per task. STPN instead predicts a few video prompts from neighbouring frames and prepends them to the current frame’s patch embeddings, so a standard backbone, with no task-specific module, produces robust features for detection (ImageNet VID), instance segmentation (YouTube-VIS) and tracking (GOT-10k).
The problem
Video frames rarely stay clean. The object moves fast and blurs, the camera loses focus, something passes in front, or the animal turns into an unusual pose. A network that looks at one frame at a time loses exactly the evidence it needs, and the fox in Fig. 1 is a typical case: sharp in a neighbouring frame, a smear in the current one.
The usual remedy is to borrow from neighbouring frames. Methods such as SELSA, MEGA and TransVOD run every frame through the backbone and then fuse the features in an attention-based integration module. That works, but the modules are heavy and complex, and each one is built for one task: a module designed for detection does not carry over to instance segmentation or tracking (Fig. 1a).1
STPN asks a different question. Instead of repairing features after the backbone, can we change what goes into it, so that one standard backbone produces robust features for every video task?
The idea
Fig. 2 rebuilds the pipeline with real pixels from the paper’s figure: a sharp support frame and the motion-blurred current frame. The frame is cut into patch tokens, and STPN puts five prompt tokens in front of them. Pick any patch to see what it can read inside the backbone, then switch to the prior pipeline and change the task.
Prompting the backbone
This interactive figure needs JavaScript. In words: the prior pipeline sends 40 patch tokens of the blurred current frame through the backbone and fuses the support frames afterwards, in an integration module built for one task. STPN predicts 5 prompt tokens from the support frames and prepends them, so 45 tokens enter a standard backbone with no new layers and every patch can read the prompts. Each task keeps its own framework and head, with no integration module.
How it works
Panel (b) of the paper’s figure has five parts. Select a number to highlight its note.
- Support frames. Other frames of the same video: neighbours within ±9 frames in training; at test time, 14 frames spread over the whole video, sampled once.
- DVP predictor. Predicts dynamic video prompts that summarise spatio-temporal information from the support frames.
- Prepend. The prompts are concatenated in front of the current frame’s patch embeddings.
- Backbone. A standard backbone with no new layers (Swin-T in the released detector) processes prompts and patches together; self-attention lets every patch read the prompts.
- Tasks. The same prompting design plugs into detection, instance segmentation and tracking frameworks, with no task-specific integration module.
At test time, the released model samples 14 reference frames spread evenly over the whole video and passes them through the backbone without prompts. The DVP predictor takes their last-stage features (768 channels in Swin-T) and projects them to 96 dimensions, the width of Swin-T’s patch embeddings. Five learned queries then attend over every spatial position of those features, and each query becomes one prompt token.
The prompts are prepended to the current frame’s patch embeddings and travel through all four Swin stages. In every shifted-window attention layer, each 7×7 window attends to the same five prompts, so any patch can read a summary of the video; when patches merge between stages, the prompts are carried along. Before the feature pyramid the prompts are dropped again, so the neck and the task heads are the unchanged still-image ones.
Two things follow. The predictor is small, about 0.11 M parameters,2 and the released code computes the prompts once per video, at its first frame, and reuses them for every later frame, so each further frame costs only five extra tokens. And because nothing after the backbone changes, the same design carries over to segmentation and tracking frameworks without a new integration module.
Results
On ImageNet VID the released Swin-T checkpoint reaches 85.2 AP50. The numbers below come from the authors’ released code (the vfe.pytorch README), not the paper’s tables, and the baselines use ResNet-101, so read them as context rather than a controlled test.
- ImageNet VID val, Swin-T
- 85.2AP50
- VOD · VIS · VOT, no task-specific module
- 3tasks
- prompt tokens (released config)
- 5prompts
Source: vfe.pytorch README (AP50 of the released checkpoint; may differ slightly from the paper’s table) and released config stpn_swint_adam_9x.py (prompts); tasks as evaluated in the paper.
| Method | AP50 |
|---|---|
| Faster R-CNN (R101) | 76.7 |
| SELSA (R101) | 81.5 |
| MEGA (R101) | 82.9 |
| MAMBA (R101) | 83.8 |
| STPN (Swin-T) | 85.2 |
Backbones differ; see the paper for STPN against a same-backbone baseline.
Source: the authors’ released code, not the paper’s tables. vfe.pytorch README: STPN and MAMBA are the released checkpoints (STPN may differ slightly from the paper’s table); Faster R-CNN and MEGA are taken from the MEGA repository, SELSA from MMTracking.
| Speed | MEGA | MAMBA | STPN |
|---|---|---|---|
| Slow | 89.4 | 89.5 | 91.4 |
| Medium | 81.6 | 83.8 | 84.1 |
| Fast | 62.7 | 65.3 | 64.1 |
STPN (Swin-T) scores highest on slow and medium objects; MAMBA (ResNet-101) scores highest on fast ones. Backbones differ (MEGA and MAMBA: ResNet-101), so this is not a controlled comparison.
Source: the authors’ released code, not the paper’s tables. vfe.pytorch README: STPN and MAMBA are the released checkpoints; MEGA is taken from the MEGA repository.
Segmentation and tracking
Results on YouTube-VIS and GOT-10k: see the paper.
| Method | AP | AP50 |
|---|---|---|
| Baseline (from the paper) | — | — |
| + STPN | — | — |
Source: paper (to be copied).
| Method | AO |
|---|---|
| Baseline (from the paper) | — |
| + STPN | — |
Source: paper (to be copied).
Reproducibility
vfe.pytorch 2.0 re-evaluates the released checkpoint and retrains STPN from scratch in plain PyTorch.
| Run | AP50 |
|---|---|
| Released checkpoint, evaluated in vfe.pytorch 2.0 | 85.15 |
| Trained from scratch, vfe.pytorch 2.0 | 84.54 |
| Originally published | 85.15 |
Training lands 0.61 low; likely run-to-run variance, not confirmed with a second seed. Source: vfe.pytorch README. Measured on 4× GH200.
Where it led
- 2021MAMBAa memory bank that enhances features after the backbone
- 2023STPNvideo context as a few tokens at the backbone’s input
- DirectionPhysical AImemory for agents that act
MAMBA kept a memory of features and read from it after the backbone; STPN compressed a video’s context into five tokens that the backbone reads at its input. My current direction, Physical AI, takes the same question, what to remember and where to feed it in, to agents that act in the physical world.
Code
STPN is maintained in vfe.pytorch, a plain-PyTorch rewrite that also holds MAMBA; the original ICCV code is at guanxiongsun/STPN. To evaluate the released checkpoint (CHECKPOINT is the stpn_swint_adam_9x model; data setup is in the README):
uv venv --python 3.12 && uv sync
python -m vfe.cli.test configs/vid/stpn/stpn_swint_adam_9x.py \
CHECKPOINT --work-dir WORK_DIR
Cite
@inproceedings{sun2023stpn,
title = {Spatio-temporal Prompting Network for Robust Video Feature Extraction},
author = {Sun, Guanxiong and Wang, Chi and Zhang, Zhaoyu and Deng, Jiankang and Zafeiriou, Stefanos and Hua, Yang},
booktitle = {ICCV},
year = {2023}
}
Guanxiong Sun, Chi Wang, Zhaoyu Zhang, Jiankang Deng, Stefanos Zafeiriou, Yang Hua. Spatio-temporal Prompting Network for Robust Video Feature Extraction. ICCV 2023. doi:10.1109/ICCV51070.2023.01250