All work

AAAI 2021 · Video object detection

MAMBA

Multi-level Aggregation via Memory Bank for Video Object Detection

Guanxiong Sun, Yang Hua, Guosheng Hu, Neil Robertson

Affiliations as printed in the paper

Fig. 1From the paper
MAMBA’s memory bank as the video advances: each read samples a small key set from the bank, and single features, not whole frames, are replaced as new frames arrive. Source: method animation by the paper’s authors.

In one paragraph

Video detectors enhance each frame with features from other frames, but sliding windows and memory queues attend to everything they store and evict whole frames at a time. MAMBA’s memory bank samples a light-weight key set for each read and updates memory feature by feature, so it keeps knowledge from the whole video at a fraction of the cost. It aggregates pixel-level and instance-level features through one generalized enhancement operation (GEO).

The problem

A detector that looks at one frame at a time struggles exactly where video is hardest: motion blur, occlusion, objects caught in a rare pose. State-of-the-art video detectors therefore enhance the current frame with features from other frames, using attention over a memory. That memory is either a sliding window of nearby frames or a memory queue of recent ones.

Both structures carry two implied operations, and the paper argues that neither is efficient or sufficient:

  1. Concatenating all features in memory for enhancement. The current frame attends to everything stored, so the cost of every read grows with the memory.
  2. Frame-wise updating. When a new frame arrives, every feature of the oldest frame is deleted at once, so the memory only ever holds a short, recent slice of the video.

One way to see why: adjacent frames are often near-duplicates, so a key set that uses everything is large without being very informative, since much of it repeats nearby frames. And because whole frames are evicted, whatever the rest of the video could have told the detector is gone by the time a hard frame needs it.

The idea

MAMBA changes both operations: read a small random sample of the memory, and write feature by feature, replacing single features at random instead of whole frames. The toy below runs both memories side by side on the same 96-frame video. Watch the coverage strips: they show which frames of the video each memory still holds.

Press Play, step, or drag the frame slider. The queue’s coverage stays a tight cluster just behind the playhead; MAMBA’s spreads across the whole video. It even holds frames the video has not reached yet (the hollow ticks), because its memory was seeded from reference frames spread over the whole video. Drag to the last frame to see what each memory has kept. Then change how many keys MAMBA reads: the cost of a read follows that number, not the size of the memory.

Fig. 2 Schematic

Queue vs. memory bank

40
8

The toy at frame 40. Frame-wise queue: 12 columns holding frames 28 to 39, all 48 features ringed as read; its coverage strip is one tight cluster just behind the playhead. MAMBA memory bank: 48 single features from 45 frames between 0 and 95, 8 of them ringed as read; its coverage strip is spread across the whole video.

This interactive toy needs JavaScript; above is its starting state. Frame 40 of a 96-frame video (frames 0–95), with 48 memory slots on each side. The frame-wise queue holds frames 28 to 39, one whole frame per column, and frame 40 reads all 48 of their features. MAMBA’s bank holds single features from 45 different frames spanning 0 to 95, because it was seeded from 14 reference frames spread over the whole video and replaces one random feature at a time; frame 40 reads a random 8 of them. Fig. 1 animates the same mechanism.

How it works

MAMBA keeps a memory bank for each video and touches it in exactly two ways, both visible in the authors’ method animation (Fig. 3).

Fig. 3From the paper
A row of 2×2 feature groups inside a blue memory band, some cells emptied (dashed) where single features were dropped; a Sampling box above it feeds a small stack of sampled features to the current frame on the right.
  1. Memory bank. Features are stored individually, drawn as 2×2 slot groups; some slots are already empty (dashed), because single features were dropped, not whole frames.
  2. Sampling. Each read draws a small random key set instead of concatenating everything.
  3. Key set. The sampled features the current frame attends to.
  4. Current frame. Enhanced by attending to the key set.
  5. Feature-wise update. New features replace individual old ones, so knowledge from the whole video survives.
A still from the authors’ method animation (Fig. 1), with its parts numbered. Source: MAMBA method animation, by the paper’s authors.

Read: light-weight key-set construction. The current frame does not attend to the whole bank. It draws a random key set and attends only to that, so the cost of a read is set by the key-set size, not by how much the bank holds. Write: fine-grained, feature-wise updating. After a frame has been enhanced, its best features go back into the bank; once the bank is full, random individual features make room for them. The bank therefore becomes a thinned-out sample of the whole video rather than a copy of its last few frames.

In the released code, the memory is primed from the first frame of each video, which receives 14 reference frames spread evenly over the whole video: the top-75 RoI features of those frames, together with the first frame’s own, seed the bank. Every later frame reads a sampled key set and writes back its top-75 enhanced RoI features. The memory resets for each video.

GEO. A detector’s features live at two levels: pixel-level feature maps and instance-level proposals. MAMBA keeps memory at both and enhances both with one generalized enhancement operation (GEO), which aggregates multi-level features in a unified way: features of the current frame attend to a sampled key set, and what they gather is added back to them.

Results

The numbers come from two places, and each carries its source: the paper (mAP and speed with ResNet-101) and the vfe.pytorch README (AP50 of the released model, which uses the instance-level memory).

at 12.6 FPS, ResNet-101
83.7mAP
at 9.1 FPS, ResNet-101
84.6mAP
released model, ImageNet VID val
83.8AP50

Source: paper abstract (mAP and FPS, ResNet-101, ImageNet VID); vfe.pytorch README (AP50 of the released checkpoint).

Fig. 4From the paper
ImageNet VID val, AP50 (ResNet-101)
MethodAP50
Faster R-CNN76.7
SELSA81.5
MEGA82.9
MAMBA83.8

Source: the authors’ released code, not the paper’s tables. vfe.pytorch README: MAMBA is the released checkpoint (instance-level memory; the paper reports 83.7 and 84.6 mAP); Faster R-CNN (the single-frame baseline) and MEGA are taken from the MEGA repository, SELSA from MMTracking.

Fig. 5From the paper
Gain over the single-frame baseline, by object speed (AP50)
SpeedMEGAMAMBA
Slow+4.5+4.6
Medium+7.5+9.7
Fast+10.4+13.0

Memory helps most when objects move fast.

Source: the authors’ released code, not the paper’s tables. Computed from the vfe.pytorch README, AP by speed (slow / medium / fast): Faster R-CNN 84.9 / 74.1 / 52.3, MEGA 89.4 / 81.6 / 62.7, MAMBA 89.5 / 83.8 / 65.3. Gain = method − Faster R-CNN; for fast objects, 62.7 − 52.3 = 10.4 and 65.3 − 52.3 = 13.0.

Reproducibility, ImageNet VID val AP50 (MAMBA)
RunAP50
Released checkpoint, evaluated with vfe.pytorch 2.083.80
Trained from scratch with vfe.pytorch 2.084.06
Originally published83.82

Source: vfe.pytorch README. Measured on 4× GH200. Reading the config literally (batch 8 throughout) scores 83.16; 84.06 follows the published model’s own schedule.

Where it led

  1. 2021MAMBAsampled reads, feature-wise writes
  2. 2022TDViTtemporally dilated attention
  3. 2023STPNprompts instead of integration modules
  4. DirectionPhysical AImemory for agents that act

MAMBA started a line of work on what a video model should remember and how cheaply it can read it back. TDViT stacked temporally dilated attention blocks to reach far back in time cheaply, and STPN moved from separate integration modules to video prompts for one shared backbone. My current direction, Physical AI, asks the same question of agents that act in the physical world.

Code

The reference implementation and the released checkpoint live in vfe.pytorch, which also holds STPN. Version 2.0 is a rewrite in plain PyTorch, with nothing from mmcv or mmdet at runtime. After downloading ImageNet VID from the Hugging Face mirror and arranging it as the README describes, this evaluates the released model on one GPU:

uv venv --python 3.12 && uv sync
python -m vfe.cli.test configs/vid/mamba/mamba_r101_dc5_6x.py \
    CHECKPOINT --work-dir WORK_DIR

Cite

@inproceedings{sun2021mamba,
  title     = {MAMBA: Multi-level Aggregation via Memory Bank for Video Object Detection},
  author    = {Sun, Guanxiong and Hua, Yang and Hu, Guosheng and Robertson, Neil},
  booktitle = {AAAI},
  year      = {2021}
}

Guanxiong Sun, Yang Hua, Guosheng Hu, Neil Robertson. MAMBA: Multi-level Aggregation via Memory Bank for Video Object Detection. AAAI 2021.