SlowFast

Abstract

We present SlowFast networks for video recognition. Our model involves (i) a Slow pathway, operating at low frame rate, to capture spatial semantics, and (ii) a Fast pathway, operating at high frame rate, to capture motion at fine temporal resolution. The Fast pathway can be made very lightweight by reducing its channel capacity, yet can learn useful temporal information for video recognition. Our models achieve strong performance for both action classification and detection in video, and large improvements are pin-pointed as contributions by our SlowFast concept. We report state-of-the-art accuracy on major video recognition benchmarks, Kinetics, Charades and AVA.

Citation

@inproceedings{feichtenhofer2019slowfast,
  title={Slowfast networks for video recognition},
  author={Feichtenhofer, Christoph and Fan, Haoqi and Malik, Jitendra and He, Kaiming},
  booktitle={Proceedings of the IEEE international conference on computer vision},
  pages={6202--6211},
  year={2019}
}

Model Zoo

Kinetics-400

config	resolution	gpus	backbone	pretrain	top1 acc	top5 acc	inference_time(video/s)	gpu_mem(M)	ckpt	log	json
slowfast_r50_4x16x1_256e_kinetics400_rgb	short-side 256	8x4	ResNet50	None	74.75	91.73	x	6203	ckpt	log	json
slowfast_r50_video_4x16x1_256e_kinetics400_rgb	short-side 256	8	ResNet50	None	73.95	91.50	x	6203	ckpt	log	json
slowfast_r50_4x16x1_256e_kinetics400_rgb	short-side 320	8x2	ResNet50	None	76.0	92.54	1.6 ((32+4)x10x3 frames)	6203	ckpt	log	json
slowfast_prebn_r50_4x16x1_256e_kinetics400_rgb	short-side 320	8x2	ResNet50	None	76.34	92.67	x	6203	ckpt	log	json
slowfast_r50_8x8x1_256e_kinetics400_rgb	short-side 320	8x3	ResNet50	None	76.94	92.8	1.3 ((32+8)x10x3 frames)	9062	ckpt	log	json
slowfast_r101_r50_4x16x1_256e_kinetics400_rgb	short-side 256	8x1	ResNet101 + ResNet50	None	76.69	93.07		16628	ckpt	log	json
slowfast_r101_8x8x1_256e_kinetics400_rgb	short-side 256	8x4	ResNet101	None	77.90	93.51		25994	ckpt	log	json
slowfast_r152_r50_4x16x1_256e_kinetics400_rgb	short-side 256	8x1	ResNet152 + ResNet50	None	77.13	93.20		10077	ckpt	log	json

Something-Something V1

config	resolution	gpus	backbone	pretrain	top1 acc	top5 acc	inference_time(video/s)	gpu_mem(M)	ckpt	log	json
slowfast_r50_16x8x1_22e_sthv1_rgb	height 100	8	ResNet50	Kinetics400	49.67	79.00	x	9293	ckpt	log	json

:::{note}

The gpus indicates the number of gpu we used to get the checkpoint. It is noteworthy that the configs we provide are used for 8 gpus as default.
According to the Linear Scaling Rule, you may set the learning rate proportional to the batch size if you use different GPUs or videos per GPU,
e.g., lr=0.01 for 4 GPUs x 2 video/gpu and lr=0.08 for 16 GPUs x 4 video/gpu.
The inference_time is got by this benchmark script, where we use the sampling frames strategy of the test setting and only care about the model inference time, not including the IO time and pre-processing time. For each setting, we use 1 gpu and set batch size (videos per gpu) to 1 to calculate the inference time.
The validation set of Kinetics400 we used consists of 19796 videos. These videos are available at Kinetics400-Validation. The corresponding data list (each line is of the format 'video_id, num_frames, label_index') and the label map are also available.

:::

For more details on data preparation, you can refer to Kinetics400 in Data Preparation.

Train

You can use the following command to train a model.

python tools/train.py ${CONFIG_FILE} [optional arguments]

Example: train SlowFast model on Kinetics-400 dataset in a deterministic option with periodic validation.

python tools/train.py configs/recognition/slowfast/slowfast_r50_4x16x1_256e_kinetics400_rgb.py \
    --work-dir work_dirs/slowfast_r50_4x16x1_256e_kinetics400_rgb \
    --validate --seed 0 --deterministic

For more details, you can refer to Training setting part in getting_started.

Test

You can use the following command to test a model.

python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [optional arguments]

Example: test SlowFast model on Kinetics-400 dataset and dump the result to a json file.

python tools/test.py configs/recognition/slowfast/slowfast_r50_4x16x1_256e_kinetics400_rgb.py \
    checkpoints/SOME_CHECKPOINT.pth --eval top_k_accuracy mean_class_accuracy \
    --out result.json --average-clips=prob

For more details, you can refer to Test a dataset part in getting_started.