A Gated Fusion Network for Dynamic Saliency Prediction

A two-stream spatio-temporal saliency network model with a gated integration mechanism.

Our two-stream dynamic saliency model uses RGB frames for spatial stream and optical flow images for temporal stream. These streams are integrated with a dynamic fusion strategy that we referred to as gated fusion. Our architecture also employs multi-level information block to fuse multi-scale features and attention blocks for feature selection.
paper thumbnail

Aysun Kocak, Aykut Erdem, and Erkut Erdem. "A Gated Fusion Network for Dynamic Saliency Prediction", IEEE Transactions on Cognitive and Developmental Systems, 2021.
Paper | Bibtex

Pre-trained models: PyTorch models


Predicting saliency in videos is a challenging problem due to complex modeling of interactions between spatial and temporal information, especially when ever-changing, dynamic nature of videos is considered. Recently, researchers have proposed large-scale datasets and models that take advantage of deep learning as a way to understand what's important for video saliency. These approaches, however, learn to combine spatial and temporal features in a static manner and do not adapt themselves much to the changes in the video content. In this paper, we introduce Gated Fusion Network for dynamic saliency (GFSalNet), the first deep saliency model capable of making predictions in a dynamic way via gated fusion mechanism. Moreover, our model also exploits spatial and channel-wise attention within a multi-scale architecture that further allows for highly accurate predictions. \revise{We evaluate the proposed approach on a number of datasets, and our experimental analysis demonstrates that it outperforms or is highly competitive with the state of the art. Importantly, we show that it has a good generalization ability, and moreover, exploits temporal information more effectively via its adaptive gated fusion mechanism.


This work was supported in part by TUBA GEBIP fellowship awarded to E. Erdem.