Gaussian Conditional Random Field Network for Semantic Segmentation
Abstract: In contrast to the existing approaches that use discrete Conditional Random Field (CRF) models, we propose to use a Gaussian CRF model for the task of semantic segmentation. We propose a novel deep network, which we refer to as Gaussian Mean Field (GMF) network, whose layers perform mean field inference over a Gaussian CRF. The proposed GMF network has the desired property that each of its layers produces an output that is closer to the maximum a posteriori solution of the Gaussian CRF compared to its input. By combining the proposed GMF network with deep Convolutional Neural Networks (CNNs), we propose a new end-to-end trainable Gaussian conditional random field network. The proposed Gaussian CRF network is composed of three sub-networks: (i) a CNN-based unary network for generating unary potentials, (ii) a CNN-based pairwise network for generating pairwise potentials, and (iii) a GMF network for performing Gaussian CRF inference. When trained end-to-end in a discriminative fashion, and evaluated on the challenging PASCALVOC 2012 segmentation dataset, the proposed Gaussian CRF network outperforms various recent semantic segmentation approaches that combine CNNs with discrete CRF models.
Contributions
- Gaussian CRF for semantic segmentation: In contrast to the existing approaches that use discrete CRF models, we propose to use a GCRF model for the task of semantic segmentation. Compared to discrete CRFs, GCRFs are simpler models that can be solved optimally.
- GMF network: We propose a novel deep network by unfolding a fixed number of Gaussian mean field iterations. Using a bipartite graph structure, we ensure that each layer in our GMF network produces an output that is closer to the optimal solution compared to its input.
- Gaussian CRF network: We propose a new end-to-end trainable deep network that combines the GCRF model with CNNs for the task of semantic segmentation.
- Results: We show that the proposed GCRF network outperforms various existing discrete CRF-based approaches on PASCALVOC 2012 test set (when trained with ImageNet and PASCALVOC data).
Experimental results
Publications:
Raviteja Vemulapalli, Oncel Tuzel, Ming-Yu Liu, and Rama Chellappa, "Gaussian Conditional Random Field Network for Semantic Segmentation", CVPR, 2016. (SPOTLIGHT)
[PDF][Supplementary][Results video]
[PDF][Supplementary][Results video]
Patents:
Oncel Tuzel, Raviteja Vemulapalli and Ming-Yu Liu, "Method and System for Semantic Image Segmentation Using Gaussian Random Field Network", US Patent 9,704,257, issued July 11, 2017.