李飞飞论文整理／计算机视觉论文／斯坦福四_计算机视觉_其他

<html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta charset="utf-8"> <meta name="generator" content="pdf2htmlEX"> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> <link rel="stylesheet" href="https://static.pudn.com/base/css/base.min.css"> <link rel="stylesheet" href="https://static.pudn.com/base/css/fancy.min.css"> <link rel="stylesheet" href="https://static.pudn.com/prod/directory_preview_static/624b58668947fd5953ee5026/raw.css"> <script src="https://static.pudn.com/base/js/compatibility.min.js"></script> <script src="https://static.pudn.com/base/js/pdf2htmlEX.min.js"></script> <script> try{ pdf2htmlEX.defaultViewer = new pdf2htmlEX.Viewer({}); }catch(e){} </script> <title></title> </head> <body> <div id="sidebar" style="display: none"> <div id="outline"> </div> </div> <div id="pf1" class="pf w0 h0" data-page-no="1"><div class="pc pc1 w0 h0"><img class="bi x0 y0 w1 h1" alt="" src="https://static.pudn.com/prod/directory_preview_static/624b58668947fd5953ee5026/bg1.jpg"><div class="t m0 x1 h2 y1 ff1 fs0 fc0 sc0 ls0 ws0">DenseCap: Fully Convolutional Localization Networks for Dense Captioning</div><div class="t m0 x2 h3 y2 ff2 fs1 fc0 sc0 ls0 ws0">Justin Johnson</div><div class="t m0 x3 h4 y3 ff3 fs2 fc0 sc0 ls0 ws0">∗</div><div class="t m0 x4 h3 y2 ff2 fs1 fc0 sc0 ls0 ws0">Andrej Karpathy</div><div class="t m0 x5 h4 y3 ff3 fs2 fc0 sc0 ls0 ws0">∗</div><div class="t m0 x6 h3 y2 ff2 fs1 fc0 sc0 ls0 ws0">Li Fei-Fei</div><div class="t m0 x7 h3 y4 ff2 fs1 fc0 sc0 ls0 ws0">Department of Computer Science, Stanford University</div><div class="t m0 x8 h5 y5 ff4 fs3 fc0 sc0 ls0 ws0">{jcjohns,karpathy,feifeili}@cs.stanford.edu</div><div class="t m0 x9 h6 y6 ff1 fs1 fc0 sc0 ls0 ws0">Abstract</div><div class="t m0 xa h7 y7 ff6 fs4 fc0 sc0 ls0 ws0">We introduce the dense captioning task, which requires a</div><div class="t m0 xa h7 y8 ff6 fs4 fc0 sc0 ls0 ws0">computer vision system to both localize and describe salient</div><div class="t m0 xa h7 y9 ff6 fs4 fc0 sc0 ls0 ws0">regions in images in natural language. The dense caption-</div><div class="t m0 xa h7 ya ff6 fs4 fc0 sc0 ls0 ws0">ing task generalizes object detection when the descriptions</div><div class="t m0 xa h7 yb ff6 fs4 fc0 sc0 ls0 ws0">consist of a single word, and Image Captioning when one</div><div class="t m0 xa h7 yc ff6 fs4 fc0 sc0 ls0 ws0">predicted region covers the full image. To address the local-</div><div class="t m0 xa h7 yd ff6 fs4 fc0 sc0 ls0 ws0">ization and description task jointly we propose a Fully Con-</div><div class="t m0 xa h7 ye ff6 fs4 fc0 sc0 ls0 ws0">volutional Localization Network (FCLN) architecture that</div><div class="t m0 xa h7 yf ff6 fs4 fc0 sc0 ls0 ws0">processes an image with a single, efﬁcient forward pass, re-</div><div class="t m0 xa h7 y10 ff6 fs4 fc0 sc0 ls0 ws0">quires no external regions proposals, and can be trained</div><div class="t m0 xa h7 y11 ff6 fs4 fc0 sc0 ls0 ws0">end-to-end with a single round of optimization. The archi-</div><div class="t m0 xa h7 y12 ff6 fs4 fc0 sc0 ls0 ws0">tecture is composed of a Convolutional Network, a novel</div><div class="t m0 xa h7 y13 ff6 fs4 fc0 sc0 ls0 ws0">dense localization layer, and Recurrent Neural Network</div><div class="t m0 xa h7 y14 ff6 fs4 fc0 sc0 ls0 ws0">language model that generates the label sequences. We</div><div class="t m0 xa h7 y15 ff6 fs4 fc0 sc0 ls0 ws0">evaluate our network on the Visual Genome dataset, which</div><div class="t m0 xa h7 y16 ff6 fs4 fc0 sc0 ls0 ws0">comprises 94,000 images and 4,100,000 region-grounded</div><div class="t m0 xa h7 y17 ff6 fs4 fc0 sc0 ls0 ws0">captions. We observe both speed and accuracy improve-</div><div class="t m0 xa h7 y18 ff6 fs4 fc0 sc0 ls0 ws0">ments over baselines based on current state of the art ap-</div><div class="t m0 xa h7 y19 ff6 fs4 fc0 sc0 ls0 ws0">proaches in both generation and retrieval settings.</div><div class="t m0 xa h6 y1a ff1 fs1 fc0 sc0 ls0 ws0">1. Introduction</div><div class="t m0 xa h8 y1b ff2 fs4 fc0 sc0 ls0 ws0">Our ability to effortlessly point out and describe all aspects</div><div class="t m0 xa h8 y1c ff2 fs4 fc0 sc0 ls0 ws0">of an image relies on a strong semantic understanding of a</div><div class="t m0 xa h8 y1d ff2 fs4 fc0 sc0 ls0 ws0">visual scene and all of its elements. However, despite nu-</div><div class="t m0 xa h8 y1e ff2 fs4 fc0 sc0 ls0 ws0">merous potential applications, this ability remains a chal-</div><div class="t m0 xa h8 y1f ff2 fs4 fc0 sc0 ls0 ws0">lenge for our state of the art visual recognition systems.</div><div class="t m0 xa h8 y20 ff2 fs4 fc0 sc0 ls0 ws0">In the last few years there has been signiﬁcant progress</div><div class="t m0 xa h8 y21 ff2 fs4 fc0 sc0 ls0 ws0">in image classiﬁcation [39, 26, 53, 45], where the task is</div><div class="t m0 xa h8 y22 ff2 fs4 fc0 sc0 ls0 ws0">to assign one label to an image. Further work has pushed</div><div class="t m0 xa h8 y23 ff2 fs4 fc0 sc0 ls0 ws0">these advances along two orthogonal directions: First, rapid</div><div class="t m0 xa h8 y24 ff2 fs4 fc0 sc0 ls0 ws0">progress in object detection [40, 14, 46] has identiﬁed mod-</div><div class="t m0 xa h8 y25 ff2 fs4 fc0 sc0 ls0 ws0">els that efﬁciently identify and label multiple salient regions</div><div class="t m0 xa h8 y26 ff2 fs4 fc0 sc0 ls0 ws0">of an image. Second, recent advances in image captioning</div><div class="t m0 xa h8 y27 ff2 fs4 fc0 sc0 ls0 ws0">[3, 32, 21, 49, 51, 8, 4] have expanded the complexity of</div><div class="t m0 xa h8 y28 ff2 fs4 fc0 sc0 ls0 ws0">the label space from a ﬁxed set of categories to sequence of</div><div class="t m0 xa h8 y29 ff2 fs4 fc0 sc0 ls0 ws0">words able to express signiﬁcantly richer concepts.</div><div class="t m0 xb h8 y2a ff2 fs4 fc0 sc0 ls0 ws0">However, despite encouraging progress along the label</div><div class="t m0 xa h8 y2b ff2 fs4 fc0 sc0 ls0 ws0">density and label complexity axes, these two directions have</div><div class="t m0 xc h9 y2c ff7 fs5 fc0 sc0 ls0 ws0">∗</div><div class="t m0 xd ha y2d ff2 fs2 fc0 sc0 ls0 ws0">Both authors contributed equally to this work.</div><div class="c xe y2e w2 hb"><div class="t m0 xf hc y2f ff8 fs6 fc0 sc0 ls0 ws0">Classification</div><div class="t m0 x10 hc y30 ff8 fs6 fc0 sc0 ls0 ws0">Cat</div><div class="t m0 x11 hc y31 ff8 fs6 fc0 sc0 ls0 ws0">Captioning</div><div class="t m0 x12 hc y32 ff8 fs6 fc0 sc0 ls0 ws0">A cat </div><div class="t m0 x12 hc y33 ff8 fs6 fc0 sc0 ls0 ws0">riding a </div><div class="t m0 x12 hc y34 ff8 fs6 fc0 sc0 ls0 ws0">skateboard</div><div class="t m0 x13 hc y35 ff8 fs6 fc0 sc0 ls0 ws0">Detection</div><div class="t m0 x14 hc y36 ff8 fs6 fc0 sc0 ls0 ws0">Cat</div><div class="t m0 x14 hd y37 ff9 fs6 fc0 sc0 ls0 ws0">Skateboard</div><div class="t m0 x15 hc y38 ff8 fs6 fc0 sc0 ls0 ws0">Dense Captioning</div><div class="t m0 x16 he y39 ff8 fs7 fc0 sc0 ls0 ws0">Orange spotted cat</div><div class="t m0 x14 he y3a ff8 fs7 fc0 sc0 ls0 ws0">Skateboard with </div><div class="t m0 x14 he y3b ff8 fs7 fc0 sc0 ls0 ws0">red wheels</div><div class="t m0 x14 he y3c ff8 fs7 fc0 sc0 ls0 ws0">Cat riding a </div><div class="t m0 x14 he y3d ff8 fs7 fc0 sc0 ls0 ws0">skateboard</div><div class="t m0 x14 he y3e ff8 fs7 fc0 sc0 ls0 ws0">Brown hardwood </div><div class="t m0 x14 he y3f ff8 fs7 fc0 sc0 ls0 ws0">flooring</div><div class="t m0 x14 hc y40 ff8 fs6 fc2 sc0 ls0 ws0">label density</div><div class="t m0 x17 hf y41 ff8 fs8 fc0 sc0 ls0 ws0">Whole Image Image Regions</div><div class="t m0 x18 hc y42 ff8 fs6 fc3 sc0 ls0 ws0">label </div><div class="t m0 x19 hc y43 ff8 fs6 fc3 sc0 ls0 ws0">complexity</div><div class="t m0 x1a hf y44 ff8 fs8 fc0 sc0 ls0 ws0">Single</div><div class="t m0 x1a hf y45 ff8 fs8 fc0 sc0 ls0 ws0">Label</div><div class="t m0 x1b hc y46 ff8 fs6 fc0 sc0 ls0 ws0">Sequence</div></div><div class="t m0 xe h10 y47 ff2 fs3 fc0 sc0 ls0 ws0">Figure 1. We address the Dense Captioning task (bottom right)</div><div class="t m0 xe h10 y48 ff2 fs3 fc0 sc0 ls0 ws0">with a model that jointly generates both dense and rich annotations</div><div class="t m0 xe h10 y49 ff2 fs3 fc0 sc0 ls0 ws0">in a single forward pass.</div><div class="t m0 xe h8 y4a ff2 fs4 fc0 sc0 ls0 ws0">remained separate. In this work we take a step towards uni-</div><div class="t m0 xe h8 y4b ff2 fs4 fc0 sc0 ls0 ws0">fying these two inter-connected tasks into one joint frame-</div><div class="t m0 xe h8 y4c ff2 fs4 fc0 sc0 ls0 ws0">work. First, we introduce the dense captioning task (see</div><div class="t m0 xe h8 y4d ff2 fs4 fc0 sc0 ls0 ws0">Figure 1), which requires a model to predict a set of descrip-</div><div class="t m0 xe h8 y4e ff2 fs4 fc0 sc0 ls0 ws0">tions across regions of an image. Object detection is hence</div><div class="t m0 xe h8 y4f ff2 fs4 fc0 sc0 ls0 ws0">recovered as a special case when the target labels consist</div><div class="t m0 xe h8 y50 ff2 fs4 fc0 sc0 ls0 ws0">of one word, and image captioning is recovered when all</div><div class="t m0 xe h8 y51 ff2 fs4 fc0 sc0 ls0 ws0">images consist of one region that spans the full image.</div><div class="t m0 x1c h8 y52 ff2 fs4 fc0 sc0 ls0 ws0">Additionally, we develop a Fully Convolutional Local-</div><div class="t m0 xe h8 y53 ff2 fs4 fc0 sc0 ls0 ws0">ization Network (FCLN) for the dense captioning task.</div><div class="t m0 xe h8 y54 ff2 fs4 fc0 sc0 ls0 ws0">Our model is inspired by recent work in image captioning</div><div class="t m0 xe h8 y55 ff2 fs4 fc0 sc0 ls0 ws0">[49, 21, 32, 8, 4] in that it is composed of a Convolutional</div><div class="t m0 xe h8 y56 ff2 fs4 fc0 sc0 ls0 ws0">Neural Network and a Recurrent Neural Network language</div><div class="t m0 xe h8 y57 ff2 fs4 fc0 sc0 ls0 ws0">model. However, drawing on work in object detection [38],</div><div class="t m0 xe h8 y58 ff2 fs4 fc0 sc0 ls0 ws0">our second core contribution is to introduce a new dense lo-</div><div class="t m0 xe h8 y59 ff2 fs4 fc0 sc0 ls0 ws0">calization layer. This layer is fully differentiable and can</div><div class="t m0 xe h8 y5a ff2 fs4 fc0 sc0 ls0 ws0">be inserted into any neural network that processes images</div><div class="t m0 xe h8 y5b ff2 fs4 fc0 sc0 ls0 ws0">to enable region-level training and predictions. Internally,</div><div class="t m0 xe h8 y5c ff2 fs4 fc0 sc0 ls0 ws0">the localization layer predicts a set of regions of interest in</div><div class="t m0 xe h8 y5d ff2 fs4 fc0 sc0 ls0 ws0">the image and then uses bilinear interpolation [19, 16] to</div><div class="t m0 xe h8 y5e ff2 fs4 fc0 sc0 ls0 ws0">smoothly crop the activations in each region.</div><div class="t m0 x1c h8 y5f ff2 fs4 fc0 sc0 ls0 ws0">We evaluate the model on the large-scale Visual Genome</div><div class="t m0 xe h8 y60 ff2 fs4 fc0 sc0 ls0 ws0">dataset, which contains 94,000 images and 4,100,000 region</div><div class="t m0 xe h8 y61 ff2 fs4 fc0 sc0 ls0 ws0">captions. Our results show both performance and speed im-</div><div class="t m0 xe h8 y62 ff2 fs4 fc0 sc0 ls0 ws0">provements over approaches based on previous state of the</div><div class="t m0 xe h8 y63 ff2 fs4 fc0 sc0 ls0 ws0">art. We make our code and data publicly available to sup-</div><div class="t m0 xe h8 y2d ff2 fs4 fc0 sc0 ls0 ws0">port further progress on the dense captioning task.</div><div class="t m0 x1d h8 y64 ff2 fs4 fc0 sc0 ls0 ws0">1</div><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a></div><div class="pi" data-data='{"ctm":[1.568627,0.000000,0.000000,1.568627,0.000000,0.000000]}'></div></div> </body> </html>

<div id="pf2" class="pf w0 h0" data-page-no="2"><div class="pc pc2 w0 h0"><img class="bi x0 y0 w1 h1" alt="" src="https://static.pudn.com/prod/directory_preview_static/624b58668947fd5953ee5026/bg2.jpg"><div class="t m0 xa h6 y65 ff1 fs1 fc0 sc0 ls0 ws0">2. Related Work</div><div class="t m0 xa h8 y66 ff2 fs4 fc0 sc0 ls0 ws0">Our work draws on recent work in object detection, im-</div><div class="t m0 xa h8 y67 ff2 fs4 fc0 sc0 ls0 ws0">age captioning, and soft spatial attention that allows down-</div><div class="t m0 xa h8 y68 ff2 fs4 fc0 sc0 ls0 ws0">stream processing of particular regions in the image.</div><div class="t m0 xa h11 y69 ff1 fs4 fc0 sc0 ls0 ws0">Object Detection. Our core visual processing module is a</div><div class="t m0 xa h8 y6a ff2 fs4 fc0 sc0 ls0 ws0">Convolutional Neural Network (CNN) [29, 26], which has</div><div class="t m0 xa h8 y6b ff2 fs4 fc0 sc0 ls0 ws0">emerged as a powerful model for visual recognition tasks</div><div class="t m0 xa h8 y6c ff2 fs4 fc0 sc0 ls0 ws0">[39]. The ﬁrst application of these models to dense predic-</div><div class="t m0 xa h8 y6d ff2 fs4 fc0 sc0 ls0 ws0">tion tasks was introduced in R-CNN [14], where each re-</div><div class="t m0 xa h8 y6e ff2 fs4 fc0 sc0 ls0 ws0">gion of interest was processed independently. Further work</div><div class="t m0 xa h8 y6f ff2 fs4 fc0 sc0 ls0 ws0">has focused on processing all regions with only single for-</div><div class="t m0 xa h8 y70 ff2 fs4 fc0 sc0 ls0 ws0">ward pass of the CNN [17, 13], and on eliminating explicit</div><div class="t m0 xa h8 y71 ff2 fs4 fc0 sc0 ls0 ws0">region proposal methods by directly predicting the bound-</div><div class="t m0 xa h8 y72 ff2 fs4 fc0 sc0 ls0 ws0">ing boxes either in the image coordinate system [46, 9], or in</div><div class="t m0 xa h8 y73 ff2 fs4 fc0 sc0 ls0 ws0">a fully convolutional [31] and hence position-invariant set-</div><div class="t m0 xa h8 y74 ff2 fs4 fc0 sc0 ls0 ws0">tings [40, 38, 37]. Most related to our approach is the work</div><div class="t m0 xa h8 y75 ff2 fs4 fc0 sc0 ls0 ws0">of Ren et al. [38] who develop a region proposal network</div><div class="t m0 xa h8 y76 ff2 fs4 fc0 sc0 ls0 ws0">(RPN) that regresses from anchors to regions of interest.</div><div class="t m0 xa h8 y77 ff2 fs4 fc0 sc0 ls0 ws0">However, they adopt a 4-step optimization process, while</div><div class="t m0 xa h8 y78 ff2 fs4 fc0 sc0 ls0 ws0">our approach does not require training pipelines. Addition-</div><div class="t m0 xa h8 y79 ff2 fs4 fc0 sc0 ls0 ws0">ally, we replace their RoI pooling mechanism with a differ-</div><div class="t m0 xa h8 y7a ff2 fs4 fc0 sc0 ls0 ws0">entiable, spatial soft attention mechanism [19, 16]. In par-</div><div class="t m0 xa h8 y7b ff2 fs4 fc0 sc0 ls0 ws0">ticular, this change allows us to backpropagate through the</div><div class="t m0 xa h8 y7c ff2 fs4 fc0 sc0 ls0 ws0">region proposal network and train the whole model jointly.</div><div class="t m0 xa h11 y7d ff1 fs4 fc0 sc0 ls0 ws0">Image Captioning. Several pioneering approaches have</div><div class="t m0 xa h8 y7e ff2 fs4 fc0 sc0 ls0 ws0">explored the task of describing images with natural lan-</div><div class="t m0 xa h8 y7f ff2 fs4 fc0 sc0 ls0 ws0">guage [1, 27, 12, 34, 42, 43, 28, 20]. More recent ap-</div><div class="t m0 xa h8 y80 ff2 fs4 fc0 sc0 ls0 ws0">proaches based on neural networks have adopted Recurrent</div><div class="t m0 xa h8 y81 ff2 fs4 fc0 sc0 ls0 ws0">Neural Networks (RNNs) [50, 18] as the core architectural</div><div class="t m0 xa h8 y82 ff2 fs4 fc0 sc0 ls0 ws0">element for generating captions. These models have pre-</div><div class="t m0 xa h8 y83 ff2 fs4 fc0 sc0 ls0 ws0">viously been used in language modeling [2, 15, 33, 44],</div><div class="t m0 xa h8 y84 ff2 fs4 fc0 sc0 ls0 ws0">where they are known to learn powerful long-term inter-</div><div class="t m0 xa h8 y85 ff2 fs4 fc0 sc0 ls0 ws0">actions [22]. Several recent approaches to Image Caption-</div><div class="t m0 xa h8 y86 ff2 fs4 fc0 sc0 ls0 ws0">ing [32, 21, 49, 8, 4, 24, 11] rely on a combination of RNN</div><div class="t m0 xa h8 y87 ff2 fs4 fc0 sc0 ls0 ws0">language model conditioned on image information, possi-</div><div class="t m0 xa h8 y88 ff2 fs4 fc0 sc0 ls0 ws0">bly with soft attention mechanisms [51, 5]. Similar to our</div><div class="t m0 xa h8 y89 ff2 fs4 fc0 sc0 ls0 ws0">work, Karpathy and Fei-Fei [21] run an image captioning</div><div class="t m0 xa h8 y8a ff2 fs4 fc0 sc0 ls0 ws0">model on regions but they do not tackle the joint task of</div><div class="t m0 xa h8 y8b ff2 fs4 fc0 sc0 ls0 ws0">detection of description in one model. Our model is end-to-</div><div class="t m0 xa h8 y8c ff2 fs4 fc0 sc0 ls0 ws0">end and designed in such way that the prediction for each</div><div class="t m0 xa h8 y8d ff2 fs4 fc0 sc0 ls0 ws0">region is a function of the global image context, which we</div><div class="t m0 xa h8 y8e ff2 fs4 fc0 sc0 ls0 ws0">show also ultimately leads to stronger performance. Finally,</div><div class="t m0 xa h8 y8f ff2 fs4 fc0 sc0 ls0 ws0">the metrics we develop for the dense captioning task are in-</div><div class="t m0 xa h8 y90 ff2 fs4 fc0 sc0 ls0 ws0">spired by metrics developed for image captioning [48, 7, 3].</div><div class="t m0 xa h6 y91 ff1 fs1 fc0 sc0 ls0 ws0">3. Model</div><div class="t m0 xa h11 y92 ff1 fs4 fc0 sc0 ls0 ws0">Overview. Our goal is to design an architecture that jointly</div><div class="t m0 xa h8 y5f ff2 fs4 fc0 sc0 ls0 ws0">localizes regions of interest and then describes each with</div><div class="t m0 xa h8 y93 ff2 fs4 fc0 sc0 ls0 ws0">natural language. The primary challenge is to develop a</div><div class="t m0 xa h8 y61 ff2 fs4 fc0 sc0 ls0 ws0">model that supports end-to-end training with a single step of</div><div class="t m0 xa h8 y62 ff2 fs4 fc0 sc0 ls0 ws0">optimization, and both efﬁcient and effective inference. Our</div><div class="t m0 xa h8 y63 ff2 fs4 fc0 sc0 ls0 ws0">proposed architecture (see Figure 2) draws on architectural</div><div class="t m0 xa h8 y2d ff2 fs4 fc0 sc0 ls0 ws0">elements present in recent work on object detection, image</div><div class="t m0 xe h8 y65 ff2 fs4 fc0 sc0 ls0 ws0">captioning and soft spatial attention to simultaneously ad-</div><div class="t m0 xe h8 y94 ff2 fs4 fc0 sc0 ls0 ws0">dress these design constraints.</div><div class="t m0 xe h8 y95 ff2 fs4 fc0 sc0 ls0 ws0">In Section 3.1 we ﬁrst describe the components of our</div><div class="t m0 xe h8 y96 ff2 fs4 fc0 sc0 ls0 ws0">model. Then in Sections 3.2 and 3.3 we address the loss</div><div class="t m0 xe h8 y97 ff2 fs4 fc0 sc0 ls0 ws0">function and the details of training and inference.</div><div class="t m0 xe h12 y98 ff1 fs9 fc0 sc0 ls0 ws0">3.1. Model Architecture</div><div class="t m0 xe h11 y99 ff1 fs4 fc0 sc0 ls0 ws0">3.1.1 Convolutional Network</div><div class="t m0 xe h8 y9a ff2 fs4 fc0 sc0 ls0 ws0">We use the VGG-16 architecture [41] for its state-of-the-art</div><div class="t m0 xe h13 y9b ff2 fs4 fc0 sc0 ls0 ws0">performance [39]. It consists of 13 layers of 3 × 3 con-</div><div class="t m0 xe h13 y9c ff2 fs4 fc0 sc0 ls0 ws0">volutions interspersed with 5 layers of 2 × 2 max pooling.</div><div class="t m0 xe h8 y9d ff2 fs4 fc0 sc0 ls0 ws0">We remove the ﬁnal pooling layer, so an input image of</div><div class="t m0 xe h13 y9e ff2 fs4 fc0 sc0 ls0 ws0">shape 3 × W × H gives rise to a tensor of features of shape</div><div class="t m0 xe h13 y9f ffc fs4 fc0 sc0 ls0 ws0">C ×W</div><div class="t m0 x1e h14 ya0 ffd fsa fc0 sc0 ls0 ws0">0</div><div class="t m0 x1f h13 y9f ffb fs4 fc0 sc0 ls0 ws0">×H</div><div class="t m0 x20 h14 ya0 ffd fsa fc0 sc0 ls0 ws0">0</div><div class="t m0 x21 h13 y9f ff2 fs4 fc0 sc0 ls0 ws0">where C = 512, W</div><div class="t m0 x22 h14 ya0 ffd fsa fc0 sc0 ls0 ws0">0</div><div class="t m0 x23 h13 y9f ffa fs4 fc0 sc0 ls0 ws0">=</div><div class="t m0 x24 h15 ya1 ffe fs4 fc0 sc0 ls0 ws0"></div><div class="t m0 x25 h16 ya2 fff fsa fc0 sc0 ls0 ws0">W</div><div class="t m0 x25 h17 ya3 ff10 fsa fc0 sc0 ls0 ws0">16</div><div class="t m0 x26 h15 ya4 ffe fs4 fc0 sc0 ls0 ws0"></div><div class="t m0 x27 h13 y9f ff2 fs4 fc0 sc0 ls0 ws0">, and H</div><div class="t m0 x28 h14 ya0 ffd fsa fc0 sc0 ls0 ws0">0</div><div class="t m0 x29 h13 y9f ffa fs4 fc0 sc0 ls0 ws0">=</div><div class="t m0 x2a h15 ya4 ffe fs4 fc0 sc0 ls0 ws0"></div><div class="t m0 x2b h16 ya5 fff fsa fc0 sc0 ls0 ws0">H</div><div class="t m0 x2b h17 ya3 ff10 fsa fc0 sc0 ls0 ws0">16</div><div class="t m0 x2c h15 ya4 ffe fs4 fc0 sc0 ls0 ws0"></div><div class="t m0 x2d h8 y9f ff2 fs4 fc0 sc0 ls0 ws0">.</div><div class="t m0 xe h8 ya6 ff2 fs4 fc0 sc0 ls0 ws0">The output of this network encodes the appearance of the</div><div class="t m0 xe h8 ya7 ff2 fs4 fc0 sc0 ls0 ws0">image at a set of uniformly sampled image locations, and</div><div class="t m0 xe h8 ya8 ff2 fs4 fc0 sc0 ls0 ws0">forms the input to the localization layer.</div><div class="t m0 xe h11 ya9 ff1 fs4 fc0 sc0 ls0 ws0">3.1.2 Fully Convolutional Localization Layer</div><div class="t m0 xe h8 yaa ff2 fs4 fc0 sc0 ls0 ws0">The localization layer receives an input tensor of activa-</div><div class="t m0 xe h8 yab ff2 fs4 fc0 sc0 ls0 ws0">tions, identiﬁes spatial regions of interest and smoothly ex-</div><div class="t m0 xe h8 yac ff2 fs4 fc0 sc0 ls0 ws0">tracts a ﬁxed-sized representation from each region. Our</div><div class="t m0 xe h8 yad ff2 fs4 fc0 sc0 ls0 ws0">approach is based on that of Faster R-CNN [38], but we</div><div class="t m0 xe h8 yae ff2 fs4 fc0 sc0 ls0 ws0">replace their RoI pooling mechanism [13] with bilinear</div><div class="t m0 xe h8 yaf ff2 fs4 fc0 sc0 ls0 ws0">interpolation [19], allowing our model to propagate gra-</div><div class="t m0 xe h8 yb0 ff2 fs4 fc0 sc0 ls0 ws0">dients backward through the coordinates of predicted re-</div><div class="t m0 xe h8 yb1 ff2 fs4 fc0 sc0 ls0 ws0">gions. This modiﬁcation opens up the possibility of predict-</div><div class="t m0 xe h8 yb2 ff2 fs4 fc0 sc0 ls0 ws0">ing afﬁne or morphed region proposals instead of bounding</div><div class="t m0 xe h8 yb3 ff2 fs4 fc0 sc0 ls0 ws0">boxes [19], but we leave these extensions to future work.</div><div class="t m0 xe h11 yb4 ff1 fs4 fc0 sc0 ls0 ws0">Inputs/outputs. The localization layer accepts a tensor of</div><div class="t m0 xe h13 yb5 ff2 fs4 fc0 sc0 ls0 ws0">activations of size C × W</div><div class="t m0 x2e h14 yb6 ffd fsa fc0 sc0 ls0 ws0">0</div><div class="t m0 x2f h13 yb5 ffb fs4 fc0 sc0 ls0 ws0">× H</div><div class="t m0 x30 h14 yb6 ffd fsa fc0 sc0 ls0 ws0">0</div><div class="t m0 x31 h8 yb5 ff2 fs4 fc0 sc0 ls0 ws0">. It then internally selects</div><div class="t m0 xe h13 yb7 ffc fs4 fc0 sc0 ls0 ws0">B regions of interest and returns three output tensors giving</div><div class="t m0 xe h8 yb8 ff2 fs4 fc0 sc0 ls0 ws0">information about these regions:</div><div class="t m0 x32 h13 yb9 ff2 fs4 fc0 sc0 ls0 ws0">1. Region Coordinates: A matrix of shape B × 4 giving</div><div class="t m0 x33 h8 yba ff2 fs4 fc0 sc0 ls0 ws0">bounding box coordinates for each output region.</div><div class="t m0 x32 h13 ybb ff2 fs4 fc0 sc0 ls0 ws0">2. Region Scores: A vector of length B giving a con-</div><div class="t m0 x33 h8 ybc ff2 fs4 fc0 sc0 ls0 ws0">ﬁdence score for each output region. Regions with</div><div class="t m0 x33 h8 ybd ff2 fs4 fc0 sc0 ls0 ws0">high conﬁdence scores are more likely to correspond</div><div class="t m0 x33 h8 ybe ff2 fs4 fc0 sc0 ls0 ws0">to ground-truth regions of interest.</div><div class="t m0 x32 h13 ybf ff2 fs4 fc0 sc0 ls0 ws0">3. Region Features: A tensor of shape B × C × X × Y</div><div class="t m0 x33 h8 yc0 ff2 fs4 fc0 sc0 ls0 ws0">giving features for output regions; is represented by an</div><div class="t m0 x33 h13 yc1 ffc fs4 fc0 sc0 ls0 ws0">X × Y grid of C-dimensional features.</div><div class="t m0 xe h11 yc2 ff1 fs4 fc0 sc0 ls0 ws0">Convolutional Anchors. Similar to Faster R-CNN [38],</div><div class="t m0 xe h8 y92 ff2 fs4 fc0 sc0 ls0 ws0">our localization layer predicts region proposals by regress-</div><div class="t m0 xe h8 y5f ff2 fs4 fc0 sc0 ls0 ws0">ing offsets from a set of translation-invariant anchors. In</div><div class="t m0 xe h13 y60 ff2 fs4 fc0 sc0 ls0 ws0">particular, we project each point in the W</div><div class="t m0 x34 h14 yc3 ffd fsa fc0 sc0 ls0 ws0">0</div><div class="t m0 x35 h13 y60 ffb fs4 fc0 sc0 ls0 ws0">× H</div><div class="t m0 x36 h14 yc3 ffd fsa fc0 sc0 ls0 ws0">0</div><div class="t m0 x37 h8 y60 ff2 fs4 fc0 sc0 ls0 ws0">grid of</div><div class="t m0 xe h13 y61 ff2 fs4 fc0 sc0 ls0 ws0">input features back into the W × H image plane, and con-</div><div class="t m0 xe h13 y62 ff2 fs4 fc0 sc0 ls0 ws0">sider k anchor boxes of different aspect ratios centered at</div><div class="t m0 xe h13 y63 ff2 fs4 fc0 sc0 ls0 ws0">this projected point. For each of these k anchor boxes,</div><div class="t m0 xe h8 y2d ff2 fs4 fc0 sc0 ls0 ws0">the localization layer predicts a conﬁdence score and four</div><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a><a class="l" rel='nofollow' onclick='return false;'><div class="d m1"></div></a></div><div class="pi" data-data='{"ctm":[1.568627,0.000000,0.000000,1.568627,0.000000,0.000000]}'></div></div>