Learning representations by back-propagating errors. Liu.Įxploring the limits of transfer learning with a unified text-to-textĭavid E. Unsupervised learning of visual representations by solving jigsawĭeeplearningexamples/pytorch/classification.Īdam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zacharyĭevito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.Ĭolin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Sgdr: Stochastic gradient descent with warm restarts, 2016. In Proceedings of the European Conference on Computer Vision
Curran Associates, Inc., 2012.Ī simple weight decay can improve generalization.Ĭhenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Liįei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Weinberger,Įditors, Advances in Neural Information Processing Systems 25, pagesġ097–1105. Imagenet classification with deep convolutional neural networks. Technical report, University of Toronto, 2009.Īlex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Learning multiple layers of features from tiny images. Mobilenets: Efficient convolutional neural networks for mobile visionīatch normalization: Accelerating deep network training by reducingĪdam: A method for stochastic optimization, 2014. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Proceedings of the 2015 IEEE International Conference onĬomputer Vision (ICCV), ICCV ’15, pages 1026–1034, Washington, DC, USA,ĭeep residual learning for image recognition.Īndrew G. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.ĭelving deep into rectifiers: Surpassing human-level performance on Pages 249–256, Chia Laguna Resort, Sardinia, Italy, 13–. Proceedings of Machine Learning Research , The Thirteenth International Conference on Artificial Intelligence and In Yee Whye Teh and Mike Titterington, editors, Understanding the difficulty of training deep feedforward neural The lottery ticket hypothesis: Finding sparse, trainable neural Imagenet: A large-scale hierarchical image database. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ProxylessNAS: Direct neural architecture search on target task and Yoshua Bengio, Nicholas Léonard, and Aaron Courville.Įstimating or propagating gradients through stochastic neurons for We thank Jesse Dodge and Nicholas Lourie for many helpful discussions, Gabriel Ilharco for valuable feedback, and Sarah Pratt for ( n 2 ).
WEIGHTED STANDARD DEVIATION DECAY FULL
When Conv6 is wide enough, a subnetwork of the randomly weighted model (with % = 50) performs just as well as the full model when it is trained. number of channels) of Conv4 and Conv6 for CIFAR-10. Figure 4: Going Wider: Varying the width ( i.e. As standard we also use weight decay 1e-4, momentum 0.9, and batch size 128. For the SGD baseline we find that training does not converge with learning rate 0.1, and so we use 0.01. The Adam baseline uses the same learning rate and batch size as in 3 3 3Batch size 60, learning rate 2e-4, 3e-4 and 3e-4 for Conv2, Conv4, and Conv6 respectively Conv8 is not tested in, though we use find that learning rate 3e-4 still performs well. We also often run both an Adam and SGD baseline where the weights are learned. On CIFAR-10 we train our models with weight decay 1e-4, momentum 0.9, batch size 128, and learning rate 0.1. When we optimize with SGD we use cosine learning rate decay. When we optimize with Adam we do not decay the learning rate. In every experiment we train for 100 epochs and report the last epoch accuracy on the validation set.
WEIGHTED STANDARD DEVIATION DECAY UPDATE
We never update the value of any weight in the network, only the score associated with each weight. On the backward pass we update the scores of all the edges with the straight-through estimator, allowing helpful edges that are “dead” to re-enter the subnetwork. On the forward pass we choose the top edges by score. Figure 2: In the edge-popup Algorithm, we associate a score with each edge. In short, we validate the unreasonable effectiveness of randomly weighted neural networks for image recognition. Moreover, a randomly weighted ResNet-101 with fixed weights contains a subnetwork that is much smaller, but surpasses the performance of VGG-16. On ImageNet, we find a subnetwork of a randomly weighted Wide ResNet50 which is smaller than, but matches the performance of a trained ResNet-34. On CIFAR-10 we empirically demonstrate that as networks grow wider and deeper, untrained subnetworks perform just as well as the dense network with learned weights. We experiment on small and large scale datasets for image recognition, namely CIFAR-10 and Imagenet.