深度神经网络中注意力模块的对抗鲁棒性分析
1. 大连交通大学, 自动化与电气工程学院, 辽宁大连 116028
2. 天津轨道交通集团有限公司, 天津 300392
| 摘 要： | 注意力机制源自于对人类视觉的研究，通过将值得注意的部分提高权重，进而提高模型的准确率，现有的注意力有通道注意力，空间注意力以及混合模型。注意力在提高模型的准确率上的发展很充分，但在对抗攻击的影响下，注意力对模型的对抗鲁棒性的影响并未得到充分的研究。本文的目的是探究注意力对模型鲁棒性的影响。本文将常用的注意力SENet、CBAM、ECANet、SGE、SKNet等与常用的分类模型ResNet相结合，从头训练模型权重，对比结合模型的对抗鲁棒性，此外还通过与高斯、椒盐噪声的对比，体现注意力对普通噪声的影响。本文在前人的鲁棒性公式基础上，提出了对抗鲁棒性公式，对不同网络不同扰动参数的对抗样本进行统一测评。证明了注意力的结构对模型鲁棒性是有影响的，SKNet、SGE、CBAM在18层模型中会提高模型的鲁棒性，随着模型深度的加深，注意力对模型鲁棒性有负向的提升，注意力对高斯、椒盐噪声有很好的鲁棒性，但也随网络的加深，体现出负向提升。 |
| 关 键 词： | 深度神经网络; 注意力机制; 对抗样本; 对抗鲁棒性 |
| DOI： | 10.57237/j.se.2023.01.005 |
View:
Downloads:
收稿日期: 2023-01-05
接受日期: 2023-02-16
在线出版日期: 2023-03-02
Adversarial Robustness Analysis of Attention Modules in Deep Neural Networks
1. School of Electric Engineering and Automation, Dalian Jiaotong University, Dalian 116028, China
2. Tianjin Rail Transit Group Co., Ltd, Tianjin 300392, China
| Abstract: | The attention mechanism is derived from the study of human vision, which improves the accuracy of the model by increasing the weight of the noteworthy parts, and the existing attention is channel attention, spatial attention, and hybrid models. The development of attention in improving the accuracy of the model is sufficient, but under the influence of adversarial attacks, the influence of attention on the adversarial robustness of the model has not been fully studied. The purpose of this paper is to explore the effect of attention on model robustness. In this paper, the commonly used attention SENet, CBAM, ECAnet, SGE, SKNet, etc. are combined with the commonly used classification model ResNet to train the weights of the model from scratch, and compare the robustness of the combined model against confrontation. In addition, the influence of attention on ordinary noise is also reflected by comparison with Gaussian and salt and pepper noise. Based on the robustness formula of the predecessors, this paper proposes an adversarial robustness formula to uniformly evaluate the adversarial samples with different disturbance parameters of different networks. It is proved that the structure of attention has an impact on the robustness of the model, and SKNet, SGE, and CBAM will improve the robustness of the model in the 18-layer model, with the deepening of the model, attention has a negative improvement on the robustness of the model. Attention has a good robustness to Gaussian and salt and pepper noise, but also reflects a negative improvement with the deepening of the network. |
| Keywords: | Deep Neural Networks; Attention Mechanisms; Adversarial Samples; Fight Robustness |
参考文献
| [1] | He K, Zhang X, Ren S, et al. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification [C] // Proceedings of the IEEE international conference on computer vision. 2015: 1026-1034. |
| [2] | Wang S, Pei K, Whitehouse J, et al. Efficient formal safety analysis of neural networks [J]. Advances in Neural Information Processing Systems, 2018, 31. |
| [3] | Vincent P, Larochelle H, Bengio Y, et al. Extracting and composing robust features with denoising autoencoders [C] // Proceedings of the 25th international conference on Machine learning. 2008: 1096-1103. |
| [4] | Zhang H, Cisse M, Dauphin Y N, et al. mixup: Beyond empirical risk minimization [J]. arXiv preprint arXiv: 1710.09412, 2017. |
| [5] | Wang H, Huang Z, Wu X, et al. Toward Learning Robust and Invariant Representations with Alignment Regularization and Data Augmentation [J]. arXiv preprint arXiv: 2206.01909, 2022. |
| [6] | Liu K, Liu X, Yang A, et al. A robust adversarial training approach to machine reading comprehension [C] // Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34 (05): 8392-8400. |
| [7] | Deng J, Dong W, Socher R, et al. Imagenet: A large-scale hierarchical image database [C] // 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009: 248-255. |
| [8] | Goodfellow I J, Shlens J, Szegedy C. Explaining and harnessing adversarial examples [J]. arXiv preprint arXiv: 1412.6572, 2014. |
| [9] | Moosavi-Dezfooli S M, Fawzi A, Frossard P. Deepfool: a simple and accurate method to fool deep neural networks [C] // Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 2574-2582. |
| [10] | Carlini N, Wagner D. Towards evaluating the robustness of neural networks [C] // 2017 ieee symposium on security and privacy (sp). Ieee, 2017: 39-57. |
| [11] | Madry A, Makelov A, Schmidt L, et al. Towards deep learning models resistant to adversarial attacks [J]. arXiv preprint arXiv: 1706.06083, 2017. |
| [12] | Xu K, Zhang G, Liu S, et al. Adversarial t-shirt! evading person detectors in a physical world [C] // European conference on computer vision. Springer, Cham, 2020: 665-681. |
| [13] | Chen J, Zhang H, He X, et al. Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention [C] // Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval. 2017: 335-344. |
| [14] | Jhamb Y, Ebesu T, Fang Y. Attentive contextual denoising autoencoder for recommendation [C] // Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval. 2018: 27-34. |
| [15] | Tay Y, Luu A T, Hui S C. Multi-pointer co-attention networks for recommendation [C] // Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018: 2309-2318. |
| [16] | 朱张莉, 饶元, 吴渊, 等. 注意力机制在深度学习中的研究进展 [J]. 中文信息学报, 2019, 33 (6): 1-11. |
| [17] | Zhang S, Yao L, Sun A, et al. Deep learning based recommender system: A survey and new perspectives [J]. ACM Computing Surveys (CSUR), 2019, 52 (1): 1-38. |
| [18] | 黄立威, 江碧涛, 吕守业, 等. 基于深度学习的推荐系统研究综述 [J]. 计算机学报, 2018, 41 (7): 1619-1647. |
| [19] | 方钧婷, 谭晓阳. 注意力级联网络的金属表面缺陷检测算法 [J]. 计算机科学与探索, 2021, 15 (7): 1245. |
| [20] | Mnih V, Heess N, Graves A. Recurrent models of visual attention [J]. Advances in neural information processing systems, 2014, 27. |
| [21] | Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks. arXiv e-prints [J]. arXiv preprint arXiv: 1709.01507, 2017. |
| [22] | Woo S, Park J, Lee J Y, et al. Cbam: Convolutional block attention module [C] // Proceedings of the European conference on computer vision (ECCV). 2018: 3-19. |
| [23] | Wang Q, Wu B, Zhu P, et al. Supplementary material for ‘ECA-Net: Efficient channel attention for deep convolutional neural networks [C] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, WA, USA. 2020: 13-19. |
| [24] | Wu W, Zhang Y, Wang D, et al. SK-Net: Deep learning on point cloud via end-to-end discovery of spatial keypoints [C] // Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34 (04): 6422-6429. |
| [25] | Li X, Hu X, Yang J. Spatial group-wise enhance: Improving semantic feature learning in convolutional networks [J]. arXiv preprint arXiv:1905.09646, 2019. |
| [26] | He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] // Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778. |
| [27] | Hendrycks D, Dietterich T G. Benchmarking neural network robustness to common corruptions and surface variations [J]. arXiv preprint arXiv: 1807.01697, 2018. |
