通过色彩校正矩阵学习实现低光图像恢复
深圳大学, 计算机科学与软件工程学院, 广东深圳 518060
| 摘 要： | 在弱光下捕获的图像不仅可见度低，而且质量差，可能会显著降低多媒体和计算机视觉任务的性能。目前大量技术已经被提出以增强低光图像。大多数低光技术使用sRGB数据集，但是在极低光图像增强方面表现不佳。一些技术使用原始数据进行极低光增强，但输出图像会遇到色彩校正和细节损失的问题。为了解决色彩校正的问题，本文中提出了一种新的基于双MultiResUnet的网络。在所提出的方法中，原始数据和sRGB数据用作输入，而sRGB图像是使用传统方法从原始数据生成的。双卷积神经网络的第一个分支从原始数据中学习结构细节，另一个分支从sRGB数据中学习空间变化的颜色变换矩阵以进行颜色校正。通过将学习到的颜色变换矩阵应用于第一个分支产生的结构信息来产生最终结果。实验结果表明，所提出的方法表现更好，并且很好地调整了极低光图像的颜色信息。 |
| 关 键 词： | 图像处理; 图像增强; 图像去噪; 图像恢复; 图像色彩分析 |
| DOI： | 10.57237/j.cst.2023.03.003 |
View:
Downloads:
收稿日期: 2023-07-17
接受日期: 2023-09-05
在线出版日期: 2023-09-08
Towards Low-light Image Restoration via Color Correction Matrix Learning
College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China
| Abstract: | Images captured in low-light not only suffer from low visibility but also result in poor quality, which may significantly degrade the performance of multimedia and computer vision tasks. A large number of techniques have been proposed to enhance the low-light images. The majority of the low-light techniques use sRGB datasets and do not perform well in extreme low-light image enhancement. A few techniques use the raw data for extreme low-light enhancement but the output images suffer color correction and details loss. In this paper, to address this color correction problem a new dual MultiResUnet based network is proposed. In the proposed approach, raw and sRGB data is used as input, whereas sRGB images are generated from raw data by using the traditional pipeline. One branch of the dual CNN learns the structural details from raw data and the other branch learns a spatially variant color transformation matrix from sRGB data for color correction. The final result is produced by applying the learned color transformation matrix to the structural information produced by the first branch. The experimental results demonstrate that the proposed approach performs better and achieved well-adjusted color information for extremely low-light images. |
| Keywords: | Image Processing; Image Enhancement; Image Denoising; Image Restoration; Image Color Analysis |
基金
| 1. | 中国科技部 (MOST) 新一代人工智能2030重大计划 (项目编号: 2018AAA0102200) |
| 2. | 国家自然科学基金重大项目 (项目编号: 61827814, U22A2097) |
| 3. | 深圳市科创委 (SZSTI) 项目 (项目编号: JCYJ20190808153619413) |
参考文献
| [1] | MT. Rasheed, G. Guo, D. Shi, H. Khan, and Q. C, “An Empirical Study on Retinex Methods for Low-Light Image Enhancement,” Remote Sensing, vol. 14, no. 18, 2022, doi: 10.3390/rs14184608. |
| [2] | X. Fu, Y. Liao, D. Zeng, Y. Huang, X. P. Zhang, and X. Ding, “A Probabilistic Method for Image Enhancement with Simultaneous Illumination and Reflectance Estimation,” IEEE Trans. Image Process., vol. 24, no. 12, pp. 4965–4977, 2015, doi: 10.1109/TIP.2015.2474701. |
| [3] | MT. Rasheed, D. Shi, and H. Khan, “A comprehensive experiment-based review of low-light image enhancement methods and benchmarking low-light image quality assessment,” Signal Processing, vol. 204, 2023, doi: 10.1016/j.sigpro.2022.108821. |
| [4] | H. Ibrahim and N. S. P. Kong, “Brightness preserving dynamic histogram equalization for image contrast enhancement,” IEEE Trans. Consum. Electron., vol. 53, no. 4, pp. 1752–1758, 2007, doi: 10.1109/TCE.2007.4429280. |
| [5] | H. D. Cheng and X. J. Shi, “A simple and effective histogram equalization approach to image enhancement,” Digit. Signal Process. A Rev. J., vol. 14, no. 2, pp. 158–170, 2004, doi: 10.1016/j.dsp.2003.07.002. |
| [6] | G. Xu, J. Su, H. Pan, Z. Zhang, and H. Gong, “An image enhancement method based on gamma correction,” Isc. 2009 - 2009 Int. Symp. Comput. Intell. Des., vol. 1, pp. 60–63, 2009, doi: 10.1109/ISCID.2009.22. |
| [7] | J. S. Yoo, J. H. Choi, K. S. Choi, D. Y. Lee, H. Y. Kim, and J. O. Kim, “Fast search of a similar patch for self-similarity based image super resolution,” IEICE Trans. Inf. Syst., vol. E99D, no. 8, pp. 2194–2198, 2016, doi: 10.1587/transinf.2016EDL8049. |
| [8] | S. Rahman, M. M. Rahman, M. Abdullah-Al-Wadud, G. D. Al-Quaderi, and M. Shoyaib, “An adaptive gamma correction for image enhancement,” Eurasip J. Image Video Process., vol. 2016, no. 1, pp. 1–13, 2016, doi: 10.1186/s13640-016-0138-1. |
| [9] | M. Tanaka, “Gradient-Based Low-Light Image Enhancement,” pp. 11–12. |
| [10] | M. Afifi, K. G. Derpanis, B. Ommer, and M. S. Brown, “Learning Multi-Scale Photo Exposure Correction,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 9153–9163, 2021, doi: 10.1109/CVPR46437.2021.00904. |
| [11] | K. Lu, “TBEFN : A Two-Branch Exposure-Fusion Network for Low-Light Image Enhancement,” vol. 23, pp. 4093–4105, 2021. |
| [12] | Y. Wang, R. Wan, W. Yang, H. Li, L.-P. Chau, and A. C. Kot, “Low-Light Image Enhancement with Normalizing Flow,” 2021, [Online]. Available: http://arxiv.org/abs/2109.05923. |
| [13] | F. Lv, F. Lu, J. Wu, and C. Lim, “MBLLEn: Low-light image/video enhancement using CNNs,” Br. Mach. Vis. Conf. 2018, BMVC 2018, no. May, 2019. |
| [14] | W. Wang, C. Wei, W. Yang, and J. Liu, “GLADNet: Low-light enhancement network with global awareness,” Proc. - 13th IEEE Int. Conf. Autom. Face Gesture Recognition, FG 2018, pp. 751–755, 2018, doi: 10.1109/FG.2018.00118. |
| [15] | Z. Zhao, B. Xiong, L. Wang, Q. Ou, L. Yu, and F. Kuang, “RetinexDIP: A Unified Deep Framework for Low-Light Image Enhancement,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 3, pp. 1076–1088, 2022, doi: 10.1109/TCSVT.2021.3073371. |
| [16] | L. W. Wang, Z. S. Liu, W. C. Siu, and D. P. K. Lun, “Lightening Network for Low-Light Image Enhancement,” IEEE Trans. Image Process., vol. 29, pp. 7984–7996, 2020, doi: 10.1109/TIP.2020.3008396. |
| [17] | S. C. Huang, F. C. Cheng, and Y. S. Chiu, “Efficient contrast enhancement using adaptive gamma correction with weighting distribution,” IEEE Trans. Image Process., vol. 22, no. 3, pp. 1032–1041, 2013, doi: 10.1109/TIP.2012.2226047. |
| [18] | D. J. Jobson, Z. Rahman, and G. A. Woodell, “Properties and Performance of a Center / Surround Retinex,” vol. 6, no. 3, pp. 451–462, 1997. |
| [19] | S. Wang, J. Zheng, H. M. Hu, and B. Li, “Naturalness preserved enhancement algorithm for non-uniform illumination images,” IEEE Trans. Image Process., vol. 22, no. 9, pp. 3538–3548, 2013, doi: 10.1109/TIP.2013.2261309. |
| [20] | D. J. Jobson, Z. U. Rahman, and G. A. Woodell, “A multiscale retinex for bridging the gap between color images and the human observation of scenes,” IEEE Trans. Image Process., vol. 6, no. 7, pp. 965–976, 1997, doi: 10.1109/83.597272. |
| [21] | C. Li, J. Guo, F. Porikli, and Y. Pang, “LightenNet: A Convolutional Neural Network for weakly illuminated image enhancement,” Pattern Recognit. Lett., vol. 104, pp. 15–22, 2018, doi: 10.1016/j.patrec.2018.01.010. |
| [22] | Y. Jiang et al., “EnlightenGAN: Deep Light Enhancement without Paired Supervision,” IEEE Trans. Image Process., vol. 30, pp. 2340–2349, 2021, doi: 10.1109/TIP.2021.3051462. |
| [23] | Y. S. Chen, Y. C. Wang, M. H. Kao, and Y. Y. Chuang, “Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 6306–6314, 2018, doi: 10.1109/CVPR.2018.00660. |
| [24] | Y. Zhang, X. Di, B. Zhang, Q. Li, S. Yan, and C. Wang, “Self-supervised Low Light Image Enhancement and Denoising,” 2021, [Online]. Available: http://arxiv.org/abs/2103.00832. |
| [25] | C. Guo et al., “Zero-reference deep curve estimation for low-light image enhancement,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 1777–1786, 2020, doi: 10.1109/CVPR42600.2020.00185. |
| [26] | A. Zhu, L. Zhang, Y. Shen, Y. Ma, S. Zhao, and Y. Zhou, “ZERO-SHOT RESTORATION OF UNDEREXPOSED IMAGES VIA ROBUST RETINEX DECOMPOSITION School of Software Engineering, Tongji University, Shanghai, China School of Computer Information Engineering, Jiangxi Normal University, China Department of Computer and I,” Icme, pp. 3–8, 2020. |
| [27] | C. Chen, Q. Chen, J. Xu, and V. Koltun, “Learning to See in the Dark,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 3291–3300, 2018, doi: 10.1109/CVPR.2018.00347. |
| [28] | K. G. Lore, A. Akintayo, and S. Sarkar, “LLNet: A deep autoencoder approach to natural low-light image enhancement,” Pattern Recognit., vol. 61, pp. 650–662, 2017, doi: 10.1016/j.patcog.2016.06.008. |
| [29] | L. Tao, C. Zhu, G. Xiang, Y. Li, H. Jia, and X. Xie, “LLCNN: A convolutional neural network for low-light image enhancement,” 2017 IEEE Vis. Commun. Image Process. VCIP 2017, vol. 2018-January, no. 2013, pp. 1–4, 2018, doi: 10.1109/VCIP.2017.8305143. |
| [30] | S. H. Park, H. S. Kim, S. Lansel, M. Parmar, and B. A. Wandell, “A case for denoising before demosaicking color filter array data,” Conf. Rec. - Asilomar Conf. Signals, Syst. Comput., no. 4, pp. 860–864, 2009, doi: 10.1109/ACSSC.2009.5469990. |
| [31] | R. Wang, Q. Zhang, C. W. Fu, X. Shen, W. S. Zheng, and J. Jia, “Underexposed photo enhancement using deep illumination estimation,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2019-June, pp. 6842–6850, 2019, doi: 10.1109/CVPR.2019.00701. |
| [32] | T. Huynh-The and T. Le-Tien, “Brightness preserving weighted dynamic range histogram equalization for image contrast enhancement,” Int. Conf. Adv. Technol. Commun., vol. 53, no. 4, pp. 386–391, 2013, doi: 10.1109/ATC.2013.6698142. |
| [33] | X. Guo, Y. Li, and H. Ling, “LIME : Low-light Image Enhancement via,” IEEE Trans. Comput. Imaging, vol. 26, no. 2, pp. 1–12, 2017. |
| [34] | B. Cai, X. Xu, K. Guo, K. Jia, B. Hu, and D. Tao, “A Joint Intrinsic-Extrinsic Prior Model for Retinex,” Proc. IEEE Int. Conf. Comput. Vis., vol. 2017-October, pp. 4020–4029, 2017, doi: 10.1109/ICCV.2017.431. |
| [35] | X. Fu, D. Zeng, Y. Huang, X. P. Zhang, and X. Ding, “A Weighted Variational Model for Simultaneous Reflectance and Illumination Estimation,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2016-December, pp. 2782–2790, 2016, doi: 10.1109/CVPR.2016.304. |
| [36] | M. Gharbi, J. Chen, J. T. Barron, S. W. Hasinoff, and F. Durand, “Deep bilateral learning for real-time image enhancement,” ACM Trans. Graph., vol. 36, no. 4, 2017, doi: 10.1145/3072959.3073592. |
| [37] | C. Wei, W. Wang, W. Yang, and J. Liu, “Deep retinex decomposition for low-light enhancement,” Br. Mach. Vis. Conf. 2018, BMVC 2018, no. 61772043, 2019. |
| [38] | A. Ignatov, N. Kobyshev, R. Timofte, K. Vanhoey, and L. Van Gool, “DSLR-Quality Photos on Mobile Devices with Deep Convolutional Networks,” Proc. IEEE Int. Conf. Comput. Vis., vol. 2017-October, pp. 3297–3305, 2017, doi: 10.1109/ICCV.2017.355. |
| [39] | J. Park, J. Y. Lee, D. Yoo, and I. S. Kweon, “Distort-and-Recover: Color Enhancement Using Deep Reinforcement Learning,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 5928–5936, 2018, doi: 10.1109/CVPR.2018.00621. |
| [40] | X. Yang, K. Xu, Y. Song, Q. Zhang, X. Wei, and R. W. H. Lau, “Image Correction via Deep Reciprocating HDR Transformation,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 1798–1807, 2018, doi: 10.1109/CVPR.2018.00193. |
| [41] | W. Yang, S. Wang, Y. Fang, Y. Wang, and J. Liu, “From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 3060–3069, 2020, doi: 10.1109/CVPR42600.2020.00313. |
| [42] | K. Xu, X. Yang, B. Yin, and R. W. H. Lau, “Learning to restore low-light images via decomposition-and-enhancement,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 2278–2287, 2020, doi: 10.1109/CVPR42600.2020.00235. |
| [43] | T. Ehret, A. Davy, P. Arias, and G. Facciolo, “Joint demosaicking and denoising by fine-tuning of bursts of raw images,” Proc. IEEE Int. Conf. Comput. Vis., vol. 2019-October, pp. 8867–8876, 2019, doi: 10.1109/ICCV.2019.00896. |
| [44] | K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, "Color Image Denoising via Sparse 3D Collaborative Filtering with Grouping Constraint in Luminance-Chrominance Space," 2007 IEEE International Conference on Image Processing, San Antonio, TX, USA, 2007, pp. I - 313-I - 316, doi: 10.1109/ICIP.2007.4378954. |
| [45] | L. Condat and S. Mosaddegh, "Joint demosaicking and denoising by total variation minimization," 2012 19th IEEE International Conference on Image Processing, Orlando, FL, USA, 2012, pp. 2781-2784, doi: 10.1109/ICIP.2012.6467476. |
| [46] | D. Khashabi, S. Nowozin, J. Jancsary, and A. W. Fitzgibbon, “Joint demosaicing and denoising via learned nonparametric random fields,” IEEE Trans. Image Process., vol. 23, no. 12, pp. 4968–4981, 2014, doi: 10.1109/TIP.2014.2359774. |
| [47] | F. Kokkinos and S. Lefkimmiatis, “Deep image demosaicking using a cascade of convolutional residual denoising networks,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 11218 LNCS, pp. 317–333, 2018, doi: 10.1007/978-3-030-01264-9_19. |
| [48] | K. Hirakawa and T. W. Parks, “Joint demosaicing and denoising,” Proc. - Int. Conf. Image Process. ICIP, vol. 3, no. 8, pp. 306–309, 2005, doi: 10.1109/ICIP.2005.1530390. |
| [49] | X. Xu, Y. Ma, and W. Sun, “Towards Real Scene Super-Resolution with Raw Images Supplementary Material,” Cvpr, no. c, pp. 1723–1731, 2019. |
| [50] | N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, “Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015: 18th International Conference Munich, Germany, October 5-9, 2015 proceedings, part III,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 9351, no. Cvd, pp. 12–20, 2015, doi: 10.1007/978-3-319-24574-4. |
| [51] | N. Ibtehaz and M. S. Rahman, “MultiResUNet: Rethinking the U-Net architecture for multimodal biomedical image segmentation,” Neural Networks, vol. 121, pp. 74–87, 2020, doi: 10.1016/j.neunet.2019.08.025. |
| [52] | W. Shi et al., “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2016-December, pp. 1874–1883, 2016, doi: 10.1109/CVPR.2016.207. |
| [53] | Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, 2004, doi: 10.1109/TIP.2003.819861. |
| [54] | Y. Zhang, J. Zhang, and X. Guo, “Kindling the darkness: A practical low-light image enhancer,” MM 2019 - Proc. 27th ACM Int. Conf. Multimed., pp. 1632–1640, 2019, doi: 10.1145/3343031.3350926. |
| [55] | Z. Ying, G. Li, and W. Gao, “A Bio-Inspired Multi-Exposure Fusion Framework for Low-light Image Enhancement,” vol. 14, no. 8, pp. 1–10, 2017, [Online]. Available: http://arxiv.org/abs/1711.00591. |
