OVERVIEW OF METHODS FOR IMPROVING THE VISUAL QUALITY OF IMAGES AND VIDEOS IN ADVERSE WEATHER CONDITIONS

In the modern world, image processing is used in various fields of human activity. Due to various interference factors, image and video deterioration seriously reduces the accuracy and efficiency of target tracking and recognition. Hence, image restoration is becoming a major problem in the field of computer vision.

Matrials and methods, results and discussions This study provides an overview of methods for improving the visual quality of images and videos when they are distorted by weather phenomena. The methods are classified according to the types and differences of weather phenomena. Examples of their approbation on images and video are given. Conclusions are drawn about each of the types of algorithms.

Conclusion The article explored various methods to improve the visual quality of images and videos in adverse weather conditions. Each method was studied in detail, their advantages and disadvantages were considered, which made it possible to come to the following conclusions: 1) methods that use physical models are very efficient, but are computationally complex, in this regard, this method is more appropriate to use for image and video post-processing. 2) Methods based on histograms are simpler, but they are only suitable for static weather conditions (fog, haze, haze). 3) The most promising are methods based on learning. Neural networks allow solving more complex problems, due to the possibility of parallelizing information and further learning. Also, these methods do an excellent job of improving the quality of both images and videos. Learning-based methods are applicable to both static weather conditions (fog, haze, haze) and dynamic ones (snow, rain, hail), which makes them more versatile for solving this problem.

1. Kshitiz Garg, Shree K. Nayar. Detection and Removal of Rain from Videos. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (1), July 2004, pp. 528–535.

2. G. Doretto, A. Chiuso, Y.N. Wu, and S. Soatto. Dynamic textures.IJCV, 51(2):91– 109, 2003.

3. K. Garg and S. K. Nayar. Photometric Model for Raindrops. Columbia University Technical Report, 2003.

4. Peter Barnum, Takeo Kanade, Srinivasa G Narasimhan. Spatio-Temporal Frequency Analysis for Removing Rain and Snow from Videos. International Journal of Computer Vision (86), Issue 2, January 2010, pp 256-274.

5. K. Garg and S. K. Nayar. When does a camera see rain? In ICCV, 2005.

6. K. Garg and S. K. Nayar. Photorealistic rendering of rain streaks. In SIGGRAPH, 2006.

7. Z. Xu, X. Liu and N. Ji, "Fog Removal from Color Images using Contrast Limited Adaptive Histogram Equalization," 2009 2nd International Congress on Image and Signal Processing, 2009, pp. 1-5, doi: 10.1109/CISP.2009.5301485.

8. Robby T. Tan, ―Visibility in Bad Weather from a Single Image‖, Computer Vision and Pattern Recognition, 2008, pp. 1-8.

9. Robby T. Tan, Niklas Pettersson and Lars Petersson, ―Visibility Enhancement for Roads with Foggy or Hazy Scenes‖, Proceedings of the 2007 IEEE Intelligent Vehicles Symposium, 2007, pp.19-24.

10. Dongjun Kim, Changwon Jeon, Bonghyup Kang and Hanseok Ko, ―Enhancement of Image Degraded by Fog Using Cost Function Based on Human Visual Model‖, Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, 2008, pp. 64-67.

11. Nicolas Hautière, Jean-Philippe Tarel. Rain or Snow Detection in Image Sequences Through Use of a Histogram of Orientation of Streaks. International Journal of Computer Vision 93(3), July 2011, pp 348-367.

12. N. Jacobs, B. W., N. Fridrich, A. Abrams, K. Miskell, B. Brswell A. Richardson, R. Pless, The global network of outdoor webcams: Properties and apllications, in: ACM International Conference on Advances in Geographic Information Systems, 2009.

13. P. Barnum, S. Narasimhan, T. Kanade, Analysis of rain and snow in frequency space, International Journal of Computer Vision 86 (2-3) (2010) 256–274.

14. J.-P. Tarel, N. Hautière, Fast visibility restoration from a single color or gray level image, in: IEEE International Conference on Computer Vision, 2009, pp. 2201–2208.

15. Y. Lu, J. Cai, H. Zheng and Y. Zeng, "A Deep Meta-Learning Neural Network for Single Image Rain Removal," 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), 2020, pp. 231-237, doi: 10.1109/CISPBMEI51763.2020.9263524.

16. S. Gupta and R. K. Sunkaria, "Real-time salt and pepper noise removal from medical images using a modified weighted average filtering," 2017 Fourth International Conference on Image Information Processing (ICIIP), Shimla, 2017, pp. 1-6.

17. L.-F. Shi et al., ―Removing haze ppapers from single image via exponential inference with support vector data description,‖ IEEE Trans. Multimedia, vol. 20, no. 9, pp. 2503–2512, Sep. 2018.

18. K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising," in IEEE Transactions on Image Processing, vol. 26, no. 7, pp. 3142-3155, July 2017.

19. H. Zhang and V. M. Patel, "Density-Aware Single Image De-raining Using a Multistream Dense Network," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 695-704.

20. W. Yang, R. T. Tan, J. Feng, Z. Guo, S. Yan and J. Liu, "Joint rain detection and removal from a single image with contextualized deep networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 6, pp. 1377-1393, 1 June 2020.

21. L. Shen, Z. Yue, Q. Chen, F. Feng and J. Ma, "Deep joint rain and haze removal from a single image," 24th International Conference on Pattern Recognition (ICPR), Beijing, 2018, pp. 2821-2826.

22. X. Fu, J. Huang, X. Ding, Y. Liao and J. Paisley, "Clearing the skies: a deep network architecture for single-image rain removal," IEEE Transactions on Image Processing, vol. 26, no. 6, pp. 2944-2956, June 2017.

23. X. Fu, J. Huang, X. Ding, Y. Liao, and J. Paisley, "Removing rain from single images via a deep detail network," Proc. IEEE Conf. Comput. Vis. Pattern Recognit, Jul. 2017, pp. 1715–1723.

24. Li X,Wu J,Lin Z,et al. "Recurrent squeeze-and-excitation context aggregation net for single image deraining," Proceedings of the European Conference on Computer Vision( ECCV) , 2018, pp.254-269. Authorized licensed.

25. V. Santhaseelan and V. K. Asari, ―Utilizing local phase information to remove rain from video,‖ International Journal on Computer Vision, vol. 112, no. 1, pp. 71–89, 2015

26. P. C. Barnum, S. Narasimhan, and T. Kanade, ―Analysis of rain and snow in frequency space,‖ Int. J. Comput. Vis., vol. 86, no. 2–3, pp.256–274, 2010.

27. Automatic Single-Image-Based Rain Streaks Removal via Image Decomposition, IEEE Transactions on Image Processing (21), No. 4, April 2012, pp. 1742-1755.

28. X. Zhang, H. Li, Y. Qi, W. K. Leow, and T. K. Ng, “Rain removal in video by combining temporal and chromatic properties,” in Proc. IEEE Int. Conf. Multimedia Expo., Toronto, ON, Canada, Jul. 2006, pp. 461–464.

29. N. Brewer and N. Liu, “Using the shape characteristics of rain to identify and remove rain from video,” Lecture Notes Comput. Sci., vol. 5342/2008, pp. 451–458, 2008.

30. J. Bossu, N. Hautière, and J. P. Tarel, “Rain or snow detection in image sequences through use of a histogram of orientation of streaks,” Int. J. Comput. Vis., vol. 93, no. 3, pp. 348– 367, Jul. 2011.

31. M. S. Shehata, J. Cai, W. M. Badawy, T. W. Burr, M. S. Pervez, R. J. Johannesson, and A. Radmanesh, “Video-based automatic incident detection for smart roads: The outdoor environmental challenges regarding false alarms,” IEEE Trans. Intell. Transp. Syst., vol. 9, no. 2, pp. 349–360, Jun. 2008.

32. M. Roser and A. Geiger, “Video-based raindrop detection for improved image registration,” in IEEE Int. Conf. Comput. Vis.Workshops, Kyoto, Sep. 2009, pp. 570–577.

33. J. C. Halimeh and M. Roser, “Raindrop detection on car windshields using geometric–photometric environment construction and intensitybased correlation,” in Proc. IEEE Intell. Veh. Symp., Xi’an, China, Jun. 2009, pp. 610–615. Google Goggles [Online]. Available: http://www.google.com/mobile/goggles/

34. D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vis., vol. 60, no. 2, pp. 91–110, Nov. 2004.

35. H. Baya, A. Essa, T. Tuytelaarsb, and L. V. Gool, “Speeded-up robus features (SURF),” Comput. Vis. Image Understand., vol. 110, no. 3, pp. 346–359, Jun. 2008.

36. Zhen Jia, Hongcheng Wang, Rodrigo Caballero, Ziyou Xiong, Jianwei Zhao, Alan Finn. A Two-Step Approach to See-Through Bad Weather for Surveillance Video Quality Enhancement. Proceedings of the IEEE International Conference on Robotics and Automation (23), Issue 6, November 2012, pp 1059-1082.

37. R. Dale-Jones and T. Tjahjadi. A study and modification of the local histogram equalization algorithm. Pattern Recognition, 26(9):1373– 1381, 2007. D. Forsyth and J. Ponce. Computer Vision: A Modern Approach. Prentice Hall, 2003.

38. A. Golan and A. Levy. Method of adaptive image contrast enhancement. US Patent 20070031055.

39. R. C. Gonzalez and R. E. Woods. Digital Image Processing. Prentice Hall, 2001.

40. K. He, J. Sun, and X. Tang. Single image haze removal using dark channel prior. In Proceedings of 2009 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), July 2009.

41. Z. Jia, H. Wang, R. Caballero, Z. Xiong, J. Zhao and A. Finn. Realtime content adaptive contrast enhancement for see-through fog and rain. In Proceedings of 2009 IEEE International Conference on International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 1378–1381, March 2010. H. Malm, M. Oskarsson, E. Warrant, P. Clarberg, J. Hasselgren, and C. Lejdfors. Adaptive enhancement and noise reduction in very low light-level video. In Proceedings of IEEE 11th International Conference on Computer vision, pages 1–8, October 2007.

42. S. Narasimhan and S. Nayar. Contrast Restoration of Weather Degraded Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(6):713–724, Jun 2003.

43. M. Piccardi. Background subtraction techniques: a review. In Proceedings of 2004 IEEE International Conference on Systems, Man and Cybernetics, pages 3099–3104, October 2004.

44. A. M. Reza. Realization of the contrast limited adaptive histogram equalization (clahe) for real-time image enhancement. The Journal of VLSI Signal Processing, 38(1):35–44, 2004.

45. S. Roth and M. J. Black. Fields of experts: A framework for learning image priors. In Proceedings of 2005 IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 860–867, June 2005.

46. R. S. Szeliski. Locally adapted histogram equalization. US Patent 6650774.

47. R. T. Tan. Visibility in bad weather from a single image. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2008), June 2008.

48. Y. Tian, Q. Wan, and F. Wu. Local histogram equalization based on the minimum brightness error. In Proceedings of The Fourth International Conference on Image and Graphics, pages 58–61, August 2007.

49. Q. Wang and R. Ward. Fast image/video contrast enhancement based on weighted thresholded histogram equalization. IEEE Transactions on Consumer Electronics, 53(2):757–764, May 2007.

50. X. Zhang, H. Li, Y. Qi, W. K. Leow, and T. K. Ng. Rain removal in video by combining temporal and chromatic properties. In Proceedings of 2006 IEEE International Conference on Multimedia and Expo, pages 461–464, July 2006.