《计算机应用研究》|Application Research of Computers

用于虚拟现实的360度视频编码技术新进展

Advance of 360-degree video coding for virtual reality: a survey

免费全文下载 (已被下载 次)  
获取PDF全文
作者 艾达,董久军,林楠,刘颖
机构 1.西安邮电大学 公安部电子信息勘查应用技术重点实验室,西安 710121;2.圣塔克拉拉大学 计算机工程系,美国 圣塔克拉拉 95053
统计 摘要被查看 次,已被下载
文章编号 1001-3695(2018)06-1606-07
DOI 10.3969/j.issn.1001-3695.2018.06.002
摘要 随着虚拟现实技术不断发展,360度视频编码开始成为研究热点。应用于虚拟现实的360度视频与传统的视频相比分辨率更高,编码数据量更大,在实际应用中面临传输带宽有限的瓶颈,编码效率问题有待解决。归纳分析了国际标准组织联合视频研究组(Joint Video Exploration Team,JVET)正在制定的360度视频编码投影变换技术及其编码优化方法,综合对比了各变换技术的编码性能。根据当前最新研究成果,对下一步研究需要解决的问题进行了探讨总结,给出未来的研究方向和思路。
关键词 虚拟现实技术;360度视频;投影方式;运动估计;视频编码
基金项目 国家公安部科技强警项目(2016GABJC51)
国家留学基金资助项目
本文URL http://www.arocmag.com/article/01-2018-06-002.html
英文标题 Advance of 360-degree video coding for virtual reality: a survey
作者英文名 Ai Da, Dong Jiujun, Lin Nan, Liu Ying
机构英文名 1.KeyLaboratoryofElectronicInformationApplicationTechnologyofSitesurveyoftheMinistryofPublicSecurity,Xi'anUniversityofPosts&Telecommunications,Xi'an710121,China;2.Dept.ofComputerEngineering,SantaClaraUniversity,SantaClara95053,USA
英文摘要 With the development of virtual reality technology, 360-degree video coding has become a hot topic in the current research.Compared with the traditional video, 360-degree video for virtual reality requires higher resolution and more data volume, which needs much higher coding efficiency on account of the limited bandwidth.JVET is developing future standards for 360-degree video coding.This paper classified and analyzed the latest projection formats and coding schemes of 360-degree video, and compared the coding performance of each method synthetically.On the basis of the latest research, it put forward the issues that need to be addressed and potential directions in the next research.
英文关键词 virtual reality technology; 360-degree video; projection format; motion estimation; video coding
参考文献 查看稿件参考文献
  [1] Routhier P P. Virtually perfect:factors affecting the quality of a VR experience and the need for a VR content quality standard[C] //Proc of SMPTE Annual Technical Conference and Exhibition. Piscataway, NJ:IEEE Press, 2016:1-20.
[2] Kopf J. 360° video stabilization[C] //Proc of ACM SIGGRAPH Asia. 2016.
[3] 张洋, 李庆忠, 臧风妮. 一种多摄像机全景视频图像快速拼接算法[J] . 光电子·激光, 2012, 23(9):1821-1826.
[4] Budagavi M, Furton J, Jin Guoxin, et al. 360 degrees video coding using region adaptive smoothing[C] //Proc of IEEE International Conference on Image Processing. Piscataway, NJ:IEEE Press, 2015:750-754.
[5] Jrvinen A. Virtual reality as trend contextualising an emerging consumer technology into trend analysis[C] //Proc of Future Technologies Conference. Piscataway, NJ:IEEE Press, 2016:1065-1070.
[6] Dai Feng, Shen Yanfei, Zhang Yongdong, et al. Selection of the most efficient tile size in tile-based cylinder panoramic video coding and transmission[J] . Visual Computer, 2007, 23(9-11):892-893.
[7] Ng K T, Chan S C, Shum H Y. Data compression and transmission aspects of panoramic videos[J] . IEEE Trans on Circuits and Systems for Video Technology, 2005, 15(1):82-95.
[8] 张洪彬, 伏长虹, 苏卫民, 等. 3D-HEVC深度图像帧内编码单元划分快速算法[J] . 电子与信息学报, 2016, 38(10):2523-2530.
[9] 田华伟, 肖延辉, 冯文刚, 等. 用于头戴式虚拟现实眼镜的视频水印[J] . 科学技术与工程, 2016, 16(5):193-195.
[10] Jin Guoxin, Ankur S, Madhukar B. Motion estimation and compensation for fisheye warped video[C] // Proc of IEEE International Conference on Image Processing. Piscataway, NJ:IEEE Press, 2015:2751-2755.
[11] Schfer R, Kauff P, Skupin R, et al. Interactive streaming of panoramas and VR worlds[J] . SMPTE Motion Imaging Journal, 2017, 126(1):35-42.
[12] Zheng Jiali, Zhang Yongdong, Shen Yanfei, et al. Panoramic video coding using affine motion compensated prediction[C] //Lecture Notes in Computer Science, vol 4577. Berlin:Springer, 2007:119-121.
[13] De la Fuente Y S, Skupin R, Schierl T. Video processing for panoramic streaming using HEVC and its scalable extensions[J] . Multimedia Tools and Applications, 2016, 76(4):5631-5659.
[14] Zhu Zhigang, Xu Guangyou, Edward M R, et al. Fast construction of dynamic and multi-resolution 360° panoramas from video sequences[J] . Image and Vision Computing, 2006, 24(1):15-17.
[15] Zheng Jiali, Shen Yanfei, Zhang Yongdong, et al. Adaptive selection of motion models for panoramic video coding[C] //Proc of IEEE International Conference on Multimedia and Expo. Piscataway, NJ:IEEE Press, 2007:1319-1322.
[16] Norkin A, Yang H, Ohm J R, et al. Call for test materials for future video coding standardization[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[17] 赵沁平. 虚拟现实综述[J] . 中国科学F辑:信息科学, 2009, 39(1):2-46.
[18] 周忠, 周颐, 肖江剑. 虚拟现实增强技术综述[J] . 中国科学F辑:信息科学, 2015, 45(2):157-180.
[19] Moreau G. Visual immersion issues in virtual reality:a survey[C] //Proc of the 26th Conference on Graphics, Patterns and Images Tutorials. Washington DC:IEEE Computer Society, 2013:6-14.
[20] Hosseini M, Swaminathan V. Adaptive 360 VR video streaming:divide and conquer[C] //Proc of IEEE International Symposium on Multimedia. Washington DC:IEEE Computer Society, 2016:107-110.
[21] Ashfaq A, Diane G, Xin Tong, et al. Immersion in cardboard VR compared to a traditional head-mounted display[C] //Proc of International Conference on Virtual, Augmented and Mixed Reality. Cham:Springer, 2016:269-276.
[22] Choi K P, Vladyslav Z, Choi M, et al. Test sequence formats for virtual reality video coding[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[23] Alshina E, Boyce J, Abbas A, et al. JVET common test conditions and evaluation procedures for 360° video[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[24] Huang Kaichen, Chien P Y, Chien C A, et al. A 360-degree panoramic video system design[C] //Proc of International Symposium on VLSI Design, Automation and Test. Piscataway, NJ:IEEE Press, 2014:1-4.
[25] Youvalari R G, Aminlou A, Hannuksela M M , et al. Efficient coding of 360-degree pseudo-cylindrical panoramic video for virtual[C] //Proc of IEEE International Symposium on Multimedia. Piscataway, NJ:IEEE Press, 2016:525-528.
[26] 王广生, 高新, 孙树文. 全景视频与个性化分发在数字博物馆中的应用[J] . 北京联合大学学报, 2015, 29(3):5-9.
[27] Lin H C, Lin Jianliang, Chang Shenkai, et al. AHG8:an alternative arrangement for cube format (double-cross layout)[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[28] He Yuwen, Vishwanath B, Xiu Xiaoyu, et al. AHG8:InterDigital’s projection format conversion tool[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[29] Kuzyakov E, Pio D. Next-generation video encoding techniques for 360 video and VR[EB/OL] . (2016-07-15). https://code. facebook. com/posts/1126354007399553/next-generation-video-encoding-techniques-for-360-video-and-vr/.
[30] Choi K, Zakharchenko V, Choi M, et al. Test sequence formats for virtual reality video coding[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[31] Lin H C, Li C Y, Lin Jianliang, et al. AHG8:an efficient compact layout for octahedron format[C] // Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[32] Zakharchenko V, Alshina E, Choi K P, et al. AHG8:icosahedral projection for 360-degree video content[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[33] Fadlallah F A, Khalifa O O, Abdalla A H. Video streaming based on frames skipping and interpolation techniques[C] //Proc of International Conference on Computer and Communication Engineering. Piscataway, NJ:IEEE Press, 2016:475-479.
[34] Zhang Chuanyi, Lu Yao, Li Jisheng, et al. AHG8:segmented sphere projection (SSP) for 360-degree video content[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[35] Li Jisheng, Wen Ziyu, Li Sihan, et al. Novel tile segmentation scheme for omnidirectional video[C] // Proc of IEEE International Conference on Image Processing. Piscataway, NJ:IEEE Press, 2016:370-374.
[36] Auwera G V, Coban M, Hendry, et al. AHG8:truncated square pyramid projection (TSP) for 360 video[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[37] Sreedhar K K, Aminlou A, Hannuksela M M, et al. Viewport-adaptive encoding and streaming of 360-degree video for virtual reality applications[C] //Proc of IEEE International Symposium on Multimedia. Washington DC:IEEE Computer Society, 2016:583-586.
[38] Jeon S, Kim N, Khanh D Q, et al. New GoPro test sequences for virtual reality video coding[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[39] Zhou Minhua. AHG8:a study on compression efficiency of cube projection[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[40] Sun Yule, Lu Ang, Lu Yu. AHG8:a study on the influence of different projection schemes[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[41] Hosseini M, Swaminathan V. Adaptive 360 VR video streaming based on MPEG-DASH SRD[C] //Proc of IEEE International Symposium on Multimedia. Piscataway, NJ:IEEE Press, 2016:407-408.
[42] Sreedhar K K, Aminlou A, Hannukselae M M, et al. Standard-compliant multiview video coding and streaming for virtual reality applications[C] //Proc of IEEE International Symposium on Multimedia. Washington DC:IEEE Computer Society, 2016:295-300.
[43] Zhou Minhua. AHG8:a study on compression efficiency of icosahedral projection[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[44] Bao Yanan, Wu Huasen, Ramli A A, et al. Viewing 360 degree videos:motion prediction and bandwidth optimization[C] //Proc of the 24th IEEE International Conference on Network Protocols. Washington DC:IEEE Computer Society, 2016:1-2.
[45] Yang Hua, Kenneth R. Optimizing motion compensated prediction for error resilient video coding[J] . IEEE Trans on Image Processing, 2010, 19(1):108-118.
[46] Ma Xiang, Yang Haitao, Zhao Zhijie, et al. Co-projection-plane based motion compensated prediction for cubic format VR content[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[47] Liu Tsuming, Ju Chicheng, Huang Yuhao, et al. A 360-degree 4K×2K panoramic video recording over smart-phones[C] //Proc of IEEE International Conference on Multimedia & Expo Workshops. Piscataway, NJ:IEEE Press, 2016.
[48] He Yuwen, Vishwanath B, Xiu Xiaoyu, et al. AHG8:algorithm description of InterDigital’s projection format conversion tool(PCT360)[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[49] He Yuwen, Ye Yan, Hanhart P, et al. AHG8:geometry padding for 360 video coding[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[50] Bao Yanan, Wu Huasen, Zhang Tianxiao, et al. Shooting a moving target:motion-prediction-based transmission for 360-degree videos[C] //Proc of IEEE International Conference on Big Data. Piscataway, NJ:IEEE Press, 2016:1161-1170.
[51] Argyriou L, Economou D, Bouki V, et al. Engaging immersive video consumers:challenges regarding 360-degree gamified video applications[C] //Proc of the 15th International Conference on Ubiquitous Computing and Communications and International Symposium on Cyberspace and Security. Washington DC:IEEE Computer Society, 2016:145-152.
[52] Hanhart P, He Yuwen, Ye Yan. AHG8:reference samples derivation using geometry padding for intra coding[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2016.
[53] Yu M, Lakshman H, Girod B. A framework to evaluate omnidirectional video coding schemes, in mixed and augmented reality[C] //Proc of IEEE International Symposium on Mixed and Augmented Reality. Washington DC:IEEE Computer Society, 2015:31-36.
[54] Rec. ITU-T H. 265 and ISO/IEC 23008-2, High efficiency video coding (HEVC)[S] . 2013.
[55] Sullivan G J, Ohm J R, Han W J, et al. Overview of the high efficiency video coding (HEVC) standard[J] . IEEE Trans on Circuits and Systems for Video Technology, 2012, 22(12):1649-1668.
[56] Misra K, Segall A, Horowitz M, et al. An overview of tiles in HEVC[J] . IEEE Journal of Selected Topics in Signal Processing, 2013, 7(6):969-977.
[57] Snchez Y, Skupin R, Schierl T. Compressed domain video processing for tile based panoramic streaming using HEVC[C] //Proc of IEEE International Conference on Image Processing. Piscataway, NJ:IEEE Press, 2015:2244-2248.
[58] Chen J, Alshina E, Sullivan G J, et al. Algorithm description of joint exploration test model 5[C] //Joint Video Exploration Team of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11. 2017.
[59] Karczewicz M, Chen J, Chien W J, et al. Study of coding efficiency improvements beyond HEVC[C] //SPIE Optical Engineering + Applications. 2015.
[60] An J, Chen Y W, Zhang K, et al. Block partitioning structure for next generation video coding, MPEG doc. m37524 and ITU-T SG16 doc COM16-C966[K] . 2015.
[61] 单娜娜, 周巍, 段哲民, 等. 高性能视频编码帧间预测的单元划分优化算法[J] . 电子与信息学报, 2016, 38(5):1194-1201.
[62] Chen J, Chien W J, Karczewicz M, et al. Further improvements to HMKTA-1. 0, ITU-T SG16/Q6 doc. VCEG-AZ07[K] . 2015.
[63] Alshina E, Alshin A, Min J H, et al. Known tools performance investigation for next generation video coding[C] //SPIE Optical Engineering+Applications. 2015:95991B.
[64] Choi K, Alshina E, Alshin A, et al. Information on coding efficiency improvements over HEVC for 4K content, MPEG doc. m37043[K] . 2015.
[65] Chen Jianle, Chen Ying, Karczewicz M, et al. Coding tools investigation for next generation video coding, ITU-T SG16 doc. COM16-C806[K] . 2015.
[66] Said A, Zhao Xin, Chen Jianle, et al. Position dependent intra prediction combination, MPEG doc. m37502 and ITU-T SG16 doc. COM16-C1016[K] . 2015.
[67] Filippov A, Rufitskiy V. Reference sample adaptive filtering for intra coding, MPEG doc. m37526 and ITU-T SG16 doc. COM16-C983[K] . 2015.
[68] Chien W J, Karczewicz M. Extension of advanced temporal motion vector predictor, ITU-T SG16/Q6 doc. VCEG-AZ10[K] . 2015.
[69] Lin S, Chen H, Zhang H, et al. Affine transform prediction for next generation video coding, MPEG doc. m37525 and ITU-T SG16 doc. COM16-C1016[K] . 2015.
[70] Liu H, Chen Y, Chen J, et al. Local illumination compensation, ITU-T SG16/Q6 doc. VCEG-AZ06[K] . 2015.
[71] Zhao Xin, Chen Jianle, Karczewicz M. Mode-dependent non-separable secondary transform, ITU-T SG16/Q6 doc. COM16-C1044[K] . 2015.
[72] Lan C, Xu J, Wu F. Enhancement of HEVC using signal dependent transform (SDT), MPEG doc. m37503 and ITU-T SG16/Q6 doc. VCEG-AZ08[K] . 2015.
[73] Atena N. Challenges of 360 degree video[EB/OL] . 2016. https://atenanguyen. wordpress. com/2016/04/20/challenges-of-360-degree-video/.
[74] Atena N. 360-degree video compression:challenges, related work and motivations[EB/OL] . (2016). https://atenanguyen. wordpress. com/2016/04/21/360-degree-video-compression-challenges-related-work-and-motivations/.
收稿日期 2017/4/28
修回日期 2017/6/13
页码 1606-1612
中图分类号 TN911.73
文献标志码 A