Mobile image format research

The picture is usually the most mobile terminal consumption and occupies important visual space. Reasonable picture formatting and optimization can save you bandwidth and enhance visual effects. In this article I will analyze the characteristics, performance analysis, parameter tuning, and the selection of the relevant open source libraries.

Index

Several picture format introduction

Mobile image type support

Static picture encoding and decoding

JPEG

PNG

WEBP

BPG

Dynamic picture coding and decoding

Gif

APNG

WEBP

BPG

Map performance comparison

Introduction to several picture formats

Let’s first talk about several old picture formats that everyone familiar with:

JPEG is currently the most common picture format. It was born in 1992 and is an old format. It only supports lossless compression, and its compression algorithm can accurately control compression ratio, and store space in image quality. Since it is too common, so that the CPU of many mobile devices support hard coding and hard solutions for it.

PNG was born in 1995, a few years later than JPEG. Its itself is designed to replace the GIF format, so it has more similar places with GIF. PNG only supports lossless compression, so its compression ratio is capable. Compared with JPEG and GIF, its biggest advantage is to support complete transparent channels.

GIF was born in 1987, with the initial generation of the Internet. It has a lot of disadvantages, such as only 256 colors, and only 1 bit of the transparent channel, the file compression is not high. The only advantage is to support multi-frame animations, with this feature, it is popular from the Windows 1.0 era, and is still very popular.

After the birth of these pictures, there are also many companies or groups to try to improve them, or create other more excellent picture formats, such as JPEG 2000, Microsoft JPEG-XR, Google WEBP, Personal Developers BPG, FLIF, etc. They have made great progress relative to the old picture format, but for a variety of reasons, only a few formats can be popular. The following is the current new format of the current strength:

APNG is a picture format in Mozilla released in 2008 to replace a poor GIF animation. It is actually just an extension equivalent to the PNG format, so Mozilla has always wanted it to merge it into the PNG standard. However, the PNG development group did not accept this expansion, but it has been advancing its own MNG animation format. The MNG format is too complex that there is no system or browser support, and the APNG format is simple and easy to implement. It is currently popular. Mozilla’s own Firefox first supports APNG, and then Apple’s Safari has also begun, Chrome has also tried to start support, and it can be said that the future prospect is very good.

WEBP is the picture format Google released in 2010, and it is desirable to replace JPEG with higher compression. It uses VP8 video frame encoding as its algorithm foundation, has achieved a good compression effect. It supports lossy and lossless compression, supporting complete transparent channels, also supports multi-frame animation, and no copyright issues are a very ideal picture format. WEBP has been widely used in several years in a few years. Look at the app in your mobile phone: Weibo, WeChat, QQ, Taobao, Netease News, etc., each app has a webp figure. Facebook further, with the webp to display the sticker animation of the chat interface.

BPG is a famous programmer Fabrice Bellard in a super high compression ratio published last year (2014). Some of this programmer may feel born, but say that his work ffmpeg, QEMU wants to know. BPG uses HEVC (ie H.265) intra code as its algorithm basis, in this regard, it is undoubtedly the most advanced picture compression format. Compared to JP2, JPEG-XR, WebP, the BPG can provide higher image quality. In addition, it is benefited from the characteristics of the video coding algorithm itself, which can save multi-frame animations with very small file volume. Fabrice Bellard smart place is that he knows that he can’t get support from major browser manufacturers, so he also developed a JavaScript version of the decoder. Any browser can load this 76KB size JS file, you can directly Displays a picture in the BPG format. The reason that currently hindering its popularity is the copyright issue of HEVC and its longer coding decoding time. Although this picture format has just been released, many of the factories have begun to try, such as Ali and Tencent. Mobile image type support

What is the support of the mainstream mobile terminal to the image format? Let’s take a look at Android and iOS’s current picture codec scheme:

Android’s picture encoding decoding is responsible by the Skia graphic library, SKIA implements a common image format for codec support by hooking a third-party open source library. At present, Android native support format has only JPEG, PNG, GIF, BMP and WEBP (Android 4.0 joining), and only JPEG, PNG, webp can only be called directly. At present, Android does not support direct moving graphic decoding.

The IOS underlayer is a picture codec implemented by imageio.framework. At present, IOS native support has: JPEG, JPEG2000, PNG, GIF, BMP, ICO, TIFF, PICT, from 8.0 from iOS, iMageIO has also added support for APNG, SVG, RAW format. At the upper layer, developers can directly call ImageIO to encode and decode above these pictures. For action, developers can decode animation GIF and APNG, which can encode animation GIF.

When the two platforms are imported into third-party codec, there are some modifications to them, such as the adjustment of Android to Libjpeg, etc. With better control memory, iOS modifies the libpng to support APNG, and add more The characteristics of thread decoding. In addition, iOS develops AppleJpeg.framework for JPEG codes, which enables higher performance hardcodes and hard solutions. Only when the hard coded decoding fails, libjpeg will only be used.

Static picture encoding and decoding

Since I am mainly IOS development, the following performance evaluation is based on iPhone, and the main test code can be seen here. There are few test materials, only two:

The first is the LOGO of DRIBBBLE, including an Alpha channel for testing images of simple, graphics classes.

The second classic Lena diagram for testing photos of the photo class, with a wide detail.

Each image has four resolutions of 64×64, 128×128, 256×256, 512×512.

Test material can lead to certain tests not accurate enough, but as a reference is roughly no problem.

JPEG

There are currently three more well-known JPEG libraries:

LibJPEG: The first development time, the most widely used JPEG library. Since the JPEG standard is too complex and blurred, there is no other person to achieve, so this library is the factual standard for JPEG.

Libjpeg-Turbo: A JPEG library dedicated to improving the codec speed. It has been renovated based on libjpeg, rewritten part of the code with the SIMD instruction set (MMX, SSE2, Neon), and the official website said that there is 2 to 4 times the performance improvement relative to Libjpeg.

Mozjpeg: A JPEG library dedicated to enhancing compression ratio. It is a library-based library based on Libjpeg-Turbo released in 2014, which has 5% to 15% compression ratio with respect to libjpeg, but the corresponding coding speed is also slow.

In addition to the three libraries above, Apple has also developed an applejpeg, but there is no open source. It calls the DSP hard coding and hard solution of the chip. Although it is not as good as the three libraries above, its performance is very high. In my test, its codec is usually 1 to 2 times the libjpeg-turbo. It is a pity that the current developer does not directly access this library. The following is the codec performance on iMageio (AppleJpeg / libpng) on ??iPhone 6:

It can be seen that the smaller the Quality in the JPEG encoding, the smaller the picture volume, the worse the quality, the shorter the encoding time. The decoding time does not have a big gap, which may be that most of its time consumption is called, hardware calls. The default value of the Quality provided by Apple in his album Demo is 0.9, near this value, image quality and volume, coding decoding time can achieve a good balance.

PNG

Compared with JPEG, the PNG standard is more clear and simple, so many companies or individuals have their own PNG coding decoding implementation. But the most widely used Libpng library is available in the official PNG. Both the IOS and Android bottoms are PNG codes that call this library implementation.

Below is the codec performance of PNG on the iPhone 6:

It can be seen that the PNG and JPEG gap are not large when codec size (less color, less detail) is not large; but for pictures of the type (color and detail), PNG is in the file volume, codec It is a lot of JPEG.

Unlike JPEG, PNG is non-destructive compression, which does not provide a compression ratio option, and its compression ratio is capable. There are many tools and services for PNG optimized for PNG, which is intended to improve the compression ratio of PNG. Here are the performance comparisons of several PNG compression tools:

Pngcrush is the PNG compression tool comes with Xcode, which can achieve a good compression effect relative to the image generated by the designer with Photoshop. ImageOptim further, compares each diagram with a variety of interstitial algorithms, selecting a higher compression ratio, further reduces the file volume. Tinypng.com relative to other tools, compression ratio is not like. It enables color index tables like GIF to compress the PNG, so it will cause a colorful picture to lose part of the details. If you use TinyPNG, it is best to let the designer look at whether the color effect is acceptable after compression.

WEBP

The WEBP standard is custom-made, so far there is only the libWebp published by Google to achieve this codec. So this library is also the factual standard for this format.

WEBP encoding has several parameters:

Lossless: YES: Loss code NO: lossless code. The main advantage of WebP is that there is a loss-like code, and its performance and compression ratio performance is general.

Quality: [0 ~ 100] Image quality, 0 indicates the worst quality, the file size is the smallest, and the detail loss is severe, and 100 represents the highest image quality, the file is large. This parameter has a significant effect on loss of loss. Google’s official advice is 75. Tencent is also given to WebP evaluation is also 75. Near this value, WEBP can achieve better balance in compression ratio and image quality.

Method: [0 ~ 6] Compression ratio, 0 indicates rapid compression, time consumption, compression quality, 6 denotes extreme compression, time consumption, compression quality. This parameter also has a significant effect on loss of loss. Adjusting the highest compression ratio of 20% to 40% of the highest parameters, but the corresponding encoding time will increase by 5 to 20 times. The value recommended by Google is 4.

For encoding non-destructive pictures, Quality0, Method0 ~ 3 is relatively appropriate, and can save encoding times, and there is also a good compression ratio. Lossless coded pictures, Quality75, Method2 ~ 4 is a suitable parameter, which can have a good balance between encoding time, picture quality, and file volume.

Webp decoding has three parameters:

Use_threads: Whether to enable PTHRead multi-threaded decoding. This parameter acts only on the width greater than 512. The multi-threaded decoding will be used inside, and the CPU will take up higher, and the decoding time can be reduced by 10% to 20%.

BYPASS_FILTERING: Whether to disable filtering. This parameter only works on the loss of the picture, which can be shortened by 5% to 10% of the decoded time, but there is a zone that is smoothed in some color transitions.

NO_FANCY_UPSAMPLING: Whether to disable it. This parameter only works on the damaged picture. In my test, after opening this parameter, the decoding time will increase by 5 to 25%, which will cause some image details, and the edges of the line will increase, which is unnatural. Typically, these three parameters are set to NO, and if you want to pursue a higher decoding speed, you can try to turn on both parameters of use_threads and bypass_filtering. NO_FANCY_UPSAMPLING is unnecessary under any circumstances.

Since the Webp test data is more, only a portion of the test results of the 512×512 size are passed, which is interested in seeing the Excel attachment at the end of the article.

For simple graphic types (such as various UI materials within App), WEBP lossless compressed file volume and decoding speed is also ideal than PNG, if you want to optimize APP installation package volume You can try a WebP.

For complex images (such as photos), the WEBP lossless coding performance is not good, but the lossless coding is very good. Picture of Picture Decoding Speed ??WEBP has been different from JPEG, while file compression is much better.

BPG

BPG is currently known for the best lossless compression format, which can reduce the volume of 50% lower than JPEG. Below is a comparison of classic Lena charts, you can also see a large number of other pictures of BPG, JPEG, JPEG2000, JPEG-XR, WEBP compression effect, very obvious.

BPG currently only has libbpg released by authors. However, the author compiled a JavaScript decoder based on libbpg, which extends greatly. BPG can encode two ways of lossless and lossless compression. When it is damaged, the compression ratio can be controlled by the Quality parameter, and the optional range is from 0 to 51, the larger the value, the higher the compression ratio. In general, 25 is a good choice, the BPG’s official tool default value is 28.

Libbpg is currently not optimized for ARM Neon, so its performance performance in the mobile terminal is generally. Here is the performance test on the iPhone 6:

Since BPG encoding time is too long, I don’t put the data in the form. Under the same quality, the decoding speed of the BPG is too much JPEG, approximately slow 3 to 5 times. Currently, BPG is suitable for those who are very sensitive to traffic, but are not sensitive to decoding time. From the news on the Internet, mobile phone Taobao and mobile QQ have already tried, but it is not clear whether they have optimized BPG decoding.

Dynamic picture coding and decoding

The animation is very popular on the network, it approximates videos, but usually simply realizes, small file size, very wide application range. The initial ancestor is GIF, which is popular on the Internet from the Windows 1.0 era, and it is still difficult to replace it by other formats today. Although it is very old, the principles used are almost the same as those of today’s emerging formats.

Here is a GIF format QQ big expression:

This expression consists of 6 static charts, each picture has a certain survival time, and the coherent play has formed an animation:

In these pictures, most of the content is similar. In order to compress file volume, the usual map format supports some special ways to cut similar pictures, only the different parts of the front and rear frames:

When decoding the animation, the decoder is usually rendered in the so-called “canvas mode”. Imagine: Playing the area is a canvas, first empty the canvas before playing, then completely draw the first frame map; play the second frame, no longer clear the canvas, but only and the first frame Different areas are covered to the canvas, just like the creation of oil painting.

A first frame like this is referred to as a critical frame (i.e., I frame, intra encoded frame), and subsequent, those obtained by compensating, are referred to as predicted coded frames (P frames). In a compressed comparison moving, there is usually only a small number of keyframes, while the rest are predicted coded frames; one of the moving motives for a poor compression tool, basically critical frames. Different movit map compression tools can usually get different results.

In addition, the animation format usually has a more detailed parameter to control the drawing process of each frame, the following is several parameters for GIF / APNG / WEBP general:

Disposal Method

Do Not Dispose: Draw the current frame increment to the canvas, no empty canvas.

RESTORE to Background: Draw the current frame, clear the canvas as the default background color.

RESTORE TO Previous: Draw the next frame, first restore the canvas to the previous frame of the current frame

Blend Mode

Blend None: When drawing, all channels (including Alpha channels) overwritten to the canvas, equivalent to clearing the designated area of ??the canvas before drawing.

Blend Over: When drawing, the Alpha channel will be synthesized to the canvas, namely the effect of two images overlap.

The above techniques are the foundation of the common animation format, and the features of different moving formats are described below.

Gif

GIF defects are very obvious: it usually only supports 256 color index colors, which leads it to simulate more rich colors through jitter, difference, etc.; its Alpha channel is only 1 bit, which means that a pixel can only be complete Transparent or completely opaque.

Above this is a demo map in Tencent blog, you can see GIF due to the problem of Alpha channels, has created a serious “fever” phenomenon. The currently usual solution is to add a circle on the edge of the picture to alleviate this visual effect:

You can take a closer look at the animation expressions in the app of QQ, WeChat, etc., almost every expression is surrounded by a circle, and you have to say a very helpless solution.

GIF’s production tools have a lot, but the effect is good, and the compression is very small. For GIFs that have been made, the file volume can be compressed in a lot of volume. If you need to turn a video to GIF, Cinemagraph Pro is a good foolish tool. Here is an article describes how to compress GIF with FFMPEG, although the parameter adjustment is a bit trouble, but the effect is ideal.

Below is an optimized GIF and the GIF optimized by FFMPEG, you can see that the gap is very large.

APNG

APNG is currently not accepted by PNG, so libpng does not directly decode APNG. However, because APNG is just a simple extension based on PNG, the APNG can be easily implemented on a platform that has already supported PNG. Chromium intends to support APNG playback, only less than 600 lines of code, I also implemented a simple APNG codec tool with approximately 500 line C code. Also, on the browser that supports Canvas, you can display the APNG animation with APNG-Canvas. The best tool for APNG is currently APNGASM, most graphical tools such as Tencent’s IsParta is based on this tool.

For now, APNG is the best alternative to GIF: simple realization, wide range of range, compression ratio, good display effect.

WEBP

WEBP did not support moving at the time of 2010. When LibWebp V0.2, Google began to try to support animation, but in fact, there are many problems, performance is very poor, so that the Chrome team has not accepted. Until 2013, when libWebp V0.4, the animation format was stable and received by Chrome.

The webp animation actually packed multiple single frame webp data into a file, not to extends the single frame webp, so that the moving map is not compatible with the static chart. If you want to support the moving, you need to add a Demux module when compiling libWebp. When you decode the webp, you need to unpack it with WebPDemuxer, then decode the demolished single frame. For the convenience of compilation, I wrote a script for packaging iOS static libraries, joined MUX and DEMUX modules.

Google offers two simple command line tools for making animats: GIF2WEBP can convert GIF to webp, webpmux packs multiple webp images as dynamic diagrams, and many parameters can be adjusted. These two tools are not ideal for compression of similar frames, so that the compression ratio is not as good as APNG, but there is no better tools other than this.

BPG

The BPG itself is based on HEVC (H.265) video coding, and the implementation of the animation is taken into account when designing. Since it takes advantage of the high compression ratio of HEVC and video encoding, its dynamic compression is much better than other formats. Here and there are several BPG animation examples, you can see that the BPG animation is only one-third of the APNG / WEBP / GIF in the same quality.

I have written a simple way to use libbpg decoding animation, if necessary.

Map performance comparison

I transfer the following GIF to the WebP, APNG, BPG, and decodes all of the frames on the iPhone 6.The evaluation results are as follows:

APNG is slightly advantageous than GIF, and the decoding time is much different.WEBP has a large advantage in volume and decoding time.BPG is the largest in volume, but the decoding time is also the longest.This seems that APNG and WEBP are nice choices, and BPG has to be optimized.

Finally, make a small advertisement: If you are the developer of the iOS platform, you can try the YywebImage I develop, it supports asynchronous loading and playback, encoding and decoding of APNG, WEBP, GIF animation, support progressive image loading, can replaceSDWEBIMAGE, PinRemoteImage, FlanimatedImage and other open source libraries.

Evaluate data

All the evaluation data sheet mentioned above: image_benchmark.xlsx recommended to open it with Excel.