Welcome back to AI Model Optimization with Intel Neural Compressor. Model quantization reduces the word lengths of parameters. Not only does this reduce the model size, allowing for deployment to smaller edge devices, it can also speed up inference by reducing memory bottlenecks and by taking advantage of AI-optimized instruction sets and accelerators. For instance with shorter word lengths, the AMX technology in 4th Generation Intel Xeon Scalable processors can process more parameters through i
ts tiled multiplies. Of course reducing word length reduces the accuracy of the model by some amount. So how do you choose the right model quantization approach for your application’s needs? It helps to start with some first principles of quantization. Most models are developed and trained with 32-bit floating-point parameters. Moving to 16-bit floating-point reduces both the exponent and the mantissa. So now the representable precision is more coarse-grained, and we lose some of the representab
le dynamic range. BFloat16 was developed to preserve the range of representable values, while giving up a bit more of the precision. It makes for simpler conversion from FP32, truncating the mantissa. And the precision is usually good enough for deep learning, making it an easy way to optimize both inference and training. Moving to integer formats like INT8 can optimize models even further, but this requires mapping all the possible data points to a finite range of integer values. So not only do
you lose granularity, you also have to choose what range of floating point values to map to these 255 discrete integer values. So the quantization process needs to figure out the right range of values for a given set of parameters and that might include clipping some outliers to control the size of the range and based on that calculate the scale factor for the mapping. Figuring this all out, along with the accuracy effects on a given model with a given dataset, requires some extra effort, which
is why there is a variety of model quantization techniques. To fully quantize to INT8 and maintain the highest accuracy possible, use quantization-aware training. This technique actually trains using FP32 data, but the FP32 values are rounded to mimic the precision of INT8. Hence the training is aware that the parameters will be quantized to INT8 afterward, so it can properly tune the weight values. This comes at a cost of extra time and effort in the training phase. Post-training static quanti
zation avoids that training overhead, just adding a calibration step. This runs inference passes on a subset of data, observing what the ranges are for the parameters so it can map to the integer range as well as possible, then it evaluates the accuracy and possibly iterates until it meets its criteria. Post-training dynamic quantization is simpler – it only quantizes the weights ahead of time. The activations are quantized during inference based on their ranges, so the matrix operations can be
performed with INT8. There’s some overhead with that dynamic conversion, plus they’re written to and read from memory as FP32. This technique is good for models whose runtime is dominated by loading weights from memory, such as transformers or LSTMs with small batch size. Finally, mixed-precision is used more in training because it maintains good enough floating point accuracy while optimizing the model, increasing your training capacity. But it can be a really simple way to achieve some model o
ptimization during inference without much effect on accuracy. All of these techniques are available in Intel Neural Compressor. Subsequent videos in this series will show how to use each of them. To get started now, check out the resources linked below, or scan the QR code here to learn more and download. And be sure to check out the other videos in the AI Model Optimization series.
Comments