What is Model Compression?

What Is Model Compression?

Model compression refers to a collection of techniques that reduce the size, memory footprint, and computational requirements of trained AI models. A large neural network with billions of parameters may deliver excellent accuracy but requires expensive GPU hardware to run and responds slowly. Model compression aims to create a smaller, faster version of that model that retains most of its capabilities at a fraction of the cost.

This is one of the most practically important areas of AI infrastructure because it directly impacts deployment costs, response latency, and the range of hardware that can run AI models. For businesses in Southeast Asia deploying AI across diverse environments, from cloud servers in Singapore to mobile devices and edge hardware across the region, model compression makes AI accessible where it otherwise would not be feasible.

Model Compression Techniques

Several established techniques are used to compress AI models:

Quantisation

Quantisation reduces the numerical precision of model parameters. A standard model uses 32-bit floating-point numbers for its weights and calculations. Quantisation converts these to lower precision formats:

• INT8 (8-bit integer): Reduces model size by 4x with minimal accuracy loss for most models
• INT4 (4-bit integer): Reduces model size by 8x with moderate accuracy impact, increasingly popular for large language models
• FP16/BF16 (16-bit floating point): A common middle ground that halves model size with negligible accuracy loss

Quantisation is the most widely used compression technique because it delivers significant size and speed improvements with minimal effort. Tools like ONNX Runtime, TensorRT, and llama.cpp make quantisation accessible to non-specialist teams.

Pruning

Pruning removes unnecessary connections or neurons from a neural network. Research has shown that many trained models contain redundant parameters that contribute little to performance. Pruning identifies and removes these parameters:

• Unstructured pruning: Removes individual weights, creating sparse matrices
• Structured pruning: Removes entire neurons, attention heads, or layers, resulting in a genuinely smaller model that runs faster on standard hardware

Structured pruning is generally more practical because it produces models that can use standard hardware and libraries without specialised sparse computation support.

Knowledge Distillation

Knowledge distillation trains a smaller "student" model to mimic the behaviour of a larger "teacher" model. Rather than training the student on raw data, it learns to match the teacher's outputs and internal representations. The result is a compact model that captures much of the teacher's knowledge:

• The student model can be 10-100x smaller than the teacher
• Performance typically retains 90-99% of the teacher's accuracy
• The student can use a completely different, more efficient architecture

This technique is behind many practical deployments, including DistilBERT (a compressed version of BERT that is 60% smaller and 60% faster while retaining 97% of its language understanding capability).

Architecture Optimisation

Designing inherently efficient model architectures rather than compressing large ones:

• MobileNet and EfficientNet: Vision models designed for mobile and edge deployment
• TinyLlama and Phi: Language models designed to be small yet capable
• ONNX optimisation: Converting models to the ONNX format and applying graph optimisations that eliminate redundant operations

Why Model Compression Matters for Business

Model compression has direct business impact across several dimensions:

Cost Reduction

Running a large language model in production can cost $10,000 or more per month in GPU infrastructure. A properly compressed version of the same model running on cheaper hardware can reduce this to $1,000-2,000 per month while maintaining acceptable quality. For organisations running multiple AI models in production, compression can reduce total AI infrastructure costs by 50-80%.

Faster Response Times

Compressed models generate predictions faster because they perform fewer calculations. For customer-facing applications where response time directly impacts user experience and conversion rates, this improvement is commercially significant. Reducing an AI response from 2 seconds to 200 milliseconds can meaningfully improve user engagement.

Edge and Mobile Deployment

Many AI use cases in Southeast Asia require running models on devices with limited resources. Retail point-of-sale systems, agricultural sensors, mobile banking apps, and manufacturing quality inspection cameras all benefit from compressed models that can run on modest hardware without cloud connectivity.

Environmental Impact

Smaller models consume less energy for every prediction. For organisations with sustainability commitments, model compression reduces the carbon footprint of AI operations.

Implementing Model Compression

A practical approach to model compression:

1. Start with quantisation: It is the easiest technique to apply and often delivers the biggest immediate improvements. Most models can be quantised to INT8 with negligible accuracy loss using standard tools.
2. Benchmark before and after: Measure model size, inference latency, and accuracy on your specific evaluation dataset before and after compression. Ensure the accuracy trade-off is acceptable for your use case.
3. Consider knowledge distillation for large models: If quantisation alone is not sufficient, distill a large model into a smaller architecture designed for your specific task.
4. Use optimised inference runtimes: Deploy compressed models using optimised runtimes like TensorRT (NVIDIA), ONNX Runtime (cross-platform), or Core ML (Apple devices) to maximise performance.
5. Test in production conditions: A model that performs well on a benchmark may behave differently in production with real-world data diversity. Validate compression results under realistic conditions.
6. Iterate on compression levels: Start with moderate compression and increase until you find the optimal trade-off between size, speed, and accuracy for your specific application.

Model compression is not a one-time optimisation but an ongoing practice. As your models evolve and hardware capabilities change, regularly reassessing compression strategies ensures you maintain the best balance of performance and efficiency.