New technique makes AI models leaner and faster while they’re still learning

AI Models Streamlined During Training with CompreSSM

Scientists from MIT CSAIL and collaborating institutions have developed a new approach called CompreSSM, which reduces the size and training time of AI models as they learn, rather than after the training is complete. This technique is especially impactful for a class of architectures known as state-space models, which are widely used in applications such as language processing, audio generation, and robotics.

How CompreSSM Works

CompreSSM leverages concepts from control theory to pinpoint which parts of a model contribute the most to its performance. By analyzing a mathematical measure known as Hankel singular values, the approach can determine soon after training begins which internal states are essential. Unnecessary components are then removed early on, allowing the rest of the training to proceed much faster with a leaner model.

Benefits Over Traditional Methods

• Integrated Compression: Models shrink and accelerate while training, eliminating the need for a separate post-training trimming phase.
• Maintained Accuracy: Compressed models achieve nearly the same results as full-scale versions but train up to 1.5 times faster.
• Efficiency: Unlike pruning and knowledge distillation, CompreSSM avoids the high computational cost of training large models entirely before compression.

For instance, a model reduced to a quarter of its original dimension achieved 85.7% accuracy on the CIFAR-10 dataset, outperforming models trained at the same smaller size from scratch. The method also delivered a 4x training speedup on the Mamba architecture.

Theoretical Foundation and Applications

CompreSSM builds on solid mathematical grounding, ensuring compression decisions are both stable and reversible if needed. The technique is particularly effective on models where state dimension strongly affects performance, such as multi-input multi-output (MIMO) systems.

A Step Toward More Efficient AI

The MIT-led team's work opens new pathways for building efficient AI systems, especially as models continue to grow in size and complexity. Their conference paper on the approach will be presented at the International Conference on Learning Representations 2026.

Learn more in the original article by MIT News.