Machine learning keeps getting more capable, and more expensive to run. Deep learning models, and large language models in particular, demand serious amounts of memory, computation, and energy. Optimizing them is not just about saving money. It is about making AI fast enough to be usable, cheap enough to be sustainable, and small enough to run where it is actually needed.
This matters in two very different settings. On the edge, deploying models to phones, embedded systems, and IoT devices runs straight into hardware limits. In the cloud, the cost of serving large models at scale can quietly dominate a product's economics. With LLMs, that cost is denominated in tokens, so optimization and token efficiency go hand in hand, a relationship I explore in Why Tokens Matter.
The techniques below let you run models more efficiently without giving up the accuracy you depend on. Most apply to classic ML and to modern LLMs alike.
1. Feature Selection
Not every available variable earns its place in a model. Feature selection identifies the most relevant inputs, reducing the computational load and often improving generalization by cutting overfitting.
Example: In a real estate price model, redundant features like "total area" and "number of rooms" can be dropped if "living area" already captures the signal. Univariate analysis, Recursive Feature Elimination (RFE), and the Lasso method rank feature importance and filter out the noise, while Principal Component Analysis (PCA) compresses high-dimensional data into a smaller set of meaningful components. The same instinct, feeding the model only what it needs, also applies to LLMs, where trimming irrelevant context reduces both cost and the accuracy loss described in LLM Context Window Limitations.
2. Model Pruning
Pruning removes weights or entire units from a neural network without meaningfully hurting performance. The result is a smaller model with faster inference.
Example: In a convolutional network for image classification, pruning can remove rarely used filters while keeping accuracy high. Structured pruning drops whole neurons, channels, or layers, while unstructured pruning eliminates individual low-impact weights. Iterative pruning followed by retraining lets the remaining model compensate for what was removed. Pruning is increasingly applied to transformer models too, where attention heads and layers can be thinned to shrink large language models.
3. Quantization
Quantization reduces the numerical precision of model weights, moving from 32-bit to 16-bit, 8-bit, or even 4-bit representations. This shrinks the model and speeds up inference on low-power devices, with lower energy use.
Example: A speech recognition model in 8-bit can run in real time on a phone while sparing the battery. Post-training quantization, dynamic quantization, and quantization-aware training all trade a little accuracy for large efficiency gains. Quantization has become one of the most important levers for LLMs specifically: 4-bit and 8-bit quantized models make it possible to run capable language models on a single consumer GPU, which is what allows many teams to self-host instead of paying per token.
4. Knowledge Distillation
Distillation trains a smaller "student" model to mimic a larger "teacher," transferring most of the capability at a fraction of the cost. This is ideal for resource-constrained deployment.
Example: A large BERT model can be distilled into a lighter DistilBERT that keeps strong performance with far fewer resources. MobileNet and TinyBERT follow the same idea. Distillation is also how many of today's small, fast LLMs are produced: a large frontier model teaches a compact one, yielding models that are cheap to serve and often good enough for focused tasks like classification, extraction, or routing inside a larger RAG pipeline.
5. Hyperparameter Optimization
Hyperparameters strongly influence training efficiency. Smart search strategies reduce the number of experiments needed to find a good configuration, saving both time and energy.
Example: Rather than random search over learning rates, Bayesian optimization finds strong configurations in fewer trials. Grid search and genetic algorithms help as well, while early stopping of unpromising runs avoids burning compute on experiments that will not pan out.
6. Distributed Inference and Edge Computing
Instead of routing every computation to a central data center, inference can run directly on the device. This cuts latency, reduces network load, lowers cloud costs, and improves privacy.
Example: A facial recognition feature on a phone can run locally instead of uploading images to the cloud. Federated learning extends the idea to training, keeping data decentralized and more secure. For LLMs, on-device and edge inference is exactly what quantization and distillation make possible, bringing small language models to laptops and phones without a network round trip.
7. Use of Specialized Hardware
AI-optimized hardware such as TPUs and modern GPUs delivers far better energy efficiency than general-purpose CPUs for these workloads.
Example: A team training large models can cut operational costs by moving from CPUs to accelerators built for deep learning. Edge AI accelerators like Google Coral and NVIDIA Jetson enable efficient on-device inference at low power. For LLM serving, the choice of accelerator and the use of optimized inference runtimes often matters as much as the model itself.
8. Efficient Optimization Algorithms
The optimizer you choose affects how quickly a model converges, and therefore how much compute training consumes.
Example: A model trained with Adam and learning-rate decay typically reaches good accuracy in fewer epochs than plain Stochastic Gradient Descent. Momentum-based methods, second-order approaches, and gradient clipping further improve training stability and speed.
9. Model Compression Techniques
Compression methods such as low-rank factorization, weight clustering, and Huffman coding reduce model size while preserving effectiveness, easing storage and computation on constrained hardware.
Example: Weight clustering groups similar weights together, shrinking storage and computational complexity, which is particularly valuable for embedded AI. Low-rank techniques also underpin parameter-efficient fine-tuning methods like LoRA, which adapt large language models by training a small number of additional weights instead of the whole network.
10. Adaptive Computation Techniques
Adaptive computation lets a model adjust its effort to the difficulty of the input, instead of running the full architecture every time.
Example: Early-exit networks let easy inputs leave the model after only a few layers. The same philosophy shows up in modern LLM serving through mixture-of-experts architectures, which activate only a fraction of the network per token, and through model routing, where simple requests go to a small cheap model and only hard ones reach a large expensive one.
Conclusion
Optimizing ML models is what makes AI scalable, affordable, and sustainable. Pruning, quantization, and knowledge distillation reduce resource consumption without sacrificing accuracy, while feature selection, efficient hardware, hyperparameter tuning, and adaptive computation compound those savings.
In the LLM era these techniques are more relevant than ever. They are the difference between a feature that is too expensive to ship and one that runs efficiently at scale. Whether you are deploying a small model to the edge or trying to keep a large language model's serving costs under control, the principle is the same: spend compute only where it earns its keep. Pair these methods with disciplined token usage, and you have the foundation for AI that is both powerful and sustainable.
Cost is usually what decides whether an AI feature survives its first budget review, so this work is rarely just an engineering concern. If you are at the stage of deciding which use cases are worth building at all, the sequencing matters more than the technique, and I wrote about that separately in how to integrate AI into business processes without disrupting everything.