When AI Models Get Too Big: Why Mixture-of-Experts Efficiency Matters for IoT

As AI models grow, the challenge is no longer only how capable they are, but how much hardware is needed to run them. Nota AI is researching ways to reduce the memory and GPU requirements of Mixture-of-Experts models, an approach that could also matter as more AI processing moves closer to connected devices.

A Short Excursion Into Mixture-of-Experts

Large language models normally contain billions of parameters that have to be stored and processed. Mixture-of-Experts, or MoE, takes a different approach: instead of using every part of the model for every input, the architecture contains multiple specialized sections, known as experts.

For each input token, a routing mechanism decides which of these experts should be activated. Only a subset performs the actual computation. In simple terms, the model tries to send each task to the parts best suited to handle it.

This can reduce the computation required for an inference. There is, however, an important limitation: although only some experts are active at any given moment, the complete model still has to be available in memory. Very large MoE models can therefore continue to require substantial GPU and memory resources.

Quantization Reduces the Hardware Requirement

One method for reducing this footprint is quantization. Instead of representing model parameters with comparatively high numerical precision, they are stored using fewer bits, for example 4 bit or even 3 bit.

This can significantly reduce memory requirements and make inference more efficient. But with MoE models, a small numerical change can have an unusually large effect.

The reason lies in the routing mechanism. If quantization slightly changes the scores used to select an expert, the model may choose a different expert than it would have selected before quantization. The result can be a different processing path and potentially a different answer.

Nota AI Targets the Routing Problem

Nota AI has developed two approaches that address this problem. Both have been accepted for EMNLP 2026, an international conference for natural language processing.

The first method, MENDS-MoE, is designed to preserve the ranking of experts during quantization, particularly when several experts have very similar routing scores. According to Nota AI, experiments with 4-bit and 3-bit quantization across three MoE models achieved higher average accuracy and better language-model performance than the evaluated comparison methods in most scenarios.

The second approach, OPERA, focuses the optimization on changes in expert selection that actually affect the model's final output. Instead of correcting every routing difference, the method attempts to concentrate resources on the differences that matter for answer quality.

Together, the two approaches address a central MoE question: How far can a large model be compressed before efficiency gains begin to change the decisions made inside the model?

What Does This Have to Do With Wireless IoT?

At first glance, research into trillion-parameter language models and GPU clusters seems far removed from RFID, Bluetooth, LPWAN or wireless sensor networks.

The direct connection is indeed limited. Nota AI's current work primarily targets large AI models and GPU-based AI infrastructure, not tiny battery-powered IoT sensors.

The underlying challenge, however, is highly relevant to Wireless IoT: bringing intelligence closer to where data is generated requires AI models that can operate within constrained computing, memory and energy budgets.

Wireless IoT systems increasingly generate data at sensors, machines, gateways and edge computers. Sending every data point to a central cloud is not always desirable because of latency, bandwidth, connectivity, cost or data-governance requirements. More local processing can therefore become attractive.

The smaller and more efficient AI models become, the more options system designers have for distributing intelligence between the cloud, data center, edge server and gateway.

Quantization, pruning and other model-optimization techniques are consequently becoming part of the broader Edge AI discussion. The MoE research presented by Nota AI operates at the much larger end of this spectrum, but the engineering objective is familiar from IoT: achieve useful computation with fewer resources.

From 24 GPUs to Four

Nota AI is also applying its optimization technology to large-model deployment. According to the company, it enabled the operation of the more-than-one-trillion-parameter Qwen3.8-Max model on four NVIDIA B300 GPUs, compared with 24 GPUs in the original configuration.

The company also reports running Moonshot AI's Kimi K3 on four B300 GPUs instead of eight and Solar Open 2 from Upstage on two NVIDIA H100 GPUs instead of eight.

Earlier in 2026, Nota AI placed third among around 40 teams in the Efficient Qwen Competition at the ICML AdaptFM workshop. Using Qwen3.5-4B on a single NVIDIA A10G GPU, the company reports an average 6.978-fold inference speedup over the baseline while maintaining model performance.

Efficiency Is Becoming Part of AI Architecture

The significance for industrial and IoT architectures is broader than any individual benchmark. As AI becomes another processing layer alongside sensing, connectivity and cloud infrastructure, the resources required to run models become an architectural consideration.

For Wireless IoT, this does not mean that trillion-parameter models will suddenly run inside sensors. It means that the same question already familiar from embedded systems is moving deeper into AI: How much intelligence can be delivered with the available memory, compute power and energy?

Mixture-of-Experts optimization is one example of how researchers are trying to answer that question.