What does an AI developer do, and how do you become one?
Until now, we’ve primarily associated artificial intelligence with massive data centers. That’s where models were trained and subsequent user queries were processed. A phone, camera, or robot often served as a terminal, collecting data, sending it over the network, and waiting for the results of calculations performed hundreds of kilometers away.
This model isn’t going away, but it’s no longer the only option. More and more AI tasks are being handled directly on devices, ranging from smartphones and home appliances to drones and production lines. Behind this shift is Edge AI—that is, performing AI-related computations close to where the data is generated. Specialized NPU chips play a key role in this, handling high-performance inference while consuming as little energy as possible.
There is more at stake here than just a more convenient assistant on your phone. The race for Edge AI is about who will control the computing layer of future devices, the tools for running models, and the entire semiconductor supply chain. A machine’s local autonomy, therefore, turns out to be closely linked to global dependence on silicon.
The cloud is incredibly powerful, but it doesn't solve every problem
Data centers will remain the backbone for training the largest models and handling the most demanding tasks. However, their continued expansion comes at a cost. According to data cited bythe European Commission, global data centers consumed approximately 415 TWh of electricity in 2024, and by 2030, demand could rise to about 945 TWh. AI-related workloads are expected to be one of the main drivers of this growth.
Shifting part of the inference to devices will, of course, not make energy costs disappear. However, it will change where those costs are generated and how they are managed. Instead of transmitting every fragment of video, audio, or telemetry to a server, the device can first determine locally which data is important. The network will then receive either the result of the analysis or a small snippet of the data, rather than a continuous stream of raw information.
This approach is important in situations where response time is critical. A drone avoiding an obstacle cannot base its decision on the quality of the connection. An industrial robot should not wait for a response from the server when a camera detects a person in the work zone. In these applications, network latency is not a minor inconvenience but a factor that affects safety.
Local processing is also changing the data protection model. Voice recordings, images of the interior of a home, or production process parameters can be analyzed without ever leaving the device. This does not automatically guarantee complete privacy, as the application may still store or transmit the results. However, Edge AI offers the ability to limit data transmission at the system architecture level, rather than relying solely on a service provider’s promise.
How does an NPU differ from a GPU?
The CPU must handle a wide variety of tasks. It executes operating system logic, runs applications, and efficiently executes sequential instructions. The GPU was designed for massively parallel graphics processing, and its architecture has also proven to be very effective for training neural networks.
The NPU is even more specialized. Its design is tailored to operations that recur in AI models, primarily the multiplication and addition of large sets of numbers. The chip contains multiple MAC units, memory located close to the computing unit, and mechanisms for efficient processing of tensors and vectors. The NPU thus sacrifices some of the CPU’s versatility in favor of a better performance-to-power ratio.
This does not mean that the entire model must run exclusively on a single silicon chip. A modern system-on-a-chip is more like a team of specialists. The CPU manages the task flow, the GPU handles selected parallel operations, the signal processor processes data from microphones or cameras, and the NPU performs the most common neural network computations. Qualcomm, for example, describes its Hexagon NPU precisely as part of a heterogeneous architecture in which scalar, vector, and tensor units collaborate with a shared memory.
The biggest change, therefore, occurs at the level of the overall system design, rather than the number of cores. Designers must decide where to place the data, how to minimize copying, and which block to entrust with each part of the model. Performance is increasingly determined by the cost of transferring information between memory and computing units, rather than by the sheer number of mathematical operations. IBM Research identifies this data movement as one of the fundamental limitations of modern AI architectures.
TOPS does not say how smart the device will be
Manufacturers like to describe NPUs in terms of TOPS, or trillions of operations per second. The number sounds impressive and, at first glance, makes it seem easy to compare two chips. The problem is that the result depends, among other things, on the precision of the calculations, the type of operations, and how zero elements in the model are handled.
| Format | What does that mean? | Precision | Memory usage | Speed | Typical Application |
| FP16 | 16-bit floating-point number | High | The Largest | Lower | Training and Carefully Deploying AI Models |
| INT8 | 8-bit integer | Medium/High | About 2× smaller than FP16 | Higher | Running AI Models on GPUs, CPUs, and Mobile Devices |
| INT4 | 4-bit integer | Lower | About 4× smaller than FP16 | Potentially the highest | Large language models on hardware with limited memory |
This is because the same circuit can produce different results for INT4, INT8, and FP16 data. The result may also take into account the sparsity of the weights, which allows certain operations to be skipped. Qualcomm explicitly distinguishes between performance for dense and sparse computations, and the performance for a model with appropriately organized sparsity can be twice as high.
The raw TOPS figure also does not reflect memory bandwidth, power consumption during extended operation, or which operations (layers) of the model are actually handled by the NPU. If a part of the network has to be offloaded back to the CPU, some of the theoretical advantage is lost. A reliable test should therefore include the complete application, response time, energy consumed per inference, and the quality of the result.
To give a simple example, this can be compared to evaluating a car based solely on engine power. The figure is accurate, but without information on weight, gear ratios, and fuel economy, it says very little about how the vehicle handles on the road. It’s similar with an NPU: the value of the chip is determined by the entire system, not by the most appealing number in a table.
The model needs to be scaled down, but its quality must not be compromised
The largest language models do not fit easily into a phone’s memory, and the device cannot devote its entire battery to a single AI function. That is why Edge AI is developing in parallel with model compression methods. One of the most important of these is quantization, which involves storing weights and performing some calculations with lower precision.
Changing the representation from 16 or 32 bits to 8 or 4—and in experimental approaches, even fewer—reduces the model size and the amount of data transferred from memory. Research on the AWQ method has shown that appropriate selection of the most important weights (those that are most critical in terms of activation/quantization error) can reduce the model size by a factor of four while simultaneously accelerating its operation on edge devices.
However, this optimization comes at a cost. Overly aggressive quantization can degrade the quality of responses, especially in layers that are sensitive to loss of precision. A longer context, in turn, increases the size of the model’s cache, so even a small assistant may hit the RAM limit after a while. Then there’s the issue of temperature. A smartphone can perform a short AI demo very quickly, but during long inference runs, it must reduce its clock speed to stay within thermal limits.
Edge AI therefore requires the co-design of hardware and software. The model should be aware of the limitations of a specific architecture, the compiler must be able to assign operations to the appropriate blocks, and the system should select the precision appropriate for the task. The best result does not always come from the largest available model. In a device performing a single, well-defined function, a smaller, specialized, and constantly available network may be more useful than a powerful general-purpose model.
From a washing machine to a factory: one idea, different scales
On a smartphone, a local model can summarize notifications, enhance a photo, transcribe a recording, or classify on-screen content. The benefit is obvious: the feature works faster, is sometimes available offline, and doesn’t have to send all the data to an external service. At the same time, the device has only a few or a dozen watts available for a short time, and its memory is shared by all apps.
In a washing machine, Edge AI can analyze drum vibrations, motor noise, and changes in power consumption. It doesn’t need a conversational model with billions of parameters to do this. All it takes is a small neural network capable of detecting an unusual pattern, an improperly distributed load, or gradual bearing wear. Such an algorithm can run on a microcontroller with a small NPU without generating a continuous stream of data about daily household operations.
In a drone, local awareness is a prerequisite for autonomy. The camera and inertial sensors provide data continuously, and the system must assess its position, obstacles, and possible route in real time. A loss of connection must not result in a loss of situational awareness. The cloud can help train the model beforehand or analyze the mission log afterward, but the control loop must remain on board.
In a factory, an edge device can monitor machine vibrations, temperature, and camera footage, and then halt the process upon detecting a dangerous anomaly. Aggregated data will still be sent to the central system, where multiple production lines can be compared and the model updated. The initial response, however, should occur locally. In this setup, Edge AI becomes an additional layer of control, embedded directly into the automation system.
Local AI depends on the global supply chain
A device that operates without the internet may seem technologically self-sufficient. However, the chip that enables this self-sufficiency is created through collaboration among companies that design architecture, EDA tool providers, lithography equipment manufacturers, semiconductor fabs, packaging facilities, and software developers. No single country currently controls all of these stages at the highest level.
Silicon independence should therefore not mean complete self-sufficiency. A more realistic goal is resilience—that is, in-house design capabilities, access to multiple production sources, the ability to test and package chips, open-source development tools, and a pool of expertise that cannot be replicated in a matter of months.
Europe is trying to build this position through the European Chips Act. The Digital Decade goal called for increasing the EU’s share of the global semiconductor market to 20% by 2030, but the latest data show that the EU currently accounts for about 9% of the market. The Commission’s own projections had previously indicated that, without additional measures, this share would reach only about 12% (11.7%) by 2030. The Chips Act 2.0, presented in June 2026, therefore places greater emphasis on specific design and manufacturing capabilities and supply chain resilience than on the market share figure itself. However, doubling that share to 20% remains the EU’s goal.
In this race, it’s not just state-of-the-art factories that matter. Technical universities can develop architectures for energy-efficient accelerators, compilers, quantization algorithms, embedded systems, photonics, and methods for the secure implementation of models. This is particularly important for countries that will not immediately build factories for the most advanced processors but may become indispensable in a specific segment of the value chain. In Poland, the Taiwanese company Foxconn will build an AI factory in Miękinia, though the exact timeline has not yet been specified.
The future will be hybrid
Will Edge AI beat the cloud? It’s hard to say definitively, because this isn’t a battle between two mutually exclusive models. A local system performs best on repetitive tasks that are sensitive to latency and privacy. The cloud retains the advantage when the largest model, long context, access to extensive knowledge bases, or joint analysis of data from multiple devices is required.
The most useful systems will therefore dynamically distribute the work. A phone can recognize the intent locally and remove sensitive data, and only later pass the more difficult task on to the server. An industrial machine will respond to an anomaly on its own, but will use the cloud to learn from the history of the entire fleet of devices. Home devices will perform simple classification on a microcontroller but will download a new version of the model after passing security tests.
The true measure of progress, then, will not be the number of devices labeled “AI.” What will prove more important is whether we can make informed decisions about which data and computations should remain at the edge of the network and which are worth passing on. The battle for Edge AI is, in essence, a battle over the architecture of the digital world—and thus over energy, response time, privacy, and control over technology that is increasingly making decisions alongside us.
See other news
PJAIT 2026 Orientation Day!
AI & CYBERSECURITY – PJAIT AI SUMMIT 2026!
