Edge GenAI: A new chapter for generative AI

Download Listen

Generative AI (GenAI) has captured global attention since the launch of ChatGPT, ushering in a new era of content creation powered by advanced algorithms. While the cloud has traditionally been the backbone of GenAI due to its immense computational demands, a shift is underway. This article explores the emerging role of Edge GenAI and the deployment of generative models on local devices such as sensors, smartphones and edge servers.

What is generative AI (GenAI)?

Generative AI (GenAI) refers to a category of artificial intelligence models that can create original content such as text, images, audio, code, simulations and video, by learning from vast datasets. Tools like OpenAI’s ChatGPT and DALL·E are leading examples of this technology, with ChatGPT producing fluent, human-like responses and DALL·E generating images from written prompts. Since its public release in November 2022, ChatGPT has rapidly gained global traction. By mid-2025, it was receiving approximately 5.2 billion visits per month, highlighting its integration into both every day and professional use.

The influence of GenAI cannot be understated and extends beyond the realms of consumer interest. A 2025 McKinsey survey found that although only 1 percent of organisations currently consider themselves fully mature in AI adoption, 92 percent plan to increase their investment in AI over the coming year. Much of this interest is driven by the promise of GenAI to enhance productivity and creativity. As its capabilities become more refined and accessible, GenAI is poised to play a central role in the future of digital transformation across sectors.

Cloud vs edge GenAI

Typically, the overwhelming majority of processing for GenAI, including the training of large language models (LLMs) and chatbot inferencing, occurs in the cloud. This is due to the immense computational demands of GenAI as algorithms require significant resources to train, process and generate insights from large datasets. Cloud computing provides the scalability, flexibility, and cost-efficiency necessary to meet these requirements. Nonetheless, there are emerging use cases driving the need to deploy this technology at the edge, where models can operate closer to the source of data. This will cater to situations where GenAI models need to interface with data that simply cannot be transported to the cloud.

What is edge GenAI?

Edge GenAI is the deployment and execution of generative artificial intelligence models on edge computing devices rather than relying solely on centralised cloud systems. This convergence brings AI capabilities closer to where data is generated on devices such as sensors, smartphones and edge servers, and eliminates reliance on cloud computing. Deploying GenAI at the edge enables real-time content generation and decision-making in environments where transmitting data to the cloud is impractical or undesirable, such as on factory floors or in autonomous vehicles.

Why deploy GenAI at the edge?

As enterprises adopt GenAI across operations, the question of where these models run becomes more and more strategic. Increasingly, enterprises are turning to the edge for GenAI deployments and four key drivers explain this growing momentum:

1. Cost and bandwidth constraints: Enterprises are aiming to cut down on unnecessary data transfers to the cloud to reduce costs and ease network strain. As GenAI tasks often require large volumes of local data, constantly sending this to the cloud can lead to high backhaul costs.

2. Privacy and regulatory requirements: In sectors like healthcare, finance and defence, strict data rules mean sensitive information must stay on site. Running GenAI models at the edge allows organisations to process this data locally while staying compliant and gaining valuable insights.

3. Latency and reliability: Real-time applications rely on fast and consistent responses. Whether it’s interacting with customers, controlling machinery or managing emergencies, edge deployment ensures GenAI delivers results instantly without relying on cloud connections.

4. Connectivity issues and resilience: In areas with poor or unstable internet access, GenAI still needs to work reliably. Deploying it at the edge ensures that essential operations such as diagnostics or customer support continue uninterrupted.

Key obstacles and enablers for edge GenAI

Deploying generative AI at the edge presents clear advantages but also introduces significant challenges. One major obstacle is the high compute demand of LLMs, which are typically memory-intensive and require substantial processing power. Compressing these models to run on constrained edge devices is often impractical. In addition, the edge environment is highly fragmented compared to the cloud. Diverse hardware types and operating conditions mean that deploying, managing and updating GenAI models consistently is complex. Interoperability frameworks such as ONNX and Margo are emerging to address this, but they are still evolving and not yet standardised across the ecosystem.

Despite these challenges, recent developments are making edge-based GenAI increasingly viable. Advances in hardware — such as Nvidia Jetson devices — now support more powerful local AI workloads. At the same time, more efficient model architectures, like DeepSeek’s R1, are reducing training and inference requirements, making edge deployment more realistic. The rise of Small Language Models (SLMs) is another enabler. Models like Microsoft’s Phi and Falcon 3B use significantly fewer parameters yet deliver strong performance on focused, domain-specific tasks. This makes them well-suited for enterprise use cases at the edge, where tasks are often narrow in scope and constrained by privacy, latency or cost considerations.

Source: STL Partners

The rise of hybrid GenAI architectures

It is likely that hybrid architectures using both edge and cloud environments will proliferate. Here, smaller models will run on edge infrastructure but may interface with the cloud to complete operations that require more compute resources. Illustrative of this architecture is that adopted by Apple for its Apple Intelligence portfolio. In this instance, some GenAI functions run on-device (in most cases, an iPhone), yet for some features, tasks are escalated to Apple’s private cloud infrastructure or even to Azure’s cloud infrastructure. For example, in the case of Apple’s AI writing tools, proofreading and rewriting tools run on-device, summarisation tasks are escalated to Apple’s private cloud and the composing of entirely new text is escalated to ChatGPT. The diagram below demonstrates how GenAI may be deployed in a hybrid architecture at the edge, with levels of escalation based on the parameter size of the LLM needed for a given task.

Source: STL Partners

Edge GenAI in action: Real-world examples

Examples of edge GenAI are emerging across diverse industries:

• In manufacturing, Nokia’s MX Workmate Nokia’s MX Workmate is a generative AI assistant deployed at the industrial edge to support frontline workers in operational environments such as factories. Running on Nokia’s MX Industrial Edge platform, it uses generative AI to produce real-time guidance, generate safety alerts, and suggest context-specific actions. By analysing live video feeds, IoT sensor data, and operational inputs locally, it enables hands-free, AI-driven decision support that improves worker safety and operational efficiency in high-risk scenarios.
• Another novel example of enterprise AI at the edge is Fujitsu’s Private GPT. This enterprise-grade generative AI platform is designed to run entirely within an organisation’s infrastructure. It enables businesses to generate secure and tailored outputs such as automated customer responses, draft legal documents, and internal reports using their own proprietary data. Since all processing takes place locally, organisations can adopt large language models while maintaining complete control over data privacy, regulatory compliance, and system security. This brings the benefits of generative AI into highly regulated or data-sensitive sectors.

Final thoughts

Edge GenAI represents a powerful and emerging opportunity to redefine where and how generative AI operates. As demand grows for more secure, responsive, and cost-effective AI, moving GenAI closer to the source of data opens new doors for innovation across industries. While technical hurdles remain, such as hardware constraints and model optimisation, the rapid progress in small language models, efficient AI architectures, and edge-capable hardware is making this shift increasingly viable. For enterprises looking to future-proof their AI strategies, now is the time to explore how the edge can enable smarter, faster and more context-aware intelligence.