Each step of the AI workflow — from data preparation, to analytics, task execution and workflow orchestration — depends entirely on processing hardware.
But not all processing power is created equal. Processors have evolved and diversified to offer powerful features that support specific, critical functionalities not found in traditional central processing units (CPUs), such as efficient data handling and complex mathematical capabilities. The rapid emergence of AI has accelerated this diversification, and there’s no end in sight.
Leaders committed to deploying AI need to understand the current suite of AI chip architectures to optimize AI platform performance while managing costs across their organizations.
Why are chip architectures diversifying?
A microprocessor or processor is an “engine” that fetches instructions from memory, decodes them, performs the functions and calculations specified by those instructions and returns results to software, OSes and applications. The processor is composed of millions of microscopic transistors arranged to execute every instruction from start to finish, input to output. Every instruction can correspond to millions of transistors, and a processor capable of handling hundreds of unique instructions can involve tens of billions of transistors, each using power and generating heat.
Early processors were general-purpose. They could perform a little logic, basic math and simple data handling. As workloads required more features and functions, the instruction sets and transistors swelled. Then, two things happened: first, the processors became too complex, resulting in excessive power consumption for too little performance; second, the processors became too cumbersome and expensive to manufacture. The processors’ evolutionary tree had to split.
Graphics co-processors, known as graphics processing units (GPUs), followed, specializing in floating-point math originally suited to graphics-rendering tasks such as gaming. Today, GPU mathematical capabilities are commonplace and crucial elements of AI training and inference. Then followed neural processing units (NPUs) and tensor processing units (TPUs) to accelerate AI and machine learning (ML) workloads, including LLMs, autonomous vehicles and healthcare use cases.
For other use cases, established processors are still too expensive or complex for specific tasks, and application-specific integrated circuits (ASICs) have emerged to meet a wide array of task-specific needs, providing narrow, efficient computing for systems and servers.
But there are some downsides to this diversification. A processor can only operate on the instructions it’s designed to support. This means advanced processors, like GPUs and ASICs, are rarely able to perform general-purpose work. Their complexity also results in greater power use and heat generation. Consequently, processors can work well together and are frequently used in tandem to share the complex demands of modern AI workloads.
A closer look at processor architectures
Several computing choices can now support modern AI systems. Whether implemented on local servers or on cloud computing instances, the choice of processor configuration significantly affects AI training and inference performance. Here is an overview of the basic types of architecture for AI chips.
CPUs
CPUs are generalists. They are well-suited to most types of software applications, such as CRM platforms, databases and word processing. CPUs are also central to AI platforms as the workload manager or orchestrator, task scheduler, system component coordinator and information manager.
However, CPUs are not strong at advanced mathematical functions and have relatively limited capabilities for the kinds of sequential or logical thinking that AI training and inference demand. This is why CPUs are a critical bottleneck for AI systems, and tasks involving complex math or massive parallel mathematical equations are delegated to advanced processors.
CPUs can train AI in test or development settings, but they’re inadequate on their own for production AI platforms. It could take days, weeks or even months to train an AI using CPU-based computing. This might be fine for offline testing where budgets are tight, but advanced processors could complete the same task in hours. Still, CPUs remain essential AI components and work well with other advanced processors.
CPUs are typically the least expensive computing infrastructure. Every computer has CPUs, and cloud-based CPU computing instances are many, diverse and readily accommodate most traditional software workloads.
GPUs
Where a CPU can adeptly process a few tasks sequentially, GPUs can make advanced calculations across vast amounts of data. This originally made GPUs vital in graphics tasks such as rendering, but their mathematical prowess and sheer execution scale have catapulted GPUs to prominence as AI accelerators.
GPUs are incredibly effective for a narrow range of tasks. While CPU architecture and instruction sets are intended for low latency, sequential processing and decision-making, GPU architecture and instruction sets are built for high-throughput parallel processing, where the same limited instructions can apply to vast amounts of data. GPUs offer high memory bandwidth and can move enormous amounts of data with great efficiency.
AI model training needs to process billions of parameters across enormous data sets. Since GPUs can divide and organize these mathematical tasks across thousands of highly optimized cores, they can complete training tasks in hours. For AI inference tasks, the AI can use GPUs to process new data and formulate responses to user queries almost instantly. Modern GPUs for AI tasks can also include specialized tensor cores that enhance the efficiency of ML equations.
However, GPUs are specialized devices and are extremely poor at performing the general-purpose tasks needed to handle computer operations, such as booting a computer, running an OS, handling I/O and launching and running ordinary software applications. GPUs always have an associated CPU to manage them.
From a cost perspective, GPUs are considerably expensive. Standard CPUs range from $100 to $600, while high-performance GPUs can cost anywhere from $300 to $2,000. In cloud computing, using a server instance with a top-tier GPU costs $1.50 to $7.00 per hour or more, whereas CPU-only instances often cost pennies per hour. The value is in time saved. Completing AI training in hours rather than months or processing an inference in milliseconds rather than hours. Advanced processors such as GPUs make modern AI systems practical.
ASICs
ASIC is a broad term that encompasses an array of “customized logic” devices. The chips that operate the office coffee maker, a bedside alarm clock, a network switch or IoT devices are typically ASICs, each custom-designed and manufactured to suit a specific purpose or task.
ASICs fill vital computing roles that other, more traditional processors are simply too complex, power-hungry, inefficient or costly to support. Where a CPU might have billions of transistors and support hundreds of instructions, an ASIC can act as a customized processor stripped to support only a handful of key instructions with a fraction of the power and far better performance.
ASICs have found ready acceptance as powerful processor accelerators in AI tasks. Because ASICs can accomplish a narrow set of tasks with extremely high performance and enhanced efficiency, they can perform key AI tasks, such as mathematical operations or incorporate advanced accelerator features such as NPUs or TPUs.
However, AI ASICs are highly proprietary and are typically reserved for high-volume deployments through hyperscaler providers such as Google, AWS and Meta. An everyday enterprise wouldn’t buy a server with ASIC accelerators for AI use. Instead, a hyperscaler would spearhead the design and commission the manufacture of ASIC chips, then manage the integration of those chips into custom computing systems that would be deployed across the provider’s infrastructure for internal use or offered as another specialized AI cloud service.
On a per-chip basis, an ASIC is cheaper, delivers better performance for its intended tasks and uses far less power than a commercial processor. But the broader cost considerations for ASICs can be tricky. ASIC design can be costly, and manufacturing ASIC chips at a contract fabrication facility can represent an enormous investment for an enterprise.
The resulting upfront investment in time and capital is huge. That’s a major reason why ASICs are the realm of the largest hyperscalers that can afford them. This enormous ASIC investment also carries risk, which is why hyperscalers generally don’t offer commercial versions of their complete ASIC systems. Hyperscalers treat their ASIC investments as intellectual property and a competitive advantage.
Field-programmable gate arrays (FPGAs)
While specialized chips like ASICs have a fixed, unchangeable configuration, an FPGA can interconnect various elements to create a chip with highly customizable functionality using major hardware description languages (HDLs) like VHDL or Verilog. This type of programmability provides two significant benefits for AI developers:
- The same FPGA chip can provide various AI functions in areas like edge AI, industrial IoT and processor acceleration devices in AI data centers. This means one chip can do a variety of jobs.
- The FPGA chip can be reprogrammed as required. This can be powerful when a bug is found in the chip’s operation. The chip can be reprogrammed to fix the bug without making any hardware changes.
FPGAs also provide compelling benefits for AI infrastructures. They are reconfigurable, so they can be updated, optimized or changed to perform entirely different jobs without further hardware investment. They also enable ultra-low-latency operations, processing data immediately without batching. Finally, as FPGAs are customized to the precise needs of their ML model, they typically consume far less power than generic CPUs, GPUs and other AI accelerator chips.
An FPGA can also operate at the edge, collecting and processing data from multiple sensors and running inference on the same FPGA chip in real time. This makes FPGAs ideal for use in robotics and autonomous vehicles, medical and industrial devices and servers.
Despite the various advantages of FPGAs, there are still numerous limitations and tradeoffs to consider. Compared to ASICs, FPGAs generally demand 20 to 35 times more silicon area, run three to four times slower, and use about 10 times more power. This tradeoff comes in the form of FPGA flexibility and programmability. When extreme performance and power savings are needed at scale, ASICs can be the better choice. FPGA chips also require a well-designed circuit board to accommodate their physical installation and practical use in a working AI system. Even though FPGAs can be programmed and reused, the enterprise still needs to develop and acquire circuit boards and have systems capable of incorporating those circuits. Developing, testing and validating FPGA code can also take up to 10 times longer than conventional software, requiring experienced talent.
NPUs
An NPU is a specialized processing accelerator that efficiently runs ML and AI tasks. NPUs rarely appear as separate, standalone chips but are typically integrated into the main system processor, such as a system-on-a-chip (SoC) device. Consequently, NPUs are almost always system-centric, accelerating AI-related functionality in devices such as smartphones and laptops. This provides sophisticated AI features while emphasizing low latency and significant power savings.
NPUs are most closely associated with endpoint devices such as smartphones, tablets and laptops, where the SoC can add an NPU to enable AI capabilities such as background blur, voice transcription, noise cancellation and image enhancement while maintaining low power consumption in these battery-powered mobile systems.
Today, more dedicated NPUs are appearing in edge computing and server-class systems to assist with rapid, high-volume AI inference tasks. The same power savings that make NPUs attractive for mobile devices also apply to edge and data center environments.
NPUs typically present several key limitations to consider. NPUs only support and accelerate specific AI models. When new models emerge, the AI workload must resort to GPUs, reducing AI performance. Their dedication to specific math functions also makes them useless for traditional computing tasks. NPUs can’t be used to train AI models; they can only support inference tasks and supplement CPUs and GPUs as AI accelerators. Finally, NPU software is often proprietary, requiring specialized tools and skills, and is designed only for local or on-device AI processing. This makes NPUs more troublesome to integrate.
While NPU-capable devices incur an additional acquisition cost, the reduction in API calls, server load and power utilization are typically attractive factors for NPU deployment. However, NPUs don’t work unless the application is specifically designed to use them. NPUs are best used with AI-enhanced creative suites, applications requiring strict privacy for local data processing and developers building local ML models and AI systems.
TPUs
TPUs are specialized processor accelerators that optimize ML and AI tasks by supporting massive parallel processing for matrix multiplication, the principal mathematical operation behind neural networks such as Google Gemini. The TPU is a highly optimized ASIC dedicated to AI math operations, using a fraction of the power of conventional processors such as GPUs. The TPU was originally designed by Google and is now in its eighth generation with the TPU 8t for training and the TPU 8i for inference.
Google’s TPU isn’t readily available for local systems or private data centers. Instead, TPUs are accessed through Google Cloud as a service, where they can be used to train massive AI models, support post-training and reinforcement learning, enable high-performance inference and perform powerful data analytics.
While TPUs provide powerful acceleration for matrix multiplication tasks, they can’t be used for general computing and offer limited benefits for workloads not focused on matrix math. TPUs are a poor choice for precision scientific computing because TPUs sacrifice precision for performance. Furthermore, TPUs depend on Google’s software stack, requiring additional skills and investment to build, refine and deploy AI apps that use them. The fact that TPUs are a Google Cloud service introduces a degree of vendor lock-in that some businesses might find undesirable. TPUs offer high bandwidth memory, but limited memory space can impose bottlenecks, and they can provide reduced latency for non-batch tasks, such as a single prompt.
Language Processing Units (LPUs)
LPUs are another specialized processor accelerator that assists with AI inference tasks, focusing primarily on token generation for LLMs. LPUs rely on high-speed local memory to keep model weights readily accessible for extremely low-latency and decision-making operations. Because LPUs focus on token generation, they are intended for inference and not suited for AI model training.
LPUs such as OpenAI’s Jalapeno chip offer predictable performance and minimize the lag spikes often found in server-based language models. This makes LPUs excellent for real-time AI agents and chatbots, financial data processing and market trading, real-time voice transcription for healthcare and voice assistance, educational and adaptive learning environments, and edge computing tasks.
The strength of an LPU is in its massive onboard memory and fast, predictable operation. Still, business leaders should consider the numerous limitations of LPUs. Onboard memory can be a constraint. Even though LPUs offer significant onboard memory, this finite resource can be exhausted by large models, requiring LPU clusters. Furthermore, LPUs rely on static execution pathways that must be carefully planned and meticulously implemented in software. Compiling LPU software to drive a new or updated model can take days, making LPUs inadequate for testing or experimentation.
Ultimately, LPUs can be the best choice when processing smaller models at sub-millisecond token speeds, when high efficiency and throughput with predictable token decoding are needed, and when the fastest AI responses are required.
As with TPU chips, LPU chips are typically unavailable individually. Instead, they’re offered as integrated components of large, enterprise-class systems such as the Nvidia Groq 3 LPX Rack, which packs 256 interconnected LPU accelerators with the Vera Rubin platform, or as a cloud service such as GroqCloud, which is a generally more cost-effective, pay-as-you-go option for business users.
Selecting an AI processor architecture
Selecting processor architecture for AI projects can be a mission-critical decision. Mistakes can be costly in performance and investment. The final choices should involve executives and their technical team. The selection criteria typically involve several broad categories, including the following:
- AI workload type. AI training builds models from vast data sets and requires massive parallel processing, high memory bandwidth and high scalability. This makes GPUs the best overall choice for AI training. By contrast, AI inference runs trained models on new data and benefits from extremely low latency, rapid token generation and power efficiency. This makes LPUs best for LLM inference, while ASICs or TPUs are better for more general inference work.
- AI model maturity. Consider the stability or maturity of the models. Processor accelerators can be powerful, but often only accelerate a specific model’s code. Changing the model can reduce the effectiveness of high-end accelerators like ASICs or FPGAs, resulting in lower performance or an outright fallback to a more general accelerator. Invest in top acceleration only when a model is fully vetted and ready for reliable production use. Models in development or subject to frequent change are best supported by general GPUs or reprogrammable FPGAs.
- AI deployment environment. AI processors can be deployed at the edge or in local or cloud data centers. Edge deployments include laptops, smartphones and IoT devices and are best served by low-power NPUs. Data center deployments are large-scale deployments that offer raw compute power for the heaviest AI tasks. This can include various accelerators, such as GPUs and cloud provider-based offerings such as TPUs, LPUs, FPGAs and ASICs. Make sure the deployment environment best matches the workload.
- Programming and development skills. Choosing a processor isn’t enough. Businesses need the coding and toolchain skills to back it up. Carefully evaluate the coding skills needed to support the processor choice. FPGAs and ASICs demand extensive and highly specialized coding expertise. GPUs need more general, well-standardized coding. Cloud instances require solid experience with the provider’s APIs and nuances.
- Decide on meaningful performance metrics. Choose and implement meaningful model performance metrics and have the tools in place to track them to verify actual performance and measure improvements once processor accelerators are implemented. This provides cost justification and lets the business recognize performance degradation or unexpected reductions in model health, which might need scaling, optimization or troubleshooting.
- Effect on local data centers. Don’t overlook the secondary effects of AI processor deployments on local data center operations. New servers mean changes to cooling and rack space demands. High-efficiency processor accelerators can save power per chip or per core, but are used in such large numbers that overall data center power consumption increases dramatically. Other factors include networking changes and vast storage resources. Consider the resources needed to implement processor accelerators in the data center, and the associated costs.
Stephen J. Bigelow, senior technology editor at TechTarget, has more than 30 years of technical writing experience in the PC and technology industry.
Leave a Reply