AI’s Second Wave: From Training Hype to Inference Reality
Bill Koss - CEO and President of Corespan Systems
The first wave of AI was about chasing the biggest models and the most FLOPS. The next wave is about something very different: deploying those models efficiently, everywhere, and at the lowest cost, most efficient energy use and the highest resource utilization.
The Next Wave of AI Deployment
AI economics are pivoting from model training to large-scale inference, with inference already representing the majority of compute and growing fast. Cost-per-token and energy-per-operation are becoming the real performance metrics, replacing raw peak FLOPS as the yardstick that matters.
To win this phase, infrastructure must support heterogeneous architectures with GPUs, FPGAs, CPUs, and tiered memory, so each workload runs on the right silicon at the right time. Optical interconnects are moving mainstream, enabling disaggregation and shared pools of accelerators instead of fixed, monolithic servers.
At the same time, edge inference is resurging as enterprises push compute closer to data to cut latency and cloud spend. Hardware-agnostic orchestration layers are emerging to hide silicon diversity and let operators continuously optimize for cost, latency, or power without getting trapped in any one vendor’s stack.
What Customers Want
Enterprises are discovering a harsh reality: it is not that they don’t have enough GPUs, it is that their GPUs are trapped. Rigid server boundaries strand capacity away from the workloads that need it, creating a CAPEX paradox where organizations overspend on hardware while utilization remains frustratingly low.
Static, server-centric designs pin accelerators to specific hosts, so workloads on the “wrong” server cannot access idle GPUs elsewhere. This leads to ghost capacity, GPUs sitting idle while adjacent jobs starve and this forces teams to over-provision entire clusters just to meet worst-case demand.
Customers want to break these GPU islands and turn the data center into a unified, dynamically composable resource pool. They want to fluidly assign the right mix of GPUs, xPUs, storage, and network to any application, on demand, without rewriting workloads or re-architecting the entire stack.
How Corespan’s DynamicXcelerator Delivers
Corespan’s DynamicXcelerator is built for this second wave of AI: it disaggregates and then recomposes resources so customers can finally use the hardware they paid for. Instead of eight fixed GPUs per server, DynamicXcelerator enables 24–32 GPUs per host—3–4× the density of a traditional 8-way GPU system, while keeping hosts simpler and cheaper.
A photonic and PCIe-based fabric connects host nodes, storage, and Photonic Resource Units (PRUs) into a single, dynamic pool of GPUs, RDMA, and NVMe. GPU-to-GPU communication can occur via PCIe within a host or via RDMA across multiple hosts, and GPUs can access storage directly using GPU Direct Storage (GDS) over RDMA for high-throughput data paths.
Corespan Composer, the software control plane, exposes this fabric to applications and ML/AI workbenches, constructing resource groups on demand instead of statically at rack design time. Customers can allocate fine-grained combinations all from a vendor-agnostic, heterogeneous pool.
Because DynamicXcelerator is host-vendor and accelerator-vendor agnostic, it supports multiple GPU and xPU types without locking customers into a single server or interconnect supplier. The result is higher utilization, fewer stranded islands of capacity, and an infrastructure tuned for the true metrics of the AI deployment era: cost-per-token and energy-per-operation.