AI clusters have gotten a shared infrastructure. Neoclouds, enterprise AI platform groups, monetary providers organizations, life sciences groups, and analysis teams must share GPU capability. This shared infrastructure can undergo from decrease monetization, elevated operational complexity, and restricted management and visibility throughout tenants, workloads, hosts, and the community material.
EVPN/VXLAN is the sensible community basis. It offers tenant-scoped overlay segmentation utilizing VRFs, VNIs, route distinguishers, and route targets. Nonetheless, tenant-aware segmentation will not be job-aware segmentation. The scheduler understands jobs; the community sometimes understands routes, interfaces, queues, drops, and flows.
Why AI clusters want multitenancy
Devoted GPU clusters are easy to isolate, however they’re inefficient to function at scale. As GPU estates develop, organizations need a shared useful resource pool that may serve a number of groups, clients, and workload lessons with out forcing each group into its personal bodily cluster. In any other case, one group can have stranded GPUs in a devoted island whereas one other waits in queue.
The requirement seems in a number of patterns:
- A GPU-as-a-Service supplier maps every tenant to an exterior buyer with its personal tackle and coverage area (per-customer isolation whereas protecting the GPU pool shareable).
- An enterprise platform group maps tenants to growth, testing, manufacturing fine-tuning, mannequin analysis, or regulated analytics (constant setting boundaries with out constructing separate clusters).
- A monetary service division separates fraud analytics, danger modeling, and analysis workloads on one coaching cluster (stronger management boundaries and auditability with out duplicating GPU islands).
- A analysis group assigns shared GPU capability to unbiased analysis teams (clearer quota, utilization, and troubleshooting accountability throughout competing initiatives).
That is why multitenancy can not cease at compute allocation. Distributed coaching is dependent upon east-west GPU communication, sometimes over Ethernet materials, so the community turns into an integral a part of the isolation and efficiency boundary.
How business solves it right this moment
Present AI multitenancy is normally carried out throughout three layers:
- Orchestration and scheduler layer. Kubernetes-based platforms, GPU cloud orchestration programs, and Slurm schedulers outline the logical possession mannequin for the cluster. They monitor tenants or initiatives, customers, queues or namespaces, job requests, node placement, and GPU allocation. For instance, Tenant A would possibly submit Job 100 requesting eight GPUs throughout two servers, whereas Tenant B submits Job 200 requesting 4 GPUs on a special set of nodes. As an illustration, in an orchestration platform like Rafay, the platform can personal tenant onboarding and infrastructure intent, whereas the precise job scheduling might occur in Kubernetes, Slurm, or a tenant-operated scheduler.
- Host isolation layer. The host enforces the native machine boundary for every workload. If a tenant receives entire servers, isolation is less complicated as a result of the server, GPU set, and NIC set might be handled as one tenant-owned unit. If a number of tenants or jobs share GPUs throughout the identical server, the runtime should expose solely the assigned GPU gadgets and bind the workload’s communication libraries, corresponding to NCCL or UCX, to the supposed NICs. This host-side mapping issues as a result of a GPU server might have a number of NICs related to totally different switches or tenant-facing community segments. Cloth segmentation can isolate visitors as soon as it enters the community, but it surely can not right an incorrect native project the place the workload is allowed to make use of the improper GPU or NIC.
- Community segmentation layer. EVPN/VXLAN offers scalable tenant segmentation throughout the material. VXLAN encapsulates tenant visitors and makes use of VNIs to determine the overlay section or routing area. EVPN makes use of BGP to promote endpoint and prefix reachability and to manage which VTEPs import a tenant’s routes by means of route targets. In a routed AI material, a tenant generally maps to a VRF and a number of VNIs, with route distinguishers protecting tenant routes distinctive and route targets controlling import-export coverage. This enables overlapping tenant tackle house and scoped reachability throughout a shared underlay.
ACLs or safety group ACLs can add supply and vacation spot coverage, particularly in brownfield L3 designs or the place the material can not but devour richer workload id. Their limitation is operational scale: static or manually up to date ACL and VRF insurance policies don’t naturally comply with fast-changing AI job placement.
Collectively, these layers present a workable tenant-level mannequin. The remaining hole is job context: the community can normally see tenant context, interfaces, routes, queues, and counters, however not the particular scheduler job operating inside a tenant. Tenant segmentation itself doesn’t mechanically isolate Job 100 from Job 101 inside the identical tenant until job id can be carried, derived, or programmed into community coverage.
Cisco Nexus One integration with AI iorchestration platforms
Cisco Nexus One is properly positioned because the broader basis for making tenant-aware AI materials extra deterministic. On this structure, Nexus One is the entire material automation, integration, and visibility floor for all the material.
Nexus One can present material topology context to an AI infrastructure orchestration platform corresponding to Rafay by means of integration workflows or APIs. That lets groups map tenant VRFs, VLANs, and port assignments on to a tenant, relatively than managing them solely as an summary tenant label.
As well as, Nexus One extends the mannequin past provisioning. Tenant-level visibility can present the tenant’s material path and related well being indicators corresponding to congestion, drops, and so forth. This enhances AI job observability: job-aware views can correlate scheduler, topology, optics, GPU telemetry, analytics, and anomalies, whereas tenant-specific Job-ID enforcement stays a separate future-facing coverage functionality.
Tenant-aware will not be job-aware
Tenant segmentation solutions the query, “Which buyer or group owns this visitors?” AI operations typically want, “Which coaching job is creating or experiencing this visitors inside a tenant?”
This distinction issues for segmentation in addition to throughout troubleshooting. A scheduler can determine the job, allotted nodes, GPUs, and runtime state. The community can determine interfaces, routes, queues, drops, ECN marks, PFC occasions, optics well being, and paths. With out correlation, operators should manually join these two views.
The result’s a standard operational downside: the material exhibits a scorching uplink or lossy interface, whereas the platform group sees a sluggish coaching job. The lacking hyperlink is the workload id within the community working mannequin.
Future course: AI Job-ID-aware segmentation
Job-ID-aware segmentation course—patent-pending expertise from Cisco—is the logical subsequent step. (Word that this describes our architectural course, not a delivery characteristic.) The purpose is for infrastructure orchestrator (corresponding to Rafay) and scheduler (corresponding to Slurm) intent to hold each tenant id and job id into the material management and data-plane mannequin.
In that mannequin, the material controller interprets job intent into coverage. The swap information airplane carries or derives a job ID, for instance by means of VXLAN GPO bits, and enforces that solely endpoints in the identical approved tenant and job can change RoCEv2 visitors.
The anticipated advantages are operationally vital:
- Less complicated operations, as a result of groups can motive in tenants and jobs as an alternative of translating each become static community objects—essential for NetOps and material operations groups.
- Deeper visibility, as a result of drops, congestion, paths, and optics might be correlated to workload context relatively than solely to interfaces or tenant VRFs—useful for platform engineering and SRE groups.
- Extra granular segmentation, as a result of coverage can comply with the lifecycle of a job relatively than stopping on the tenant boundary—vital for safety, compliance, and tenant governance groups.
This method is constructed on open requirements—not a proprietary overlay. EVPN/VXLAN is IETF-defined, and the Group Coverage Choice (GPO) follows the identical path: an IETF-defined mechanism that encodes a bunch/coverage identifier within the VXLAN header alongside the VNI, which Cisco NX-OS implements in alignment with the open specification. Tenant scope (VNI) and workload/job scope (GPO) are subsequently expressed in constructs a standards-compliant material can interpret—letting operators evolve from tenant-aware to job-aware enforcement and not using a material forklift.
Technical instance: tenant and job boundaries
Contemplate a GPU-as-a-Service setting with two clients, Tenant A and Tenant B. Every tenant is mapped to its personal VRF/VNI within the EVPN/VXLAN material. Tenant-level segmentation prevents Tenant B from reaching Tenant A.
Now assume Tenant A runs two concurrent coaching jobs. Job 100 makes use of GPUs on servers 1 and a pair of. Job 101 makes use of totally different GPUs on the identical shared material. Tenant-level EVPN/VXLAN nonetheless treats each jobs as Tenant A visitors. Job-ID-aware segmentation would add one other enforcement dimension: Job 100 endpoints might change RoCEv2 visitors with different Job 100 endpoints, however not with Job 101 endpoints, even inside the identical tenant.
That’s the architectural shift: EVPN/VXLAN stays the tenant basis, whereas Job ID turns into the longer term workload-level coverage and observability attribute.
Advancing safety from tenant-level to job-level segmentation
AI information heart multitenancy begins with EVPN/VXLAN tenant segmentation, but it surely doesn’t finish there. The stronger working mannequin combines topology-aware provisioning, tenant-level enforcement, and end-to-end visibility right this moment, then evolves towards Job-ID-aware segmentation as scheduler and orchestrator integration matures.
If you’re designing a shared AI cluster right this moment, tenant-aware EVPN/VXLAN is the inspiration. Job-aware enforcement and observability are the subsequent frontier.
*Particular because of Ramesh Ponnapalli and his group, whose engineering management has been instrumental in bringing this expertise to life.
Extra assets:
