Neoclouds
Neocloud Users
AI Data Centers
Enterprise Data Centers
Visibility & Posture
Verify Rented Capacity
Risk Prioritization & Hardening
Red Teaming & Pen Testing
Runtime Detection & Response
Compliance & Audit Readiness
Blog
Learn
FAQ
BMC Exposure
Blog
Some GPU cloud and neocloud providers use InfiniBand to connect thousands of GPUs into high-performance clusters. It enables extremely fast communication between machines, making it essential for many large-scale AI training workloads.
But InfiniBand was originally designed for environments where the machines sharing a network were generally trusted. In a multi-tenant GPU cloud, that assumption no longer holds.
During our security research, we identified two concerning configurations in shared InfiniBand environments. In one case, different tenants shared the same network partition. In another, tenant-controlled machines could access management information belonging to other devices on the fabric.
Neither finding required exploiting a previously unknown vulnerability.
What Is InfiniBand, and Why Does It Matter?
InfiniBand is a high-speed networking technology commonly used to connect GPUs, servers, and storage in AI and high-performance computing environments.
Unlike traditional Ethernet networks, InfiniBand supports Remote Direct Memory Access (RDMA), allowing machines to transfer data directly between application memory regions with minimal CPU involvement.
This makes it particularly useful for AI training, where thousands of GPUs may need to exchange large amounts of data continuously.
However, this performance comes with an important security consideration.
When multiple customers share the same InfiniBand fabric, the provider must ensure that one tenant cannot access, interfere with, or learn sensitive information about another tenant’s infrastructure.
InfiniBand includes mechanisms to enforce these boundaries, but they must be configured correctly.
How InfiniBand Separates Tenants
One of InfiniBand’s primary isolation mechanisms is partitioning.
Each partition is identified by a Partition Key (P_Key), which determines which devices are allowed to communicate over the fabric.
In a properly segmented multi-tenant environment, different customers should be assigned separate partitions, preventing unauthorized communication between their devices.
InfiniBand also includes a management component called the Subnet Manager (SM), which discovers devices, assigns addresses, configures routing, and manages the fabric.
The SM works alongside the Subnet Administrator (SA), which provides information about the fabric to connected devices.
These components are necessary for operating large GPU clusters, but they also introduce sensitive management capabilities.
If tenant isolation or management access is configured incorrectly, customers may gain visibility into infrastructure that should remain outside their environment.
Finding #1: Different Tenants Shared the Same InfiniBand Partition
During our research into shared GPU cloud environments, we examined how InfiniBand partitions were configured.
In one environment, we discovered that devices belonging to different customers were assigned to the same partition.
This meant that the InfiniBand partition configuration was not enforcing separation between those tenants.
The issue is important because partitioning is one of the primary controls used to restrict communication between devices on a shared fabric.
When tenants share a partition, the network no longer uses that mechanism to prevent communication between their devices.
Why Is This Dangerous?
InfiniBand supports RDMA operations that can read or write registered memory regions on remote machines, subject to additional access controls.
Sharing a partition does not automatically give one tenant access to another tenant’s memory. RDMA operations still depend on other requirements, including valid connections, memory addresses, and access keys.
However, a shared partition removes an important layer of protection between customers.
This creates several potential risks:
Cross-tenant communication: Devices belonging to different customers may be able to communicate over the shared fabric.
Reduced isolation: A key network-level security boundary is no longer enforced.
Infrastructure visibility: Tenants may be able to discover information about other devices connected to the fabric.
Increased attack surface: Additional weaknesses in RDMA applications or access controls could have consequences beyond a single tenant.
In a multi-tenant environment, these boundaries should not depend solely on the applications running above the network.
Each tenant should have its own properly configured fabric partition, rather than relying on a shared default partition.
Finding #2: Tenants Could Access Fabric Management Information
The second finding involved the InfiniBand management plane.
InfiniBand relies on management protocols to discover devices, monitor their status, and configure the network.
These operations are intended to be controlled by trusted infrastructure administrators.
However, in one environment we examined, a tenant-controlled machine could query management information from devices belonging to other customers.
We confirmed that management requests could retrieve configuration information and traffic counters from devices outside the tenant’s own environment.
Why Is This Dangerous?
Management information can reveal details about how infrastructure is configured and used.
For example, traffic counters can provide visibility into network activity. If collected over time, this information could potentially reveal when other tenants are running workloads or transferring large amounts of data.
Our research confirmed that this information was accessible, but we did not continuously monitor other tenants’ activity.
The more significant concern is the separation between tenant workloads and the fabric’s control plane.
InfiniBand includes management keys and access controls intended to protect sensitive configuration operations. If these protections are missing or improperly configured, the consequences can extend beyond information disclosure.
Depending on the configuration, an attacker who gains unauthorized control over fabric management functions could potentially:
Disrupt workloads: Interfere with communication between GPUs and servers.
Modify network configuration: Change routing or partition settings.
Impact other tenants: Affect workloads sharing the same fabric.
Compromise availability: Cause broader disruptions across a GPU cluster.
These are potential consequences of additional management-plane weaknesses, not actions we demonstrated during our assessment.
Our confirmed finding was that tenant-controlled infrastructure could access management information belonging to other devices on the shared fabric.
In another environment, access to certain low-level management interfaces was restricted, demonstrating that these boundaries can be enforced.
What Should Providers Check?
Securing a shared InfiniBand environment requires protecting both tenant communication and the underlying management plane.
1. Separate tenant partitions: Assign appropriate P_Keys to different tenants and avoid unrestricted membership in shared default partitions.
2. Restrict management access: Prevent tenant-controlled devices from accessing privileged fabric management operations.
3. Protect management credentials and keys: Configure appropriate management keys and access controls rather than relying on default settings.
4. Limit infrastructure visibility: Ensure tenants cannot unnecessarily discover information about other customers’ devices or network activity.
5. Validate isolation from the tenant’s perspective: Test what customers can actually discover, communicate with, and access across the fabric.
These checks are particularly important in environments where multiple organizations share the same high-performance networking infrastructure.
The Bigger Picture
Both findings highlight a challenge in securing modern AI infrastructure.
In the first environment, different tenants shared an InfiniBand partition that should have provided network-level separation. In the second, tenant-controlled devices could access management information belonging to other devices on the fabric.
Neither required a new vulnerability in InfiniBand itself. The risks came from how existing security controls were configured in multi-tenant environments.
InfiniBand provides mechanisms for partitioning, access control, and management-plane protection. But those mechanisms must be configured with untrusted tenants in mind.
This research relates to #2: Network & Interconnect Vulnerabilities, as well as #3: Unsafe Multi-Tenant Isolation & Resource Reuse, in the OWASP Top 10 Neocloud & AI Data Center Security Risks.
High-performance networking should never come at the expense of tenant isolation.
Inside the Neocloud is a three-part research series by Lava exploring security risks uncovered during real-world assessments of GPU clouds and AI infrastructure.
Explore the previous articles in the series:
