If you could isolate one component that slows down AI and HPC performance, it would be storage. Storage technology has experienced exponential speed improvements. Among those advancements are the creation of SSD technology and the move from a SATA connection type to NVMe, which is about 15x faster than their SATA counterparts.
Despite these improvements, the file architectures that run on storage devices have seen little change to accommodate the advancements in storage hardware. Why is this important? Modern computing workloads continually need more data. Consider genomics, which benefits from the abundance of rich data sets; the more abundant, the better. To house and process sufficient data, teams must leverage both cloud storage and on-premise infrastructure. However, traditional file systems aren’t built for this.
WEKA Provides a Truly Parallel File System
GPU-powered servers are allowing teams to do what would have been impossible before. These systems can run algorithms that process incredible amounts of data at breakneck speeds. However, as data density increases and the use of hybrid systems along with it, teams may still notice system performance suffering. This is due to the disparity between processing speeds and file storage architecture.
As file densities increase, it’s more critical for local and cloud-based file systems to have interoperability. WekaFS allows parallel file access via POSIX, NFS, SMB, S3, and GPUDirect Storage. These connections enable users to connect directly with any S3 system like AWS S3, providing speeds that rival or surpass local performance.
WekaFS doesn’t use typical block storage, which uses a layered architecture, impacting performance. Instead, it’s a distributed, parallel file system that enables a cloud-like experience whether working on-premise, in the cloud, or, more realistically, in a hybrid environment. How does it achieve this? WekaFS leverages low-level Linux container techniques to run its own real-time operating system alongside the Linux kernel, which minimizes context switching and shortens the IO path. These optimizations, among others, reduce system latency by simplifying how data is handled.
Another benefit of WekaFS is its ability to scale out systems easily. Traditional NAS systems have a more complex scale-out process since they require the full isolation of user groups and the creation of new file systems and namespaces. Consequently, teams would have to manage separate pockets of user data instead of treating all the user groups as parts of the whole. On the other hand, with WekaFS, a single file system can support billions of directories and trillions of files providing room for near limitless expansion.
Powering AI and HPC With NVIDIA GPUDirect
Normal GPU operations require GPUs to communicate with system memory constantly. GPUDirect creates a direct path between storage and GPU memory. This streamlined approach eliminates unnecessary memory copies, simplifying system paths and significantly improving performance.
When GPUDirect is combined with the WekaIO solution, it can achieve a performance of up to 5x greater than using WekaIO on its own. Since AI and HPC workloads perform better with denser data, the impact of even minor performance improvements can be significant — GPUDirect allows for radical performance improvements that help remove computational bottlenecks.
Deploy WEKA Powered GPUDirect Systems With Ease
Data requirements continue to grow, and traditional file systems can’t keep pace. Companies that want to future-proof their infrastructure need a file system that can scale with ever-increasing data densities. WekaIO solutions provide a file infrastructure designed for performance, scale, and flexibility.
Equus helps its clients deploy preconfigured technology stacks based on Intel and AMD hardware and leverage WekaIO and GPUDirect technologies. To start, we learn about your needs and then guide you through the selection process. From there, we design, assemble, configure, test, and deploy your customized solutions.
Both AMD and Intel systems leverage the latest PCIe Gen4 servers providing the following benefits to performance and efficiency:
- AMD and Intel systems access data faster with twice the NVMe storage and I/O utilizing PCIe 4.0
- Up to 40 cores per socket, higher instructions per clock, and two 512-bit FMA units
- Up to 20% power efficiency
WEKA provides a modern files system for today’s HPC requirements — one that is optimized for hybrid computing use cases. Contact us to learn more about how WekaIO and GPUDirect software can help you create a more scalable infrastructure.
