The first official performance benchmarks for AMD’s Instinct MI300X accelerator have surfaced, showcasing mixed results in MLPerf Inference v4.1. This industry-standard benchmarking tool evaluates AI systems with workloads designed to assess AI accelerator training and inference performance.
AMD released benchmarks comparing the performance of its MI300X with Nvidia’s H100 GPU to showcase its Gen AI inference capabilities. For the LLama2-70B model, a system with eight Instinct MI300X processors reached a throughput of 21,028 tokens per second in server mode and 23,514 tokens per second in offline mode when paired with an EPYC Genoa CPU.
Key findings from the benchmarks include:
The results show that the MI300X trails the Nvidia H100 in AI performance. The H100 achieved higher throughput in both server and offline modes, despite the MI300X’s higher memory capacity. These findings align with Intel’s recent claims that its Blackwell and Hopper chips offer massive performance gains over competing solutions, including the AMD Instinct MI300X.
While the MI300X shows promise, it still lags behind the Nvidia H100 in AI performance. However, its higher memory capacity and peak memory bandwidth make it a viable option for certain applications.
The latest benchmarks for AMD’s Instinct MI300X accelerator are in, and the results are mixed. While AMD’s new chip shows promise in certain areas, it ultimately trails behind Nvidia’s H100 in AI performance.
The MLPerf Inference v4.1 benchmarks reveal that the MI300X struggles to keep up with the H100 in terms of throughput. In server mode, the MI300X reached 21,028 tokens per second, while the H100 achieved 21,605 tokens per second. In offline mode, the MI300X fared slightly better, but still fell short of the H100’s score.
One area where the MI300X excels is in memory capacity. With 192 GB of HBM3 memory and a peak memory bandwidth of 5.3 TB/s, the MI300X has the potential to run larger models without the need for model splitting. This could be a significant advantage in certain use cases.
What does this mean for the future of AI?
As the AI landscape continues to evolve, it will be interesting to see how these two powerhouses continue to compete. Will AMD be able to close the gap and challenge Nvidia’s dominance? Only time will tell.