» Current | 2020 | 2019 | 2018 | Subscribe

Linley Newsletter

AI Benchmarks Remain Immature

January 29, 2019

Author: Linley Gwennap

Several popular benchmark programs evaluate CPU and graphics performance, but even as AI workloads have become more common, comparing AI performance remains a challenge. Many chip vendors quote only peak execution rate in floating-point operations per second or, for integer-only designs, operations per second. But like CPUs, deep-learning accelerators (DLAs) often operate well below their peak theoretical performance owing to bottlenecks in the software, memory, or some other part of the design. Everyone agrees performance should be measured when running real applications, but they disagree on what applications and how to run them.

To solve this problem, several leading vendors have joined to develop a broad set of benchmarks covering AI training and inference. MLPerf started as a collaboration between academia and industry involving Google, Stanford, Harvard, and Baidu. The list of supporters has since grown to include data-center-processor vendors AMD, Intel, and Nvidia as well as startups such as Cerebras, Esperanto, SambaNova, and Wave; mobile-processor developers Huawei, MediaTek, Qualcomm, and Samsung; intellectual-property (IP) vendors Arm, Cadence, and Synopsys; and five of the “Super 7” cloud-service providers. Amazon, Apple, and Ceva are notably absent from the official supporters. The group recently completed its first benchmark suite, which measures the training of large networks; Google, Intel, and Nvidia posted initial results.

Addressing a different aspect of AI, researchers at ETH Zurich—a leading technical university—developed a set of tests to measure AI inference on smartphones. The generically named AI-Benchmark employs small neural networks and is available as an Android app, generating thousands of results. Many of the tests use Android’s Neural Network API (NNAPI, pronounced “nappy”) to take advantage of integrated accelerators for which the processor vendor supplies drivers. Results are available for all popular mobile processors except Apple’s, since the app doesn’t run on iOS.

Subscribers can view the full article in the Microprocessor Report.

Subscribe to the Microprocessor Report and always get the full story!

Purchase the full article

Free Newsletter

Linley Newsletter
Analysis of new developments in microprocessors and other semiconductor products
Subscribe to our Newsletter »


Linley Fall Processor Conference 2020
October 20-22 and 27-29, 2020 (All Times Pacific)
Register Now!
More Events »