» Current | 2020 | 2019 | 2018 | Subscribe

Linley Newsletter

MLPerf Debuts Inference Benchmarks

November 26, 2019

Author: Bob Wheeler

Start with an unbounded problem and you’ll never find a solution. Such was the MLPerf group’s quandary in developing machine-learning-inference benchmarks. Pragmatism won out, and the group released a modest set of five benchmarks, focusing heavily on vision processing. It also defined client, edge, and cloud use cases, however, for 20 total combinations. It included tightly controlled and user-defined implementations as well, trading reproducibility against innovation.

Among the first results are submissions from 14 organizations comprising cloud-service providers (CSPs), server vendors, chip vendors, and professional-services companies. Smartphone OEMs are notably absent, but AI-Benchmark also serves that segment. The MLPerf submissions include Alibaba, Google, Habana, Intel, and Nvidia data-center accelerators; Intel Core and Xeon processors; and Nvidia and Qualcomm client/edge SoCs. The newest designs stood out in vision-processing performance, but incumbents demonstrated broader support across neural-networks types.

MLPerf Inference v0.5 carries over a problem we noted with the group’s Training v0.5 benchmark: it doesn’t require submissions to include minimum sets of benchmarks. As a consequence, the initial results form a sparse matrix that limits direct product comparisons. Still, the benchmark should enable more-reproducible performance testing and more-verifiable vendor claims.

Subscribers can view the full article in the Microprocessor Report.

Subscribe to the Microprocessor Report and always get the full story!

Purchase the full article


Linley Spring Processor Conference 2020
Coming April 7-8, 2020
Hyatt Regency, Santa Clara, CA
Register Now!
Linley Fall Processor Conference 2020
Coming October 28-29, 2020
Hyatt Regency, Santa Clara, CA
More Events »


Linley Newsletter
Analysis of new developments in microprocessors and other semiconductor products
Subscribe to our Newsletter »