Machine learning is becoming an increasingly important workload for embedded systems, but its requirements on hardware are not well-understood. This keynote presentation will discuss what product teams have been asking for, and how hardware can help them build new features, and new applications. It will cover topics like cascaded networks to help duty-cycle high-accuracy, high-power models by gatekeeping them with smaller models on low-power hardware, how accelerators can be useful, and thinking about the whole system design when optimizing solutions.

After Centaur introduced the industry’s first integrated AI Coprocessor for x86 systems, continued software advancements have more fully unlocked the heterogeneous SoC’s capabilities. Adoption of MLIR has enhanced the software and compiler infrastructure, and creation of multiple levels of intermediate representations has enabled more advanced optimizations than were present in our previous system. These advancements are likewise advantageous for other developers that turn to MLIR in hopes of finding a "Holy Grail" compiler technology for heterogeneous SoC solutions.

The dataflow flexibility of spatial architectures allows for the possibility of a higher "sustained to peak" TOPs/TFLOPs ratio. In this presentation we will describe the architecture of the Stratix 10 NX family, AI optimized FPGAs which contain a matrix of almost 4,000 AI tensor blocks, providing 130 TOPs/TFLOPs INT8/FP16 or 260 TOPs/TFLOPs INT4/FP12 peak performance. Large on board HBM stacks enable single node capability, and multiple banks of high-speed transceivers allow distributed or unrolled algorithms across the datacenter.

Training deep-learning models requires large compute resources and is performed in the cloud or data centers. Yet the data is generated at the edge, and transporting it to the cloud leads to unsustainable network bandwidth, high cost, slow responsiveness and compromises data privacy. This presentation will disclose Deep-AI's accelerated deep-learning solution for the edge, which implements training at fixed-point INT8 coupled with high sparsity, to enable deep learning at a fraction of the cost and power of GPU systems.