ONNX API now available in Windows

By Peter McGuiness

At the recent Windows Developer Day, Microsoft executive Kevin Gallo announced a number of new products for developers, the most interesting of which was what he called “the intelligent edge” and the Windows AI Platform. In a nutshell, this means that while Microsoft has previously focused mainly on cloud-based AI, it is now turning its attention to enabling the use of machine learning at the edge. The platform provides a single workflow across all of Microsoft’s developer products, including Azure and Windows.

For the Windows developer, Windows has a new ML architecture that makes it easy to add neural network-based intelligence into Windows apps. Accessible from both native and managed code, the API gives access to an inference engine that runs on whatever compute resources are available to it, including any hardware acceleration that is available. As Gallo’s colleague Kam VedBrat pointed out, machine learning is already used extensively throughout the windows operating system for handwriting and voice recognition, as well as behind the scenes for threat detection. This new API delivers the same type of functionality into the hands of developers so that they, too, can take advantage of Microsoft’s machine learning expertise.

VedBrat called out the fact that DX12 compute shaders are fully integrated so that inference is already accelerated on the Intel GPU on the surface Pro and the Nvidia GPU on the Surface book. (As an aside, Intel simultaneously announced that machines equipped with their Movidius MyriadX accelerator will have the option of using that hardware as well.)

The API is available now, it accepts trained networks in the new ONNX transfer format, developed jointly with Facebook and others, which gives access to a fairly broad range of frameworks. In the next update, Microsoft will supply a library of its own pre-trained networks that developers can simply pick up and use off-the-shelf.

Acknowledging the importance of visual applications in this field, VedBrat also pointed out that the camera pipeline through DirectX has been optimized to reduce latency and power-hungry memory copies.

Why is this important?

Machine learning is hard and there are multiple barriers for developers who are not specialists to make use of it in their apps so any initiative that helps to break down those barriers is a good thing. The example app given is also significant: a visual classification app that can find manufacturing defects in a printed circuit board does not sound sexy, but it establishes that Microsoft’s priority lies in exploiting machine learning in enterprise settings. No fluff at Microsoft these days: it’s all business so we can expect to see these apps cropping up to augment or replace intellectual as well as manual skills in the workplace.

Creating and training networks from scratch is a costly, time consuming exercise that depends for success on the availability of huge amounts of high quality data. This is the reason that the big cloud companies like Google, Baidu, and Facebook are so prominent in the field: they have the data. The important move by Microsoft is their statement of intent to provide trained networks to their developers. They are one of the companies with data and, like Google and all the rest, their future prosperity depends on leveraging its value. This is one of the ways they intend to do that and in the process stimulate a new wave of automation that will be felt far beyond the factory floor.

What do we think?

Windows ML will dynamically determine the most suitable hardware for any given AI workload and intelligently distribute across multiple hardware types—now including the Intel Movidius Myriad X VPU, a dedicated neural compute engine for hardware acceleration of deep learning inference at the edge.

The WinML and Intel VPU combination has the potential to enable more intelligent client applications and core OS features like personal assistants, enhanced biometric security, smart music and photo search and recognition.

It has been speculated that Intel would incorporate its Movidius chip into Windows platforms as a result of the Microsoft AI extensions. I think the only practical route into a laptop is via a camera module but that’s a few generations away. For desktops and enterprise, who cares? There’s the GPU and it’s plugged in. That’s where Microsoft is focused: they couldn’t care less about drones—that’s a purely Intel obsession.