Engineers at Stanford University have created a more efficient and flexible AI chip that could bring the power of AI into tiny edge devices.

AI-powered edge computing is already pervasive. Devices such as drones, smart wearables and industrial IoT sensors are equipped with AI-enabled chips so computing can occur at the edge where the data originate. This allows real-time processing and guarantees data privacy.

However, AI functionalities on these tiny edge devices are limited by the energy provided by a battery. Therefore, improving energy efficiency is crucial. In today’s AI chips, data processing and data storage happen at separate places – a compute unit and a memory unit. The frequent data movement between these units consumes most of the energy during AI processing, so reducing the data movement is the key to addressing the energy issue.

Stanford University engineers have come up with a potential answer: a resistive random-access memory (RRAM) chip that does the AI processing within the memory itself, thereby eliminating the separation between the compute and memory units. Their compute-in-memory (CiM) chip, called NeuRRAM, is about the size of a fingertip and does more work with limited battery power than what current chips can do.

“Having those calculations done on the chip instead of sending information to and from the cloud could enable faster, more secure, cheaper and more scalable AI going into the future, and give more people access to AI power,” said Philip Wong, a professor in the university’s School of Engineering.

Weier Wan, a recent graduate at Stanford leading this project, added: “The data movement issue is similar to spending eight hours in commute for a two-hour workday. With our chip, we are showing technology to tackle this challenge.”

They presented NeuRRAM in a recent article in the journal Nature. While compute-in-memory has been around for decades, this chip is the first to demonstrate a broad range of AI applications on hardware, rather than through simulation alone.

The memory technology that NeuRRAM used is resistive random-access memory (RRAM). It is a type of non-volatile memory – memory that retains data even once power is off – that has emerged in commercial products.

RRAM can store large AI models in a small area footprint, and consume very little power, making them perfect for small-size and low-power edge devices.

Even though the concept of CiM chips is established, and the idea of implementing AI computing in RRAM isn’t new, “this is one of the first instances to integrate a lot of memory right onto the neural network chip and present all benchmark results through hardware measurements”, said Wong, who is a co-senior author of the paper.

The architecture of NeuRRAM lets the chip perform analogue in-memory computation at low power and in a compact footprint. It was designed in collaboration with the lab of Gert Cauwenberghs at the University of California, San Diego, who pioneered low-power neuromorphic hardware design.

The architecture also enables reconfigurability in dataflow directions, supports various AI workload mapping strategies, and can work with different AI algorithms, all without sacrificing AI computation accuracy.

To show the accuracy of NeuRRAM’s AI, the team tested how it functioned on different tasks. They found it was 99% accurate in letter recognition from the MNIST dataset, 85.7% accurate on image classification from the CIFAR-10 dataset, 84.7% accurate on Google speech command recognition, and showed a 70% reduction in image-reconstruction error on a Bayesian image recovery task.

“Efficiency, versatility and accuracy are all important aspects for broader adoption of the technology,” said Wan. “But to realise them all at once is not simple. Co-optimising the full stack from hardware to software is the key.”

Wong added: “Such full-stack co-design is made possible with an international team of researchers with diverse expertise.”

Right now, NeuRRAM is a physical proof-of-concept but needs more development before it is ready to be translated into actual edge devices. But this combined efficiency, accuracy and ability to do different tasks showcases the chip’s potential.

“Maybe today it is used to do simple AI tasks such as keyword spotting or human detection, but tomorrow it could enable a whole different user experience,” said Wan. “Imagine real-time video analytics combined with speech recognition all within a tiny device. To realise this, we need to continue improving the design and scaling RRAM to more advanced technology nodes.”

Priyanka Raina, assistant professor of electrical engineering and a co-author of the paper, added: “This work opens up several avenues of future research on RRAM device engineering, and programming models and neural network design for compute-in-memory, to make this technology scalable and usable by software developers.”

If successful, RRAM CiM chips such as NeuRRAM could be embedded in crop fields to do real-time AI calculations for adjusting irrigation systems to current soil conditions. Or they could turn augmented reality glasses from clunky headsets with limited functionality to something more akin to Tony Stark’s viewscreen in the Iron Man and Avengers movies, without intergalactic or multiverse threats.

If mass produced, these chips would be cheap, adaptable, and low power enough they could be used to advance technologies already improving lives, said Wong, such as in medical devices that allow home health monitoring. AI-enabled sensors could also play a role in tracking and addressing climate change.

“By having these kinds of smart electronics that can be placed almost anywhere, you can monitor the changing world and be part of the solution,” Wong said. “These chips could be used to solve all kinds of problems from climate change to food security.”

Additional co-authors include researchers from University of California San Diego (co-lead), Tsinghua University, University of Notre Dame, and University of Pittsburgh. Former Stanford graduate student Sukru Burc Eryilmaz is also a co-author.

The research was funded by the National Science Foundation Expeditions in Computing, SRC Jump Ascent Center, Stanford SystemX Alliance, Stanford NMTRI, Beijing Innovation Center for Future Chips, National Natural Science Foundation of China, and the Office of Naval Research.