Editor | 2049

In the era of rapid artificial intelligence development, computational efficiency has become a key bottleneck restricting further progress. Neuromorphic computing, with its innovative architecture inspired by the biological brain, demonstrates significant potential in terms of energy efficiency and real-time processing. However, the long-standing lack of unified evaluation benchmarks in this field has made it difficult to objectively measure and effectively compare technological innovations.

Recently, a research team led by Harvard University and comprising researchers from multiple top universities, research institutions, and companies worldwide proposed NeuroBench, the first unified benchmarking framework for neuromorphic computing. By employing a dual-track parallel evaluation approach covering both algorithms and systems, this framework provides a common assessment standard for the development of neuromorphic computing technologies.

NeuroBench is a neuromorphic computing benchmark framework initiative launched through international multi-party collaboration, aiming to establish unified evaluation standards for emerging neuromorphic computing technologies. The project involves the joint participation of more than 60 institutions from industry and academia, including leading technology companies and research organizations such as Intel, SynSense, and Harvard University.

This study uses「The neurobench framework for benchmarking neuromorphic computing algorithms and systems」published in Nature Communications on February 11, 2025.

The field of neuromorphic computing research currently faces three major challenges:

First, there is a lack of a formal definition, with significant variations in the understanding and implementation of “neuromorphic” across different approaches. Early research primarily focused on simulating the physical properties of biological neurons on silicon-based hardware, whereas current research has expanded to encompass a broader range of biomimetic computational methods, including innovations at the algorithmic, hardware, and system levels.

Second, the diversification of tools has led to low portability and standardization, as various frameworks are optimized for different objectives. While some tools focus on neuroscience simulations, others are dedicated to automated SNN training. Although this diversity fosters the exploration of different methods, it also increases the difficulty of evaluation.

Third, the rapid evolution of research progress necessitates that evaluation systems be updated in sync with technological advancements. These challenges make it difficult to objectively measure technological innovation, thereby constraining the healthy development of the field.

To address these challenges, the NeuroBench framework introduces systematic innovations by adopting a dual-track evaluation architecture that runs algorithmic and system assessments in parallel.

Algorithm Evaluation Track

The algorithm evaluation track primarily assesses hardware-agnostic algorithm performance, encompassing four benchmark tasks: Few-Shot Class-Incremental Learning (FSCIL), event camera-based object detection, motor cortex decoding, and chaotic function prediction.

Each task is designed with multi-dimensional evaluation metrics, including accuracy, parameter count, connection sparsity, and activation sparsity.

For example, in the FSCIL task, the M5 ANN baseline and the SNN baseline achieved accuracies of 97.09% and 93.48%, respectively, on 100 base classes. During the incremental learning phase, the prototypical network method enabled the ANN and SNN to achieve average accuracies of 89.27% and 75.27%, respectively. In terms of system scale, the SNN baseline had a parameter size of 1.36 × 10^7 bytes and an execution rate of 200 Hz, whereas the ANN baseline had a parameter size of 6.03 × 10^6 bytes and an execution rate of 1 Hz.

The system evaluation track focuses on the actual performance of fully deployed systems.

The acoustic scene classification task requires the system to perform real-time classification of 1-second audio samples from the DCASE 2020 dataset. The system implemented by the research team based on the Xylo neuromorphic chip employs a feedforward Spiking Neural Network (SNN) with multiple synaptic time constants.

Compared to the Arduino baseline, the Xylo system demonstrates significant energy efficiency advantages while achieving comparable classification accuracy (Xylo: 79.9% vs. Arduino: 79.6%). Dynamic power consumption decreases from 21.32 mW on Arduino to 0.341 mW on Xylo, and dynamic energy consumption per inference drops from 0.934 mJ to 0.028 mJ.

In QUBO optimization tasks, the research team compared the parallel simulated annealing algorithm implemented on the Loihi 2 neuromorphic chip with traditional algorithms on CPUs.

The results show that under tight time constraints (within 10^-2 seconds), Loihi 2 can handle problem sizes four times larger than those handled by CPUs. More importantly, the power consumption of Loihi 2 is only 1/37th of that of the best CPU solver, demonstrating a significant energy efficiency advantage. This result proves the potential of neuromorphic computing in combinatorial optimization problems.

To ensure the objectivity and reproducibility of the evaluation, NeuroBench has standardized the evaluation process and metrics.

Algorithm evaluation requires providing a complete complexity analysis, including static metrics such as parameter count, connection sparsity, and activation sparsity, as well as dynamic metrics based on workloads.

System evaluation emphasizes end-to-end performance, requiring measurements of execution time and power consumption across the complete processing pipeline, including data preprocessing. All benchmark tasks and evaluation tools are openly hosted with comprehensive documentation to ensure experimental reproducibility.

Figure: Performance comparison results of QUBO optimization tasks. (Source: Paper)

The launch of NeuroBench marks a new phase in the evaluation of neuromorphic computing. By employing a dual-track parallel evaluation framework for both algorithms and systems, it supports foundational research exploration while driving the development of practical systems.

The framework will continue to expand, with plans to incorporate new tasks such as closed-loop benchmarking to more comprehensively evaluate the adaptability and robustness of neuromorphic systems in real-world scenarios.

Source: ScienceAI

For academic sharing only. If there is any infringement, please leave a message, and the infringing content will be removed immediately!

   Join the Community   

Welcome to the Brain-Computer Interface Community Group,

Exploring Topics in the Field of Brain-Computer Interfaces: Real-Time Tracking of Frontiers in Brain-Computer Interface Technology.

Join the WeChat group:

Add WeChat: RoseBCI [Note: Name + Industry/Profession].

Join QQ Group: 913607986

   Call for Papers   

1. Submissions are welcome. For submission inquiries, please contact us via WeChat: RoseBCI

Click to Submit:Submission Guidelines for Academic News in the Brain-Computer Interface Community

2. To join the community as a part-time creator, please contact us on WeChat: RoseBCI

One-click Triple Action: “Share,” “Like,” and “Mark as Read”

Stay Ahead of Every Breakthrough in Brain-Computer Interface Technology