When a self-driving car approaches an intersection, the onboard neural network receives a command such as "turn left." Yet upon receiving this command, the inference model still computes, uniformly, the channels for extracting straight-driving features, the channels for right turns, and the channels dedicated to braking decisions—all of them. This is simply how multi-task inference models are designed: one backbone is shared across all tasks, and the same computation repeats regardless of which task is actually active.
This "uniform execution across all channels" wastes energy and latency in edge inference. A research team consisting of Afzal Ahmad and Wei Zhang of the Hong Kong University of Science and Technology (HKUST), Gaoyu Mao, Shoubo Hu, Hui-Ling Zhen, and Mingxuan Yuan of Huawei Noah's Ark Lab, and Xinyu Chen of HKUST (Guangzhou), devised a method to eliminate this waste at its root. They used the task command itself as a signal for "which computations to skip," and designed hardware that physically refrains from executing masked-tile computations on an FPGA. The results were published on arXiv in July 2026 and accepted to the 59th IEEE/ACM International Symposium on Microarchitecture (MICRO 2026, October 31–November 4, 2026, Athens), a top-tier conference in computer architecture.
"Conditional Computation," Born 35 Years Ago, Has Long Been Blocked by a Hardware Wall
The idea of dynamically varying a neural network's computational load depending on the input dates back to 1991. Mixture of Experts (MoE), proposed by Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton, was a framework in which a gating mechanism selects expert networks based on the input, and non-selected experts are not computed at all. This lineage of "conditional computation" was applied to large language models in 2017 by Noam Shazeer and colleagues as the Sparsely-Gated MoE layer, and it is now a standard building block in foundation models such as GPT-4 and Gemini.
However, conventional dynamic network methods, including MoE, share a common limitation: even if software decides at the software level "which parts not to compute," there is no guarantee that the hardware can actually stop that computation. Take GPUs as an example. The NVIDIA RTX 4090 D has a peak performance of 1,321 TOPS in INT8 operations. Yet in batch-size-1 inference, the network operates at less than 1% of peak performance. The dominant factor in latency is not arithmetic computation but the overhead of kernel launches and memory transfers across 13 layers. Even if software judges "this channel is unnecessary," the GPU's CUDA cores still load all the weights, execute convolution across all channels, and simply multiply the unneeded results by zero. Rather than "skipping" the computation, it ends up being "wastefully executed and then discarded."
Even among FPGA-based accelerators, existing support for sparsity has been limited to static schemes. HPIPE from 2020 (Hall and Betz, FPGA'20) fixes sparsity from weight pruning at compile time, and cannot change what is skipped at runtime depending on the task. SCNN from 2017 (Parashar et al., ISCA'17) handles element-level sparsity, but index-matching logic and irregular memory access complicate the datapath. No design existed that used the task command as a signal to switch sparsity patterns at runtime.
Tile-Level "Instruction-Level Masking" That Halts Both Computation and Memory Transfer with a Single-Bit Check
The research team's approach consists of a three-stage co-design spanning the software/hardware boundary.
The first stage is a lightweight gating MLP that predicts a tile-level binary execution mask from the task command. This MLP's parameter count is less than 0.12% of the entire backbone. It runs only once before inference, and the resulting mask is reused for as long as the command remains active (spanning hundreds of frames in autonomous driving). Training follows a three-stage pipeline that gradually transitions from a soft mask to a hard binary mask.
The second stage is an extension of the instruction set architecture (ISA). Each instruction carries a per-tile bitmask field, and the hardware skips masked tiles without any software intervention. Control overhead is zero.
The third stage is the tile-based inference accelerator itself, implemented on an FPGA. It features an INT8 datapath, double-buffered memory, and configurable parallelism. The tile manager checks the least significant bit of a shift register before iterating on each output tile. If the bit is 1, it proceeds through the normal pipeline (DMA transfer, computation, accumulation); if the bit is 0, it advances to the next tile in a single cycle, issuing neither a DMA request nor any computation.
A structural feature of this design is that skipping propagates across layers. An output feature tile masked at layer $L$ no longer needs to be read as an input feature at layer $L+1$ either. A single mask thus reduces computation on both the output side and the input side simultaneously.
The Same Mask Backfires on GPUs but Delivers a 2.4x Speedup on FPGAs: A Behavioral Fork
Evaluation was conducted in closed-loop coordination with the CARLA autonomous driving simulator. The backbone is a 13-layer CNN supporting six types of driving commands (lane keeping, left turn, right turn, straight, lane change, and braking). The prototype was implemented on an AMD/Xilinx Alveo U50 FPGA, operating at 287 MHz.
Let's look at the numbers. Task-conditional sparsity reduced FLOPs by 66–76%. Inference latency dropped from 9.12 ms to 3.74–4.44 ms, corresponding to a 2.1–2.4x speedup. Energy per inference decreased from 263 mJ to 108–128 mJ. Throughput increased from 110 inferences/sec to 226–267 inferences/sec, and route completion rate remained 100% across all commands.
| Metric | Dense Execution (Baseline) | Task-Conditional Sparse Execution | Change |
|---|---|---|---|
| FLOPs | 100% | 24–34% | 66–76% reduction |
| Latency | 9.12 ms | 3.74–4.44 ms | 51–59% reduction (2.1–2.4x) |
| Energy/inference | 263 mJ | 108–128 mJ | 51–59% reduction |
| Throughput | 110 inf/s | 226–267 inf/s | 2.1–2.4x |
| Route completion rate | 100% | 100% | No change |
On the other hand, when the same model was run on an NVIDIA RTX 4090 D (1,321 TOPS in INT8, 425 W TDP), dense execution latency was 0.44 ms—21 times faster than the FPGA—but applying tile masking made it 22% slower, at 0.54 ms. This is because the mask merely multiplies by zero after the convolution, while the GPU continues to load all weights and keep all CUDA cores occupied. The research team went further and built a small dense network retaining only the masked channels to measure the best case on GPU, but even after removing roughly 75% of the convolutions, the speedup averaged only 1.03x.
What this contrast shows is that for tile-level sparsity to yield benefits, hardware must be able to simultaneously abandon both the fetch and the computation. On the FPGA, both the DMA transfer and computation for a masked tile are halted before they even launch. Since latency is proportional to the number of tile iterations, the mask translates directly into cycle reduction. On the GPU, batch-size-1 inference is bottlenecked by memory bandwidth and kernel launches, so reducing computation barely changes latency.
From an energy efficiency standpoint, the FPGA's advantage becomes even more pronounced. The FPGA, with a board power consumption of 28.8 W, consumes 108–128 mJ per inference under sparse execution. The RTX 4090 D, with a 425 W TDP, consumes approximately 187 mJ/inference under dense execution—46–73% more than the FPGA's sparse execution. In absolute speed, the GPU is overwhelmingly faster, but in power-constrained edge environments, the FPGA's sparse execution surpasses it in energy efficiency.
Why Hasn't a Design Where "Sparsity Changes by Task at Runtime" Existed Until Now?
The table below compares this research against existing FPGA-based inference accelerators from the perspective of sparsity.
| Design | Conference/Year | Sparsity Type | Granularity | Pattern Origin | Runtime Adaptation | HW Overhead | Datapath Change |
|---|---|---|---|---|---|---|---|
| Eyeriss (Chen et al.) | ISCA'16 | Activation zeros | Element | Input data | Per input | Gating logic | None |
| SCNN (Parashar et al.) | ISCA'17 | Weight + activation | Element | Training | None | Index matching | Yes |
| Cambricon-S (Zhou et al.) | MICRO'18 | Weight (coarse-grained) | Filter | Training | None | Index decoder | Yes |
| HPIPE (Hall and Betz) | FPGA'20 | Weight sparsity | Element | Pruning | None | Skip logic | Yes |
| HASS (Yu et al.) | FPL'24 | HW-aware sparsity search | Variable | Search | None | Variable | Yes |
| This work | MICRO'26 | Task-conditional tile | 16 channels/tile | Task command | Per task | 1-bit check | None |
In all existing designs, the sparsity pattern originates from either the input data or trained weights, and is determined either at compile time or per inference input. This is the first study to use the task command—a signal that is known before inference and fixed across hundreds of frames—to control sparsity. Moreover, it requires no changes to the datapath whatsoever, achieved solely through the instruction's bitmask field and a single-bit check on a shift register.
Extension to ViT-Based Backbones and Remaining Conditions
Beyond validation on CNN-based driving control models, the research team confirmed that the same gating mechanism and bitmask mechanism can be applied to tiles in MLP linear layers of a ViT-Base backbone as well. Masking at the granularity of attention head groups is mentioned as a complementary extension, but its implementation remains future work.
Two conditions must be met for this method to apply. First, there must exist a discrete, low-dimensional task descriptor known before inference—such as driving commands in autonomous vehicles or grasp types (pinch, scoop, push) in robotic manipulation. Second, execution must take place on an accelerator that uses fixed-size output channel groups as its scheduling unit. If these two conditions are not met—for example, if the task descriptor is continuous, or if execution is GPU-only—the benefits of this method cannot be realized.
The closed-loop evaluation on the CARLA simulator achieved 100% route completion, but no validation was performed in a real vehicle environment. Robustness against sensor noise, weather changes, and unexpected events remains unconfirmed. Furthermore, the evaluation used a relatively small number of six task commands, and how mask overlap and interference behave as the number of tasks increases has not been investigated. The paper also does not mention any fallback mechanism for cases where the gating MLP's prediction accuracy degrades.
The competition to reduce energy consumption in edge inference has been diversifying methods, moving from model compression (pruning, quantization, distillation) to dynamic networks (early exit, token pruning, MoE). "Sparse by Command" adds a new axis to this landscape: directly leveraging the task command—a signal available for free—at the hardware instruction level. Whether this axis influences GPU architecture design, or remains a benefit unique to FPGAs and ASICs, will be tested by the community's response following its presentation at MICRO 2026.