On November 18, Architect Labs announced research results for Frostbite, a CPU in which AI co-designed everything from the compiler down to the circuits. In the company's pre-silicon evaluation, performance per watt across 25 tasks was a geometric mean of 4.59 times that of an existing CPU of similar area. Frostbite finds work within a program that can proceed at the same time and hands it to idle processing units. Efficiency also improved on the 15 tasks that were not used to tune the design, but the size of the gain differed widely between convolution and Fibonacci computation.

AD

What is inside the 4.59x average

The unit of comparison for Frostbite is one quad, a 2×2 arrangement of tiles (the processing units), combined with four L3 cache nodes. Its area is 0.792 mm². The comparison target, "TCPU," is a single core of an in-order CPU that can issue two instructions at once, with an area of 0.895 mm². According to the evaluation conditions in the official announcement, both use the same manufacturing process. TCPU has a manufacturing track record, but its model name is withheld for legal reasons.

Frostbite is designed for a 7 nm process. The assumed operating frequency at logic synthesis was 800 MHz, and timing evaluation of the final manufacturing layout, known as GDS, gave 500 MHz. The layout diagram in the announcement shows 16 tiles in a 4×4 arrangement, but the main performance comparison uses 4 tiles, a portion of that. The figures are the company's pre-silicon design evaluations, and should be read as distinct from results measured on a physical chip running.

Performance per watt is the amount of work that can be processed in a given time divided by average power. If the same amount of work is finished, it corresponds to the amount of work divided by the energy consumed. The 4.59x figure is therefore a comparison of processing efficiency, not a claim that every workload finishes 4.59 times faster.

The company split the 25 tasks into 10 used to tune the design and 15 held back for evaluation. The geometric mean of performance per watt was 6.02x for the former and 3.83x for the latter, giving 4.59x overall. Of the 15 tasks, 5, including Kalman filter and principal component analysis, were added to the evaluation group later. Because the two groups cover different computations, the gap between the averages alone cannot tell us how far the design is biased toward the tasks it was tuned on.

Sorting the 15 tasks not used for tuning by performance-per-watt ratio, the median is radix sort at 3.96x. We identified the evaluation group from Figure 3 of Architect Labs' November 18 announcement and took the 8th of the 15 light-colored bars in Figure 11, sorted in descending order.

Evaluation task Performance-per-watt ratio
Convolution 23.12x
Longest common subsequence 6.35x
k-d tree 6.19x
Kalman filter 5.87x
Principal component analysis 5.31x
Huffman decoding 4.93x
Finite element analysis 4.50x
Radix sort 3.96x
Random forest 3.07x
LU solve 2.46x
LZ4 decompression 2.44x
emeans 2.32x
Breadth-first search 2.20x
Quicksort 2.09x
Fibonacci 1.15x

Values are relative to TCPU's performance per watt, set to 1. Source: Architect Labs, Figures 3 and 11.

The range of efficiency gains is wide, from 23.12x for convolution, the largest, to 1.15x for Fibonacci, the smallest. The median of 3.96x is a number for reading the center of the distribution without relying on the highest value; it is not a prediction for typical applications. The inputs are also specific: convolution uses a 64×64 input with 2 input and 2 output channels and a 3×3 kernel, radix sort uses 16,384 elements, and Fibonacci uses n=20. FFT, radix sort, and LU are based on SPLASH-2 implementations, but the company explains that it scaled them down to 1/64 of their original size for design exploration.

Representing computation as work that can proceed at once

Frostbite uses "interaction nets," which represent computation as a collection of nodes and connections. Each node has one principal port, and two nodes whose principal ports are connected are rewritten locally according to rules. Pairs that can be executed independently do not share nodes, so their rewrites can proceed at the same time.

The relationship of which value is used where is also made explicit in the net's connections. Frostbite converts programs written in the experimental language Vine into "Ivy," an intermediate representation that preserves the net's structure. Unlike a design that simply makes units execute the same instruction in lockstep, the places where work can be executed change depending on inputs and branch outcomes.

Figure 1 of the announcement takes as its example a computation that places 5 queens on a 5×5 board and searches for arrangements in which none attack each other. The total number of rewrites is 3,183, but the longest path following dependencies is 262 steps. Dividing the total amount of work by this length gives about 12.1. That is a guide to the ideal parallelism implied by dependencies alone, compared with processing everything sequentially.

In practice, however, tiles also need time to hand off work and access memory. The figure of about 12.1 is not a predicted speedup for execution on 4 tiles. Note also that this 5-queens example is for explaining the mechanism; the N-queens benchmark used for design tuning has n=10.

Interaction nets themselves are a computation model that Yves Lafont presented at POPL in 1990. With Frostbite, how to connect this model to an instruction set and circuits became the subject of the design. Even if the parts that can run in parallel can be expressed, a CPU will not be highly efficient unless that work can be distributed well across a limited number of processing units.

AD

How idle tiles pick up work

Each tile prioritizes its own work and, when that runs out, takes work from neighboring tiles. If there is still no work, it rests. Rather than allocating all work from a single central point, each tile acts according to how busy or idle things are.

In the execution examples in Figures 8 and 9, when a computation branches, the processing to resume after the branches complete is saved in a local record called a "frame." Newly generated work is pushed onto a hardware queue, and the tile handles the most recent work itself. If another tile takes on a piece of work, the completion notification is sent back to the original tile. When the count of outstanding work reaches zero, the saved continuation is started.

"Work stealing," in which an idle processor takes work from another, has a long body of research behind it. The 1999 original paper by Robert D. Blumofe and Charles E. Leiserson analyzed execution time and communication volume for computations such as those where the relationship of returning a child computation's result to its parent is constrained. What AI designed in Frostbite are the queues for handing off work and the circuits that process completion notifications from distant tiles.

Bringing AI design proposals closer to circuit-level estimates

The human researchers first chose the tasks and evaluation conditions from categories such as parallel, sequential, and branching computations. The AI then read documents and existing benchmarks and modeled multiple design candidates. Whereas the company's previous work Redwood implemented and verified an inference accelerator with a default configuration, with Frostbite it says it explored the compiler, instruction set, and computation mechanism together, covering the RTL (the circuit description) and the final layout.

A physically viable design was not obtained from the start. An early 141-page design proposal contained unrealistic estimates and was discarded after the first circuit implementation. Even a configuration that looks promising on a computation model changes in the energy it needs once wiring and memory behavior are estimated as a circuit.

To verify the design's behavior, the company wrote the behavior each process must satisfy in its own specification language, "ASL." This was checked against the model "Arcmodel" and the RTL, and also converted to the Lean theorem prover. Energy estimation, by contrast, was calibrated separately. The energy estimates of the initial model and the RTL differed by up to about 250%.

The company therefore aligned the model's estimates with the circuit side using activity counts such as SRAM reads and writes and communication. The estimation gaps shown in Figure 5 came down to roughly 1–5% after calibration. What came closer here are the estimates from two design stages. This is not a figure showing the error against power consumption measured after manufacturing.

Once the circuit burden could be estimated this way, 37 of 58 candidates were adopted over four rounds of exploration and a final ablation experiment. They include clock gating, which stops the clock supply to circuits that are not operating, and circuit sharing. Verifying behavioral contracts, and confirming power and speed in an implementation that includes wiring, were each advanced in their own stages.

AD

The circuits that distribute work also need energy

In the energy breakdown in Figure 12, the share taken by instruction fetch, decode, and issue falls from 27% on TCPU to 6% on the quad. Memory goes from 57% to 38%. On the other hand, the internal network and work distribution account for 26% on the quad. The mechanism for distributing computation itself also requires a corresponding amount of energy.

These are averages over 20 tasks, with the energy for clock distribution excluded from each chip's breakdown. The energy used to deliver the clock signal to the parts of the chip accounts for 49% of the total on TCPU and 41% on the quad. The set of tasks aggregated and the denominator differ from those of the performance-per-watt figure, which uses 25 tasks, and the decline in a share does not let us calculate how much the absolute consumption of each part fell. With 16 tiles, the network and work distribution accounted for 23% and memory for 40%.

Frostbite's results show a design that reduces the burden on circuits that handle instructions while spending energy on the circuits and memory that distribute work. However, what caused the difference in ratios between convolution and Fibonacci cannot be pinned down from this average breakdown. Efficiency by task and energy by component need to be evaluated separately.

The company plans next to move on to comparisons with out-of-order CPUs, which reorder instruction execution, and with multicore configurations, and to examine SPEC, PARSEC, and others as well. It also plans to prepare an environment that can be tried on FPGAs and an early OS, and to release the RTL and instruction set. If reproduction on a physical chip and comparisons across larger and broader workloads follow, it will be easier to judge for which uses a design that builds in work distribution can save energy.