On October 28, 2026, developer Nolan Lawson published a comparison in which he used Claude Code to generate the same Pokémon Pokédex app in 12 different configurations, five times each. The median cost, converted to API pricing, ranged from $2.67 for Hono+htmx to $4.97 for Angular. The app, a Pokédex with search and filtering by type and generation, was used to compare the cost and time of generating a new app and the quality of the finished product. The dollar figures are the API-equivalent amounts shown by Claude Code. They are not additional charges paid by Lawson, who uses the $100-per-month Max plan. Costs differed by configuration, but spec compliance, display speed and ease of later modification are separate evaluation items. To use the results in technology selection, you also need to check the conditions for counting a build as complete and how display performance was measured.

AD

How much did costs differ when the same Pokédex was generated five times?

Lawson's published conditions were Claude Code with Opus 5.5 and the reasoning setting at high effort. This is an individual experiment that began with the question, "Where is the benchmark?" It is not peer-reviewed research. It repeatedly generated a small app under the same model conditions to see how results varied with the framework and other parts of the configuration.

The agent was told to use the latest stable version of each configuration, and official MCP servers and skills were provided where possible. Whether to render on the server or in the browser was not specified. The use of Tailwind, Bootstrap and similar libraries was prohibited. The agent could see the tests that checked spec compliance, but it was not told what the Lighthouse display-performance evaluation would measure.

Below are the costs Lawson reported for all 12 configurations. Figures are in US dollars and show the API-equivalent cost across five runs per configuration. For Angular as well, all five runs, including those that failed to work, are counted in the cost, time and token totals.

Configuration Median cost Min–max
Hono+htmx 2.67 2.23–3.19
Astro 2.76 2.16–3.63
Nuxt 2.82 2.39–4.25
Vanilla 2.98 2.24–3.37
Next.js 3.14 2.58–4.13
Qwik 3.27 2.35–3.80
TanStack Start 3.54 3.11–3.96
SolidStart 3.79 2.84–4.81
React Router 3.81 2.82–3.97
SvelteKit 4.28 3.59–4.65
Gracile+Lit 4.65 3.92–5.07
Angular 4.97 3.19–6.82

The gap between the cheapest median, Hono+htmx, and the most expensive, Angular, is $2.30, or about 1.86 times. However, costs varied from run to run, and the min–max ranges of the configurations overlap. This result alone cannot show whether the differences are statistically significant, but for this task, generation cost did differ by configuration.

On time, Astro had the shortest median at 8.2 minutes, followed by Hono+htmx at 8.6 minutes, Nuxt at 9.0 minutes and Angular at 17.2 minutes. Hono+htmx was cheapest and Astro was fastest. The cost and time rankings do not necessarily match, so it is hard to judge which configuration suits AI-driven development using a single metric.

What token counts and documentation lookups show

The median token count for Nuxt was 3.85 million, lower than Astro's 4.13 million and Hono+htmx's 4.26 million. Angular reached 9.76 million. Yet the lowest cost belonged to Hono+htmx, not Nuxt, so the ranking by total tokens does not match the ranking by API-equivalent cost.

These token counts include input read from cache, uncached input and output. Not all of them are newly generated text or code. Reading total tokens simply as "how much the AI wrote" would count the documentation and existing information the agent read while working as generated output.

In fact, in SvelteKit all five runs consulted the migration guide to version 3, and with Gracile the agent fetched the repository and inspected source code inside node_modules. Lawson speculates that the cause may be that the model's training data does not include enough of the newer specifications. However, this experiment did not isolate how much documentation lookups contributed to the cost differences.

Use of the support tools also varied. Nuxt's MCP was not used. Next.js's skill file was consulted in one of five runs and AGENTS.md in two of five. React Router's skill, by contrast, was used in all five runs. Providing tools the agent can use does not guarantee they will be used.

In its November 3 article, The Register suggested that standardized implementation steps, API stability and simplicity may matter. But in this comparison, the use of support tools was not uniform, and no controlled experiment tested how much using the tools reduced costs. What was measured was the whole process of the model consulting materials as needed, choosing implementation methods and completing the app.

AD

Handling dynamic APIs and the conditions for measuring display performance

As of the fixed commit in the published repository, the spec required data to be fetched over HTTP at runtime, and embedding data at build time or pre-generating it was prohibited. The rendering method was left to the agent. Whether HTML is assembled on the server (server-side rendering, SSR) or drawn by browser JavaScript (client-side rendering, CSR), the requirement to fetch data at runtime is the same.

To check spec compliance, the app is built against the normal API, and then tested against a switched API with some data changed and new Pokémon added. This verifies that the app still works correctly when the source data changes. Caching API responses in memory at runtime is allowed, however, so the app does not have to refetch all the data every time.

In two of five Angular runs, the generated SSR app rejected requests from anything other than localhost and did not meet the requirement to run behind a reverse proxy. In an addendum, Lawson explained that cost, time and token counts use all five runs, including failures, while quality and lines of code are aggregated only over the three runs that worked. So in the quality comparisons below, only Angular is measured on three apps, and this result cannot be read as a failure rate for Angular in general.

The README of the currently published repository describes a procedure in which only the agent-generated app/ directory is moved into a fresh container and rechecked with the original tests. Before Lighthouse measurement, the server-side cache is also warmed up. However, this describes the current procedure and is not a record fixing the execution conditions of every run performed on October 28.

According to the README, no artificial delay is set on the API, and communication from the server to the API runs at localhost speed. Network throttling is applied when the browser fetches from the API. Network conditions therefore differ depending on whether data is fetched on the server or in the browser. This experiment did not establish that SSR or caching is the cause of the cost differences, but it is an important measurement condition when comparing display performance.

The median homepage JavaScript transfer size Lawson reported was 0 KiB for Astro (min–max 0–1 KiB), 18 KiB for Hono+htmx, 84 KiB for Nuxt and 142 KiB for Next.js (136–143 KiB). Largest Contentful Paint (LCP), which measures how long it takes for the main content to appear, was 810 ms for Astro and 2,090 ms for Next.js. These are initial-display measurements and do not measure the time taken by operations after display, such as search and screen transitions.

Measurements used a 2025 Framework 13 running Ubuntu 24.04. Lighthouse was run on Chromium with 4x CPU throttling and mobile conditions equivalent to Slow 4G. Each app was measured five times to get a median, those values were aggregated over the five generated apps, and a median was taken again. For Angular's quality measurements, only the three apps that worked correctly were included.

Lighthouse Performance scores ranged from 97 to 100 and Accessibility scores were all 100, yet JavaScript transfer size and LCP still differed. Chrome's official documentation likewise says the Performance score is a weighted average of several metrics and recommends looking at the distribution of measurements, not just a single number. Results also change with device and network conditions, the element targeted by LCP and lazy loading of images, so both the overall score and the individual metrics need to be checked.

Evaluate generation cost, spec compliance, display performance and maintainability separately

To use these results in actual technology selection, you need to distinguish what each metric measured, beyond the ranking by generation cost. Cheap generation does not mean the app works to spec, displays quickly and is easy to change later.

The table below organizes the items to check additionally when adopting a stack, based on Lawson's results, the published spec, the current README and Chrome's explanation of the score. It is not a new measurement. It summarizes what each metric covers and its limits under the comparison conditions: the same Pokédex spec, Opus 5.5 at high effort and five generations each. Only Angular's quality and lines-of-code aggregates cover the three runs that worked.

Evaluation axis What this experiment shows Conditions to align and additional checks for selection
Generation cost Differences in API-equivalent cost and time to generate a new app to the same spec The same business problem, model conditions, number of repetitions, how support tools were actually used, and the number of runs counted in cost totals, including failures
Spec compliance Whether the app meets requirements for handling runtime data changes and running behind a reverse proxy Compare with the same tests and deployment conditions. Note also that cost totals including failures and quality evaluation of working apps only cover different numbers of runs
First display How quickly the homepage displays under the specified device and network conditions Align the network path to the API, cache state, and device and network conditions, and also measure interaction performance such as search and screen transitions
Maintainability proxy How many lines of source code (excluding blank lines) were generated Give the same change tasks and measure additional token consumption, human review effort and ongoing maintenance cost

On maintainability in particular, the proxy Lawson used was the number of source lines excluding blank lines. Medians were 1,370 lines for Hono+htmx, 1,415 for Nuxt, 1,461 for Astro and 2,009 for Angular. The count includes comments but excludes JSON, lock files, test code, generated artifacts and the like.

Lawson points out that fewer lines of code may also affect the number of tokens needed for future edits. However, what was measured here was generating an app from scratch, not repeatedly modifying existing code. The differences in line counts cannot directly show differences in the burden of human review or in maintenance cost.

Going forward, one could generate several apps closer to real work multiple times, record the cost of failed runs and the use of support tools, and then have both AI and humans perform the same change tasks, which would allow a comparison of cost differences in ongoing development. Whether the low cost at initial generation persists afterward, evaluated together with interaction performance and review effort, is the next question for putting these results to use in actual technology selection.