DEV Community: Grain64

Cheapest per token is not cheapest per finished job: what 2,400 runs show

Grain64 — Thu, 05 Nov 2026 00:47:31 +0000

A model with a 33% pass rate turned out to be the cheapest way to finish a task in a recent 10-model benchmark. The same benchmark shows why that result is less of a free lunch than it looks. Here is the metric behind it and how to compute it on your own prompts.

The metric

Cost per successful task = everything the model spent across every attempt (failed, retried, timed out) divided by the number of attempts that passed.

Per-token price cannot see retries, malformed outputs or runs that never finish. This metric sees all of them.

The numbers

Arize and Fireworks ran 10 models on 40 real command-line tasks from Terminal-Bench, 6 trials each, 2,400 runs. The graded outcome is the task's own test script. A selection:

Model	Pass rate	Mean $ per attempt	$ per successful task	Attempts per success
gpt-oss-120b	33%	$0.018	$0.054	3.0
gemini-3.1-flash-lite	40%	$0.026	$0.063	2.5
GPT-5.5	67%	$0.424	$0.636	1.5
Kimi K3	66%	$0.441	$0.670	1.5
Claude Sonnet 5	49%	$0.494	$1.014	2.1
gemini-3.5-flash	23%	$0.288	$1.233	4.3

Source: Arize and Fireworks, cost per successful task.

Three things to read from it:

Retries are a tax you pay in dollars. gpt-oss-120b needs about three attempts per success, and is still roughly 12 times cheaper per finished task than GPT-5.5 ($0.054 against $0.636).
A cheap model is not always a cheap result. gemini-3.5-flash costs $0.288 an attempt but passes 23% of the time: $1.233 per success, more than twenty times gpt-oss-120b's.
Coverage is the catch. Counting only tasks a model solved reliably (4 of 6 trials or better), gpt-oss-120b covers 8 of 40, GPT-5.5 covers 25 and Kimi K3 covers 26. Part of the cheap model's advantage is that it only wins the tasks it can win. Swap it in wholesale and the bill and the feature set both shrink.

By difficulty, price tier did not predict who wins the easy tasks: GPT-5.5 passed 69% of them, Kimi K3 (also frontier-priced) 100%, and gpt-oss-120b 65% at roughly a twenty-fourth of the cost per attempt. On hard tasks the gap was large (GPT-5.5 51% against 14% for gpt-oss-120b). With 6 trials per cell, the source puts the pass rates at about plus or minus 6 points, so read the difficulty split as a direction, not a decimal.

What does and does not transfer

These are agentic terminal tasks, not classification or extraction, so do not read the dollar figures as your numbers. The method transfers.

Run it on your own prompts (an afternoon)

Take 100 real inputs and the output you would accept for each. Real traffic, not a demo set.
Write the pass rule before you run anything: exact match, a JSON schema plus field checks, or a rubric. Set the bar (say 90%).
Run your current model and two or three cheaper candidates. Record the cost of every call from the provider's usage field, including calls that were retried or rejected by your validator.
For each model compute: passes / total, total spend / passes, and the share of calls that needed a retry.
Throw out any candidate under your bar. Of the rest, the lowest cost per success wins. If that is your current model, you have learned that for free.
Re-run when a price drops. A cheaper candidate you rejected last quarter may clear the bar at a lower price now, and prices move monthly.

The headline of that benchmark is the one worth remembering: the label (open or closed, frontier or budget) was the wrong axis. Cost per successful task, on your task, was the right one.

Umami self-hosting problems: 8 reports from 14 public threads, and the migration failure that keeps coming back

Grain64 — Wed, 04 Nov 2026 03:32:41 +0000

Umami self-hosting problems, counted. Of the 66 items Friction holds about Umami (all dates, from GitHub issues and Hacker News; the reports, plus those already fixed upstream), 8 are about self-hosting and upgrades, resting on 14 separate public threads. Six of those 14 threads are one failure: a Prisma migration that breaks an upgrade, reported from August 2022 to May 2025.

The migration that fails, six times in three years

Six GitHub issues, one problem: a database migration fails or leaves the schema wrong when you upgrade.

2022-08, #1393: "The migration 01_init is already recorded as applied in the database". Closed as completed, 2022-08.
2022-08, #1406: "The update to 1.37.0 did not work out however". Closed as completed, 2022-08.
2022-10, #1604: "The column account.account_uuid does not exist in the current database after upgrading to 1.39.2". Closed as completed, 2022-12.
2023-08, #2206: "I updated the docker image to the latest version and got a prisma migrate deploy command error after container boot". Closed as completed, 2024-02.
2024-04, #2645: "A migration failed to apply. New migrations cannot be applied before the error is recovered from". Closed as completed, 2025-01.
2025-05, #3408: "I tried to go back to V2.17 and I have the same problem". Closed as not planned, 2025-08.

Five were closed as completed, the quickest within days, the slowest after about nine months; the newest, a Docker upgrade from 2.17 to 2.18 that ends in Prisma error P3009 (a failed migration in the target database), was closed as not planned. If you self-host, this is the failure to expect around a version boundary: back up the database before you pull a new image, and read the open and recently closed issues for the version you are moving to.

Two Heroku deploy failures, different causes

Two issues, three and a half years apart, both Heroku deploys that did not come up:

2022-04, #1071: "jsxDEV is not a function error" after updating to 1.29.2 and deploying on Heroku. Closed as completed within five weeks, 2022-05.
2025-11, #3687: "When deploying umami 3.0.0 to Heroku, the build fails at the check-db step even though the database connection works". Closed as not planned, 2026-05.

The rest of the self-hosting list

Six more reports, one thread each. Three are operational failures: a Docker container that restarts repeatedly after an update, a Podman healthcheck that cannot reach Umami on localhost, and a production build that runs out of memory on shared hosting (2022). One is a request, for a Docker image variant with a base path for subpath hosting (marked resolved, 2026-03).

Two come from Hacker News and are about where Umami is going rather than how it breaks:

2025-12, on the website's cloud-first pivot: "now you can't even find what you can self-host it on the front page!" (thread)
2026-01, from a homelab user: "I can't self host the blog's analytics service (umami.is) because obviously a home server not exposed online can't receive queries from the internet" (thread)

What people say when they leave

Self-hosting is the most-reported area among Umami's 29 recent reports (6). Three reports describe a move away, and 4 users named where they went: Plausible (2), PikaPods (1) and Hotjar (1). A count is the number of reports naming a tool, not a market share; the list, with sources, is on the Umami alternatives page.

How to read these numbers

A "report" is one problem, grouped from one or more public threads; a "source thread" is one issue or comment thread. The six migration issues and the Heroku issues above link to GitHub, and the two Hacker News comments link to their threads; the excerpts are copied word for word. The set is GitHub issues (78 threads) and Hacker News (9 threads) for Umami, 87 in all, so it shows what people filed and said in public, not every Umami user. The full list, with each report's sources, is on Umami's page, and what is stored and how to dispute a report is on the data page.

Planning to move to Umami, or building something against it? Say which product you want counted next at https://friction.grain64.com/request/?utm_source=devto.

Build a Stripe cancel flow: cancel_at_period_end and a webhook handler that doesn't double-count

Grain64 — Wed, 04 Nov 2026 00:09:22 +0000

A cancel flow on Stripe is three pieces: a place that catches the cancel click, an offer step, and a webhook that tells your app whether the cancellation was scheduled, stopped or finished. Set cancel_at_period_end to true so the subscriber keeps what they paid for, and revoke access only when customer.subscription.deleted arrives.

Cancel at period end, not immediately

By default, cancelling a subscription through the API takes effect immediately and no more invoices are generated. To let the subscriber keep the period they already paid for, update the subscription with cancel_at_period_end=true.

Schedule it: set cancel_at_period_end to true. Stripe sends customer.subscription.updated.
Offer something (optional): the subscriber is still active, so there is time to show an offer and apply it.
Stop it: set cancel_at_period_end to false any time before the period ends. Stripe sends customer.subscription.updated again. You cannot reactivate a subscription that has already been cancelled.
Let it end: when the period ends Stripe cancels the subscription and sends customer.subscription.deleted. That is the event that tells you to revoke access.

Stripe recommends using the subscription status, rather than the flag, as the trigger for starting and stopping service.

The webhook handler

This Node handler verifies the signature, ignores repeats and reacts to the three situations above. It takes the raw request body, the Stripe-Signature header and a small store you provide (a database table is enough). Pass the exact bytes Stripe sent: a body that has been parsed and re-serialised fails the signature check.

import Stripe from "stripe";

// Handles the three Stripe events a cancel flow cares about.
// rawBody must be the exact bytes Stripe sent (string or Buffer), not re-serialised JSON.
export function createCancelWebhook({ webhookSecret, store }) {
  return async function handle(rawBody, signatureHeader) {
    let event;
    try {
      event = Stripe.webhooks.constructEvent(rawBody, signatureHeader, webhookSecret);
    } catch {
      return { status: 400, body: "bad signature" };
    }
    // seen() must record the id in the same transaction as the writes below, so a failed write is retried, not skipped.
    if (await store.seen(event.id)) return { status: 200, body: "duplicate" };

    const sub = event.data.object;
    const prev = event.data.previous_attributes ?? {};

    if (event.type === "customer.subscription.updated") {
      if (sub.cancel_at_period_end === true && prev.cancel_at_period_end === false) {
        // A cancellation was scheduled: the subscriber keeps access until the period ends.
        await store.cancelScheduled({
          subscription: sub.id,
          customer: sub.customer,
          reason: sub.cancellation_details?.feedback ?? null,
          comment: sub.cancellation_details?.comment ?? null,
        });
      } else if (sub.cancel_at_period_end === false && prev.cancel_at_period_end === true) {
        // The scheduled cancellation was stopped: a candidate save, counted after your waiting period.
        await store.reactivated({ subscription: sub.id, customer: sub.customer });
      }
    } else if (event.type === "customer.subscription.deleted") {
      // The real end, from a direct cancel or from reaching the end of the paid period: revoke access here.
      await store.accessEnded({ subscription: sub.id, customer: sub.customer });
    }
    return { status: 200, body: "ok" };
  };
}

The dedupe contract: seen() must record the event id in the same database transaction as the writes the handler makes, so that if a write fails Stripe's retry is processed again instead of being skipped as a duplicate.

The handler ran in six tests with stripe-node 23.1.0, using events signed the way Stripe signs them: a wrong secret and a tampered body are rejected, a scheduled cancellation records the reason and comment, a stopped one is reported as a reactivation, an unrelated update does nothing, a deleted subscription revokes access, and a repeated event id is processed once. Before you deploy it, send a test event from your Dashboard or the Stripe CLI and watch it arrive end to end.

Where the reason lives: in the subscription object it sits under cancellation_details (fields comment, feedback and reason). The handler stores whatever string arrives and does not depend on a fixed list of values.

Applying an offer

Once the reason is known, the three standard offers map to three Stripe changes.

Discount: a coupon with percent_off or amount_off, lasting once (the default), repeating for a set number of months, or forever. Then set cancel_at_period_end back to false.
Downgrade: switch the subscription item to a cheaper price and set cancel_at_period_end to false.
Pause: pause_collection with a behavior (keep_as_draft, mark_uncollectible or void) and an optional resumes_at. The subscription stays active and the customer keeps access, so decide in your app what a paused subscriber may use.

Checklist before you ship

The webhook endpoint reads the raw body and checks the signature.
Event ids are stored, so a retry changes nothing.
Access ends on customer.subscription.deleted or on status, not when the cancel button is clicked.
Every offer path ends with cancel_at_period_end set to false, and a "Cancel anyway" button is always one click away.
Every cancel attempt is logged with the reason, the offer shown and what the subscriber chose, so you can count saves later.

Choosing which offer to show is the harder part. If you want to see one decided from a subscriber's own words, the sample cancel session runs in 90 seconds with no sign-up, and shows the offer, the wording and the Stripe change it would make.

Full guide with sources: stayput.grain64.com/guides/build-a-cancel-flow-with-stripe

What users of Jira and Asana keep complaining about, counted by thread, and where they say they went

Grain64 — Mon, 02 Nov 2026 08:47:55 +0000

Building against a named tool, or choosing where to position against it? Here is the count Friction holds for 54 B2B products, with every report linked to the comment it came from: for Jira and Asana, which complaint keeps coming back and where people say they went.

Jira: five of its 21 recurring themes (in order of reports; all dates; 154 Hacker News threads read; theme names as written on https://friction.grain64.com/p/jira/?utm_source=devto)

"Slow Jira page and board performance": 26 reports across 33 threads, from 2008 to 2026 (one report can rest on several threads). Typical: "have to wait minutes for the program to respond" (HN, 2025-07).
"Time-consuming Jira configuration": 8 reports in 8 threads. Typical: "the power to actually configure it to be sane is always reserved by a few people who don't want to bother" (HN, 2026-05).
"Jira feels too burdensome for simple tracking": 7 reports in 7 threads, 2013 to 2026.
"High Jira subscription costs": 7 reports in 7 threads. Typical, from a team that moved to Plane: "Jira wanting 26k per year, per customer we have was unacceptable" (HN, 2025-07).
"Complex Jira navigation and workflows": 6 reports in 7 threads. Typical: "historically i have been quite critical of Jira - because its always felt so complicated for me" (HN, 2024-09).

Asana: four of its 22 recurring themes (in order of reports; all dates; 92 threads; theme names as written on https://friction.grain64.com/p/asana/?utm_source=devto)

"Slow page loads and navigation": 10 reports in 9 threads, 2015 to 2020.
"High memory and CPU usage": 6 reports in 9 threads, 2013 to 2023.
"Flexible task and workflow structure": 6 reports in 6 threads, 2011 to 2024.
"High pricing for paid features": 4 reports in 4 threads. Typical: "columns and some sorting options behind a paywall" (HN, 2024-03).

Which tool should be counted next?

Name the product you build against, replace or compete with, in the comments here or at https://friction.grain64.com/request/?utm_source=devto. A product gets a page when public sources hold at least 10 sourced reports about it; otherwise the request is declined with the reason and the coverage numbers.

Where people say they went

26 Jira reports describe leaving and 16 Asana reports do (https://friction.grain64.com/alternatives/jira/?utm_source=devto, https://friction.grain64.com/alternatives/asana/?utm_source=devto).
Where a destination is named, Jira users name Linear most (5 reports), then GitHub Issues (2) and GitLab Issues (2), and five more tools with one report each, Plane among them; for Asana, Workflowy (2) and Clubhouse (2, from 2018) lead, with six more at one report each. A count is the number of reports naming a tool, not a market share.

Across the category (all dates, four tools; items, not recent reports; https://friction.grain64.com/state-of/project-and-issue-tracking/?utm_source=devto)

Area	Items	Jira	ClickUp	Plane	Asana
UI & usability	118	64	11	6	37
Performance	72	31	15	1	25
Customization	34	17	0	6	11
Integrations	33	16	1	4	12
Reliability & bugs	26	8	8	6	4
Pricing & plans	19	9	2	3	5

How to read these numbers

Sources: Hacker News (plus GitHub issues for open-source tools), so the counts are the developer-tooling view of each product. A count is reports, not users: a report counts once per product however many people agreed. Each product page states how many source threads it rests on (Jira 154, Asana 92), so you can judge the depth before you rely on it.

Check or dispute a report

Each report has a link to flag it, and https://friction.grain64.com/data/ says what is stored and how to ask for removal. Vendors can claim a page for free and reply on any report. Friction covers 54 products.

Written with AI assistance. Posted by Friction's publishing account (Grain64).