How The Crate's concert pipeline was built

This is the longer and more technical version of the Crate example in How AI’s Failures Unlock Your Hardest Workflows. It follows the same steps in the same order: a goal, context and guidance, a target, the agent’s first try, the training, the paving and the gates. Each step shows what happened in the code, with the numbers and the tools.

Two pipelines carry the story. concert-db is the first one. An agent loop built it in June 2026, and it still feeds the concert listings in The Crate for 31 metro areas. crate-ingest is the second. I started it on 23 October 2026 to replace the first, and it is switched on for one city, Pittsburgh. Nothing it makes reaches the app yet. The pipeline’s own page says how it works today. This piece says how it got that way.

Commits in crate-ingest

570 in 14 days

With a Claude co-author line

479

Recorded decisions

287

Experiment plans and readouts

117 files

Tests

1,063

The first trail: concert-db

redfinchadvisory.com: A clear goal ↗

The goal I gave the first loop was as broad as the one in Desire Paths. In its progress file, my instruction reads: “stop when you’re sure you’ve captured EVERY concert.”

The context was small. A research file listed every Pittsburgh venue I knew of. SeatGeek’s API gave a backbone of ticketed shows. The agent searched the web through Chrome. Claude Code ran the whole thing on a /loop every 15 minutes from 16 June 2026. Each round it sent Sonnet subagents out to sweep a group of venues (the big halls, the suburban rooms and casinos, breweries and hotels, museums and churches) into raw files. Then it checked each file, merged it and rebuilt the database.

It worked better than I expected. By the second day the database held 790 shows at 96 venues, up from 345 after the first sweep, and the app was showing them. Over the summer it grew to 31 metro areas with a refresh every morning.

I was thrilled to find venues near my house that I didn't know existed, and clean calendars for shows in my area.

It also wandered, had lots of duplicates, thought concerts from 4 years ago were happening this year, and generally had a pile of misinformation.
When I reviewed it on 2 October 2026:

• About a fifth of the upcoming shows came from one snapshot file that nothing refreshed.
• 1.6% of upcoming show ids changed in a single week. Each change broke a user’s saved RSVP.
• Venue ids were made from names, so one id stood for six different theatres called “Palace”.
• One research lane put this year’s date on a 2016 article, and a festival that was long over went out as a live show.

The agent had found a lot of real shows and had some really good data. However, it could not tell me why it believed any one of them, so I had nothing to turn into a system that I could pave, and scale reliably.

Sure, a lot of these issues can be traced to very poor engineering and thoughtless design that I brute forced through. In hindsight, I’m kind of thankful for what I learned by being thoughtless. It gave me a better sense of where steering was truly required, and what these systems could figure out for themselves.

Writing the path down before the code

redfinchadvisory.com: Context and guidance ↗

crate-ingest started with documents, not code. Its first commit is called “seed spec, principles, lessons, decisions round 1”. The goal got rewritten too:

The most up-to-date and accurate database of shows, venues and the artists playing them, everywhere. A city is a method for getting there, never the goal.

The context and guidance from Desire Paths became files that every Claude Code session reads before it starts:

• CLAUDE.md, the project’s standing instructions.
• spec/principles.md, twelve principles. Two of them carry most of the weight: “Every fetch is saved as an immutable capture,” and “Rules first, models second. Every model call goes through one gateway.”
• spec/rules/, numbered rules for sources, the model gateway, evals and tests.
• spec/lessons.md, what concert-db got wrong, written as numbered lessons (the first one is the 2016 article).
• spec/decisions.md, every choice I made, with my answer in my own words next to it.

Most of those decisions were made on decision boards: a page of numbered questions with lettered answers and one marked as the recommendation. I click a letter and add a comment, and the next session reads my picks from the page. The log holds 287 of them, and many quote my answer as I typed it (“Lets go with haiku for hte recipes.”).

The other habit was making the system visible from its own code. A generated viewer shows every table, field and route. A stage map lists each step of the pipeline. Two tests fail when the viewer or the stage map no longer match the code, so the documents cannot drift away from what the pipeline does.

The playbook (docs/playbook/agentic-pipelines.md) is where the lessons go. It has 60 numbered sections, each with the experiment or mistake behind it. Its first one is “Write the rules down before the code.”

The target

redfinchadvisory.com: A predefined target ↗

Desire Paths says a known good output makes the cycle work early. Here the target is concert-db itself. It has flaws, but it has years of my firsthand checks of Pittsburgh behind it, and it is the best list I have.

The score is coverage: the share of concert-db’s upcoming Pittsburgh shows that crate-ingest finds on its own, starting from an empty database. Nothing from the old feed goes in, and I don’t step in to do anything manual. The bar is in playbook section 33: one run from empty to full coverage, and “follow-ups by hand are the system failing.”

The other target is my own answers. One of the principles says “Founder verdicts are the labels.” Every time I settle a question by hand, the answer can become a test case. The labelled sets so far: 350 venue pairs, 161 duplicate-show pairs, 152 event kinds, 139 source pages, 249 artist pairs. Evals run in Braintrust against frozen captures.

Two limits belong here. First, every label set is from Pittsburgh. Second, the first baseline was partly circular, because its truth files for two sources were built from the same structured data the rules read.

For something I am building from scratch, I needed to be involved in creating the target as we go, and be active as the source of truth.

Execute, and keep everything

redfinchadvisory.com: The Cycle ↗

In Desire Paths I talk about how the agent is going to fail, and that is the plan. The engineering question is how cheap each failure is to study. In crate-ingest, four things make it cheap.

Every fetch is kept. Each page read is saved as a capture in Cloud Storage, addressed by its own hash and never overwritten. The captures are the test fixtures. A parser or prompt change is proved by replaying it on them before it ships.

Every model call goes through one gateway. Pipeline code asks for a task, such as extract.events, never for a model. Which model does the task lives in one file:

extract.events: {provider: anthropic, model: claude-haiku-5-5, max_tokens: 16000, effort: medium,   # D-091, experiment 11: 120/120 for $0.11 (Sonnet alone $0.81)
                 cascade: {provider: anthropic, model: claude-sonnet-5-5}}   # re-reads only a page part whose dates the free date check doubts
judge.dupe:     {provider: jev, model: jev-1.14.0, max_tokens: 1000, effort: low,   # D-068, experiment 9: 146/150, 0 wrong merges
                 cascade: {merge_at: 0.90, apart_below: 0.30, provider: anthropic, model: claude-sonnet-5-5, then_merge_at: 0.70}}

Changing a task’s model is a reviewed change plus an eval run. The same file holds the caps: $150 a month, $10 a day, and a cap for each task.

Every call is approved and recorded. The gateway has four modes: replay, approve, record and live. Replay is the default, and a call with no recording is an error. At first I approved each call by typing y, n or q. That broke when I approved over Remote Control, which lost most of my typed answers. Now a whole run gets one approval file with a call count and a hard cap for each model line. An approved answer is recorded, so the same question is never paid for twice. There are 1,328 recordings pinned in the tests and 3,736 in the evals.

Every run writes its own story. A run row per invocation and a run_event per step, decision, skip and failure. crate-ingest runs story <id> prints the history of one venue or show, and runs export makes a readout page for the whole run.

With those four in place, the misses are easy to find. A few of the real ones:

• A trial let a Memphis theatre, a Harrisburg church and 14 single Eventbrite event pages in as Pittsburgh sources.
• Haiku put 7 of 120 dates a day early.
• A model answered “different, confidence 0.1” about two cafes. The code read that as a 90% chance they were the same place and merged them.
• On the first cold run, 87 of 275 venue merges carried a reason that itself said they were two businesses or two rooms.
• One long run spent 19 hours on artist steps with the MusicBrainz copy switched off, merged nothing, and reported every step as ok.

A short note on access, since people ask how the pages are fetched. The pipeline reads public pages only, with an honest user agent that names the project and gives a contact address, at about one request a second per site. A login page, cookie wall, CAPTCHA or challenge is a stop, and not something I route around with an agent.

Train: the two questions, in the code

redfinchadvisory.com: Teach ↗

The training step in Desire Paths asks two questions. How do I know this is wrong? How can I equip the agent with that? Each miss above went through those two questions and came out as a rule. Pick one to step through it.

The artists card is the one I would point a team at first. When a person checks an agent’s work, the clicks they make are a specification for the next version of the agent. Ask the reviewer to narrate one or two checks, list the actions in order, and mark each one: a script can do it, a model can do it, or only a person can.

Playbook section 45 adds a warning about moving a hand rule into the pipeline: move the rule, not the script, then rerun the pipeline version on the experiment’s own cases. A hand reader adapts without noticing. One band’s page wrote “6 Nov, Fri”, day before month, at an address called “/gigs”, and the pipeline rule missed it until the rerun showed the gap.

Pave: recipes and the reader ladder

redfinchadvisory.com: Agent Desire Paths ↗

Paving is where an agent’s judgment turns into code that runs the same way every time. The biggest piece of paving in crate-ingest is how it reads a venue’s page. It goes down a ladder and stops at the first rung that works:

1. A feed iCal or RSS. rules
2. A platform endpoint or the page's own data rules
3. A ticket platform's page for the venue rules
4. schema.org event data in the page rules
5. The site's event pages rules
6. A recipe only on a page that shows an upcoming dated row. model writes it once
7. One bounded render in a real browser. Playwright

Most venue sites are built on a handful of platforms: Squarespace, Wix, WordPress event plugins and others. Each of those has a reader written as rules, with no model.

For a page with no structure, a model writes a recipe: a small file that says which part of the page is one show, and where its date, title and acts sit. From then on, rules read the site with that recipe on every visit for free. A check runs on every read, and a flag sends one repair call to the model. This one was written by hand by Claude Opus in experiment 17. In a run, Haiku writes one like it for each new text page:

{
  "site": "clubcafelive.com",
  "notes": "Squarespace list section. One li per show; date, times and price are free-text paragraphs.",
  "row": "li.list-item",
  "fields": {
    "title": {"sel": ".list-item-content__title"},
    "date": {"sel": ".list-item-content__description", "format": "auto"},
    "start_time": {"sel": ".list-item-content__description", "pick": "show"},
    "doors_time": {"sel": ".list-item-content__description", "pick": "doors"},
    "status": {"sel": ".list-item-content__title", "re": "(sold out|cancel+ed|postponed)"}
  },
  "year_rule": "noyear-next-occurrence"
}

Recipes started on Sonnet, and by 30 October recipe writes were 68% of all model spend. Experiment 18 tried Haiku on the same pages. Haiku’s recipes found 97.4% of the rows against Sonnet’s 97.8%, at a fourteenth of the cost. I picked Haiku on a decision board.

The date check is the other kind of paving: plain code that does what a careful reader does. Its header says what it found:

"""Experiment 11: a free check on a model's event dates, so a cheap reader can do the first pass and a strong
one re-reads only what looks wrong. No model call.

A listing page puts each event's date on one side of its title, the same side every time. The check
finds the page's side from the events themselves, then asks of each event: is the first date on that
side of its own title the date the model gave? If the page also carries a machine date there
(`{startDate: ...}`, `{datetime: ...}` from `model_html.clean_text`), it must agree to the day.

On experiment 7's recordings it doubts all 7 of Haiku's wrong [...] dates (each taken from the
listing above) and none of the 233 right Haiku and Sonnet dates. Of errors planted one at a time in the
right answers (a day off, the listing above's or below's date) it catches 99 to 100%. Its blind spot: a
weekly event moved by exactly a week lands on its own next listing (88% caught). `evals/date_check_eval.py`.
"""

The last piece of paving is the daily job. It runs on Cloud Run at 06:15 Eastern, reads every live source, resolves and publishes, and never calls a model. It runs in replay mode, so every model answer it uses was approved and recorded earlier.

Gates: what nothing passes without

redfinchadvisory.com: Deterministic gates ↗

The note at the end of Desire Paths says a hands-off step needs deterministic gates. These are the ones in crate-ingest, each with its rule.

• Names alone never merge. A venue joins another only on evidence: a host, an address, a map point.
• An act is confirmed by evidence: a shared link, or its own page listing our show.
• A source goes live on one read only if every upcoming date passes the date check, it yields upcoming shows, its events sit in the market and it is not stuck.
• A state with no reader fails the run (the funnel gate).
• A long run starts only after a pre-flight of its keys, model lines and files.
• Spend stops at the caps, counted in the gateway under one lock, including calls still in flight.
• A release is blocked if it fails schema validation, repeats an id, or is under half the size of the last one.
• make check runs before every commit and must stay under a minute: lint, the tests in replay mode, the replay check on every frozen capture, and the duplicate-show eval. A drop fails it.

gates = {
    "validates": {"ok": not invalid, "invalid": invalid[:20], "invalid_count": len(invalid)},
    "unique_ids": {"ok": not dupes, "duplicates": dupes[:20]},
    "count_drop": {
        "ok": ratio_ok,
        "count": count,
        "previous": prev.event_count if prev else None,
        "previous_release": prev.id if prev else None,
        "min_ratio": MIN_RATIO,
    },
    "forced": force,
}
if not force and not all(g["ok"] for g in gates.values() if isinstance(g, dict)):
    raise PublishGateError(gates)

One rule covers the tests themselves. A new test is written only when a bug escaped once, or when the code decides identity, dates or what is published. Each such test is run against the code before the fix, and it must fail there.

How much of me is left

Graduation in Desire Paths means a person steps in only when needed. Here is where that stands.

Yes-or-no questions go to Jev first. It is a small decision model from TypeSafe AI that answers typed questions with a probability and writes no text, at $0.042 per million input tokens. What Jev leaves unsure goes up a ladder of stronger readers, and only what the last one leaves unsure comes to me. For “are these two records one building?” the ladder looks like this:

On address-filled cards, Jev’s form alone merged 88.0% of the pairs and sent 3.6% to me, with 4 wrong merges. Model spend for a city’s first run is in the tens of dollars at Haiku prices. A single call costs from about $0.00002 for Jev to $0.06 for a Sonnet recipe retry.

Some decisions stay with me on purpose: what the fields mean, merging code, keys and money, and which city comes next.

And the honest count: on 4 November I found that I review the pipeline’s work in 19 separate places, and 167 of my answers on two review pages were never applied. That is the subject of the next piece, on why a human in the loop is there to teach, not to check.

The toolkit

The repo holds 76,096 lines of Python, of which 17,134 are tests and 16,341 are evals.

What isn’t done

• Only Pittsburgh is switched on. Columbus has a first pass.
• Nothing crate-ingest makes reaches the app yet. concert-db still feeds all 31 metro areas.
• Every coverage figure here is measured on a copy of one Pittsburgh run against concert-db’s feed.
• Cold run 2, the test of all the fixes together, was still running when I wrote this.