Crate ingest pipeline

What it is

The Crate shows each user upcoming concerts by the artists in their crate, and other shows nearby they might like. Ticketing feeds such as Ticketmaster and SeatGeek cover many of them, but plenty of shows in bars, breweries, cafes and small clubs never reach those feeds. This pipeline is meant to find every concert in a city, from the venues' and promoters' own pages as well as the ticketing feeds.

What comes out is one clean record per show, with its venue, date, start time and the acts on the bill. Each fact in a record keeps a note of the page it came from. When two pages list the same show, they become one record.

Its first user will be the Shows feature in The Crate. Those listings come today from an older pipeline I built first, which refreshes a feed for each of 31 metro areas every morning. When I reviewed that pipeline on 2 October 2026, about a fifth of its upcoming shows came from one snapshot file that nothing refreshed. Its show ids also shifted, and 1.6% of upcoming ones changed in a single week, which broke saved RSVPs. The new pipeline runs apart from the app, and its daily job compares what it holds with the live feed. There is no date yet for switching the app over to it.

Only Pittsburgh is switched on so far. I score coverage against the old pipeline, as the share of its upcoming Pittsburgh shows that the new one finds when it starts from an empty database. Nothing from the old feed goes in. On 31 October 2026 that share was 37.8%. Measured on a copy of a later test run, it reached 66.1% on 2 November 2026. My bar is full coverage from one run with no follow-up by hand, and the fixes so far should bring it near 85 to 90%.

How it works

A new city starts as one entry in a settings file, with its center point and a radius of 30 miles. A run works through these steps in order, then repeats them on what the last round found, for up to eight rounds, until one adds nothing new.

1. Find placesCollects every place in the area that could host a show, starting from the ticketing feeds and open map data.Ticketing feeds, Overture Maps
2. Tidy placesGives each place a street address and a map point, then merges records that turn out to be the same building.US Census geocoder, rules, then judge models
3. Check websitesOpens each place's website and decides whether it lists real upcoming events and whether it is the venue's own page.Rules, then Jev (a small decision model), Playwright (a browser) for pages that need one
4. Pick a readerUses the page's own event data if it has any, and if not, has a model write a recipe that shows where each date and title sits.Rules, Claude Haiku
5. TrialReads each new page once and keeps it only if its dates pass a free check and its shows are in the right city.Rules, no model
6. Merge showsTurns each listing into a claim, one page's statement about a show, and matches claims so two pages about one show become one record.Rules, then Jev and Claude Sonnet
7. Match artistsSplits each bill into acts, ties each act to one artist record, then reads that artist's own tour page, which names more venues for the next round.MusicBrainz (an open music database), rules, Claude Sonnet and Haiku
8. ReviewSends what the rules and models could not settle to the judge models first, and what they leave unsure to me.Jev, Claude, then me
9. PublishWrites a checked snapshot of upcoming music shows, which the app does not read yet.Cloud Storage, a daily Cloud Run job

Rules do as much of the reading as they can. Many venue sites carry their events in a standard format that code reads for free. For a page without it, Claude Haiku writes the recipe once, and code reads the page with that recipe on every visit after that. The model is asked again only when the recipe's own check fails. A recipe costs about a third of a cent to write.

Where a model has to read, the cheaper one goes first. Claude Haiku reads pages that are plain text. A free check then compares each date with the words around it, and only the parts it doubts go to Claude Sonnet, the larger model. On five test venue pages on 26 October 2026, that found all 120 upcoming shows with the right dates for $0.11, against $0.81 for Sonnet alone. Yes-or-no questions, such as whether two listings are the same show, go first to Jev. It answers with a probability, for a few thousandths of a cent a call.

Every model call goes through one gateway that records what it cost. I approve each call by hand, or approve a whole run in advance as a batch with a call count and a hard cap for each model. The daily job in the cloud never calls a model. It replays answers that were approved and recorded earlier, so the same question is never paid for twice.

Where a person decides

I am the last step for every judgment the rules and models cannot settle. When the judge models cannot decide whether two venue records are one building, or what type an event is, the question comes to a review page. My pick there binds every rule and model after it. I also approve model spending, call by call or a whole run at a time.

By a count on 4 November 2026, the pipeline waits on me in 19 separate places, and many of my answers were never fed back into it. Two review pages hold 167 answers I gave that were never applied, so none of them became a label the models could learn from.

The fix I picked that day is not built yet. Every question will land in one inbox, on one kind of card. Each card carries all the evidence, a reason, a note and a box for what I needed that was not there. Then each kind of question moves to the models in steps, once my own past answers show it is safe. A random 2% of their answers still come back to me. Same-venue merges in a city, for example, go automatic only after 150 checked cases with no wrong merge.

Why build it, and who else could use it

I plan to offer this data to others one day, so I need sources that allow it. Ticketmaster's developer terms bar selling or sublicensing its API. Bandsintown opens its API only to artists and the people working for them. So a dataset I could license to someone else has to be my own. The rule I set on 24 October 2026 is that anything I sell must come from a venue's, promoter's or artist's own pages.

The product I have planned is an API and data files that serve only what may be sold. Beside the shows they would carry each venue's published booking and press contacts, and each artist's own links. One idea I logged on 3 November 2026 is a free tool for small venues that post their shows as images. A venue would enter each show once, and the tool would make the poster, the social posts, the web listing and a print order from that record. Each venue that used it would license its calendar to me. I have not tested the idea.

JamBase is the one concert-data API I know of with a public price list. Its annual plans run from $6,000 to $30,000 a year, and it names Google and Spotify among its customers. PredictHQ sells event data to businesses for demand forecasting.

Nothing is for sale yet, and there is no buyer and no price. The code that would cut a sellable set out of everything the pipeline holds is not written. Under my own rule, shows at venues with no website of their own could not be sold, and about a third of small venues have none.

The hard parts

What it runs on

commits on main

557

lines of Python

76,096

recorded decisions

285

experiment write-ups

104

Counts read from the repository on 4 Nov 2026.