WhoFits

WhoFits is OCTYN's own product. We built it, we run the crawl, and we operate the dashboard. Everything below is drawn from the repo and its own progress log.

1. The situation

Agencies pick creators by scrolling, then have to defend the pick to a client who asks why. Follower count is the only number in the room and it is the easiest number to fake. The premise was that influence shows up in network position, in who replies to whom and how often, and that a recommendation should arrive with its evidence attached. The constraint was platform access. X does not hand out its graph, so the graph has to be assembled from the outside without getting an account banned.

2. Why it was hard

Two things fight each other. A network graph is only useful at scale, and everything that produces one at scale looks like abuse to the platform. Pre-qualification through a browser needs a five to fifteen second delay per profile and a 160 second cooldown every twenty profiles, which is why 5,000 profiles took several days across two machines. And the graph is the asset: interaction edges cannot be recollected cheaply, because the window that produced them has passed.

3. What was built

A pnpm monorepo on Turborepo, split into a Next.js 16 site and dashboard, a crawler service, and shared packages for types and database access. Nothing creates its own Supabase client, which is what keeps credentials and row rules in one place.

Ingestion has two lanes. TwitterAPI.io covers profiles through an API. Playwright covers what the API cannot, slowly and with human-shaped delays. A local Ollama model sits between the two as a quality gate on replies, so low-signal interactions never become edges.

Storage is split by job. Supabase holds the canonical rows with row-level security. A local Postgres holds the analytics schema and its materialised views, because network aggregation over an OLTP database run by everyone else's writes is the wrong shape.

Crawling runs on more than one machine, coordinated by atomic task claiming keyed on a per-device worker ID, so two machines never take the same profile.

4. What it does under load

12,000 interactions in the crawler database at the 2026-07-05 reading. 2,875 creators were marked qualified in a single migration on 2026-03-22. A parallelisation change on 2026-03-08 moved the queue from a boolean lock to an active-call counter, two concurrent model calls on a GPU machine and one on CPU, and processed 73 replies from two profiles in the verification run.

5. What happens when it breaks

The most instructive failure was silent. Both foreign keys on the interaction-edge table cascaded on delete, so removing a creator quietly removed their edges. Edges were disappearing and nobody could say why. On 2026-03-22 both keys were changed to restrict, which turns a silent data loss into a blocked delete that says what it is blocking. For data that is expensive to collect, deletion should be loud.

The standing safety rule is that every script writing to the database defaults to dry run, and tests read production but write their results to a local file, never back.

Crawls fail constantly and are expected to. Rate limits, dead profiles and blocked requests are normal, so failed profiles go to a reprocess queue rather than stopping the run. A health check validates a device before it starts, and a status command reports where the pipeline is.

There is no uptime monitor and no alerting on the crawler. It is a batch system, and a stalled crawl is noticed by a person running the status command. That is the honest state, and the first thing we would fix before putting a client's campaign on it.

6. Who maintains it, and what that costs in attention

Two people, and the attention is unevenly distributed. The dashboard and database need almost nothing. The crawler needs a human regularly, which is inherent rather than a defect: it depends on a third-party API, on page structure that changes without warning, and on machines the team owns and has to keep awake.

The cost is not hours of debugging. It is the periodic check that the crawl is still moving and still returning sensible profiles, because detection against a crawler is silent by design and the failure mode is empty results rather than an error.

7. What changes when requirements change in six months

Configuration, no deploy. Qualification thresholds, model choice through the Ollama environment variables, crawl pacing and per-device worker identity are all environment values. Swapping the quality-gate model is one variable.

A code change, contained. New qualification signals, new scoring in the card, new dashboard views, or a change to what the quality gate asks the model. The card is generated from stored evidence, so adding a criterion is a read-side change.

A code change, larger. A second platform. The edge model and the scoring are platform-agnostic by design, but ingestion, rate-limit handling and the quality gate's prompt are all written against X and would be built again per platform.

A rebuild. Changing what counts as an edge. Interaction edges are accumulated over a window that cannot be replayed, so a different definition of influence means starting the collection again. This is the honest constraint in the product and it is worth stating plainly: the graph is cheap to query and expensive to re-earn.

8. What we would do differently

Set the delete rules before the first bulk load, not after losing edges to them.

Put a monitor on the crawl from the start. The status command answers the question, but only when somebody thinks to ask it.

Separate the analytics database earlier. It was the right decision and it arrived after the queries had already been written the other way.