Case study · Build pipeline
The build pipeline behind every catalog
In short
Every brand ships its product data differently, and a catalog built by hand breaks on the next update. alfa built one pipeline: each brand gets a thin adapter, and a shared core indexes the products, matches the models, expands the variants, verifies them and writes the import file. 14 watch brands, 4 sports brands and a jewelry catalog run on that core today.
Counts as of 28.10.2026. SKUs and pages are summed over 6 client catalogs. Dollar values: master data cleansing at $5,000 to $15,000 per 10,000 records; product copy and data entry at a market minimum of $10 to $15 per SKU.
The problem
Brand data arrives in every shape: live brand sites behind Akamai or Cloudflare, platform exports, Excel masters and archives. Each source needs its own reading. Building each catalog by hand does not scale, and it breaks on the next update.
What alfa built
Two layers. The adapter is the only code written per brand: where the products are, how to read a product page and how to page through a listing. Everything else is shared core: the crawl layer, normalization, mapping to the target template, and the build.
The crawl layer runs as several processes, with resumable runs, a hard timeout per page and a live control board. A run that stops picks up where it stopped instead of starting over.
- One adapter per brand
- Shared core
- Resumable multi-process crawl
- Hard timeout per page
- Live control board
One run, stage by stage
The stage names and agent roles below come from a recorded run of one brand catalog.
- 01IndexHERMES and ARGUS read the brand sitemap and index every product on the site.
- 02Match modelsATHENA resolves the products into models and matches every model.
- 03Expand variantsAPOLLO expands each model into its colorway and size grid, one row per variant.
- 04VerifyPLUTUS reads live price and stock. THEMIS runs the safety checks and reports any blocking issue.
- 05Write workbookHEPHAESTUS assembles the output workbook.
The result
A new brand is configuration, not a project. 14 watch brands, 4 sports brands and a jewelry catalog run on the same core, and every file the pipeline writes goes through the validation gate before it reaches a store. Images follow their own route, described in every image on the right variant.