On-Site Automated Parts Catalog Data Extraction

The challenge

On-Site needed structured product data extracted from two large manufacturer parts catalogues — one in agricultural machinery, one in powersports — on behalf of their own clients. These catalogues are deep, hierarchical systems: a machine is looked up by serial number, then broken down level by level into assemblies, sub-assemblies and individual parts, each with its own exploded-view diagram and part number table.

Copying that by hand is not an option. On-Site needed a repeatable, verifiable extraction process that produced clean Excel exports plus all associated diagram images — first as a controlled pilot on a small dataset, with the option to scale to the full catalogue afterwards.

Our approach

We built a custom crawler in C# on top of Selenium WebDriver, driving a real Chromium browser so that the catalogue's JavaScript-heavy navigation behaves exactly as it does for a human user. The crawler walks the full hierarchy per serial number, captures the diagram for every level and writes the result to a structured Excel workbook — one tab per level, ordered by level, with part numbers, descriptions and the reference back to the matching image.

The pilot ran from a single controlled environment so the client could review data quality before committing to volume. For the second project we integrated Bright Data, moving the collection layer onto a managed proxy network with rotating IP addresses. That gives On-Site controlled, distributed crawling that scales to a complete catalogue without hammering the source site from one address.

What we delivered

• A C#/Selenium crawler that navigates multi-level parts catalogues by serial number, to any depth
• Excel exports in the client's exact required format — tabs sorted by level, consistent columns, ready for import into their own systems
• Complete image sets of exploded-view diagrams, matched one-to-one against the export rows
• Bright Data integration for distributed, controlled crawling at full-catalogue volume
• Incremental crawling of only the missing model years, so previously collected data was never re-processed
• A validation pass on every delivery: missing images, gaps in the hierarchy and format deviations caught before hand-over

The result

On-Site received test datasets fast enough to validate the approach with their own client within days, then a production-ready pipeline for the full extraction. Data arrives in the exact structure their systems expect, which removes the manual clean-up step entirely — and the same crawler architecture can be pointed at further catalogues with only the navigation logic swapped out.