Built a Python vessel‑data scraper (MarineTraffic, Maritime‑Database) and imported roughly 700 maritime companies to seed the platform's core reference data.
Data Engineer
Situation. A maritime networking‑and‑reviews site is dead on arrival if it’s empty. Nobody joins a directory with no companies in it. So before any of the social features mattered, the platform needed a real body of maritime companies and vessels already sitting there, ready to be found.
Task. Go and get that data — real companies, at enough scale to feel populated — and get it into the database in a way that could be rerun, not a one‑off scrape nobody could reproduce.
Action. The scrapers are written in Python. One drives MarineTraffic with Playwright; another pulls from Maritime‑Database over async httpx; there’s a ClassNK fetcher in there too. They write out CSVs, and an import step cleans and normalises those and loads them into the Postgres schema through a task (docker‑dat‑populate‑companies), so seeding the database is a single command rather than an afternoon of manual work. The company set came out to around 700.
Result. The platform launched with roughly 700 real maritime companies in it instead of empty tables — a directory that actually looked like something on day one, and a base of reference data the networking, jobs and review features could all build on top of. And because the pipeline is repeatable, refreshing or extending it later is just running it again.