A self-driving intelligence pipeline for European public affairs

The problem

Public-affairs analysts needed to continuously track what every major EU institution was publishing, national governments, political parties, media outlets, and advocacy groups across 9 countries and ~11 languages, but the sources were dozens of incompatible website layouts, many behind anti-bot firewalls, publishing at all hours.

The build

The solution is one codebase running four parallel pipelines (stakeholders, national parties, governments, media), unified by a shared processing engine that routes each article to the right destination by source type. A tiered crawler extracts content by whatever method each site allows; a multi-stage pipeline then cleans, dates, deduplicates, and classifies every page before storage. A nightly N8N workflow chains five Cloud Run jobs, queries the results, and emails a formatted daily report.

Results

859

sources monitored across 9 countries

13161

articles collected & structured

99.3%

published with a verified date