How would you design a data retrace (data lineage) system for your data?

Designing a data retrace (data lineage) system involves capturing, storing, and visualizing the flow of data from source to destination. Key components include: 1) Ingestion layer that hooks into data sources (databases, files, streams) and emits metadata events whenever data is read, transformed, or written. 2) Metadata extraction engine that parses job logs, ETL scripts, or schema definitions to identify source tables, columns, transformations, and target destinations. 3) Lineage graph store – a graph database (e.g., Neo4j, JanusGraph) that persists nodes (datasets, jobs, columns) and edges (reads, writes, transforms). 4) Query and API layer that allows users to traverse the graph, ask “where did this column come from?” or “what downstream jobs consume this table?”. 5) Visualization UI that renders lineage paths, highlights impacted datasets, and supports drill‑down. 6) Security and governance controls to enforce access policies on lineage data. 7) Integration with existing data catalog or governance tools. The system should be event‑driven, scalable, and tolerant of schema evolution. By decoupling metadata capture from storage, it can handle large volumes of data and provide real‑time lineage for compliance and debugging.