A client has four source systems: two databases, a daily flat‑file drop, and a RESTful API. They want all data in one warehouse, refreshed before business opens with a dashboard. Nothing exists yet. How would you build it?
💡 Model Answer
First, identify the four source types: two relational databases, a daily flat‑file drop, and a RESTful API. I’d choose a cloud‑native data warehouse like Snowflake or Redshift for the target, and use a managed ingestion service (e.g., AWS Glue, Azure Data Factory, or dbt Cloud) to orchestrate the pipeline. For the databases, set up CDC or incremental pulls using native connectors; for the file, use an S3 bucket or Blob storage with a scheduled ingestion job; for the API, schedule a daily fetch that writes to a staging table. All sources feed into a staging layer where raw data is stored in its native format. Next, run a transformation layer that cleans, normalizes, and merges the data into a unified schema. Use a DAG orchestrator (Airflow, Prefect) to enforce dependencies: extraction → staging → transformation → load. Schedule the entire pipeline to run nightly, finishing before business opens, and trigger the dashboard refresh after the load completes. Add data quality checks (record counts, checksum, null‑value thresholds) and alerting. For scalability, partition the warehouse tables by date and use incremental loads. Finally, document the lineage and expose the warehouse to BI tools via secure connections.
This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.
🎤 Get questions like this answered in real-time
Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers on a discreet on-screen overlay.
Get Assisting AI — Starts at ₹500