Data Engineer
Mission
Section titled “Mission”Design, build, and maintain the data pipelines that let a client ingest, transform, and use their data - across different types and formats - to make valuable business decisions. The Data Engineer turns raw, scattered data into trustworthy, queryable, decision-ready assets.
Responsibilities
Section titled “Responsibilities”- Design data architecture: ingestion, storage, transformation, and serving layers for batch and/or streaming workloads.
- Build pipelines that ingest from varied sources and formats (databases, files, events, APIs) and transform them (ELT/ETL) into clean, modelled datasets.
- Maintain & operate pipelines - scheduling, orchestration, backfills, and recovery - so data stays fresh and reliable.
- Own data quality & correctness: validation, tests, contracts, lineage, and handling of schema evolution.
- Model data for analytics/BI and expose it to downstream consumers (warehouses, lakehouses, dashboards, ML).
- Manage cost, performance, and governance of data platforms (partitioning, clustering, access, PII handling).
Inputs
Section titled “Inputs”| Input | From |
|---|---|
| Data sources, formats & access | Stakeholders (PO), client systems |
| Business questions & metrics to enable | Stakeholders (PO), BA |
| Target architecture & NFRs | Solutions Architect / Dev Lead |
| Platform / environment | Cloud Engineer (DevOps) |
Outputs
Section titled “Outputs”| Output | To |
|---|---|
| Ingestion & transformation pipelines (ELT/ETL) | Whole team, client |
| Modelled, documented datasets / marts | Analysts, BI, ML, Stakeholders (PO) |
| Data-quality tests, contracts & lineage | Quality Engineer, Stakeholders (PO) |
| Pipeline run health & cost reports | Architect, Cloud Engineer, PM |
AI tools & how we use them
Section titled “AI tools & how we use them”Data work is largely SQL, transformation logic, and configuration - a strong fit for AI assistance, provided correctness is verified against real data, not assumed.
- Cloud data services + AI (AWS Glue/Redshift, Azure Synapse/Fabric Copilot, GCP BigQuery + Gemini): generate and explain SQL, suggest partitioning/clustering, and surface anomalies in datasets.
- Databricks Assistant / dbt + AI / Snowflake Copilot: scaffold dbt models, macros, and tests; draft notebook transformations; and explain or optimize queries on the lakehouse/warehouse.
- Claude / ChatGPT / Claude Code (research & code gen): research approaches faster, explore multiple solution paths for a pipeline design, generate ETL/transformation code, and debug complex errors (stack traces, skew, failing joins) by reasoning over logs and schemas.
- AI for documentation: write clearer data dictionaries, pipeline READMEs, and lineage/run-book docs from the code and schema.
flowchart LR
SRC[Raw sources<br/>DB · files · events · APIs] --> ING[Ingest]
ING --> AIX[AI drafts transforms / SQL / dbt models]
AIX --> REV[Data Engineer reviews & validates]
REV --> DQ{Data-quality tests pass?}
DQ -->|no| REV
DQ -->|yes| SERVE[Modelled, served datasets]
SERVE --> BI[Analytics · BI · ML · decisions]
Collaboration
Section titled “Collaboration”The Data Engineer takes data sources and target questions from the Stakeholders (PO) and BA, aligns architecture and NFRs with the Architect/Dev Lead, and relies on the Cloud Engineer (DevOps) for the underlying platform, environments, and CI/CD. The Quality Engineer consumes data-quality contracts and tests. On data projects, the Data Engineer often sits alongside Developers building data-facing application features.
Definition of Done for this role
Section titled “Definition of Done for this role”- Pipelines are version-controlled, orchestrated, and reproducible - no manual one-off loads in shared environments.
- Data-quality tests and contracts exist and pass; schema changes are handled deliberately, not by surprise.
- Datasets are modelled, documented, and discoverable by their consumers.
- Pipeline cost, freshness, and reliability are visible and within agreed targets.