What it is
AI data engineering is the work of getting data ready for agents, models and reports to use. It covers pipelines (scheduled jobs that copy, test and clean data) from your source systems, and a warehouse or lakehouse to hold the result. It also covers reconciliation between systems, matching of customers and suppliers across systems, and lineage, which is the record of where each figure came from.
An agent that reads a customer balance from the wrong system gives the wrong answer with full confidence. AI projects in banks, insurers and manufacturers stall on this layer more often than on the model.
The problem
A wealth manager holds client records in a CRM, holdings on the platform, fees in the ledger and documents in a file share. A manufacturer holds orders in the ERP, supplier invoices in accounts payable and quality records in spreadsheets. Each system names the same customer or supplier differently, and the systems disagree on totals. An agent that reads from one of them answers from that one view. A month end report built across them needs a person to reconcile it by hand first.
Architecture
Airflow or Dagster schedules each pipeline and runs the steps in order. Where a source system publishes changes as events, Apache Kafka carries them into the warehouse as they happen. Spark loads large histories that are too big for a single database query.
Transformations are dbt models. A dbt model is a SQL query kept in your repository that builds one table from others. dbt tests run at every load and check row counts, totals, unique keys and formats. The pipeline stops when a test fails, and the bad load does not reach a report or an agent.
The data lands in a warehouse or lakehouse in your own environment, such as PostgreSQL. We size the design to how the data is queried. A daily finance report and an agent that answers questions all day need different things.
History is stored as it stood on each date with dbt snapshots. A report for 30 June 2025 run today returns what the business knew on 30 June 2025, with later restatements kept separate.
Entity resolution and reconciliation
Entity resolution matches one customer, supplier or counterparty across systems by name, ABN, address, account number and other identifiers. Matches below a set confidence go to a person. Each entity gets one identifier that every downstream system uses.
Reconciliation compares systems that must agree, such as the subledger and the general ledger, or the platform and the CRM. It lists each break with the records behind it.
Lineage, access and monitoring
dbt records lineage from each table back to its sources, and each row keeps the key of its source record. An auditor can trace a reported figure to the record and the step that produced it. Access is granted by role in the warehouse, and columns that hold personal information are masked for roles that do not need them. Grafana shows freshness and test results for every pipeline and alerts your team when a load is late or fails.
Where agents and reporting fit
The warehouse is the layer that LLM integration and RAG, workflow automation and your reports read from. Document text and embeddings sit in pgvector, the vector search extension for PostgreSQL, and the same pipeline keeps them in sync with the source. AI payables matches an invoice to the purchase order and the receipt. That match needs one supplier entity across the ledger, the purchasing system and the inbox.
For a Sydney builder we made about 10,000 job files, site notes and tenders machine readable on a private server they rent before we built any automation. The estimating automations that followed all read from that layer.
Handover
Your team owns the pipelines at the end. The code sits in your repositories, and the tests, the alerts and the break report run without us. The runbook says what to do when a source system changes its format. Pipelines and storage stay in your own environment, hosted as described under on-prem AI deployment.