A data engineer at a Singapore logistics software company inherits a reporting pipeline built over three years by four different people. It consists of around a hundred and forty scheduled queries in BigQuery, transformation logic embedded in two BI dashboards, and a Python script that runs on a colleague’s laptop on Sunday nights. Nothing documents which query depends on which. When the finance team reports that a revenue figure looks wrong, it takes two days to find the query responsible, because changing it might break something downstream and nobody can say what.
The engineer’s job, in practice, is not building anything new. It is stopping the existing system from falling over. This is the situation dbt was designed for, and it is why the tool has become a standard part of how data teams work. This article explains how dbt data pipelines differ from ad hoc SQL, how dbt works day to day, what it does not do, and how a team can adopt it without rewriting everything at once. It draws on the approach used by Engine Analytics, a data analytics company in Singapore that builds data infrastructure for growing businesses.
In most organisations, the logic that turns raw data into reporting tables is scattered. Some lives in scheduled queries, some in BI tool calculations, some in scripts, and some in the head of whoever built it. None of it typically has the disciplines software teams take for granted: dependency tracking, automated tests, version control, and review before changes go live. The consequences are predictable — silent errors, slow debugging, fear of change, and reports that drift apart over time.
Two shifts made a better approach possible. Cloud warehouses such as BigQuery made it practical to load raw data first and transform it inside the warehouse using SQL, a pattern known as ELT. And a large population of analysts already write SQL fluently. dbt — short for data build tool — brings engineering practice to that SQL, so that the transformation layer can be built, tested, and maintained like software.
In dbt, each transformation is a model: a SQL SELECT statement saved in a file. Models reference each other using a simple reference function rather than hard-coded table names, and dbt uses those references to build a dependency graph, so it always runs models in the right order and knows exactly what depends on what. Each model can be materialised as a view, a table, or an incremental table that processes only new data, which matters for cost on large event datasets. Reusable logic can be written once as a macro using Jinja templating, and raw source tables are declared explicitly so their freshness can be monitored.
A typical project is layered. A staging layer cleans and standardises each source. An intermediate layer applies shared business logic. A marts layer produces the final tables that dashboards and analysts read from. dbt compiles this into SQL and runs it inside the warehouse. It does not extract data from source systems or load it into the warehouse — that remains the job of ingestion tools — and it does not move data out of the warehouse. Its scope is deliberately narrow, and that focus is much of why it works.
The change most engineers notice first is testing. dbt ships with generic tests that check a column is unique, is never null, contains only accepted values, or matches a valid record in another table, and teams can write custom tests as SQL queries. Source freshness checks flag when upstream data has stopped arriving. Tests run every time the project is built, so a duplicated order ID or an unexpected drop in row count is caught before it reaches a dashboard rather than after someone quotes the wrong number in a meeting.
Documentation follows the same principle. Descriptions of models and columns are written alongside the code, and dbt generates a browsable site with a lineage graph showing how every table is derived. A new team member can trace a revenue figure back to its source without asking anyone. That single change addresses much of the knowledge concentration that makes inherited pipelines so fragile, and it supports the wider goal described in our article on data engineering for business leaders: reporting that can be trusted because it can be traced.
dbt projects live in Git, which means every change is recorded, reviewed through a pull request, and reversible. Developers work in their own development schemas rather than on production tables, so experimenting cannot break live reports. Continuous integration can build and test only the models a change affects, catching problems before merge. Production runs are scheduled through dbt’s managed platform, an orchestrator such as Airflow, a scheduled container job, or a CI workflow, depending on what the team already uses. dbt Core, the open-source version, covers the modelling and testing itself; the managed platform adds scheduling, a hosted development environment, and collaboration features.
It is worth being clear about the limits. dbt does not replace ingestion, so data still needs a pipeline into the warehouse — for example the connector-based approach in our article on building a Shopify and BigQuery analytics stack. It is not a general-purpose orchestrator for tasks outside the warehouse, such as calling an external API. It does not fix poor modelling; a badly designed model tested and documented is still badly designed. And without naming conventions and a defined structure, a dbt project can become as tangled as the queries it replaced. Poorly chosen materialisations can also raise warehouse costs. The tool provides the framework, but the discipline still has to come from the team.
Return to the logistics company in the opening example. After adopting dbt, its revenue logic lives in a small number of models with clear dependencies, and the Sunday-night script has been replaced by a scheduled job. When finance next flags a figure that looks wrong, the engineer opens the lineage graph, sees the three models feeding it, and finds that a test on the billing source has been failing since a currency field changed format. The fix is a one-line change, reviewed through a pull request, and the test that caught the problem now guards against a repeat. Debugging that once took two days takes a fraction of the time, and the analyst can add new metrics in a development schema without worrying about what else might break. The gain is not speed alone; it is the confidence to change the system.
The teams that succeed with dbt rarely attempt a full migration. They pick one domain where the pain is highest — usually revenue or customer metrics — and rebuild those models first, alongside the existing pipeline, checking that the results match. They agree conventions early: how models are named, how layers are organised, which tests are mandatory. Then they migrate domain by domain, retiring old queries as the dbt versions prove reliable. Within a few months the highest-risk logic is under version control, tested, and documented, and the rest follows at whatever pace the team can sustain.
If your transformation logic is scattered across scheduled queries and dashboards, changes feel risky, or nobody can say where a number comes from, the path forward starts with a structured transformation layer. View our data and AI services, explore our engagement plans, see what we have built in our project portfolio, or get in touch and we can review your current pipeline and outline how to introduce dbt in a way that fits your team.
Engine Analytics is a data analytics company in Singapore that builds the data foundations — warehouse models, tested transformation layers, and audience-specific reporting — that businesses need to maintain their pipelines with confidence.
No. dbt handles the transformation step only, running SQL inside the warehouse after data has been loaded. Ingestion tools or custom pipelines still move data from source systems into the warehouse. The two work together: an ingestion layer lands raw data, and dbt turns it into tested, documented, business-ready tables.
For a handful of simple queries, scheduled queries are fine. As the number of models grows, the problems appear: unclear dependencies, no automated tests, no documentation, and changes made without review. dbt is worth considering once the transformation logic is important enough that an unnoticed error would affect decisions, or once more than one person maintains it.
Yes. We set up the project structure and conventions, migrate your highest-priority models, add testing and documentation, configure deployment, and train your team to maintain it. Get in touch via the Engine Analytics contact page to start with a review of your current transformation logic.
— Engine Analytics | Singapore’s data and AI consultancy — building the tested, documented data pipelines that let teams change their reporting without fear of what it might break.