A Singapore SaaS business with sixty employees has three BI dashboards. One is owned by the head of sales, built in Salesforce reports, updated when someone remembers to update it. One is owned by the product team, built in Amplitude, showing feature usage metrics that no one outside the product team fully understands. And one is a BigQuery-connected Looker Studio report that the CFO built six months ago to track MRR, which currently shows a number that is twelve percent lower than the Salesforce dashboard shows for the same month.

When the CEO asks for a single view of revenue performance in the Monday leadership meeting, one person quotes the Looker number, another quotes the Salesforce number, and the next fifteen minutes are spent trying to understand why they are different. Nobody can explain it clearly. The meeting moves on. Both numbers continue to be used by different people for different purposes, producing decisions that are made from different starting points.

This is not an unusual situation. It is, in fact, the most common state of BI reporting in SaaS businesses that have grown past thirty or forty people without making a deliberate decision about how their analytics layer should be structured. The dashboards exist. They are just set up in ways that guarantee the outcome described above: multiple sources of truth, undefined metrics, and reports that nobody fully trusts. This article covers the specific mistakes that produce this state and what a setup that actually works looks like. It draws on the approach used by Engine Analytics, a data and AI consultancy in Singapore that builds BI infrastructure for SaaS businesses across growth and enterprise stages.

Mistake One: Building Dashboards Before Defining Metrics

The single most common BI setup mistake is building a dashboard before agreeing on what the metrics it shows actually mean. Revenue is the clearest example. In a SaaS business, “revenue” could mean bookings, billed revenue, recognised revenue, or cash collected — four different numbers that move differently and matter to different stakeholders for different reasons. MRR could include or exclude free trials, one-time charges, setup fees, or discounts. Churn could be calculated on a logo basis, a revenue basis, or a seat basis, and the window for measuring it — are customers who cancel mid-month counted in that month or the next? — produces different numbers depending on what was decided.

Dashboards built before these definitions are agreed produce numbers that are technically computed but conceptually undefined. Different people read them differently, and the metric means whatever the reader assumes it means. When two people quote the same metric name but arrive at different numbers, it is almost always because they are using different definitions that neither of them has stated explicitly. The dashboard did not cause the disagreement — the undefined metric did.

The fix is a metrics document that exists independently of any dashboard and defines, precisely and without ambiguity, every metric the business tracks: what it includes, what it excludes, how it is calculated, and who owns the definition. This document should be agreed across revenue, finance, and product before any dashboard is built or rebuilt. The principles of data-driven decision-making and why this alignment matters are covered in more depth in the article on data-driven decision-making with business intelligence.

Mistake Two: Connecting Dashboards Directly to Source Systems

The second most common mistake is connecting BI dashboards directly to production databases, CRM exports, or billing system APIs and reading from those sources directly. This approach is fast to set up — there is no intermediate layer to build, and the data is current at the point of the last sync — but it creates several structural problems that become increasingly damaging as the business grows.

Direct source connections produce inconsistent results when the same metric is queried at different times. A revenue query run at 9am may produce a different result than the same query run at 11am because a billing event was processed in between. Without a layer that snapshots data at a consistent point, the dashboard’s numbers are a function of when the query ran, not of what actually happened in the period being measured. This is the most common cause of the “why is this number different from yesterday?” conversation that happens in analytics meetings across SaaS businesses.

Direct connections also mean that the transformation logic — the SQL or calculation that turns raw billing records into MRR, or raw product events into active user rates — lives inside the BI tool, repeated in every dashboard that uses those metrics. When the calculation changes, every dashboard that contains it needs to be updated individually. When a new dashboard is built by someone who was not involved in writing the original calculation, it is written slightly differently, producing a metric that is similar but not identical to the others. This is how you end up with three different MRR numbers for the same month.

Mistake Three: One Dashboard for Every Audience

The instinct to build a single “company dashboard” that shows everything to everyone is understandable. Leadership wants visibility. The dashboard should provide it. But a single dashboard built for multiple audiences inevitably makes a trade-off that serves none of them well: either it is too high-level to be useful for operational decisions, or it contains so much detail that the signal the leadership team needs is buried in metrics that are irrelevant to them.

The head of sales needs a dashboard focused on pipeline, conversion rates by stage, deal velocity, and attainment against target. The product manager needs feature adoption rates, activation cohorts, and engagement depth by user segment. The CFO needs MRR waterfall, recognised revenue, CAC, and unit economics by cohort. Building a single dashboard that tries to present all of these simultaneously produces something that is used by no one because it is designed for everyone.

The right structure is multiple audience-specific dashboards, each built around the questions a specific role actually needs to answer, all reading from the same underlying data layer where metric definitions are consistent. This way, the head of sales and the CFO can both see MRR — defined identically — in the context of the other metrics relevant to their role, without either of them wading through information that is not relevant to their decisions.

Mistake Four: No Single Source of Truth Underneath the Dashboards

All three mistakes above have a common cause: there is no single, governed analytical layer underneath the dashboards where data is cleaned, standardised, and modelled before any dashboard reads from it. Without this layer, every dashboard is effectively its own data environment — with its own connections, its own transformation logic, and its own implicit metric definitions. The piece on data engineering for business leaders covers what this layer looks like and why it is the foundation that makes everything above it trustworthy. For a SaaS business specifically, this layer typically includes a central data warehouse — BigQuery is the most common choice — where data from the CRM, billing system, product database, and any other relevant source is landed on a consistent schedule, cleaned, and modelled into the business-logic tables that dashboards read from.

With this layer in place, metric definitions live in the data model rather than in dashboard calculations. MRR is defined once, in a table that every dashboard queries. When the definition changes, it changes in one place and propagates to every dashboard automatically. New dashboards are built against the same tables, so a new team member building a product dashboard for the first time uses the same MRR definition as the CFO’s finance view. The multiple-source-of-truth problem is structurally impossible when there is only one source.

What the Right Setup Looks Like

The setup that works begins with metric alignment: a documented, agreed definition for every metric the business tracks, owned by the relevant function and reviewed when business logic changes. This document is a prerequisite for any dashboard work, not a deliverable produced after the dashboards are built.

A central data layer is built and maintained separately from the dashboards that read from it. Source data lands in a cloud warehouse on a defined schedule. Transformation models run against that raw data and produce clean, labelled, business-logic tables. Monitoring alerts when pipeline failures occur or when data freshness falls outside expected windows. The data layer is owned and maintained as infrastructure — not as a byproduct of dashboard work.

Audience-specific dashboards are built on top of the data layer, each focused on the questions a specific role needs to answer. They are connected to the warehouse, not to source systems directly, so they read from data that is consistent, snapshotted at the right time, and defined identically across every view. For teams with time-sensitive operational metrics, the architecture may extend to near-real-time data movement — the piece on real-time analytics covers when that investment is justified and what it requires.

Ready to Fix Your SaaS BI Setup?

If your current BI environment produces numbers that different people quote differently, dashboards that nobody fully trusts, or reports that raise more questions than they answer, the path forward starts with the data layer and the metric definitions — not with a new dashboard design. View our data and AI services, explore our engagement plans, see what we have built in our project portfolio, or get in touch and we can assess your current setup and identify exactly where the BI environment needs to be rebuilt to produce reporting your team can trust.

Engine Analytics is a data analytics company in Singapore that builds the analytical foundations — data layers, metric definitions, and audience-specific dashboards — that SaaS teams need to move from conflicting reports to a single version of the truth.

Ask a data engineer at a Singapore startup what they use for their analytics layer and the answer, more often than not, is BigQuery. Ask the same question at a mid-market eCommerce business. Ask at an agency managing analytics for multiple clients. Ask at an enterprise team that migrated off an on-premise data warehouse two years ago. The answer is BigQuery with a frequency that, five years ago, would have been surprising. Today it is simply the default.

Defaults happen for reasons. They are rarely accidents. BigQuery did not become the go-to analytics layer for Singapore data teams because of a single decisive feature or because of marketing spend. It became the default because a combination of technical architecture decisions, ecosystem alignment, pricing model, and timing converged in a way that made it the obvious answer to a problem — “where does our analytical data live?” — that every data-capable organisation eventually has to solve.

Understanding why BigQuery won this position matters for teams that are evaluating their options for the first time, and for teams that are already using BigQuery and want to understand what they have and where its limits apply. This article covers the technical and ecosystem reasons BigQuery became the default, what being the default actually means in practice, and where the boundaries of that position still sit. It draws on the perspective of Engine Analytics, a data analytics company in Singapore that has built BigQuery-based analytics environments across SaaS, eCommerce, healthcare, and enterprise clients.

What Singapore Data Teams Were Using Before — and Why They Moved

Before BigQuery became the default, Singapore data teams with serious analytical requirements typically used one of three approaches. On-premise data warehouses — Oracle, SQL Server, or Teradata — handled large-enterprise environments but required dedicated infrastructure teams, significant upfront capital, and upgrade cycles that lagged years behind analytical requirements. Redshift, Amazon’s cloud data warehouse, was widely adopted through 2018 and 2019 and remains in active use, but requires cluster sizing decisions and ongoing management that adds operational overhead the fully-serverless BigQuery eliminates. And many teams simply used PostgreSQL or MySQL read replicas as their analytical layer — workable for modest data volumes but increasingly painful as data grew and analytical query complexity increased.

The migration stories are consistent in their structure. An event that outgrew what an on-premise warehouse could handle affordably. A Redshift cluster that needed to be resized and then resized again. A read replica that started affecting production performance as analytical query load increased. In each case, the push away from the existing solution converged with BigQuery’s growing ecosystem maturity and Singapore’s accelerating Google Cloud adoption — and BigQuery became the natural destination.

The timing of Google’s acquisition of Looker in 2019 and the subsequent development of what became the modern Google data stack also mattered. The combination of BigQuery as the data layer, dbt for transformation, Looker or Looker Studio for reporting, and GA4’s native BigQuery export created a coherent, well-integrated architecture that data teams could adopt as a system rather than assembling from competing parts. Singapore’s strong Google Ads and Analytics adoption meant that many teams were already partly inside the Google ecosystem, which reduced the activation energy required to move the analytics layer there as well.

The Technical Reasons BigQuery Won

Serverless architecture is BigQuery’s most significant operational advantage. There are no clusters to provision, no nodes to scale, no capacity planning decisions to make before running a query. BigQuery allocates compute dynamically against each query and releases it immediately after. For a data team that wants to focus on analytics rather than infrastructure management, this removes a category of ongoing operational work that Redshift and traditional data warehouses impose by design.

The separation of storage and compute is what makes BigQuery’s pricing model work at scale. Storage is cheap and persistent. Compute is charged per query against the on-demand model, or reserved at fixed cost through slot commitments for teams with predictable, high-volume query workloads. For organisations with variable analytical load — heavy queries during business hours, minimal usage overnight — on-demand pricing is significantly more cost-efficient than a Redshift cluster running at constant cost regardless of utilisation.

Native support for nested and repeated fields allows event data to be stored in BigQuery in its natural JSON structure rather than being flattened into normalised relational tables. For SaaS and eCommerce businesses generating high-volume clickstream or product event data, this is a meaningful technical advantage: the event schema can evolve without requiring table restructuring, and queries can target specific nested fields without scanning the entire event record. Combined with partition pruning and clustering, this makes BigQuery exceptionally efficient for the analytical query patterns that growth-stage businesses actually run. The data engineering principles behind this architecture are covered in more depth in the article on what data engineering means for business leaders.

The Ecosystem That Made BigQuery Sticky

Individual technical features explain why BigQuery was adopted. The ecosystem explains why it stayed the default. The GA4 native export, which lands raw and unsampled event data directly into BigQuery at no additional cost, created an immediate, compelling use case for every business already running GA4. The alternative — using the GA4 API directly with its sampling limitations and rate constraints — produces inferior data at inferior cost for any property beyond modest scale. For Singapore businesses investing in analytics on top of GA4, BigQuery became not just a good option but the obviously correct infrastructure choice.

The dbt ecosystem grew in parallel with BigQuery adoption and made the transformation layer significantly more manageable. dbt’s BigQuery adapter is mature and well-maintained, its jinja-based templating handles BigQuery’s partitioned table syntax cleanly, and the dbt Cloud managed service integrates with BigQuery’s permissions model without significant friction. For teams building analytical data models — the clean, business-logic-bearing tables that dashboards and analysts read from — dbt on BigQuery is the combination that most Singapore data teams converge on. The reporting layer built on top, covered in the piece on data-driven decision-making with business intelligence, connects directly to the BigQuery output of these models.

The connector ecosystem that lands non-Google source data into BigQuery — through Fivetran, Airbyte, or custom pipelines — has also matured significantly. Salesforce, HubSpot, Shopify, Meta Ads, Stripe, Zendesk, and most other platforms used by Singapore businesses have maintained BigQuery connectors with predictable, documented schemas. The practical friction of getting data from a SaaS platform into BigQuery is now low enough that it is rarely a reason to choose a different analytical layer.

What Being the Default Actually Means — and Its Limits

“Default” does not mean universally correct. BigQuery is the right analytical layer for most Singapore data teams working at meaningful scale with diverse data sources. It is not necessarily the right choice for every use case. Organisations with extremely high query concurrency requirements and predictable workloads may find Redshift’s slot-based reservation model more cost-predictable at scale. Businesses deeply embedded in the Azure or AWS ecosystems may have better reasons to use Synapse or Redshift than to introduce a GCP dependency. And for very small teams with simple analytical needs, the operational overhead of a cloud data warehouse — any cloud data warehouse — may not be justified at all.

BigQuery is also not a substitute for the data engineering work that makes an analytical layer useful. A BigQuery project full of raw, unmodelled tables is not meaningfully more valuable than the source systems the data came from. The value emerges from the transformation layer, the data models, the quality monitoring, and the reporting infrastructure built on top — the full stack that turns raw data movement into analytical capability. The piece on real-time analytics covers one dimension of that full stack for teams whose use cases require data freshness that batch pipelines cannot provide.

Getting the Most Out of BigQuery as Your Analytics Foundation

The teams that get the most value from BigQuery are the ones that treat it as an analytical layer with clear architectural boundaries — not as a general-purpose database or a replacement for operational systems. Raw source data lands in BigQuery from pipelines. Transformation logic runs in a separate layer and produces clean, business-logic-bearing tables. Reporting tools read from those tables, not from raw data. Access controls are applied at the dataset and table level, not just at the BI tool level. Query cost monitoring is in place before the team grows beyond two or three analysts.

Dataset and table naming conventions, partition strategies, and documentation standards established early pay compounding dividends as the analytical environment grows. A BigQuery project that has been organised thoughtfully from the start is navigable and maintainable by any competent data engineer. One that has grown organically without structure becomes a source of confusion and technical debt that slows down every subsequent analytical project.

Ready to Set Up BigQuery the Right Way for Your Singapore Data Team?

Whether you are adopting BigQuery for the first time or working to restructure an existing environment that has outgrown its initial organisation, the decisions that matter most are architectural — dataset structure, partitioning strategy, transformation layer design, access control, and cost governance. View our data and AI services, explore our engagement plans, or get in touch and we can review your current setup and map out what a well-architected BigQuery analytics layer looks like for your specific team and data sources.

Engine Analytics is a data analytics company in Singapore that designs and builds BigQuery analytics environments — from initial architecture through transformation models, quality monitoring, and the reporting layer that stakeholders across the business rely on.

A Singapore SaaS business reaches Series B. Headcount has doubled in eighteen months. The sales team wants pipeline dashboards. The product team wants retention cohorts. The finance team wants revenue waterfall reporting. Customer success wants churn risk signals. The single analyst on the team is producing all of it manually — downloading exports, running SQL, pasting numbers into slides — and spending so much time on data assembly that there is almost no time left for actual analysis.

The instinctive response is to hire more analysts. The better response, in most cases, is to build better infrastructure. The difference between a company that needs six analysts to support a two-hundred-person business and one that needs two is almost never the complexity of the questions being asked. It is the quality of the data engineering underneath the analytics layer.

Data engineering — building the pipelines, data models, and self-serve infrastructure that analytical work depends on — is what allows SaaS companies to grow their analytical output significantly faster than their analytical headcount. This article covers exactly how that works: the specific manual work that data engineering replaces, the data models that make self-serve analytics possible, and what the operational picture looks like for a SaaS business that has built this foundation properly. It draws on the approach used by Engine Analytics, a data and AI consultancy in Singapore that builds data engineering infrastructure for SaaS businesses at growth stage and beyond.

The Headcount Trap: What Happens When Analytics Grows Linearly With the Business

Without data engineering infrastructure, analytical capacity scales with people. Every new stakeholder who wants data requires someone to produce it. Every new question requires someone to write a query, pull an export, join a spreadsheet, and format a slide. The work compounds as the business grows: more teams, more questions, more data sources, more reports to maintain. The analyst team that was one step behind at fifty employees is three steps behind at two hundred.

The deeper problem is that most of the work in this model is not analysis — it is assembly. An analyst spending four hours a week maintaining a revenue report that pulls from three systems and needs to be reconciled manually is not doing analysis for four hours. They are doing data plumbing for four hours, and the plumbing produces a result that is out of date the moment it is finished. The analysis that would actually inform a decision — the cohort comparison, the churn driver investigation, the pricing sensitivity study — never gets done because the week is already full of maintenance work.

This is the headcount trap: the business grows, the analytical demand grows, and the response is to hire people who then spend most of their time on manual data assembly rather than on the work the business actually needed them for. The trap compounds until either the analytical backlog becomes visibly damaging or someone makes the decision to invest in the infrastructure that makes the whole model work differently.

What Data Engineering Actually Does for a SaaS Business

Data engineering, at its core, is the discipline of making data move reliably and arrive in a useful form without manual intervention. For a SaaS business, this means building pipelines that pull data from every relevant source — the product database, the CRM, the billing system, the ad platforms, the support tool — and land it in a central analytical layer on a defined schedule, cleaned, standardised, and ready to query. The piece on what data engineering means for business leaders covers this foundation in detail. For a SaaS business specifically, the payoff is that every stakeholder who previously needed an analyst to produce their data can instead query it directly — or open a dashboard that is already built and already current.

The transformation layer is the part of data engineering that most directly replaces manual work. Transformation logic — the SQL or dbt models that join tables, apply business definitions, calculate derived metrics, and produce the clean, labelled datasets that dashboards and analysts read from — runs automatically on every pipeline cycle. The revenue waterfall that took an analyst four hours to assemble in Excel runs in sixty seconds and is available every morning before anyone arrives at their desk.

Scheduling and orchestration ensure that pipelines run in the right order, at the right time, with monitoring that alerts when something fails. A SaaS business with ten data sources feeding a central analytical layer needs those sources to arrive in the right sequence — billing data before revenue models, CRM data before customer metrics — and needs to know immediately if any source is late or missing. Without orchestration, pipeline failures are discovered when a dashboard shows stale data and someone asks why. With orchestration, they are caught and addressed before anyone is affected.

The Specific Manual Work That Data Engineering Replaces

The most immediate replacement is the weekly or monthly report assembly process. In most SaaS businesses without data engineering infrastructure, recurring reports are assembled manually: someone exports data from multiple sources, joins it in a spreadsheet, applies the relevant calculations, and populates a template. Data engineering replaces this entirely — the same logic runs automatically on a schedule and the report is always current without anyone touching it. The analyst who was spending a day a week on report assembly gets that day back for actual analysis.

Ad-hoc data requests — “can you pull the retention rate for enterprise customers who onboarded in Q1?” — are the second major category. Without a clean, well-modelled analytical layer, every ad-hoc request requires an analyst to locate the relevant tables, understand how they join, write the query, validate the result, and format it for the stakeholder. With a well-built data model where customer tiers, onboarding dates, and retention events are already defined and queryable in a consistent schema, the same request takes minutes rather than hours and can often be answered directly by the stakeholder without involving an analyst at all.

Data pipeline maintenance — fixing broken exports, updating queries when a source system changes its schema, reconciling discrepancies between two systems that report the same metric differently — is the third major category. This work is invisible when it is running well and highly disruptive when it fails. Proper data engineering infrastructure makes this maintenance systematic rather than reactive: schema changes are detected automatically, pipeline failures trigger alerts, and the analytical layer continues to produce accurate data even when upstream sources change. This is the same reliability principle that makes real-time analytics viable at scale — the infrastructure is robust enough to operate without constant human intervention.

The Data Models That Make Self-Serve Analytics Possible

Self-serve analytics — the ability for product managers, sales leads, and finance directors to answer their own data questions without routing requests through an analyst — depends entirely on the quality of the data models underneath the self-serve interface. A data model that accurately represents how the business works, uses consistent definitions for every metric, and covers the questions that each team actually asks allows non-technical users to explore data confidently. A poorly built data model produces results that look right but are not, which is worse than no self-serve capability at all because it generates bad decisions that nobody questions.

For a SaaS business, the core data models typically cover customers, subscriptions, product usage, and revenue. A customer model that tracks every account, its tier, its lifecycle stage, and its key attributes. A subscription model that records every plan, every change, and every renewal, with clear definitions for MRR, ARR, expansion, and churn that apply consistently across all downstream reporting. A product usage model that captures feature-level engagement events and translates them into the behavioural signals — active user rates, feature adoption, session frequency — that product and customer success teams need. Revenue models that reconcile billing records with subscription events and produce the waterfall views finance requires.

When these models are built correctly and documented clearly, an analyst can onboard a new stakeholder to self-serve reporting in an hour rather than maintaining a bespoke report for them indefinitely. The analytical team’s time shifts from data production to data interpretation — answering the harder questions that emerge once teams have reliable access to the standard metrics.

What Scaling Without Scaling Headcount Actually Looks Like

A SaaS business that has built proper data engineering infrastructure looks different in a few specific ways. Reporting meetings start with everyone looking at the same numbers from the same source, rather than each team arriving with their own version of the data and the first twenty minutes being spent reconciling discrepancies. New stakeholder requests for data are met with a dashboard link or a query against a clean data model, not with a one-to-two week wait for an analyst to build something from scratch. When a pipeline fails, the monitoring system catches it before anyone’s dashboard goes stale.

The analyst team’s work changes in character. Less time is spent on data assembly and more is spent on the questions that data engineering cannot answer automatically — the investigative work, the causal analysis, the modelling that requires human judgement about what to look for and what it means. This is the work that produces the insights that actually inform product and commercial strategy. It is also the work that is most valuable and most engaging for the analysts doing it, which has a measurable effect on retention.

Ready to Build an Analytical Layer Your Team Can Scale With?

If your analytics team is spending more time on data assembly than on analysis, or if your stakeholders are waiting too long for answers to questions that should be straightforward, the investment that changes that picture is almost always infrastructure, not headcount. View our data and AI services, explore our engagement plans, see our project portfolio, or get in touch and we can assess where your current analytical infrastructure stands and what it would take to build something your team can genuinely scale with.

Engine Analytics is a data analytics company in Singapore that builds the data engineering foundations that allow SaaS businesses to grow their analytical output significantly faster than their analytical headcount.

A Singapore fintech business hired a data analytics consultancy eighteen months ago for a twelve-week strategy engagement. The deliverable was a sixty-page document: a recommended tech stack, a data governance framework, an implementation roadmap with quarterly milestones, and a set of KPI definitions the business should be tracking. The consultants presented it, answered questions, and left.

Twelve months later, the tech stack recommendations are partially installed but not connected to each other. The governance framework is referenced in internal documentation but has not changed how anyone actually handles data. The implementation roadmap has never been formally started. The KPI definitions sit in a shared folder that nobody has opened in three months. The engagement cost six figures. The organisation has no dashboards that a decision-maker relies on daily. The ROI is, by any honest accounting, zero.

This is not an isolated story. It is the pattern that a growing number of Singapore businesses are recognising when they look back at analytics consulting engagements that produced documents but not systems, strategies but not outcomes. The rethink is not about whether data analytics is valuable — it clearly is. It is about whether the traditional consulting model is the right way to extract that value. This article covers why the conventional model underdelivers and what a better approach looks like. It draws on the perspective of Engine Analytics, a data analytics company in Singapore that builds analytics infrastructure rather than producing recommendations about it.

What Changed — and Why the Rethink Is Happening Now

The traditional analytics consulting model was designed for a different era. When data infrastructure required significant custom engineering, specialised hardware, and months of setup work, the case for a strategy-first engagement was clear: before investing that much time and money in implementation, you needed a careful plan. The strategy phase was proportionate to the implementation cost.

The modern data stack has changed that equation significantly. A cloud data warehouse, a managed ELT tool, a transformation framework, and a BI layer can be assembled and producing useful output in weeks, not months. The implementation cost and timeline have both dropped dramatically. But the traditional consulting model has not adapted to match. Businesses are still paying for twelve-week strategy phases before any implementation begins — and the strategy is still being handed off to internal teams who were not part of building it.

Singapore’s business community is also maturing in its data literacy. Leaders who commissioned analytics engagements two or three years ago have now seen enough outcomes — or lack of them — to ask sharper questions before committing budget. The questions they are now asking are covered in detail in the guide to what makes a great data analytics partner, which addresses the specific things worth evaluating before engaging an analytics consultancy.

The Specific Problems With the Traditional Consulting Model

Strategy documents do not build dashboards. This sounds obvious, but the traditional engagement model consistently treats the strategy deliverable as the primary output — the thing that demonstrates the engagement’s value. A well-structured sixty-page document with a clear technology recommendation and an implementation roadmap looks like value. It does not produce a single dashboard that a decision-maker opens at nine in the morning to understand what happened yesterday.

Recommendations made without implementation ownership are not accountable to outcomes. When the consultancy recommends a tech stack and then leaves, the success or failure of the implementation falls on an internal team that was not involved in making the recommendation. When the implementation runs into problems — data quality issues the strategy didn’t anticipate, integration complexity that the recommendation underestimated, internal capacity that proved insufficient — there is no one accountable for making it work. The consultancy delivered its contractual obligation. The business is left with a partially implemented system.

The handoff model assumes that internal teams can successfully take over systems they did not build. This assumption is frequently wrong. Data pipelines require understanding of the specific decisions made during implementation — why a particular table is structured a certain way, what edge cases the transformation logic is handling, what monitoring is in place and what it alerts on. Without that knowledge, internal teams maintaining a handed-off system are always one undocumented decision away from breaking something they cannot diagnose.

What Singapore Businesses Are Looking For Instead

The shift in what Singapore businesses want from analytics consulting is visible in how the conversations at initial meetings have changed. Three years ago, the typical brief was: “we need a data strategy.” Today, the typical brief is: “we need dashboards that our team actually uses, and we need someone who will build them rather than tell us how to build them.” The emphasis has moved from planning to delivery, and from one-time engagements to ongoing partnerships.

Embedded analytics partnership — where the consultancy continues to work with the business through implementation, iteration, and evolution — is increasingly the model businesses ask for rather than accept as an upgrade. The rationale is simple: a consultancy that remains involved through implementation is accountable for the system working, not just for the recommendation being coherent. That accountability changes the quality of the recommendation, because it is made by people who will be responsible for delivering against it.

Transparency about what is being built, for whom, and what success looks like is also something businesses are demanding more explicitly. The complete guide to data analytics consulting in Singapore covers the questions worth asking before committing to any analytics engagement — including how to evaluate whether a consultancy’s past work actually produced systems that are still in use, or documents that are no longer referenced.

How to Evaluate an Analytics Partner Before You Commit

Ask to see live dashboards built for other clients — not case study PDFs or screenshots, but reports that are currently in use and that someone in that business opens regularly. A consultancy confident in its implementation quality will have no hesitation sharing examples. A consultancy whose primary deliverable is strategic documentation will find this question difficult to answer.

Ask who builds the pipelines and writes the transformation logic. In some consulting models, senior consultants design the architecture and junior staff or offshore teams implement it — with the result that the people accountable for the recommendation are different from the people responsible for making it work. Understanding the composition of the team that will actually do the implementation work, not just the team that attends the discovery calls, matters significantly.

Ask what happens to the system when the engagement ends. A system that requires the consultancy to remain involved to function is not an asset — it is a dependency. A system that is well-documented, built on standard tools, and handed over with enough institutional knowledge that an internal team or a replacement partner can maintain it is what you are actually paying for. The answer to this question tells you a great deal about how the consultancy thinks about its relationship with clients.

What a Better Analytics Engagement Model Looks Like

It starts with a diagnostic rather than a proposal. Before recommending technology or architecture, the right first step is understanding the actual state of the data: what sources exist, what quality problems are present, what questions the business most needs to answer, and what internal capacity exists to maintain whatever gets built. A proposal made without this understanding is a guess. A proposal made with it is a plan.

The first deliverable is something the business can use — a working dashboard, a connected pipeline, a functioning report — not a document describing what those things should look like. Early delivery builds trust, surfaces the practical problems that discovery calls cannot anticipate, and gives the business something concrete to respond to. Iteration based on real use is faster and more accurate than iteration based on a strategy review meeting.

Ongoing engagement covers monitoring, iteration, and evolution as the business changes. Data infrastructure that is built and left alone degrades. Sources change. Business questions evolve. New data becomes available. The organisations that get sustained value from analytics investment are the ones with an ongoing relationship with a team that understands their systems and continues to develop them — not the ones that commissioned a strategy eighteen months ago and have been maintaining it alone since.

Ready to Work With an Analytics Partner That Builds Rather Than Advises?

If your organisation has invested in analytics consulting and found the outcomes disappointing, or if you are evaluating partners and want to understand what a build-first engagement model looks like in practice, we are happy to walk through it. View our data and AI services, explore our engagement plans, see our project portfolio, or get in touch directly.

Engine Analytics is a data analytics company in Singapore that builds analytics infrastructure — pipelines, data layers, and dashboards that decision-makers actually use — rather than producing recommendations about what that infrastructure should look like.

An operations analyst at a Singapore eCommerce business needs to combine data from Shopify, Meta Ads, Google Ads, Klaviyo, and Salesforce into a single reporting view in BigQuery. Her options are clear: build and maintain custom API connectors — and accept that every platform update will break something — or use a managed ELT tool like Fivetran that keeps the connectors maintained and lands the data automatically. She chooses Fivetran. For the five connectors she needs, it works exactly as advertised.

Six months later, the business acquires a logistics partner whose data lives in a proprietary warehouse management system with no Fivetran connector. The engineering team spends three weeks building a custom connector, discovering in the process that the custom connector development process is significantly more involved than the pre-built connector experience suggested. The Fivetran bill has also grown faster than expected, because one of the ad platform tables updates historical records retroactively and the MAR count is three times what the team projected.

Fivetran is a genuinely excellent product for the problem it was designed to solve. It is also frequently adopted by Singapore businesses without a clear understanding of where its model fits well and where it creates cost or coverage problems. This article covers what Fivetran does, where it excels, where it struggles, and how to evaluate whether it belongs in your data stack. It draws on the approach used by Engine Analytics, a data analytics company in Singapore that designs data integration architectures for businesses across eCommerce, SaaS, and enterprise sectors.

The Data Integration Problem Fivetran Was Built to Solve

Every SaaS platform exposes its data through an API. Every API has authentication requirements, rate limits, pagination logic, schema conventions, and a history of breaking changes. Building a custom connector to a single platform is a few days of engineering work. Maintaining it as the platform evolves is an ongoing cost that never goes away. Multiply that across ten, fifteen, or twenty data sources and the engineering overhead of API maintenance becomes a significant tax on a team that would rather be building analytics.

Fivetran’s answer is to maintain those connectors so you don’t have to. Their engineering team tracks API changes for five hundred-plus sources, updates the connectors when platforms release new versions, handles authentication refreshes automatically, and manages the incremental sync logic that ensures only new or changed records are moved rather than full table dumps on every run. For businesses whose data sources are all on Fivetran’s supported list, this is a genuine engineering time saving that compounds significantly as the number of sources grows.

The ELT model — extract from source, load to destination, transform in destination — also aligns well with the modern data stack. Raw data lands in BigQuery or another cloud warehouse in a predictable schema. Transformation happens in the warehouse using dbt or SQL, applied to data that is already there and queryable. This separation between movement and transformation is one of the core principles covered in the article on what data engineering means for business leaders — and Fivetran implements the movement half of that separation reliably for supported sources.

What Fivetran Does Well

Pre-built connector quality is Fivetran’s clearest strength. For major platforms — Salesforce, HubSpot, Shopify, Stripe, Google Ads, Meta Ads, Zendesk, Jira, and hundreds more — the connectors are production-grade, actively maintained, and handle the edge cases that custom connectors frequently miss: API pagination, rate limit handling, historical backfills, and incremental sync with change detection. For a business whose data sources are all on the supported list, setup time for a new connector is typically measured in minutes rather than days.

Schema normalisation means data arrives in BigQuery in a structured, queryable format without custom transformation logic at the ingestion stage. Fivetran handles type casting, nested field flattening, and basic schema evolution automatically. When a source platform adds a new field, Fivetran detects the schema change and adds the column to the destination table without breaking the existing pipeline. For analytics teams that want to focus on analysis rather than ingestion plumbing, this reduces the operational overhead of keeping data current significantly.

Fivetran’s Data Observatory provides lineage visibility — showing which downstream tables and dashboards depend on which connectors — and sync health monitoring that alerts when a connector fails or data freshness falls outside expected windows. For businesses running multiple connectors across multiple destinations, this observability layer is genuinely useful for maintaining confidence that the data arriving in your analytics environment is accurate and current.

Where Fivetran Is Not the Right Choice

Custom or proprietary data sources with no pre-built connector are the clearest limitation. Fivetran’s value proposition depends on its connector library covering your sources. When a critical data source — a proprietary ERP, a bespoke internal system, a regional platform with limited international adoption — is not on the supported list, the options are to build a custom connector using Fivetran’s connector SDK (which is significantly more engineering work than using a pre-built connector) or to handle that source with a different tool. Businesses with even one critical proprietary source need to account for this before committing to Fivetran as their primary integration platform.

High-volume data sources where MAR-based pricing becomes expensive need careful evaluation. Fivetran charges based on Monthly Active Rows — the number of rows that are synced or updated in a given month. For sources where historical records are frequently updated retroactively — ad platform data is the most common example, because performance metrics for past campaigns are recalculated as attribution windows close — the MAR count can be significantly higher than the number of new records suggests. A table with one million historical ad impressions that updates attribution data for the past thirty days on every sync can accumulate MAR counts that are multiples of what the team initially projected.

Real-time data requirements are also outside Fivetran’s primary design. Fivetran syncs on a scheduled basis — typically every hour for most plans, with more frequent options at higher tiers. For use cases where analytics need to reflect data that is minutes old rather than hours old, Fivetran’s batch sync model is not the right fit. And for very simple use cases — moving data from one or two sources that rarely change — the cost-benefit calculation often favours a lightweight alternative or a direct API call handled by a simple pipeline.

The Cost Structure You Need to Understand Before Committing

Fivetran’s MAR-based pricing model is logical but requires careful modelling before commitment. The core question is how many rows in your source tables change each sync cycle — not how many rows exist in total, but how many are new or updated. For relatively static data — customer master records, product catalogues, CRM contacts — MAR counts tend to be low and predictable. For event data, ad platform data, or any source that recalculates historical records retroactively, MAR counts can grow quickly and unpredictably.

The free tier is useful for evaluating connector quality and setup experience but does not represent production costs accurately for most businesses. The enterprise tier is negotiated rather than list-priced, which means the published pricing is a ceiling rather than a floor for higher-volume customers. If your projected MAR volume is significant, getting a specific quote and modelling costs against your actual data sources before signing an annual contract is essential.

How to Decide Whether Fivetran Is Right for Your Business

Start by listing every data source you need to move and checking whether Fivetran has a maintained pre-built connector for each. If every source is supported, Fivetran is likely the fastest path to a working integration layer. If one or more critical sources require custom connectors, factor in the engineering cost and consider whether a tool that handles custom sources more naturally — such as Airbyte, which has a lower barrier to custom connector development — would be a better fit for your specific mix of sources.

Then model your MAR costs based on your actual data. Look at how many rows in each source table change between sync cycles, whether any sources update historical records retroactively, and what the resulting monthly MAR estimate would be across your full connector set. Compare that estimate against your available budget and against what alternative approaches — purpose-built pipelines for high-MAR sources, Fivetran for the rest — would cost in total.

Not Sure Whether Fivetran Fits Your Stack?

The right data integration approach depends on your specific source mix, data volumes, latency requirements, and engineering capacity. View our data and AI services to understand how we approach integration architecture, explore our engagement plans, or get in touch and we can review your sources and model what the right integration architecture looks like for your business — including whether Fivetran is the right tool for all of it, part of it, or none of it.

Engine Analytics is a data and AI consultancy in Singapore that designs data integration architectures for businesses across eCommerce, SaaS, and enterprise sectors — matching the right tools to the right sources rather than defaulting to a single platform regardless of fit.

The head of operations at a Singapore specialist clinic group receives a monthly reporting pack assembled by a junior administrator. It shows appointment volumes, revenue by clinician, and average consultation duration. It takes two days to produce and arrives a week after the period it covers. The group has three clinics, two different EMR systems, and a growing patient base. The operations head knows the reporting is inadequate. She has seen the kind of live dashboards her peers in retail and finance are using and assumes the same approach will work for her clinics.

She engages a generic BI vendor. Three months and a significant budget later, the dashboard shows appointment volumes pulled from one EMR but not the other, revenue figures that do not match the finance team’s records because the two systems use different billing codes, and a patient satisfaction score that turns out to measure form submission rates rather than clinical outcomes. The dashboard looks impressive. It is not useful.

Healthcare BI is not a harder version of retail BI or SaaS BI. It is a structurally different problem — shaped by patient-centric data models, complex clinical coding, PDPA compliance requirements, and the interplay between clinical and operational metrics that do not exist in other sectors. This article covers what makes it different, where Singapore clinics consistently go wrong, and what a properly designed healthcare BI environment actually looks like. It draws on the experience of Engine Analytics, a data analytics company in Singapore that builds reporting infrastructure for healthcare organisations navigating exactly these challenges.

Why Healthcare BI Is Not Just Another Reporting Problem

Most business intelligence frameworks are designed around transactional data: a customer buys something, an event is recorded, the BI layer aggregates and visualises it. Healthcare data does not work this way. A single patient encounter generates clinical data (diagnoses, procedures, medications), administrative data (scheduling, referrals, correspondence), operational data (room utilisation, staff allocation, equipment usage), and financial data (fees, claims, payments) — all of which are interconnected and need to be reported both separately and together depending on the question being asked.

The time dimension in healthcare data is also more complex than in most other sectors. There is the time an event occurred clinically, the time it was recorded in the system, the time the billing was processed, and the time the data was exported for reporting. These can differ by hours or days, and which time dimension you use significantly affects the numbers a dashboard produces. Generic BI tools default to a single timestamp and rarely surface this distinction to the analyst building the report.

Clinical coding adds another layer of complexity. Diagnosis and procedure data in Singapore healthcare is typically recorded using ICD-10 codes, with some systems also using SNOMED CT or CPT codes for specific purposes. Aggregating across these coding systems, or building dashboards that need to present clinical data in plain language rather than code strings, requires domain knowledge that generic BI implementations typically do not have. The result is dashboards that accurately report what the system contains but present it in a form that clinicians and operations leaders cannot interpret.

The Ways Healthcare Data Is Structurally Different

Healthcare data is patient-centric in a way that most business data is not. Every data point — an appointment, a prescription, a billing record, a referral — links back to a patient record, and meaningful analytics often require following a patient’s journey across multiple encounters, multiple clinicians, and multiple systems. Retail analytics can aggregate across customers without tracking individual journeys. Healthcare analytics frequently cannot, because the clinical question being asked is about what happened to a specific patient population over time.

The many-to-many relationships in healthcare data are also more complex than in most business contexts. A single encounter can involve multiple clinicians. A single clinician sees hundreds of patients. A single patient may have multiple active conditions, each generating its own stream of encounters, prescriptions, and referrals. Building BI on top of this data model requires careful thought about how joins are structured, which entity is at the centre of each analytical question, and how aggregations are defined to avoid double-counting.

Many Singapore clinics operate across multiple EMR systems — either because different specialties use different tools, or because acquisitions brought in legacy systems that were never consolidated. When the same patient appears in two EMR systems under different identifiers, every cross-system report needs a reconciliation layer that matches records correctly. Without that layer, patient counts are wrong, continuity-of-care metrics are meaningless, and any analysis that spans the full patient journey is structurally unreliable.

What Singapore Clinics Get Wrong About BI

The most common mistake is treating BI as a reporting layer to be added on top of existing systems without addressing the data foundation underneath. Dashboards that connect directly to live EMR systems inherit every data quality problem in those systems — missing fields, inconsistent coding, duplicate records, entries made in the wrong time period. The dashboard looks authoritative. The numbers it shows are not.

Generic business metrics applied to clinical contexts produce dashboards that measure the wrong things. “Conversion rate” in a clinic context could mean the proportion of enquiries that became appointments, the proportion of appointments that resulted in a second visit, or the proportion of treatment plans that achieved their clinical objective. Using the term without defining which of these is being measured leads to dashboards that different stakeholders interpret differently and that cannot drive consistent decisions.

Separating clinical reporting from operational reporting from financial reporting — and building each for the audience that needs it — is one of the foundational principles of healthcare BI that generic implementations typically miss. A clinician reviewing patient outcome metrics needs a different view from an operations manager reviewing resource utilisation, and both need a different view from a finance director reviewing revenue per encounter. Building one dashboard that tries to serve all three audiences typically serves none of them well. The piece on data-driven decision-making with business intelligence covers how audience-specific reporting structure applies across sectors, including healthcare.

PDPA and Compliance: The Layer That Changes Everything

Healthcare BI in Singapore must be designed with PDPA compliance embedded in the architecture from the start, not applied as a filter at the reporting layer. Patient-identifiable data in a BI environment needs access controls that determine which staff can see which patient records — and those controls need to be enforced at the data layer, not just by dashboard permissions. A dashboard permission that hides a patient’s name does not prevent a query against the underlying data from returning it.

Row-level security — the ability to show different users different subsets of the same data based on their role — is a requirement for any healthcare BI environment where multiple clinical teams share reporting infrastructure. A GP should not be able to see a specialist’s patient panel. An administrator should not be able to see clinical notes. Implementing row-level security correctly requires it to be designed into the data layer, not bolted onto the BI tool after the environment is built.

Audit trails are a compliance requirement in healthcare contexts that most generic BI implementations do not include by default. Who accessed what patient data, when, and for what reported purpose needs to be recorded and retainable for inspection. For Singapore clinics operating under both PDPA and MOH data governance frameworks, this is not optional — and retrofitting audit capability onto a BI environment that was not built for it is a significant undertaking.

What Good Healthcare BI Actually Looks Like

A properly designed healthcare BI environment has a governed data layer between source EMR systems and reporting tools — a data warehouse where patient data is reconciled across systems, clinical codes are standardised, and access controls are enforced at the data level before any reporting tool touches the data. This separation is the same principle described in the article on what data engineering means for business operations: the quality and reliability of your reporting is determined by the quality and governance of the layer underneath it, not by the reporting tool on top.

Separate reports for separate audiences, each with clearly defined metric logic appropriate to the questions that audience actually asks. Clinical dashboards that use clinical terminology and are built around patient outcomes and care pathways. Operational dashboards built around resource utilisation, scheduling efficiency, and capacity. Financial dashboards built around revenue, collections, and cost per encounter. Each report is connected only to the data it needs and is visible only to the staff who need it.

Metric definitions documented and agreed before any dashboard is built. In healthcare, where the same word can mean different things to a clinician, an administrator, and a finance director, this step is not optional. Undocumented metric definitions produce dashboards that generate disagreement rather than decisions — and that disagreement typically surfaces in a meeting where someone important is watching.

Ready to Build BI That Actually Works for Your Clinic?

If your current reporting is slow, inaccurate, or simply not connected to the clinical and operational questions your leadership team needs to answer, the path forward starts with the data layer — not the dashboard. View our data and AI services, explore our project portfolio, or get in touch and we can walk through what a properly architected healthcare BI environment would look like for your organisation.

Engine Analytics is a data analytics company in Singapore that builds reporting infrastructure for healthcare organisations — designed for the compliance requirements, data complexity, and audience diversity that clinical environments demand.

A Singapore SaaS company’s product team wants to understand why enterprise-tier customers are churning at a higher rate than SME-tier customers. The answer requires combining product usage data from their event tracking system, subscription data from Stripe, support ticket data from Zendesk, and CRM data from Salesforce. Their analyst spends three days exporting CSVs, joining them in Excel, and producing an analysis that is outdated before it is presented. The CEO wants this kind of answer available on demand, not in three days.

They implement BigQuery. Several things get dramatically better immediately. Product usage data that previously lived in isolated event logs is queryable in seconds. Cross-source joins that took days in Excel run in minutes. Looker Studio connects directly and the dashboards update automatically. Then the first unexpected billing alert arrives, and the conversation about query governance begins.

BigQuery is a genuinely excellent tool for SaaS analytics in the right configuration. It is also widely misconfigured in ways that create cost problems, performance issues, and architectural debt that becomes expensive to unwind. This article covers what BigQuery was built for, where it struggles, and the decisions that determine whether it becomes a reliable analytics foundation or an ongoing source of operational friction. It draws on the approach used by Engine Analytics, a data analytics company in Singapore that designs and builds BigQuery-based analytics environments for SaaS and eCommerce businesses.

Why SaaS Businesses Are Turning to BigQuery

BigQuery’s appeal for SaaS analytics starts with its data model. SaaS businesses generate event-heavy data — user sessions, feature interactions, API calls, product usage logs — and BigQuery was built to handle exactly this kind of high-volume, semi-structured data at scale. Native support for nested and repeated fields means that JSON event payloads can be queried without flattening, which matters when your event data has hundreds of properties and you want to query specific ones without restructuring the entire table.

The serverless architecture removes a significant operational burden. There are no clusters to provision, no indexes to maintain, and no query planning that falls on the analytics team. You write standard SQL and BigQuery handles the execution. For businesses that want analytical power without dedicated data infrastructure engineering, this is a meaningful advantage over alternatives that require cluster management.

The Google ecosystem integration also matters. GA4’s native BigQuery export lands raw, unsampled event data into BigQuery at no additional cost — which immediately solves the sampling problem that makes direct GA4 reporting unreliable at scale. Looker Studio connects natively, removing the need for intermediate connectors. And for SaaS businesses already running on Google Cloud, BigQuery fits into an existing security and governance model. The foundational data engineering principles that make this architecture work are covered in detail in the piece on what data engineering means for business leaders.

What BigQuery Does Exceptionally Well for SaaS Analytics

Event-scale data handling is BigQuery’s clearest strength. A SaaS business generating tens of millions of product interaction events per day can query that data in seconds without any pre-aggregation. This is not the case for most relational databases, which slow dramatically as event tables grow. For product analytics use cases — funnel analysis, feature adoption rates, session reconstruction, cohort retention — BigQuery handles the query patterns that matter most without performance degradation.

Table partitioning and clustering give analytics teams precise control over query costs and speed. Partitioning by date means queries that filter to a recent time range only scan the relevant partitions rather than the full table. Clustering on high-cardinality dimensions like user ID or product ID further reduces the data scanned per query. These features, applied correctly from the start, can reduce query costs by an order of magnitude compared to unoptimised table designs.

Scheduled queries, materialised views, and BigQuery’s integration with orchestration tools make it straightforward to build automated reporting pipelines that run without manual intervention. Combined with Looker Studio or a more capable BI tool, the result is a reporting layer that updates automatically and can handle the kind of real-time data needs that most SaaS analytics teams eventually require. The piece on real-time analytics covers the infrastructure considerations involved when data freshness becomes a critical requirement.

Where BigQuery Falls Short

Cost unpredictability is the most common problem for teams that implement BigQuery without governance in place. BigQuery charges by the amount of data scanned per query in the on-demand pricing model. An analyst writing an unoptimised query that scans a large, unpartitioned table can generate a bill in a single query that exceeds what the team expected to spend in a month. This is not a theoretical risk — it is the experience of a significant portion of teams that move to BigQuery without establishing query cost controls from the outset.

BigQuery is an OLAP system, not an OLTP system. It was designed for analytical queries across large datasets, not for transactional workloads that require low-latency reads and writes at the individual record level. SaaS businesses that try to use BigQuery as a real-time operational database — powering product features directly, for example — encounter latency and concurrency constraints that make it unsuitable for that use case. The right architecture uses BigQuery as the analytical layer and keeps operational databases separate.

The free tier, while genuinely useful for evaluation, does not represent production costs accurately for most businesses. Teams that evaluate BigQuery on small datasets and limited query volumes consistently underestimate what their costs will look like once the full event stream is flowing and multiple analysts are running exploratory queries simultaneously. Building cost estimation into the architecture design phase — rather than discovering it at the first billing cycle — is essential.

The Setup Decisions That Determine Whether BigQuery Delivers

Project and dataset structure is a decision that is easy to get wrong early and expensive to restructure later. A well-organised BigQuery environment separates raw data (exactly as it arrives from source systems), staging or transformed data (cleaned and standardised), and mart or reporting tables (purpose-built for specific analytical questions) into distinct datasets with clear naming conventions. Teams that dump everything into a single dataset and rely on table naming to create structure find that the environment becomes increasingly difficult to navigate and govern as it grows.

Access control needs to be designed from the start, not added after the fact. BigQuery supports column-level and row-level security, which matters for SaaS businesses that need to control which teams or users can see which customer data. Applying IAM roles at the dataset and table level ensures that analysts can query what they need without having access to data they should not see. Retrospectively applying access controls to an environment that was built without them is a significant project that disrupts ongoing analytics work.

Query cost controls — custom quotas per user or service account, BI Engine for dashboard query caching, and cost assignment for monitoring spend by team or project — should be implemented before access is opened to a wider group of analysts. The moment multiple people are running queries against production data, uncontrolled query costs become a real risk. These operational considerations are part of what distinguishes analytics infrastructure that scales from analytics infrastructure that becomes a maintenance burden. The article on data-driven decision-making with business intelligence covers how the reporting layer built on top of this infrastructure should be structured to serve different analytical audiences effectively.

Ready to Build SaaS Analytics on BigQuery the Right Way?

Whether you are evaluating BigQuery for the first time or working to fix an existing environment that has accumulated architectural debt, the decisions that matter most are the foundational ones: dataset structure, partitioning strategy, access control, and cost governance. View our data and AI services, explore our project portfolio, or get in touch and we can review your current BigQuery setup and map out what needs to change.

Engine Analytics is a data analytics company in Singapore that designs and implements BigQuery analytics environments for SaaS and eCommerce businesses — from raw data ingestion through to governed reporting layers that analysts and leadership teams can both rely on.

A Singapore retail business builds its first AI use case — a product recommendation engine — on a data pipeline the team assembled in a few weeks. It works well. Results are good. Leadership sees the output and approves three more AI initiatives: a demand forecasting model, a dynamic pricing system, and a customer segmentation engine. Six months later, the engineering team is spending most of their time maintaining the original pipeline rather than building the new ones. Every new AI use case requires bespoke data plumbing that does not connect cleanly to what already exists.

This is the scale problem organisations hit when AI moves from experiment to programme. The first use case is built on whatever data infrastructure is available. That infrastructure was designed to serve one model, at one cadence, for one team. When the second and third use cases arrive — each with different data requirements, different freshness needs, and different feature engineering logic — the ad-hoc infrastructure collapses under the cumulative weight of what is being asked of it.

Structuring data infrastructure to support AI at scale is a different problem from building infrastructure to support a single AI project. It requires deliberate architectural decisions about how data is ingested, stored, transformed, and served to models — decisions that are expensive to undo once the infrastructure is embedded in production systems. This article covers those decisions and why they matter. It draws on the approach used by Engine Analytics, a data and AI consultancy in Singapore that designs infrastructure for businesses moving from individual AI experiments to sustained AI programmes.

Why Scaling AI Is a Data Infrastructure Problem, Not an AI Problem

When organisations talk about scaling AI, the conversation tends to focus on compute — more GPU capacity, larger models, faster inference. These are real requirements for specific use cases, but they are rarely what actually prevents AI from scaling. What prevents AI from scaling is data infrastructure that was not designed to serve multiple models simultaneously, at different cadences, with different feature requirements, without each pipeline interfering with the others.

The difference becomes concrete quickly. A single AI model needs a data pipeline that delivers clean, labelled training data on some schedule. Multiple AI models need data infrastructure that can serve different subsets of data to different models, on different schedules, with different transformation logic applied — and they need a feature layer where computed inputs can be stored and reused across use cases rather than rebuilt from scratch for each new project. Without this infrastructure, every new AI initiative becomes a bespoke project that re-solves problems the previous project already solved. This is the pattern that makes AI programmes feel disproportionately expensive relative to their output.

The businesses that scale AI efficiently are not the ones with the most sophisticated models. They are the ones that invested in the data infrastructure layer early enough that each new AI use case could be built on top of existing foundations rather than alongside them.

The Architecture Decisions That Determine Whether AI Scales

The architectural decisions that matter most for scalable AI infrastructure are made early and are costly to change later. The first is the separation between raw data, transformed data, and features. Raw data should be stored exactly as it arrives from source systems, with full history preserved. Transformation — cleaning, joining, aggregating, standardising — should happen in a separate, version-controlled layer. Features — the computed inputs models actually train on — should be stored in a dedicated layer that models can read from consistently at both training time and inference time. When these three layers are conflated, each new model creates its own version of each layer, and the infrastructure becomes an overlapping tangle of pipelines with no single source of truth.

The second critical decision is the data movement pattern. Batch processing — moving data on a schedule — works for AI use cases where the model does not need very recent data. Real-time or near-real-time data movement is necessary when the model is making decisions that depend on current state: fraud detection, live personalisation, or dynamic pricing. Getting this choice wrong means either rebuilding the pipeline when a real-time use case arrives, or forcing it onto batch infrastructure that cannot support it. As the piece on real-time analytics covers, the infrastructure requirements for real-time and batch data movement are substantially different and the choice needs to be made based on the actual latency needs of the AI systems being built, not on what the first use case happened to require.

The third is the governance model for feature definitions. When multiple AI models reference the same underlying concept — “customer lifetime value”, “days since last purchase”, “product affinity score” — but each computes it differently, the outputs of those models cannot be meaningfully compared or combined. A feature governance framework that standardises definitions and enforces consistency across all AI pipelines is not glamorous infrastructure, but it is what prevents a situation where three models in production are each defining the same business concept differently and producing outputs that no one can reconcile.

Data Quality and Governance at Scale

Data quality management becomes significantly more complex when multiple AI pipelines are reading from shared infrastructure. A quality problem in one field of one upstream table can degrade multiple models simultaneously. If monitoring is not in place at the infrastructure level, that degradation may not be detected until it surfaces in model outputs — by which point the business has been acting on unreliable AI predictions for an unknown period of time.

Scalable AI infrastructure needs data quality monitoring built into the pipeline layer, not added as an afterthought. This means automated checks on record volume, field completion rates, statistical distributions, and referential integrity — running on every data movement cycle and alerting when something falls outside expected ranges. The alternative is discovering quality problems by working backwards from unusual model behaviour, a process that can take days and that leaves degraded AI systems in production while the investigation runs.

Schema governance is equally important. When source systems change — a field gets renamed, a new category is added, a CRM migration changes the customer ID format — those changes can break multiple downstream AI pipelines simultaneously if there is no governance layer that catches and manages schema changes before they propagate. At single-model scale, a schema break is a contained problem. At multi-model scale, it is an incident affecting the entire AI programme.

Infrastructure Patterns That Do Not Scale — and What to Use Instead

Direct database connections from AI pipelines to operational systems work for the first use case. As the number of models grows, the query load on operational systems increases, and the risk that AI training jobs affect production system performance becomes real. The right pattern is a dedicated analytical data layer — a data warehouse or data lake — that mirrors operational data and serves as the read layer for all AI pipelines, completely separated from production systems. Every model reads from the analytical layer. Nothing reads directly from operational databases.

Per-model feature engineering — where each model computes its own version of each feature from raw data — creates duplication and inconsistency that compounds as the model inventory grows. A feature layer in the data warehouse, or a dedicated feature store, allows features to be computed once and reused across models. This reduces pipeline complexity, ensures consistency across use cases, and makes it possible to update a feature definition in one place rather than across multiple model codebases simultaneously.

Manual pipeline orchestration — where data movement jobs are triggered manually or through ad-hoc scheduling — does not survive the complexity of multiple concurrent AI pipelines with interdependencies. A proper orchestration layer is necessary once the infrastructure grows beyond a handful of pipelines. This is part of what foundational data engineering covers: the operational infrastructure that makes data pipelines reliable, observable, and maintainable at the scale an AI programme requires.

Ready to Structure Your Data Infrastructure for AI at Scale?

Whether you are building your first AI use case and want to start with infrastructure that will not need to be rebuilt, or managing a growing AI programme on infrastructure that is beginning to strain, the architectural decisions above are the ones that matter. View our data and AI services, explore our engagement plans, review our project portfolio, or get in touch and we can assess your current infrastructure and map out what it needs to look like to support AI reliably as your programme grows.

Engine Analytics is a data analytics company in Singapore that designs and builds the data infrastructure that makes AI programmes scalable — from raw data layers and pipeline orchestration through to feature governance and quality monitoring.

The problem every growing company recognizes

Ask any team past a certain size the same question, and you’ll get the same answer: “where do I find that?”

The answer usually isn’t a document. It’s a person. “Ask Marco, he set that up.” “Check with Sara, she handled that client.” Knowledge sits in inboxes, Slack threads, someone’s laptop, someone’s memory. Everywhere except somewhere the rest of the team can actually reach it.

This is why the idea of a “company brain AI”, an internal AI assistant that can answer questions using your own documents, procedures, and history, has moved from novelty to genuine business priority in 2026. It’s a real, well-defined technology category now, not a buzzword: 71% of organizations report using generative AI in at least one business function, and the market for the underlying technology is projected to grow nearly tenfold by 2030.

But there’s a gap worth understanding before you invest in one.

What a “company brain” actually is

Strip away the marketing language, “enterprise search,” “knowledge management,” “AI intranet”, and nearly every tool in this category runs on the same core technique: Retrieval-Augmented Generation (RAG).

Here’s the mechanism, in plain terms:

1. Your documents, wikis, tickets, and procedures get broken into searchable pieces and indexed
2. When someone asks a question, the system retrieves the most relevant pieces from that index
3. An AI model generates an answer grounded in those specific pieces, not from general internet knowledge, but from your company’s actual content
4. The answer can cite exactly which document it came from, so it’s verifiable, not a guess

This is fundamentally different from asking ChatGPT a question. A generic AI model doesn’t know your contracts, your customer history, your internal processes, or what happened in last quarter’s board meeting. A properly built company brain AI does because it’s not relying on memory, it’s retrieving and citing your real information every time.

Why most of these projects underdeliver

Here’s the uncomfortable statistic worth sitting with: while 71% of companies are already using generative AI somewhere in the business, only 17% say it’s contributed meaningfully to their bottom line.

That gap isn’t a technology problem. The underlying models are genuinely capable. The gap is almost always the same thing we see across every AI initiative we’ve worked on:

The knowledge base underneath the AI was never properly governed.

A company brain AI built on top of scattered, duplicated, outdated, or inconsistently-labeled documents doesn’t magically become organized because you added AI on top of it. It just gets faster at surfacing the mess. Ask it a question, and it might retrieve the outdated version of a policy instead of the current one, confidently, with a citation, looking completely trustworthy while being wrong.

The most common failure pattern looks like this:

• No single source of truth: three versions of the same document exist across different folders, none marked as authoritative
• No access controls: the AI surfaces information to people who shouldn’t see it, or worse, withholds information from people who should
• No maintenance plan: the system works well at launch, then quietly degrades as new documents pile in unindexed and old ones go stale
• No evaluation: nobody’s actually measuring whether the answers are accurate, so problems surface through user complaints instead of monitoring

None of these are AI problems. They’re data governance problems that AI makes visible faster than before.

What actually needs to be true first

Before layering a conversational AI interface on top of your company’s knowledge, a few foundational things need to be in place. The same principles that apply to any serious data infrastructure work:

A real source of truth. Every core piece of knowledge (policies, client information, process documentation) needs one authoritative version, not five conflicting copies competing for relevance.

Consistent structure. Tables, spreadsheets, and structured data need to be handled properly, not flattened into unstructured text that loses its meaning. A pricing table converted badly into plain text stops being useful the moment someone asks a specific question about it.

Role-based access built in from the start. The AI assistant should only ever surface what a given person is already authorized to see, this isn’t optional for anything touching client data, HR records, or financial information.

A real maintenance and monitoring plan. Documents change. Processes get updated. Without a system for keeping the index current and periodically checking answer quality, a company brain AI has a shelf life, it starts strong and quietly erodes.

Get these right, and the AI layer on top becomes genuinely transformative. Instant, accurate answers instead of hours lost searching, and less pressure on the two or three people who currently hold all the institutional knowledge in their heads. Skip them, and you’ve built an impressively fast way to retrieve the wrong answer.

What the tech stack actually looks like

There’s no single “right” stack, the best choice depends on company size, budget, and how much control you want over the underlying system. Broadly, the options fall into three tiers:

Off-the-shelf platforms (fastest to deploy, least customizable) Tools like Glean, Guru, Notion AI, Lore, and Sana connect directly to your existing tools (Slack, Google Drive, Confluence, SharePoint) and handle ingestion, indexing, and the chat interface out of the box. Best fit for companies that want something working in weeks, not months, and don’t need deep customization of how retrieval or permissions work.

Modular / build-your-own (more control, more setup) For companies that want a tailored solution, the typical stack looks like:

• Document ingestion & chunking: LlamaIndex or LangChain; frameworks that handle pulling in documents and breaking them into retrievable pieces
• Vector database: Pinecone, Weaviate, or pgvector (if you’re already on Postgres); stores the document chunks in a form the system can search by meaning, not just keywords
• Embedding model: OpenAI, Cohere, or open-source options (e.g., BGE models); converts text into the numerical representations the vector database searches against
• LLM for generation: GPT, Claude, or Gemini via API; the model that actually writes the answer, grounded in retrieved content
• Access control layer: custom-built or via an identity provider (Okta, Azure AD); mirrors your existing permissions so the assistant never surfaces what a user shouldn’t see
• Orchestration/interface: a simple internal chat UI, or increasingly, exposed via the Model Context Protocol (MCP), a newer standard that lets AI agents (like Claude or GPT-based tools) query your knowledge base directly, not just a chat window

Hybrid (most common in practice) Many companies start with an off-the-shelf platform for speed, then migrate specific high-value use cases (e.g., client-facing support, compliance-sensitive knowledge) to a custom-built pipeline once they understand exactly what they need.

The technology choice matters far less than most companies assume. Two companies using the identical stack can get completely different results — one builds it on a governed, well-structured knowledge base and gets genuinely useful answers; the other builds it on the same mess that was there before, and gets a faster way to retrieve the wrong thing.

Where it makes the biggest difference across the business

A company brain AI isn’t a single-department tool but some teams see the impact faster and more clearly than others. Worth knowing where to look first:

• Sales Instant access to the latest pricing, case studies, competitive positioning, and past deal history; instead of digging through old decks or pinging a colleague mid-call. New reps ramp faster because they’re not dependent on tribal knowledge to answer basic prospect questions.
• Customer support The most common and highest-ROI starting point. Support agents get accurate, cited answers to policy and troubleshooting questions in seconds instead of escalating or searching multiple systems; directly reducing resolution time and the load on senior staff who currently field the hard questions.
• Marketing Fast access to brand guidelines, past campaign performance, approved messaging, and product positioning; especially useful for keeping a growing or distributed marketing team consistent, without every new hire needing to be personally walked through the archive by someone senior.
• HR & onboarding New employees can self-serve answers to policy, benefits, and process questions instead of relying entirely on a person’s availability; freeing HR from repetitive queries and giving new hires faster, more consistent answers from day one.
• Engineering & product Searchable access to past technical decisions, documentation, and “why did we build it this way” context that otherwise lives only in old Slack threads or a departed engineer’s memory; genuinely valuable for reducing repeated mistakes and speeding up onboarding for technical hires.
• Legal & compliance Fast retrieval of the current, authoritative version of contracts, policies, and regulatory documentation, with the access-control and citation requirements from earlier in this article being especially non-negotiable here, given the stakes of surfacing an outdated or incorrect clause.
• Leadership & strategy A queryable record of past board materials, planning documents, and institutional history; useful when decisions need context from before a leader’s tenure, or when the person who “remembers why we did that” has left the company.

The pattern across all of these: any team currently bottlenecked by “ask the person who knows” is a strong candidate. The department that benefits most first is usually whichever one has the most repetitive, well-documented questions and the least tolerance for slow answers. Support and sales tend to see the fastest, most measurable wins, which makes them a natural place to run the first pilot.

A step-by-step action plan

1. Audit before you build. Before choosing any tool, map what knowledge actually exists, where it lives, and who currently “owns” it in their head. This usually surfaces the real problem faster than any technical discussion, most companies find the issue isn’t a lack of documentation, it’s three conflicting versions of the same document with no clear authority.
2. Pick one high-value use case to start. Don’t try to index everything on day one. Choose a single, well-bounded area, customer support FAQs, onboarding documentation, or sales enablement material, where a fast, accurate answer would create obvious, measurable value. This keeps scope manageable and gives you a clean way to prove ROI before expanding.
3. Clean and consolidate before you index. Resolve duplicate or conflicting documents, archive anything outdated, and designate a single source of truth for each core topic. This is the least exciting step and the one most commonly skipped, it’s also the one that determines whether the final system is trustworthy.
4. Build access control in from day one, not as an afterthought. Map your existing permission structure (who can see what) before connecting anything to the AI layer. Retrofitting access control after a system is live is far harder, and a permissions mistake here is a real security and compliance risk, not just an inconvenience.
5. Choose your stack based on the use case, not hype. If speed and low maintenance matter most, start with an off-the-shelf platform. If you need deep customization, sensitive data handling, or tight integration with proprietary systems, a modular build is worth the extra setup time.
6. Pilot with a small group before a company-wide rollout. Run it with one team for a few weeks. Track what questions get asked, how often the answers are actually correct, and where the system confidently gets things wrong, this will happen, and catching it early with a small group is far cheaper than catching it after a full launch.
7. Put a maintenance process in place before you scale. Assign clear ownership for keeping the knowledge base current, new documents indexed, outdated ones archived or flagged, and periodic checks on answer accuracy. Without this, even a well-built system degrades within months.
8. Expand deliberately. Once the first use case is genuinely working and trusted, extend the same governed approach to the next area, rather than rushing to index everything at once and diluting quality across the board.

The real opportunity

The technology to build this well now exists, and it’s more accessible than it was even a year ago; modern platforms let you deploy a properly governed knowledge assistant without needing an in-house data science team. The gap between the 71% experimenting and the 17% seeing real value isn’t about who has access to better AI models. It’s about who did the unglamorous work of getting the data foundation right first.

That’s the actual opportunity here not just faster answers, but finally solving the “ask him” problem for good, in a way that scales past any one person’s memory or availability.

If you’re exploring how to turn your company’s scattered knowledge into something your whole team can actually query — reliably, securely, and grounded in your real data — we’d be glad to talk through what that would take for your specific setup.

A Singapore fintech company spends six months and significant budget building an AI model to predict customer churn. The data scientists are experienced. The model architecture is sound. The compute infrastructure is ready. The model goes live — and the predictions are unreliable. Not dramatically wrong, just inconsistent enough that the business cannot act on them with any confidence. Six months of work produces a dashboard that no one trusts.

The postmortem reveals the actual problem. Customer data across three systems was never fully reconciled. Event timestamps were recorded in different time zones across different platforms. A field that should have captured subscription tier was populated inconsistently — sometimes with a product code, sometimes with a label, sometimes left blank. The model was trained on this data and learned its patterns faithfully. The predictions were inconsistent because the data was inconsistent.

This story plays out repeatedly across Singapore’s business landscape as companies accelerate AI adoption. The problem is almost never the AI. It is the data that feeds it. This piece covers what a data foundation actually means, the specific failures that sink AI projects, and what fixing the foundation first — rather than the model — looks like in practice. It draws on the experience of Engine Analytics, a data and AI consultancy in Singapore that builds the data infrastructure AI projects depend on.

What “Data Foundation” Actually Means for AI

Most conversations about AI focus on the model layer — the algorithms, the compute, the training runs. The data foundation sits a level below all of this, and it is what determines whether the model layer can produce anything useful.

A data foundation for AI has four components. Data completeness: the model needs to see enough examples of what it is trying to predict or classify. Data accuracy: the labels and features in the training data need to reflect reality, not how reality was imperfectly recorded. Data consistency: the same concept needs to be represented the same way across all records and all time periods, not differently in each source system. And data recency: AI models making operational decisions need to train on data that reflects how the business operates today, not a snapshot from eighteen months ago before a key system migration.

When any of these four components is missing, the model learns to be wrong — not randomly wrong, but systematically wrong in ways that mirror the gaps in the underlying data. A churn model trained on incomplete customer interaction data will consistently underestimate churn for customers who interact through the channels that were not captured. A demand forecasting model trained on inconsistent SKU coding will produce forecasts that look reasonable in aggregate but are useless at the product level where decisions actually get made.

The Most Common Data Foundation Failures That Sink AI Projects

Siloed data that was never designed to be joined is the most fundamental. AI models need cross-functional views: customer records alongside transaction history alongside product data alongside support interactions. When these live in separate systems with different schemas, different identifiers, and different update schedules, producing a joined training dataset requires reconciliation work that is rarely scoped into an AI project at the outset — and when it is done hastily, the joins introduce errors that the model absorbs as signal.

Inconsistent historical data is a problem that surfaces specifically when training models that need to learn from the past. If the definition of a “converted customer” changed eighteen months ago, or product categories were restructured, or a new data source was added without historical records before a certain date — the training dataset has structural breaks that the model will learn from and replicate in its predictions. The model is not malfunctioning. It is accurately reflecting an incoherent history.

Missing ground truth stops AI projects before they produce anything. Supervised learning models — which cover the majority of practical business AI use cases — need labelled examples of the outcome they are trying to predict. If the business has never systematically recorded which leads converted, which support resolutions were satisfactory, or which recommendations led to a purchase, there is no ground truth to train on. Attempting to build supervised models without it produces systems that are solving a different problem than the one the business actually has.

Poor data governance means AI systems that work at launch become unreliable as the underlying data changes. If the pipeline feeding the model is not monitored for schema changes, volume drops, or field-level quality degradation, the model silently drifts as the data it was trained on diverges from the data it is now receiving. This is one of the hardest failure modes to detect — the model continues to produce outputs, those outputs just gradually become less accurate.

Why Fixing the Data First Is Not a Delay — It Is the Strategy

The natural response to a data foundation problem in an AI project is to work around it — clean the data just enough to run a first model, learn from the results, and fix the underlying issues in parallel. This approach feels pragmatic. In practice, it consistently produces AI systems that consume ongoing maintenance effort without delivering reliable value.

A model trained on incomplete or inconsistent data does not just produce wrong answers — it produces wrong answers in ways that are genuinely difficult to diagnose. The model appears to work. It generates predictions. It may even show acceptable accuracy metrics during evaluation, because the evaluation set has the same gaps as the training set. The problems surface in deployment, or after three months when the model silently degrades, or in exactly the edge cases the business most needs it to handle correctly.

Fixing the data foundation first does slow the timeline to the first model run. It does not slow the timeline to a model that reliably works. The organisations that move fastest on AI — in a way that produces durable business value rather than impressive demos that cannot be operationalised — treat the data foundation as the first deliverable. Not a prerequisite to be resolved later. The first deliverable.

What a Foundation That Can Support AI Actually Looks Like

A data foundation that can reliably support AI has a single source of truth for each core business entity. Customer records, product data, transaction history — each should have one authoritative, reconciled version that AI pipelines read from, rather than being assembled from multiple conflicting systems at training time. This does not require all data to live in one place. It requires a defined, governed layer where the reconciled version exists and where pipelines connect consistently.

It has consistent historical records with documented schema changes. When business definitions change — when a product category is restructured, or “active user” gets redefined, or a CRM migration changes how customer IDs are formatted — that change needs to be documented and applied consistently to historical data. AI models trained across an undocumented schema change will learn the break as a meaningful signal and build it into their predictions.

It has a monitoring layer that catches data quality degradation before it reaches the model. Schema changes in upstream systems, unexpected drops in record volume, new null rates in fields the model depends on — these need to be detected automatically and flagged immediately, not discovered when model outputs start looking unusual weeks later. And it separates raw data from transformed data: raw source records preserved exactly as they arrive, transformation logic version-controlled and auditable, features stored in a layer that is reproducible from scratch. This separation is the foundation of reliable AI operations, and it is covered in more depth in the article on what data engineering actually means for business leaders.

Ready to Build the Data Foundation Your AI Actually Needs?

If your organisation has AI ambitions — or an AI project that is underperforming — the most valuable investment is almost always a better data foundation, not a better model. View our data and AI services, explore our project portfolio, or get in touch and we can assess where your current data foundation stands relative to the AI outcomes you are trying to achieve.

Engine Analytics is a data analytics company in Singapore that builds the data infrastructure AI needs to work reliably — pipelines, governed data layers, quality monitoring, and the consistency framework that keeps model inputs accurate over time.

— Engine Analytics | Singapore’s data and AI consultancy — building the data foundations that make AI deliver what it promises, not just what it demonstrates.