Centralised or distributed?

2024-09-20

We have witnessed quite a few discussions on how the data functions should be organised. The key point on centralised vs. decentralised data team. Both views have their ardent supporters.

What is often missing in such discussions is the context.

One team to rule them all

When you’re a small startup with 50 or 100 people, chances are that your whole data team is 1-3 data analysts and maybe 2 data engineers. They work closely together, handling everything from data pipelines to dashboards. Collaboration within the team is easy, people share the same goals and support each other. The focus is on moving quickly and delivering results. Distributing these few people or splitting them to data engineering team and data analytics team would result in forming teams of 1 or 2 and would do more harm than good.

But as your company grows further, things start to shift. Requests for data become more frequent and complex. Multiple teams across the company — from sales, marketing, and product departments — are all relying on data insights, often at the same time. This is when the central, single data team begin to feel stretched, and that once-smooth collaboration gets strained.

A centralised team, where all data engineers and analysts report into one function, works well with a manageable workload. But as the organisation expands, this structure can become a bottleneck. The data team gets overwhelmed, and other departments are left waiting for insights, slowing the business down.

Let’s have more

At this point, decentralising your data function may seem logical. It is realised by embedding data analysts, perhaps also data engineers, into departments (Product, Marketing, etc.). Product gets its own data experts, marketing gets theirs. They know their specific business domains. As a result the decisions on on the department level can be made faster. Teams no longer have to wait in line for a centralised data team to clear its backlog — they are getting autonomous. This agile approach can help larger organisations move faster.

However, decentralisation isn’t without risks. Scattering data teams across the business can lead to inconsistencies in how data are managed or how the metrics are defined and calculated. Without strong (data) governance, departments might use different tools or definitions, leading to confusion. You solved one problem — bottlenecks — but potentially created another.

Then, there is the question of whether to split the data function itself. In smaller companies, data engineers and analysts often work together, wearing multiple hats as needed. It fosters collaboration within the team, lets the team members understand the broader contexts and gives tangible benefits. But as your company scales, the data team grows and the complexity of the data operations increases, specialisation becomes essential. It is time to split. Data engineers focus on data infrastructure or data platform while analysts keep up with the increasing demand for insights. Creating more specialised analytics teams — for sales, marketing, and product departments — may take place at this stage as well. This move prevents bottlenecks and improves efficiency but requires coordination and comes with the price of overhead for communication between the teams.

Think before you copy

The key takeaway? The structure that works for a 100-person startup won’t work for a company with 1000 or more employees and vice versa. The data and analytics needs of a 100-person tech startup, where majority of the staff works on a sophisticated digital product, are much different from e.g. a transport company of the same size, but where most of the employees are truck or bus drivers. Trying to replicate the structure of a successful player much larger than you does not lead to success, but to costly troubles. The best setup depends on your company’s size, business domain, growth stage, and data needs. Don’t just blindly copy another company’s structure — adapt yours to your needs and change it as you grow.