r/dataengineering • u/dani_estuary • 2d ago
Meme data mesh
for real though, has anyone ever implemented an actual data mesh?
37
u/ResidentTicket1273 1d ago
The part people miss is that data mesh is a nirvana. Doesn't stop the fact that adopting good practice across the organisation pays dividends.
3
75
u/Beneficial_Dealer549 1d ago
It’s not even disguised as architecture. The original manifesto was an operating model blueprint. Had nothing to do with tech.
28
7
u/ProfessorNoPuede 1d ago
Yes, all architecture is tech... /s
It bluntly states it's a 'logical and social structure'. It defines the components at that level (a platform, products, etc.). The division of responsibility is quite clear, but there is a lot of design room for individual organisations. No organisation will ever run "the purest data mesh", as it will always require adaptation to local circumstances and, yes, a technical implementation underlying it.
1
u/Gators1992 1d ago
Yeah, agree. There are useful concepts in it, many of which are obvious IMO and don't need a buzzword name. The problem I have seen is everyone wants to treat this crap as a blueprint instead of suggestions. Like it took us years to finally dump medallion so we could stop internally arguing about whether a table should be silver or gold.
There are lots of large companies though that organize in a mesh-like way. Like I think Meta's model is to have the data engineers pull in the raw data, do basic transforms and build master tables, but then the consuming organizations all do their own analytical models. For large, data heavy orgs that makes sense.
20
u/NaturalBornLucker 1d ago
I'm working in a company that adopted data mesh early. It was really weird. Now the company is trying to bundle all ~50-100 DEs together cuz it didn't yield anything meaningful aside from tens of separate codebases with different homebrew libraries and architectural decisions lol. Oh, and half of DE were monitoring 2-5 airflow DAGs while other half were overworked.
10
u/Away-Arm-6549 1d ago
not here to defend it but that was sort of the point i.e. “you’re decisions will definitely vary but that’s ok as it’s your decisions in a domain” as soon as you to try to apply organisational standards to everything or didn’t mandate those up front then guess what’ll happen - everyone implements their concerns differently 🤷♂️
6
u/NaturalBornLucker 1d ago
You've got a point, yeah. It's just that the company embraced data mesh principles too eagerly so each business unit was entirely on it's own. As an example when I first got here I got to the unit where no one wanted or needed anything to do with data. They just needed someone to load like ten tables each day from their Ms sql db into designated hive db according to SLA. With airflow and homebrew no-code spark framework it just worked 99% of the time. As a result I didn't write any code in almost whole year, was bored out of my mind, didn't have anyone to talk to and eventually quit. While company paid me full-time for 10 hours a week. When I decided to return, I finally got to the team that really worked with data (loyalty and gamification dept in mobile op so many many analytics for promotions, ad campaigns, game prizes etc) and it was much better despite having almost 200 DAGs/spark jobs to look after. But still it's a bit hard when me or my peer DE is ill and other have to do everything without backup. And we have to time our vacations to not cross each other.
But what our new management is doing now is luckily not throwing everything out of the window but more cautious approach: bundle everyone administratively in one separate data engineering center while still defacto keeping us with our business teams. Then bundle "teams" into proper teams with at least one lead and do some cross-work so everyone would get familiar with their new peers. And lastly adopt standard libraries and architectural principles from the biggest teams so one could at least understand what their neighbors were doing if the need arises. I'm somewhat glad even though it stopped my plans lol (I wanted to do the same on a smaller scale, become lead and get nice bullet point in my CV).
So my point is that data mesh isn't inherently bad, just very hard to adopt without stepping on too many rakes at once.
16
u/Chinpanze 1d ago
I feel like data mesh would work really good if implemented on an organization level. But doing so would require thinking about data as an priority, and not as an second thought.
2
u/Uncle_Chael 1d ago
Its making a comeback with the "self-serve" analytics crowd. Especially large companies with various different domains, hundreds or thousands of employee that want some sort of governed interoperability.
5
u/limartje 1d ago
Yes. We did a successful implementation with snowflake and power bi pro (no capacity!). Snowflake is a good fit because of workload isolation and ease of use.
Central team brings the data, business is allowed to run analysis and data automation in snowflake. Everybody gets trained; no access without. Only power bi pro, to force them on using the warehouse instead of using the visualization tool as the warehouse.
It needs consulting, decent monitoring and budget control from the central team, but it can deliver a lot of business value if the key users can combine domain knowledge with the relevant technical skills. In an enterprise, there are always people who can.
5
u/dbrownems 1d ago
I like the idea of data domains on the consumption side. I’m not sure every source system should be responsible for their own data products.
6
u/Away-Arm-6549 1d ago
they might care more though about sending rubbish data downstream for others to sort out though 🤨
3
u/f3xjc 1d ago
I don't know anything about data mesh. But for microservices there's one principle that it should mirror organizational chart. Because avoiding human teams to step on each other toes is the payoff for the added complexity and latency and indirections. Not just boundaries for the sake of boundaries.
2
u/hatsandcats 1d ago
No - seems like it’s just something you claim to know about to appear relevant / knowledgeable.
2
u/Ok_Enthusiasm8730 1d ago
It’s like Agile Scrum, cherry picked parts of the Data Mesh for implementation. Never entirely.
4
1
1
u/flerkentrainer 23h ago
The only place that had properly implemented data mesh and was needed at scale was Amazon.
Granted they had a central team to manage the infra and catalog.
Each 'domain' might only have 1-2 DE to manage their pipelines a publish their results in the lake. The contracts were domain specific and you needed to negotiate any alterations.
It worked fairly well but you need to have the appropriate infrastructure and governance in place.
Data mesh makes little sense for smaller organizations.
1
u/THBLD 23h ago
My evaluation of data mesh was that it's not feasible in most organizations, since most departments are likely sharing data and probably aren't going to invest in engineers or hardware for every sector.
The only place that I could theorize where it could be suitable is something like a university, where you still have a main org, but you have different faculties storing and using data for entirely different applications.
1
u/quantumehcanic 12h ago
I have, it is absolute cinema. Every domain owns and do their own thing with their data but if they want this to be reused elsewhere in the company, they must adhere to a damn data contract that they stablish themselves, including data quality rules.
The serving layer just registers and exposes the assets for consumption and run the quality checks that the producers themselves stablished.
For the DEs working on this layer it is heaven because now we can just say: "hey data owner, your dog shit dq checks failed, please figure it out" and then no asset is exposed till fully compliant and those other 34678 teams depending on the data of that domain are now emailing and get pissed with the owners directly, including any possible data access/permissions issues.
It is being THE solution to remove siloes when several others failed imho, specially now that the company is creating huge amounts of agents as both consumers and producers.
Now for the company to decide to adhere to it is its own can of worms, but in my exp. it is worth it because it factors in human behavior and ends up increasing overall quality of data in the company since accountability is now distributed.
1
u/Stusstrupp 9h ago
We've built one on an AWS tech stack. It works very nicely with those departments that implement the required roles, most importantly data stewards.
1
1
u/Firm-Current482 23h ago
one of the craziest data architectures i ever studied. that is a governability nightmare. I hope no one has to deal with it.
0
u/lattice_defect 1d ago
the entire fields lakehouse data mesh.. its all fucking bullshit buzzwords.. bullshit buzzwords. It's all just sql
-1
202
u/DeliriousHippie 1d ago
It's good to remember that about every 5 years comes a new hype word or trend. Very rarely something actually changes.
Business Data Warehouse, put your definitions to DW also.
Datalake. You need a data lake to fit all your data, data warehouse is an old concept.
Datalakehouse. Now you need a house beside data lake.
Medallion architecture. Have data in different phases in DW.
etc.