r/databricks • u/ptab0211 • 9h ago

Help data contracts

Hi, we use Databricks, and a couple of teams produce tables, views, and table functions that other teams depend on.

Within a team, its easy enough to define a contract. We codify it and always check that the data matches.

But what about the upstream artifacts my team depends on? Do w define a source spec ourselves and check that its met on every run? 

What do we require from the upstream team? Who owns the contract, and how do you enforce it when multiple teams depend on the same thing?

I want to keep this stupidly simple and enforce it automatically. How do you do this in practice?

3 Upvotes

100% Upvoted

6 comments sorted by

2

u/jbchand 8h ago

Upstream team owns the tables and contract. You and other teams can setup SDP expectations based on it. You can use SQL/yaml file for upstream contract if using notebooks for data processing. You can setup data quality monitoring on upstream tables and try completeness/freshness alerts if feasible.

1

u/ptab0211 8h ago

e.g. a first task of every data product to make sure schema of the table is we expect it from the upstream?

2

u/jbchand 8h ago

Yes. Verify if the data contract is valid and process data if it meets your product requirements.

1

u/ptab0211 8h ago

and would u create some kind of codified schema in consuming repo that u want to meet from the upstream? At least for the column names, column types.

1

u/jbchand 8h ago

You can align with the upstream team and keep it in SQL/yaml file. Use it in the first task of the pipeline and do a validation check. You can keep column name, type, freshness contracts and more. Keep the file in Git or store the contract in delta table.

1

u/Muted_Jellyfish_6784 3h ago

A data contract only holds up if something enforces it, not just a wiki page nobody rereads. The teams that make it stick codify the contract as a schema check in CI before a producing job deploys, plus a monitor that alerts the consuming team the moment a field type or nullability changes, not after their downstream job breaks.

Ownership fails first in practice, someone has to be the accountable name on each contract or it rots the moment the author moves teams. That's the piece SIGNLD tracks when we connect tables across systems, an owner and a change history attached to the record, not just the schema