Semprini

Hello, people of the internet, browsing random IT blogs - and this one with a Gilbert and Sullivan reference! We are indeed esoteric and interesting.

Confused? Excellent. That's a great place to start. To get caught up, check out the previous blogs:

• Data Products: Save $150 million, Reduce risk, Increase speed
• Data Products: Architecting Data Autonomy

We're into the engineering here, so strap in guys, gals and non-binary pals, this one's gonna get technical!

But first, hats. As the Very Big Corporation of America research shows, we're not wearing enough hats...

"Hats? What are you on about Semprini, you disconcerting IT anecdotist?" I hear you ask.

Metaphorical hats, my dear reader. Metaphorical.

The quickest way to deliver code is a straight line, but let's put our framework hat on. Should this part use or extend a library or be written generically? Will it be quicker in the future for you and others to make this bit into a separate code repo? Is there an engineering pattern here which we can create a library for?

Next, let's put our platform hat on. Is this part suited to be a reusable platform feature? If a platform team was responsible for this bit, and we added automation of it for others, will this speed up the business? Talk to your local, friendly, architect about platform capabilities.

It's these changes in culture which should snowball the ecosystem - if the engineers are supported and guided to make things better. If we only ever wear our project hat, we can't create an ecosystem of improvement. Funding of internal improvement is rarely as sexy to the business as direct customer initiatives and when the budget gets tight, we all know what gets dropped so it's up to each of us to wear these hats with aplomb.

Data products are applications/workloads grouped into domains and deployed on platforms. So lets fire up those platforms! Here's two options for our platform which can be used for our sample domain - Digital:

I don't care what platform makeup is right for you - I'm purposely ignoring my platform hat for this blog. Today, I care about the capabilities we need for data products and having a standardised way to automate those capabilities.

A design decision I'm glossing over here is what a domain actually is for us. Data products belong to domains but is this:

• logical? (tag your data products with the domain, and use techniques like security groups)
• physical? (separate infrastructure, like I'm using here)
• hybrid? (some domain specific systems and some enterprise wide)

This decision will depend on the size, requirements and makeup of your organisation but because I want this to be repeatable by any curious engineer, i'll do this just using docker compose. Therefore in my example we have:

• Flink
• Kafka
• PostgreSQL
• Pg GraphQL
• Ducklake
• Minio
• Trino
• APICurio

Here's the repo for all this. The readme has a quickstart to fire up this platform.

This infrastructure gives us a set of capabilities for our data products in our chosen Digital domain which cover all the functions that your organisation will be doing anyway, just maybe in different teams. Consider an Inverse Conway to avoid nasty silos - if you have the right backing from senior leadership.

One important omission from the platform is the ubiquitous DBT or an equivalent, and some engineers are probably thinking this Semprini chap is a few clowns short of a circus. We will rely on DBT for creating experiences (E.g. data marts) and dimensional models which is outside the scope of this blog but here the data is available in the storage engines within half a second of the change happening in the source - it's a different cadence than many analytic experiences which will happen down-stream from this.

Hopefully, it's also obvious that this domain template is a prototype. I'll share a fully worked version another time and I enjoy prototyping like this, but cloud data providers like Snowflake, Databricks, Confluent etc are usually the right answer.

"Wait a minute Semprini, what about that Gilbert & Sullivan reference in the title?" You inquire. Very perceptive dear reader, the title comes from the fact that master/canonical data products should be declarative, so they are governed as a guardrail at the top of the hill, not as an ambulance at the bottom. The best way to do this is through model/metadata driven generation.

This declarative approach enables DATSIS: Discoverable, Addressable, Trustworthy, Secure, Interoperable, and Self-Describing foundational data products which in-turn unlocks citizen data engineer types of data products which consume these foundational ones. A declarative approach should not slow down any delivery, we're just shifting left and progressively using automation which will speed us up massively.

Because we put on our framework hat, our declarative approach will create a reusable template for future foundational data products. I.e. the 2nd foundational/canonical data product will be much easier than the 1st and all data products which use this template will automatically get features added (in each products next release) as we enhance the template. Our foundational data product template will require a standard set of artifacts to run. The template in the repo has:

- products/<slug>/
  - product.yaml: names every resource the product owns
  - odps.yaml ODPS v4 manifest for the catalogue, listing the four interfaces
  - ods/
    - 010\_schema.sql schema, pg\_graphql directive, shared trigger function
    - 020\_tables.sql one CREATE TABLE per entity, foreign keys, association tables
    - 030\_api.sql grants for the API role and the product's GraphQL RPC endpoint
    - 040\_seed.sql optional sample rows
  - graphql/
    - schema.graphql documents the reflected GraphQL surface
    - smoke.graphql query run by `test`
  - events/schemas/
    - <table>.schema.json JSON Schema of the canonical record, one per table
  - flink/
    - job.sql inbound topic -> ODS upsert, ODS CDC -> outbound topic
  - lakehouse/
    - 010\_replication.sql registers each ODS table with pg\_duckpipe
  - trino/
    - smoke.sql one count per table through Trino

To generate these artifacts, we need a system which understands a logical data model and can combine with a set of rules/standards to produce multiple physical level artifacts. Let's put on our modelling hat and write up our model for an example foundational data product which we're calling "Digital Core":

In the repo the full model is made using markdown and mermaid as it's part of an AI native modelling approach. More to come on this too but for today, I don't care what modelling tool you use, I care about the way you model.

The data product definition in the model points to the entities, modelled at a logical level, and template. I have 2 systems which use a logical model for model driven generation:

• Markdown Data Definition Language (MD-DDL)
• pyMDG

Key to our data product today is found in the repo under data_model/data_products/canonical.md

md-ddl comes with a set of agents. Agent Ontology to help you model but here we use Agent Artifact which understands how to generate the required artifacts from the model and template. Here is the Claude Code prompt I then used to generate:

/agent-product We now have a platform, a template and a data model. Use the canonical data product definition with its linked entities to create the data product in the build folder.

Then we deploy and test our freshly minted canonical data product:

uv run semprini-blog-dataproducts deploy products/canonical-party
uv run semprini-blog-dataproducts test   products/canonical-party

Your data product awaits on any of our designed interfaces.

This is indeed a model of model driven generation.

One piece of awesomeness which I want to highlight, if you'll indulge me - even though I'm not showing it in this blog is what becomes of raw to refined layers which are ubiquitous in data lakes. When working with a foundational/canonical data product, data is already in canonical 3rd normal form - this is what is enforced on the ingress interfaces. We do the raw to refined transforms in-stream which is then published to the foundational data products. This means we can get much groovier with the lake-house, by providing a set of views which flatten our 3rd normal/canonical form into dimensional or wide column format. We can even use meta-data to algorithmically transform to a dimensional model on the fly!

Yet another form of model driven generation! Huzzah!

A huge advantage of using Lakehouse storage (DuckLake, Iceberg, Deltalake and Snowflake tables too) is time travel. We get this for free in these platforms and here every pg_duckpipe flush commits a snapshot and the history is queryable. This simplifies our Slow Changing Dimensions for this data product class but downstream data products will likely use temporally tracked SCDs. Data products act like lego bricks, so design your awesome lego brick template which can generate data products which plug-in to the ecosystem.

Most organisations I see are so close to being able to unlock all the goodness. You're already doing all this, just in siloes! Your data model should be a nexus - a place for BAs, Stewards, Engineers and Modellers to collaborate.

This was the 3rd entry in a data product journey from strategy, through architecture, to implementation. We have worn a lot of hats, haberdashers would be proud! and I hope this soup to nuts vision of data products is compelling.