On data platforms · 8 min read

The Data Lake Promise Might Finally Be Coming True

The data lake was supposed to be the last data project you ever ran. One place for everything, ask any question, answer it yourself. Mostly it became the place data went to be ignored.

I spent a decade building data platforms at Google, Meta, and Airtable. The reason lakes failed had nothing to do with storage or compute. It was context, and context is finally getting cheap enough that the original promise might actually happen.

TL;DR

• Enterprise app data (Salesforce, ServiceNow) is built to be read through the app, not through SQL. Dump it in a lake and you get thousands of tables of GUIDs and integer codes only the app's own admins can read.
• The companies that got value from their lakes (Netflix, Uber, Airbnb) hand-built a context layer on top: decoded codes, defined metrics, real joins, access rules. Same storage, opposite outcome. It just cost a fifty-person platform team.
• Every app vendor now sells its own data cloud and AI agent (Salesforce Data 360, ServiceNow Workflow Data Fabric, SAP, Workday). That is proof the prize is context, not storage. It is also several clouds, each with its own version of the truth.
• AI is what finally makes the context layer cheap: it can build it and read it. Anthropic took the same model from no higher than 21% to consistently above 95% just by adding context. The promise is coming true. It just does not belong to the lake, or to any one vendor.

We called it a lake. It was a landfill.

The first time I really understood the problem, I was staring at a schema browser.

We had done the modern thing. Pointed Fivetran at Salesforce and ServiceNow, replicated it all into the warehouse, and told the analytics team the data was in the lake now. Go self-serve.

Then someone asked a simple question. How many incidents did we resolve last quarter? The answer was sitting in a column called state, and the value was 6. Not "Resolved." Just 6. The dictionary that turns 6 into "Resolved" lives in a different ServiceNow table called sys_choice [4], and nobody had told Fivetran to bring it [9]. A stock ServiceNow instance ships with thousands of tables and more than a hundred thousand field definitions before anyone customizes a thing, and we had faithfully copied the ones that were unreadable and skipped the handful that explained them.

Salesforce was worse, because Salesforce is personal. Open the replicated Opportunity table and half the columns end in __c, the suffix Salesforce puts on custom fields. One object can hold up to 800 of them [6]. Most are empty. None are labeled, because the human name ("Annual Contract Value") lives in Salesforce metadata and what lands in the warehouse is the API name [7], something like ACV_Renewal_2019__c. And whether a deal even counts as won is not in the row at all. It is in a separate object called OpportunityStage [8] that nobody joined.

We had not built a data lake. We had built a landfill of sys_ids [5] and __cs, and handed the analysts a shovel.

The meaning lived in metadata the replication left behind.

How the lake was supposed to work

The idea was beautiful, and for the record it was not a bad one. James Dixon coined "data lake" in 2010 [1]: instead of the warehouse's bottled water, cleansed and packaged before you could drink it, keep the data in its natural state and let anyone dive in. Hadoop made the storage almost free. Schema-on-read meant you could load first and work out the meaning later. One place for everything, no upfront modeling, cheaper than Teradata. For a company staring at a seven-figure warehouse renewal, it sold itself.

Gartner saw the problem before most companies had finished their rollout. In 2014, in a note titled "Beware of the Data Lake Fallacy," it warned that data lakes "typically begin as ungoverned data stores," and that meeting the needs of wider audiences requires "curated repositories with governance, semantic consistency and access controls" [2]. Read that again. In 2014 the analysts had already named the exact thing that was missing, and it was not capacity. Everyone loaded first anyway, and mostly never modeled. The word for what you got showed up fast: a data swamp. (You will hear that 85% of these projects failed. That number is a Gartner analyst's off-hand 2017 estimate that was never published as research [3], not a study you can cite. The duller, real version is that most of them underdelivered, and anyone who lived through it will tell you so without a slide.)

The problem was never the storage

The swamp taught me something that took a few years to really land. Enterprise applications are designed to be read through the application, not through SQL.

Salesforce and ServiceNow are both metadata-driven. The row you export is deliberately context-free. The meaning lives in a separate layer: the picklist that says which stage counts as won, the choice table that says 6 means "Resolved," the reference that says assigned_to = 5137153cc6... is a person named Priya [5]. Inside the app you never see any of that. The app resolves it for you. Copy the tables into a lake and you take the half without meaning and leave the half with meaning behind.

So the analyst rebuilds it by hand, one query at a time. Decode the codes. Guess the joins. Remember to filter out the soft-deleted rows the app used to hide automatically. And then argue about definitions, because there is never just one. I have sat in the meeting where Finance says ARR is 42 million, Sales says 44, and Product says 41, and all three are technically right, each pulling a different stage filter out of a different dashboard. The raw table cannot settle that argument. It does not contain the answer. The answer was context, and context is what we threw away at the door.

Why the big companies got value anyway

Some lakes did work. Mine, eventually, did. And the pattern was always the same, whether I saw it up close or read the engineering blogs of the teams doing it best.

Netflix built Metacat so meaning was searchable across dozens of stores [10], and Iceberg so the tables behaved [11]. Uber built Databook to track what every dataset meant and who owned it [12]. LinkedIn built DataHub [13]. Airbnb built Minerva, a metrics layer with more than 12,000 governed definitions [14], so a search for "revenue" returned the blessed number instead of the ninth copy of it. None of these teams had better storage than anyone else. They ran on the same S3 and the same open-source engines as the companies with swamps. What they had was a context layer, hand-built on top, and a large platform team paid to keep it alive.

That is the part that stings. For fifteen years the fix was known. It just cost a fifty-person team, which meant only a handful of companies could afford it. Everyone else got the landfill.

Same storage, opposite outcome. The winners hand-built the context layer: Metacat, Databook, DataHub, Minerva.

The vendors figured this out too

Lately you can watch the whole industry arrive at the same conclusion at once, and it is worth watching where they are taking it.

Salesforce spent three years turning its CRM into a data platform. Genie became Data Cloud became Data 360 [16]. It launched a zero-copy network in 2024 to federate over Snowflake and BigQuery, with Databricks soon after [15], bought Waii for natural-language-to-SQL [17], and put Agentforce on top so you can ask its data questions in plain English. ServiceNow ran the same play from the other side: Workflow Data Fabric in 2024 [18], a high-performance database called RaptorDB Pro, zero-copy connectors to eighteen platforms, and in 2025 it acquired the data catalog data.world [19] for exactly the governance-and-meaning layer it was missing. SAP, Workday, and Microsoft are all doing versions of the same thing [20].

Notice what every one of them is actually selling. Not storage. A governed context layer, with an AI agent on top, over your data wherever it lives. That is the data lake's promise, restated word for word by the incumbents. Which is the good news and the catch in the same breath.

There is a catch worth naming. Each of these clouds is designed to be that layer inside its own walls. Zero copy keeps your data where it is, which is genuinely better, but the definitions, the governance, and the agent all live on the vendor's platform. Adopt several and your context ends up spread across as many clouds as you have vendors, each with its own version of the truth. It is a real step up from the swamp. It is not the same as owning one layer over all of it.

Four vendor clouds, or one open layer that belongs to you.

Your definitions should belong to you, not to whoever hosts your data this quarter.

So is the promise finally coming true?

I think it is. Just not the way the 2010 slide imagined, and not for the reason the vendors want you to believe.

What changed is not storage, which was solved a decade ago, and it is not a bigger model. What changed is that context finally got cheap. An LLM can read a schema and draft the definitions, decode the picklists and the choice codes, propose the joins, and label the unstructured documents that were always most of the lake and almost none of its value. The work that used to need a fifty-person team now takes a small one and a few weeks. And the same AI that builds the context layer can read it. Anthropic reported that giving its own model a real context layer took it from answers that "didn't exceed 21%" correct to "consistently above 95%" [21]. Same model. The difference was context, not scale. Handing it raw access to thousands of old queries, for what it is worth, barely moved the number at all.

The fix was not a bigger model. Anthropic's own analytics, same Claude.

The part I care about is who owns that layer. It does not have to be the lake, and it does not have to be Salesforce. Your definitions, your access rules, your join logic, the decoded meaning behind every state = 6, are yours. They can live in your own repo, ride open standards like Iceberg [11] and MCP [22], and sit above your data wherever it happens to be, including inside several vendor clouds at once.

How Agami solves it

That layer is what we build at Agami. Not a new place to log in, and not a rip-and-replace of your lake or your warehouse. Agami is a governed context layer that sits between the AI assistants your team already uses, Claude, ChatGPT, Copilot, and the data you already have.

It does the three things the fifty-person platform team used to do by hand. It reads your schema and drafts the context layer for you: the definitions, the decoded codes, the joins, the metrics. So state = 6 comes back as "Resolved," and there is one definition of ARR instead of six. Your team reviews and signs off on it, so nothing goes live on the model's guess. That layer is governed, so every answer honors your access rules and arrives with the exact SQL that produced it. And it lives in your own repo, over open standards, which means it is yours to keep and it works over your data wherever that data sits.

Then anyone asks in plain English, right inside the assistant they already open every morning, and gets an answer that traces back to your definitions.

You do not have to take my word for any of this. Agami runs on your own machine, and you can try it in a few minutes. Point it at a sample dataset if you would rather not connect anything yet, or at your own database if you would. It introspects the schema, builds the semantic model, and starts answering questions in plain English, locally, with nothing leaving your laptop. The code is on GitHub at github.com/AgamiAI/agami-core.

The data lake was never really the problem, and it was never really the answer. It was a place to put data. The promise, the part about anyone asking anything and trusting the number, was always about context. Fifteen years later we can finally afford that context, and better still, we can own it. Your definitions, your access rules, the decoded meaning behind every state code, living in your own repo and travelling with you across every tool and every cloud. That is the layer worth building. The lake was just where we learned we needed it.

Frequently asked questions

Q: Why is Salesforce or ServiceNow data so hard to use once it is in a lake or warehouse?

A: Because those platforms are built to be read through the application, not through SQL. The exported row is deliberately context-free: the meaning lives in a separate metadata layer. In ServiceNow, an incident's state = 6 only means "Resolved" if you also have the sys_choice table, and a reference like assigned_to is a 32-character GUID, not a name. In Salesforce, custom fields are cryptic API names ending in __c, and whether a deal counts as won lives in a separate OpportunityStage object. Replication copies the data and leaves the meaning behind, so a semantic layer that re-attaches it is what makes the data answerable again.

Q: What is a data lake, versus a warehouse, versus a lakehouse?

A: A warehouse stores structured data you model before loading (trustworthy, slow to change). A data lake stores raw data of any kind and defers structure to query time (cheap, flexible, easy to turn into a swamp). A lakehouse adds warehouse-grade features (ACID transactions, schema enforcement, performance) on top of lake storage. All three solve where data lives. None of them, on their own, define what it means.

Q: Do I still need a data lake if Salesforce, ServiceNow, and SAP now offer their own data clouds?

A: Their data clouds (Salesforce Data 360, ServiceNow Workflow Data Fabric, SAP Business Data Cloud) prove the prize is a governed context layer plus AI, not storage. Each one is designed to be that layer inside its own walls, with its own semantics, agent, and pricing. Adopting several can spread your context across as many clouds as you have vendors. An independent context layer over open standards keeps the definitions and the control with you.

Q: Can an AI agent just query my data lake directly?

A: It can, and on an ungoverned lake it will confidently return wrong answers: hallucinated joins, the wrong metric, undecoded codes, and data a user was not allowed to see. The fix is not a bigger model. When Anthropic added a governed context layer, the same model went from answers that "didn't exceed 21%" correct to "consistently above 95%."

Q: How do I finally get value from a data lake I already paid for?

A: Add the layer it never had. A governed context layer supplies the definitions (what the numbers mean), the decoded codes and joins, the access rules, and the lineage. AI now makes that layer cheap to build and keep current, and lets anyone query the result in plain English, so the storage you already bought becomes a trustworthy, permissioned answer engine.

References

1. James Dixon's Blog, "Pentaho, Hadoop, and Data Lakes," 14 October 2010.
2. Gartner, "Gartner Says Beware of the Data Lake Fallacy," 28 July 2014.
3. Brian T. O'Neill, Designing for Analytics, "Failure rates for analytics, BI, IoT, and Big Data projects = 85%."
4. ServiceNow, "Incident Management State Model (KB0564465)."
5. ServiceNow Docs, "Unique record identifier (sys_id)."
6. Salesforce Help, "Custom Fields Allowed Per Object."
7. Salesforce Help, "Difference between 'API Field Name' and 'Field Label'."
8. Salesforce Developers, "OpportunityStage Object Reference."
9. Fivetran Docs, "ServiceNow connector."
10. Netflix Technology Blog, "Metacat: Making Big Data Discoverable and Meaningful at Netflix," 2018.
11. Apache Software Foundation, "Apache Iceberg."
12. Uber Engineering, "Turning Metadata Into Insights with Databook."
13. LinkedIn Engineering, "Open Sourcing DataHub."
14. Airbnb Tech Blog, "How Airbnb Achieved Metric Consistency at Scale," 2021.
15. Salesforce, "Salesforce Unveils Zero Copy Partner Network," 25 April 2024.
16. Salesforce, "Welcome to the Agentic Enterprise (Agentforce 360 and Data 360)," 13 October 2025.
17. Salesforce, "Salesforce Signs Definitive Agreement to Acquire Waii," August 2025.
18. ServiceNow, "ServiceNow Introduces Workflow Data Fabric," 23 October 2024.
19. TechCrunch, "ServiceNow acquires Data.World," 7 May 2025.
20. SAP News, "SAP and Databricks Debut SAP Business Data Cloud," 13 February 2025.
21. Anthropic, "How Anthropic enables self-service data analytics with Claude," 3 June 2026.
22. Anthropic, "Introducing the Model Context Protocol," 25 November 2024.

Try Agami

See it on your data.

Point Agami at your lake, your warehouse, or your Salesforce, and watch it answer real questions, governed and auditable, in plain English. Or run it yourself, locally on your own machine.

Book a demo → View on GitHub

About the author

Sandeep Kachru is the co-founder of Agami AI. Before Agami, he spent over a decade building data platforms at Google, Meta, and Airtable.