Building the governed agentic data foundation with Unity Catalog and Iceberg
Workday’s solution centers on two key technologies working together: Apache Iceberg™ as the open table format and Databricks Unity Catalog as the unified governance layer. Together, they create what Workday calls its “universal data layer”: a foundation where AI agents access current, governed data without creating additional replicated stores.
Apache Iceberg™ provides the transactional guarantees that enterprise AI agents demand. Its ACID compliance and snapshot isolation give agents a consistent, stable view of data while they reason and act, which is critical when multiple agents query the same datasets simultaneously. But Iceberg’s most transformative contribution, according to Majumder, was cultural: “Iceberg brought oil and water together. Data science and ML engineering teams want to innovate and experiment really fast — hypothesis testing and validation is very important to them. But platforms and data engineering teams, their work is fairly deterministic. Iceberg was a format equally embraced by two opposing schools of thought. Oil and water mixed; they fused together.”
Unity Catalog serves as the centralized governance control plane, delivering fine-grained access control, end-to-end lineage, comprehensive auditing, and consistent policy enforcement across all data and AI assets. Majumder illustrates the necessity with a concrete example: “An account executive who works in the North Territory shouldn’t be able to ask the agent to tell them the account status or health of the South Territory. The agentic interface should know the entitlement of that individual and block the information they are not entitled and authorized to see. You shouldn’t be able to prompt inject it and get information you are not authorized to see. That would be a debacle, especially when operating with highly sensitive data.”
“Unity Catalog was a key contributor to achieving true enterprise scale for us when it comes to compliance, security, observability, lineage, tracking, and monitoring,” says Majumder. The architecture also incorporates long-term memory (stored in the Iceberg layer as the persistent DNA of agents) and short-term session memory (vector stores for contextual interactions), with reasoning traces captured for auditability, ensuring that if an agent’s output is ever questioned, the team can “open the hood” and trace the rationale end-to-end.
The modular design philosophy is intentional. “We have to be very deliberate to design our systems modularly,” Majumder explains. “We have to think like we are builders of a Lego toy. You take the parts and build a toy. But equally, if a part breaks down in future, you should be able to swap it very quickly. That’s very important because we’re still in uncharted territory.”
Leave a Reply