Finance used to be bottlenecked on producing content. It is now bottlenecked on verifying it. An analyst who once spent the day building a model now spends it checking whether a machine’s output can be defended to a client, a committee, or a regulator.

Citations don’t solve this. A citation is a post-hoc audit trail — it tells you where a number allegedly came from, not that the number is right. This 22-minute talk covers the architecture we built at Kepler instead, and the argument for why it has to be architectural.

Evals Are Not Verification

The industry’s default answer is to add evaluation layers. That helps, but it doesn’t change the category of the thing you’re holding: you cannot take a non-deterministic LLM and eval your way to something deterministic. Stacking probabilistic checks on a probabilistic system gets you better odds, not proof.

The alternative is to separate reasoning from fact-production by design.

Three Pillars

Atomic provenance. The model never manipulates numbers. It identifies sources and delegates extraction to deterministic tools, and anything that can’t be confirmed against a source gets stripped. A model that never touches a numeral cannot hallucinate one.

Scope determinism. The model decides what to compute; it never does the computing. Every operation routes through deterministic code — database queries, XBRL parsing, financial algorithms. There’s no good reason for basic arithmetic to run through a billion-parameter model instead of a CPU.

Derivation chains. When a metric isn’t stated in a filing and has to be derived, the system records the full lineage from inputs to output. Any number can be replayed, re-verified, and audited — which also lets firm-specific definitions travel across desks instead of living in one analyst’s spreadsheet.

Why Finance Specifically

Other high-stakes fields layer their verification. Software has tests and staged rollouts; aviation has checklists and redundant systems; emergency medicine has protocol and second reads. Finance frequently has one overworked analyst.

Bloomberg and FactSet partly solved this socially — when everyone uses the same data, culpability is distributed. But they’re read-only. The work product still gets assembled by hand. Closing that gap without giving up verifiability is the problem we set out to solve.

Questions about verifiable AI or the architecture behind Kepler? Feel free to reach out.

Most companies hire forward deployed engineers as an extension of go-to-market: a technical body to keep a large account happy. That’s the wrong frame. FDE is a product strategy. The job is to find the real problem, ship the smallest thing that solves it, and turn what you learned into leverage for the product.

This 22-minute talk covers how we practice that at Kepler, built on what I learned running Project Frontline at Palantir and leading FDE practices at Citadel.

Proximity Discovers Truth

Building in isolation is expensive in ways you don’t find out about until production. A data store I worked on early in my career was designed without understanding that real customer data was full of null timestamps — which defaulted to the Unix epoch, generated millions of unwanted key spaces, and blew the memory budget by orders of magnitude. The information existed. Nobody was close enough to the data to see it.

Solve the Small Problem First

A shipping customer once sent a 47-page requirements document asking for custom dashboards and fourteen metrics. Four months of scoping went into it. Then an engineer asked a dispatcher on site what the first thing they did on Monday morning was, and learned the actual need was a Slack alert for late trucks. That took hours, not months — and it bought the trust to do the bigger work.

Watch Workflows, Not Words

A data quality engineer resisted a migration to Parquet for a year, insisting the format was worse. She wasn’t wrong about her experience: she was downloading CSVs and opening them in Excel, and Parquet broke that. The fix wasn’t a better argument, it was a Parquet viewer. She approved the migration two days later, and pipeline runtime went from 17 hours to two.

Define the Language

Sales says “customers,” ops says “clients,” finance says “billing entities,” and all three mean something slightly different. Finding the contested nouns and canonicalizing them is real product work — when a customer adopts your product, they adopt your vocabulary. Palantir’s ontology mattered for exactly this reason.

Every Hack Is Permanent

The most dangerous phrase in this job is “this is just temporary.” Undisclosed shortcuts end up running in production for years with no owner. Ship every fix as though it will run for eighteen months, because it will.

Questions about FDE or how we run it at Kepler? Feel free to reach out.

Most people now use ChatGPT, Claude, or Gemini every week without a clear mental model of what’s happening underneath. That’s fine right up until the model is confidently wrong and you have no way to tell.

AI Model Basics is a new 20-minute LinkedIn Learning course aimed at exactly that gap.

What the Course Covers

• How large language models actually work — enough mechanism to reason about them, without the math
• What prompting does and why it changes the output as much as it does
• Why models produce wrong answers, and what kinds of wrong to expect
• Personalization — what these tools can and can’t retain about you
• How to evaluate and improve what you get back instead of accepting the first response

Who This Is For

Anyone using AI tools at work who wants to understand them rather than just operate them. It’s beginner-level and assumes no technical background — no coding, no prior AI experience.

If you’re an engineer looking for the production-systems version of this material, Fundamentals of AI Engineering is the deeper course.

Why It’s Short

Twenty minutes is deliberate. The goal isn’t comprehensive coverage — it’s giving people enough of a model to be appropriately skeptical, which is the single highest-leverage thing a non-specialist can learn about this technology.

You can find the course on LinkedIn Learning.

Questions? Feel free to reach out.

Most AI projects don’t fail at the model. They fail somewhere between the demo that impressed everyone and the system that has to survive contact with a real customer’s data, workflows, and definitions.

I published a new LinkedIn Learning course on the role that closes that gap: Forward Deployed Engineering in the Age of AI.

Why This Course Exists

AI deployment differs fundamentally from a traditional SaaS rollout. A SaaS product is largely the same on day one for every customer. An AI system is not — its behavior depends on the customer’s data quality, their vocabulary, their edge cases, and their tolerance for being wrong. That means the work of making it succeed happens in the field, not at headquarters.

Forward deployed engineering is how that work gets done. It stopped being optional the moment companies started shipping systems whose output has to be trusted.

What the Course Covers

This 36-minute course walks through:

• Why forward deployment became essential rather than optional as AI moved from prototypes into production
• Context engineering as an FDE responsibility — and how it differs from prompt engineering
• Why FDEs have to be developed internally rather than hired off the market
• How field experience reshapes engineering judgment about data quality and evaluation

Who This Is For

Engineers moving into customer-facing AI work, technical leaders standing up an FDE function, and anyone responsible for getting an AI system past the pilot stage. The course is beginner-level and assumes no prior FDE experience.

Background

I built and led Project Frontline at Palantir, training 250+ engineers in this methodology. Alumni now work at OpenAI, xAI, Anduril, and other AI companies. I later led FDE practices at Citadel before co-founding Kepler.

If you want the longer written treatment, see The Definitive Guide to Forward Deployed Engineering.

You can find the course on LinkedIn Learning.

Questions about FDE or deploying AI in the field? Feel free to reach out.

Inside a platform that indexes 26M+ SEC filings, earnings call transcripts, IR presentations, consensus estimates, and private data across 14,000+ companies and 27 global markets, and how the team behind it built AI that validates every number to the exact filing, page, and line item.

The quick pitch

Financial firms operate in a heavily regulated environment where reporting has to be auditable and accountable. Every figure in a regulatory filing, deal pitch, or research report needs to be verifiable against source documents.

The tools the financial industry has traditionally relied on can pull data, but they still require analysts for that verification process. An analytics system can’t interpret a freeform question, decompose it into steps, or work out that a single metric requires pulling three different line items across specific fiscal periods. AI systems can do that interpretation, but they handle it in the same step as the computation, so the numbers they produce are generated by the model, which can make mistakes.

Vinoo Ganesh and John McRaven spent years at Palantir building data systems for defense, energy, and financial firms. That work shaped how they think about trust in environments where answers must be verifiable. Before founding Kepler, they spoke with 147 financial firms, including private equity, hedge funds, and investment banks, and heard the same thing at nearly all of them: everyone wanted to use AI for research, but nobody trusted the output. As one managing director told them, “How am I supposed to trust something I can’t audit?”

The duo’s answer was to build deterministic infrastructure that serves as a trust and verification layer for AI. That infrastructure, together with Claude as the reasoning and interpretation layer, powers Kepler Finance: a research platform for financial services used by analysts to ask questions in plain English and receive instantly verifiable answers.

Handling long, multi-step tasks and flagging ambiguity

Financial analysis involves complex, multi-step calculations, dense data, and overloaded terminology, and has no tolerance for error. With that in mind, Kepler needed a model that could hold a long plan together without drift and flag ambiguity.

For example, if an analyst asks for a company’s inventory days outstanding over the last eight quarters, the model needs to figure out what the answer needs: the right formula, correct fiscal periods, and any restatements that might affect the numbers.

The team benchmarked across all frontier models. They found that on straightforward queries, models performed comparably. But when it came to long, multi-step plans with interdependencies, all but Claude started taking shortcuts or losing track of constraints by the fourth or fifth step. “On our workloads, Claude was the model that consistently held the plan together,” Ganesh says. “Other models would start strong and then quietly drop a constraint by step five.”

The clearest difference was how each model handled uncertainty and kept humans in the loop. For example, in situations where one term can have two different meanings, most models picked one meaning and kept going. Claude stopped and asked the analyst to decide. “That behavior matters more than any benchmark score,” Ganesh says. “One wrong assumption early in a financial analysis breaks everything downstream.”

Engineering the context around Claude

The Kepler team found that Claude produced better results when given precisely defined tasks enhanced with structured domain knowledge, definitions, and hard boundaries on what to resolve versus what to escalate. “In finance, the model can’t be the whole system. We treat it as one stage in a pipeline whose job is to hand the model exactly what it needs to succeed at exactly that stage,” says McRaven. “Prompt engineering optimizes a call while content engineering optimizes the system around it.”

The team built deterministic execution environments that Claude can invoke for every operation that needs to be provably correct, such as computing a ratio or resolving a fiscal period. They developed a proprietary ontology that maps financial concepts to precise definitions and formulas, customizable on a per-use basis. Security and access control restrictions are enforced at every step, governing which data sources each user can pull from. On top of this, they built recurring, customizable skills for the most common workflows in their pipeline, such as enterprise value calculations across complex capital structures (e.g. handling preferred shares, convertibles, and minority interests) and segment revenue waterfall reconciliation across reporting period changes. These skills coordinate between deterministic and nondeterministic stages and are idempotent by design: the same input will always generate the same output.

Next, they decomposed their workflows into a multi-stage pipeline, matching different Claude models to different stages: Opus 4.7 for complex reasoning like decomposing intent, resolving ambiguity, and producing structured execution plans, and Sonnet 4.6 for higher-throughput stages where tasks are more constrained. They also trained their own specialized models for recall (some use Claude as the foundation, some are proprietary to Kepler), scoring 94% accuracy on tasks like mapping financial statement labels to standardized taxonomy codes, compared with the 38-46% accuracy achieved by other models.

The team tests every prompt change, model upgrade, and context modification against thousands of cases before going to production. They’ve built automated evaluation pipelines that compare Claude’s output against known-correct answers at every stage, checking both the structured plan and the final computed result. When a test fails, they can trace whether the issue was in Claude’s reasoning, the context provided, or the downstream execution. When Anthropic ships a new model version, Kepler benchmarks it within hours and knows exactly which stages improve, which regress, and which need prompt adjustments.

Scaling with Claude

Kepler Finance has indexed more than 26 million SEC filings across 14,000+ companies, 50M+ public documents, and 1M+ private documents spanning 27 global markets. Claude makes that volume of unstructured data usable, interpreting questions against the entire corpus and reconciling differences in terminology across companies and time periods. Kepler’s retrieval layer then pulls figures from verified SEC filings, computes the result, and assembles the results into the desk’s Excel template, where with a single click analysts can trace each number back to its exact line item highlighted in the source document.

The separation between Claude’s reasoning and Kepler’s deterministic infrastructure lets a small team build at this scale. Claude handles the interpretation layer that would otherwise require many domain-specific NLP engineers and Kepler’s infrastructure handles the rest. New capabilities that would take a large team months to ship can be built in weeks because the architecture is modular: the team improves the reasoning at one stage without touching the rest of the pipeline.

As financial institutions require compliance infrastructure before they engage, Kepler has built full audit logging, siloed customer environments, and end-to-end provenance from the start, and has SOC 2 Type II certification, with ISO 27001 certification underway.

Kepler’s platform is domain-agnostic by design. The team started in finance deliberately as it’s one of the most demanding environments for AI, with dense data, overloaded terminology, complex calculations, and zero tolerance for error. The architecture built to survive that scrutiny applies wherever professionals need verifiable answers from large document collections. From healthcare providers reconciling clinical trial data against treatment protocols to legal teams tracing precedent across decades of case law, the pattern is the same: Claude reasons about the question and infrastructure guarantees the answer.

“Kepler Finance is our first product,” says Ganesh. “It won’t be the last.”

Best practices from the Kepler team

Most products fail not because of bad engineering, but because teams build the wrong thing. They either overbuild features nobody asked for or underbuild solutions that miss the real problem. Forward Deployed Engineering (FDE) is the antidote.

What is FDE?

Forward Deployed Engineering is a methodology where technical teams embed directly with customers to witness how software meets reality. Instead of building from assumptions, FDEs develop customer instinct by being in the field — watching real users interact with real products under real conditions.

Companies like Palantir, OpenAI, and Anduril use this approach to achieve dramatically higher product adoption rates.

What You’ll Learn

This 30-minute lesson covers the core FDE playbook:

• Detecting real customer problems — how to separate signal from noise when users tell you what they want vs. what they actually need
• The four core FDE moves — detect problems, demonstrate through action, control the narrative, and ship fast while maintaining production quality
• A structured week-by-week playbook for developing field-based customer instinct
• How AI is changing the FDE role and what that means for engineers building products today

Background

I built and led Project Frontline at Palantir, training 250+ engineers in the FDE methodology. Alumni now work at OpenAI, xAI, Anduril, and other AI-focused companies. I also led FDE practices at Citadel before co-founding Kepler.

Who This Is For

Whether you’re a product engineer, technical founder, or engineering leader — if you ship software to customers, the FDE mindset will change how you think about building products. The lesson also covers how FDE differs from professional services and how to balance customer wants vs. actual needs.

The lesson is free and includes Q&A. You can watch it on Maven.

Questions about FDE or product engineering? Feel free to reach out.

If you work in tech, you should learn about Forward Deployed Engineering (FDE). It’s quickly become one of the most popular roles that nearly all fast-growing AI companies are looking for…at the same time, it’s the type of role very few people and companies seem to actually understand.

I designed Project Frontline at Palantir, a program that sent over 250 engineers into live customer deployments. Those engineers are now at OpenAI, xAI, Anduril, and dozens of leading companies.

The Bottom Line

The Forward Deployed Engineer (FDE) has become the most valuable credential in tech, and most people don’t understand why.

If you look at job postings at OpenAI, xAI, Anthropic, Helsing, Anduril, Scale, or Palantir, you’ll notice they’re not just looking for software engineers. They’re looking for engineers who can deploy, who can sit with customers, who can make software work in the real world.

Here’s what you need to know:

An FDE is a software engineer who owns customer outcomes. Not customer relationships, not customer satisfaction scores, but outcomes: the actual results the customer is trying to achieve with the software. This distinction sounds subtle, but it changes everything. A sales engineer’s job is to help close the deal. A solutions architect designs how the product fits the customer’s environment. An FDE’s job is to make sure the customer actually wins.

The critical insight is that you can’t easily hire your way to a true FDE organization. The skill set is too rare, too specific, too dependent on experiences most engineers never have. Traditional engineering jobs don’t build this skill set because they’re structured to insulate engineers from customer reality. You write code, you ship it, you move to the next ticket. You never feel the weight of a customer depending on your work, never see what happens when your elegant system meets messy real-world data, never have to look someone in the eye and explain why the thing they needed isn’t working.

So how do you hire forward deployed engineers? You don’t. You grow them. This was Palantir’s critical insight, and it took us years to fully understand it. The only way to build an FDE organization is to take talented people and put them through a crucible that transforms how they think about building software.

Why this matters now: Three forces are making the FDE model the default rather than the exception.

First, AI is making deployment harder, not easier. AI systems require fine-tuning, prompt engineering, evaluation, and ongoing adjustment. You can’t throw an LLM over the wall and expect customers to figure it out. OpenAI, Anthropic, xAI, and every serious AI company is building FDE-style functions, even if they don’t call them that. There’s a related point here about what it means to be an engineer at all: if AI can do much of what junior engineers used to do, the engineers who remain valuable are the ones who can do what AI can’t, which is sit with a customer, understand their real problem, and own the outcome.

Second, defense tech is exploding. Anduril, Palantir, Shield AI, and dozens of startups are building software for military and government customers who can’t tolerate deployment failure. These customers operate in austere environments with unique constraints, and you can’t support them with a helpdesk. You need engineers in the field.

Third, enterprise buyers are getting smarter. After decades of shelfware, enterprises are starting to demand implementation guarantees. They want to see engineers on site and know that someone will own the deployment, not just the sale. The vendors who can deliver this will win.

Five years from now, I expect every serious enterprise software company to have an FDE function. If you’re thinking about where to invest your career, this is the trend to bet on.

Why This Matters Now

Three forces are making the FDE model increasingly standard:

First, AI is making deployment harder. AI systems require fine-tuning, prompt engineering, evaluation, and ongoing adjustment. You cannot simply deploy an LLM and expect customers to figure it out. Companies like OpenAI, Anthropic, and xAI are building FDE-style functions, even if they use different terminology.

Second, defense tech is exploding. Companies like Anduril and Palantir build software for military and government customers who cannot tolerate deployment failure. These customers operate in austere environments with unique constraints that require engineers in the field, not helpdesk support.

Third, enterprise buyers are getting smarter. After decades of expensive software sitting unused, enterprises are demanding implementation guarantees and want to see engineers on site who own deployments, not just sales.

The Grey Track Jacket

At Palantir, everyone got a black track jacket. Standard issue. But there was another jacket, a grey one, that you couldn’t buy or request.

The mythology was that your black jacket had faded through years of experience, that you’d worn it through enough deployments and enough hard field work that the black had burned away and become grey. Leadership awarded them when you had proven yourself in the field.

Brian Schimpf, now CEO of Anduril, awarded these jackets when he was one of Palantir’s senior engineering leaders. When I got mine, I didn’t feel pride or accomplishment. I felt the weight of what it represented: the late nights, the systems that failed, the customers who’d depended on me when things went wrong. The jacket wasn’t a trophy. It was an acknowledgment that you’d been through something, and that something had changed you.

When engineers ask me how to become an FDE, they want skills to develop, certifications to get, keywords for their resume. But that’s not what this is about. The jacket was just the symbol. The transformation is what matters.

When I received my grey jacket, I did not feel pride or accomplishment. I felt the weight of what it represented: the late nights, the systems that failed, the customers who depended on me when things went wrong. The jacket was not a trophy. It was acknowledgment that you had been through something, and that something had changed you.

My Transformation

If you want concrete recommendations on how to become an FDE, skip to “The Playbook” at the end. But if you want to understand why this transformation matters and what it actually feels like, read on.

I joined Palantir on the product development side in Palo Alto. I fixed Jira tickets, wrote code, built tests, did code reviews, architected systems. I attended sprint planning and hit my deadlines. By every internal metric, I was doing good work.

But once I rotated into the field, what I saw broke how I understood my own job.

We had a product called Phoenix. It worked beautifully for the use case it was designed for. Clean architecture, solid code, good test coverage. We were proud of it.

Then I watched it fail.

The system relied on a data integration that seemed straightforward: when new data comes in, we generate a keyspace in Apache Cassandra. We’d designed it to create keyspaces in 10-minute increments for easy automated deletion. We thought it felt like a clever trick.

Then bad data came in. Well, specifically, no data came in. The date column was empty. Our system interpreted this as the epoch: January 1, 1970. So it did exactly what it was designed to do, and generated every keyspace between January 1, 1970 and October 7, 2013.

That’s 2.3 million keyspaces. Cassandra requires roughly 5MB per column family. You’d need 14 terabytes of RAM just to start up.

The system OOMed on startup. As was commonly said at Palantir back then, “bad times on boats.”

I sat there watching this happen, and I realized something that should have been obvious: we had never seen the data this system would actually process. We had never been in the room when garbage data showed up at 2am. We had never had to look a customer in the eye and explain why the system they depended on was suddenly unresponsive.

This wasn’t a one-time thing. I saw it again on other projects. We would build something elegant and technically sophisticated, ship it to the field, and watch it collide with reality in ways we never anticipated. The pattern was always the same: we built without understanding, optimized for the wrong things, and solved problems that didn’t exist while ignoring problems that did. The gap between what we were building and what users actually needed wasn’t a small gap. It was a chasm.

After that first experience, I kept going back to the field: NYC, Singapore, Afghanistan, Israel, UAE, Toronto, SF, Mumbai, and government deployments I can’t name. I spent hours in SCIFs working through data integrations across multiple networks. My laptop once overheated and nearly melted sitting in a car in the Middle East.

Each field experience taught me something new about the distance between building software and deploying it, about what happens when your code meets reality, about the difference between solving a problem in theory and solving it for a real human being who needs it to work right now.

I alternated between software engineering and forward deployed work for years, going back and forth between building and deploying. Each time I returned to product development, I built differently. I asked different questions and anticipated different failures. I understood, in a way I couldn’t have before, what the software was actually for. That oscillation changed me. And I became convinced it could change others.

Why This Model Works

The Business Case

The business case is simple: software that actually gets adopted is worth infinitely more than software that sits on a shelf.

Most enterprise software fails to deliver value not because it doesn’t work technically, but because it never gets properly deployed, configured, adopted, and integrated into customer workflows. The industry’s dirty secret is that a huge percentage of enterprise software purchases end up as shelfware. Customers buy it, try to implement it, hit friction, and give up.

Companies with an FDE model don’t have this problem. Their software gets used because there’s an engineer on site making sure it gets used. Problems get solved in real time, configurations get tuned to actual workflows, and edge cases get handled before they become blockers.

This creates a flywheel: deployed software generates feedback, feedback improves the product, better products deploy more easily, easier deployment means more customers, and more customers means more feedback.

Palantir generates over $1.1 billion in annual revenue from just their top 20 customers. That’s not because the software is marginally better than competitors. It’s because Palantir engineers own customer outcomes in a way that traditional enterprise software vendors simply don’t.

The other reason companies want FDEs is that they’re the best source of product insight. Engineers who work directly with customers see things that product managers reading survey data never see. They understand the actual workflows, the real constraints, the true pain points. The best product ideas at Palantir came from FDEs who’d spent enough time in the field to understand what needed to be built.

One deployment I worked on had over fifty data pipelines running daily, and monitoring them was a nightmare. A few engineers who’d experienced this pain firsthand built a pipeline monitoring tool that got deployed across multiple sites. They didn’t build it because a PM told them to. They built it because they’d felt the problem themselves.

That’s the FDE advantage: engineers who build from experience, not from specs.

A Correction, Not a Trend

For a long time, the software industry optimized for building at scale while pushing deployment complexity onto customers and divorcing engineering from the business problems it was supposed to solve. The prevailing model was simple: throw it over the wall, let the customer figure it out, and hire systems integrators to clean up the mess.

This worked when software was simple and customers were patient, but neither of those things is true anymore.

The companies winning right now are the ones who own the full loop of building and deploying. They don’t hand off to SI firms or blame customers for implementation failures. They send engineers into the field who own outcomes, not just outputs.

Palantir figured this out twenty years ago and got criticized for it. The criticism was relentless: “It’s not scalable.” “It’s just consultants.” “It’s not a real software company.” Now everyone is trying to copy the model.

The problem is, you can’t hire your way to an FDE organization.

Building Frontline: The Program That Scaled It

To really understand FDE, you need to understand Frontline.

At the time, Palantir had two organizations that didn’t understand each other. Product Development (PD) built the software while Business Development (BD), which included the forward deployed engineers) put it in front of customers. The two groups were culturally and operationally separate, and that separation was causing real damage.

PD thought BD lacked technical rigor. BD thought PD lacked urgency. Deploying the product was painful, sometimes taking days or weeks. Engineers in the field drowned in problems that engineers in headquarters had never seen, while engineers in headquarters built features that solved the wrong problems because they’d never felt the pain of the people using their software.

The best way to understand someone is to walk a mile in their shoes. We decided to make that literal.

Project Frontline was born from this insight. Software engineers would spend time in the field doing a forward deployed rotation before formally beginning their product development work. They would see how the software actually got used, feel the pain of deployment, and understand viscerally the gap between what we built and what customers needed. Then they would come back and build differently.

I had the opportunity to lead Frontline alongside Palantir’s senior leadership: Matt Steckman, John Garrod, Bill Ward, Lynne Lu, and Randy Schults. Shyam Sankar, Palantir’s now CTO, was directly involved, sitting in on our update meetings as we figured out what was working and what wasn’t. This wasn’t a side project. It was a strategic priority.

We learned quickly that throwing engineers into the field wasn’t enough. They needed support and structure. Every Frontliner got a mentor who had been through the experience, who could guide them through the disorientation of going from building software to deploying it. They got leads who checked in frequently, not just to track progress but to surface problems early. We created feedback loops throughout the rotation so that when someone was struggling, we caught it early, and when something wasn’t working, we heard about it fast.

We made mistakes along the way. The six month rotation timeline was supposed to be a guideline, but resourcing pressures sometimes stretched rotations to a year, and engineers felt stuck. There was friction between existing FDEs and the rotators, along with cultural clashes and different working styles.

We also discovered something crucial: often, the rotations weren’t connected to the products engineers would eventually build. Someone would spend six months deploying software they would never touch again. They learned valuable things, but the connection was abstract. So we started aligning rotations with future product work. If someone was going to build pipeline monitoring tools, we sent them to deployments where pipeline monitoring was a problem.

They became users before they became builders.

This was transformative. When engineers experienced the pain of problems they would later fix, something clicked. They built with emotion and passion, with a deep understanding of what users actually needed. You could feel it in the products. The software wasn’t just technically good; it was designed by people who cared, who had been there, who understood. Over time, the program scaled. Over 250 engineers rotated through Frontline, and it stopped being an experiment and became the way things worked.

The proof is in what happened to the people who went through it. Frontline alumni are now at OpenAI, xAI, Helsing, Anduril, Hex, and dozens of other companies building important things. Bill Ward, who co-led Frontline, started Northslope Technologies, which offers FDE as a service. Jesse Rickard started Fourth Age, which does something similar. The model we built is now being replicated across the industry.

What Makes Someone Great at This

I’ve watched hundreds of engineers go through the FDE crucible. Some became exceptional while others washed out. Here’s what separates the great ones.

Great FDEs are relentlessly curious about the customer’s world. Good FDEs solve the problems they’re given, but great FDEs understand the customer’s business well enough to surface problems the customer had given up on or hadn’t yet articulated. In other words, as I said above, they become users before they become builders. I once spent a week at a customer site and noticed that every morning, an analyst spent 45 minutes manually downloading data from three different systems and combining them in Excel before she could start her actual work. She’d been doing this for two years and had never thought to ask if it could be automated. She’d just accepted it as part of her job. A good FDE would have built what she asked for. I built a pipeline that did her morning routine automatically and had it waiting in her inbox when she arrived. That’s the difference.

Great FDEs calibrate their engineering to the situation. They know when to build a robust system and when to write a script that just works. Here’s a failure story: early in my FDE career, a customer asked for a way to deduplicate records. I spent two weeks building an elegant, configurable deduplication engine with fuzzy matching, confidence scores, and a review interface. What they actually needed was a SQL query that ran once to clean up a specific data import. They never used the engine again. I’d overengineered by 10x. The opposite failure is equally painful: I once hacked together a “temporary” data deletion script that ended up running in production for eighteen months (vinoo.groovy, a name many of my old Palantir teammates still call me).

Great FDEs communicate across audiences. They can explain complex systems to executives who don’t care about the details and dive deep with technical counterparts who need to understand exactly how something works. In one meeting, I had to explain why a migration would take three months instead of three weeks. To the CTO, I said: “The data has 15 years of accumulated edge cases that will break downstream reports if we don’t handle them carefully. Rushing it risks the quarterly board reporting.” To the engineering lead, I said: “There are 847 columns across 12 tables with undocumented interdependencies, and the existing ETL has implicit type coercions that we need to preserve or we’ll get silent data corruption.” Same problem, different language.

Great FDEs stay calm when things break. And things will break. I remember a deployment where the customer’s CEO was demoing our product to their board when it crashed. My phone rang. Instead of panicking, I talked the customer’s IT person through restarting the service (which I knew would work because I’d diagnosed the root cause earlier that week), and the demo was back up in four minutes. The CEO never knew there was an engineer on the phone. How you handle a crisis matters as much as whether you fix it.

Great FDEs own outcomes without having authority. This is the hardest skill. Once, I needed a product team to prioritize a bug fix that was blocking a major customer. I had no authority over that team. What didn’t work: “Can you please prioritize this bug?” What did work: “This customer represents $4M in annual revenue and their renewal is in six weeks. They’ve told their exec sponsor that this bug is their top complaint. If we don’t fix it, we risk losing the renewal, and I can get you a call with the customer to hear it directly if that would help.” I got the fix in two days.

Great FDEs know when to push back. They don’t just do whatever customers ask. Once, a customer demanded we build a custom feature that would have taken three months of engineering time. Instead of saying yes or no, I spent two days understanding why they wanted it. It turned out they were trying to solve a problem that our existing product could handle with a configuration change. I showed them how to do it, and they were happier with that solution than they would have been with the custom feature. Pushing back isn’t about saying no. It’s about understanding the real need and finding the right solution.

Why You Might Not Want This

I’ve spent this essay explaining why FDE is valuable. Now let me tell you why it might not be right for you.

If you need deep focus time, this isn’t your path. FDE work is interrupt-driven, with customers having emergencies and deployments hitting unexpected problems. Your calendar will look like that week I described above: chaotic, fragmented, constantly shifting. If ambiguity paralyzes you, stay away. FDEs operate with incomplete information constantly. Requirements are unclear, priorities shift, and you have to make decisions without knowing if they’re right.

If you want to go deep on a technical specialty, look elsewhere. FDEs are generalists by necessity. You’ll learn a lot about a lot of things, but you won’t become a world expert in any of them. If your goal is to be the best in the world at distributed systems or machine learning, pure product engineering is a better path.

If you take customer frustration personally, you’ll burn out. Customers get frustrated and blame you for things that aren’t your fault.

If you can’t set boundaries, you’ll work yourself to death. FDE work expands to fill all available time. I’ve seen talented people burn out because they couldn’t say no.

These aren’t weaknesses; they’re just characteristics that make FDE a bad fit. There are plenty of great engineering roles for people who have them.

The Playbook

Manufacturing Your Own Field Exposure

Most of you reading this don’t work at a company with a Frontline program. Nobody is going to hand you this experience, so you have to create it yourself. Here’s exactly how to do it.

Week 1-2: Get on customer calls. Email your PM or customer success lead today with this exact message: “I want to understand our customers better so I can build more robust features. Can I sit in on 2-3 customer calls over the next few weeks? I’ll be on mute, just observing.” Almost no one will say no to this. On the calls, don’t take notes on feature requests. Take notes on these things: What does this person’s day look like? What did they do before this call? What will they do after? What do they seem frustrated by that they’re not explicitly complaining about? What workarounds are they using?

Week 3-4: Shadow support or implementation. Find whoever handles customer escalations or implementations at your company. Ask to shadow them for three days. If you work remotely, ask to be added to their Slack channels and get on video calls with them when issues come in. Watch how problems get diagnosed. Notice the gap between what customers say is wrong and what’s actually wrong. Pay attention to which parts of your product cause the most confusion.

Month 2: Take an escalation. When a customer issue comes in, volunteer to own it end-to-end. Not just fix the bug, but talk to the customer, understand the impact, communicate the timeline, deploy the fix, and confirm it’s resolved. This is the full loop. It will be uncomfortable. You will learn more from this one experience than from months of normal development work.

Month 3: Visit a customer site (or do a deep-dive remote session). If your company has customers you can visit, ask to join a site visit. If not, ask if you can do a 2-hour screen-share session with a customer where you watch them use the product for their actual work, not a demo, not a training session, their actual job. While you’re watching, identify one papercut you can fix in under a day. Then fix it and ship it before the week is over. Tell them you did it. This builds trust and goodwill like nothing else.

Month 4-6: Build your feedback loop. By now you should have enough context to start seeing patterns. Create a simple document titled “What I’ve Learned From Customers” and update it weekly. Share it with your product team. Include specific quotes, specific workflows, specific pain points. This document will make you invaluable. It will also prepare you for FDE interviews, where you’ll need to demonstrate customer empathy.

Developing the Six Traits

Curiosity is really about empathy. It develops when you force yourself out of the codebase and into the customer’s environment. The next time you’re on a customer call or a site visit, don’t just listen for feature requests. Try to understand the customer’s day: what are they doing before they open your software, what are they doing after, and what are the three things keeping them up at night that have nothing to do with your product?

Calibration comes from getting it wrong in both directions. You need both scars. Deliberately vary your approach: on your next project, try solving it with the simplest possible thing, and on the one after, try building it properly. Pay attention to which approach the situation actually called for.

Communication is a muscle you have to exercise. Think of it like the game of telephone: every time a message passes through another person, it degrades. FDEs short-circuit that chain. Practice writing one-page summaries of technical projects for non-technical stakeholders. If you can’t explain what you built and why it matters in one page without jargon, you don’t understand it well enough.

Staying calm is about preparation and repetition, not personality. Put yourself in high-pressure situations voluntarily: volunteer for the on-call rotation, take the escalation, join the war room when production is down. By the tenth time, you’ll have a playbook in your head.

Owning outcomes without authority develops when you put yourself in situations where you need something from someone who doesn’t report to you. Learn to make requests that explain the why, not just the what.

Pushing back effectively runs on credibility. Think of credibility as points you can spend or lose. Early on, pick your battles carefully. The key is to always push back with an alternative, not just a no.

What You Should Actually Know Technically

FDEs are generalists, but that doesn’t mean you can be shallow on everything. Here’s the specific technical knowledge you need.

The XY Problem. Customers will ask you for Y when what they actually need is a solution to X. Practice by asking “what are you trying to accomplish?” before jumping to solutions.

SQL. JOINs, GROUP BY with HAVING, window functions, CTEs, subqueries. Target: write a query that finds “the second-highest value per category” without Googling. Data is objective and data analysis is your lifeblood.

Command line and Linux. Navigating the filesystem, reading logs (grep, awk, tail -f), understanding processes (ps, top, kill), basic networking (curl, netstat, ping, traceroute). Know what “out of inodes” means. Resource: OverTheWire Bandit.

Containers. What Docker does, difference between container and VM, where logs go. What Kubernetes does at a high level. Resource: Docker’s getting started guide.

Networking. DNS, HTTP status codes, TLS/SSL basics, debugging when you can ping but can’t connect. Resource: Julia Evans’ zines.

Python scripting. Read a CSV, transform data, write it somewhere. Call an API, handle pagination. Connect to a database. Be fast. Resource: Automate the Boring Stuff with Python.

Data quality. What happens with timezone mismatches, encoding issues, unexpected nulls. The Cassandra story I told? That’s a data quality issue.

Getting Hired: What Interviewers Actually Ask

FDE interviews are heavy on behavioral questions because the job is about judgment, not just technical skill.

“Tell me about a time you had to work with a difficult customer.” Bad: “The customer was being unreasonable, but I stayed calm and eventually they came around.” Good: “The customer was frustrated because our product had lost their data. They were angry, and honestly, they had a right to be. I started by acknowledging that we’d screwed up, not making excuses. Then I explained exactly what had happened, what we were doing to recover the data, and what we were doing to make sure it never happened again. I also gave them my cell phone number and told them to call me directly if anything else went wrong. It took two weeks, but by the end they renewed their contract.”

“Tell me about a time you disagreed with a customer’s request.” Bad: “The customer wanted something that didn’t make sense, so I explained why they were wrong.” Good: “The customer wanted us to build a custom dashboard that would have taken a month of engineering time. Instead of saying no, I asked them to walk me through exactly how they’d use it. It turned out they needed to answer a specific question for a weekly meeting. I showed them how to get that answer with our existing product plus a 10-minute SQL query. They were actually happier with that solution because they got it that day instead of waiting a month.”

“Tell me about a time something broke in production.” Bad: “Our service went down, and I fixed it.” Good: “I was on-call when our main API started returning 500 errors. First, I checked monitoring to understand scope: 30% of requests, not all. That told me it probably wasn’t new code. Error logs showed database connection timeouts. Database was at 100% CPU. Slow query log showed a full table scan on 50 million rows from a feature that launched that morning. I disabled the feature, fixed the API, then worked with the team to add an index. Whole thing took 45 minutes. Next day we implemented query review for large tables.”

Technical questions focus on debugging and practical coding:

• “How would you debug a slow API endpoint?” (Walk through: monitoring, logs, profiling, database queries, external dependencies)
• “Write a script that processes this data.” (Messy data problem. Handle edge cases. Python.)
• “Explain X to me like I’m not technical.” (DNS, database indexing, rate limiting. No jargon.)

Your First 90 Days

Days 1-30: Learn. Absorb context. Meet everyone. Read all documentation. Shadow every call. Build a glossary. Write down three things you learned each day. Don’t try to add value yet. The most common failure mode is engineers who come in hot with opinions before they understand the environment.

Days 31-60: Find your first win. One small thing you can own completely. A script that automates manual work. A bug fix that’s been annoying customers. A dashboard replacing a spreadsheet. Something with a clear before and after. Ship it. Tell people.

Days 61-90: Start forming opinions. Now you have context and credibility. Engage harder problems. Propose solutions. Be humble—three months in, you’re still a beginner. But you’re a beginner with context.

Traps to avoid: Overengineering. I once saw a new FDE spend three weeks building a “robust, scalable” solution for a problem that needed a bash script. The customer cared about getting the answer by Friday.

Retreating into code. When a meeting gets tense, your instinct is to hide behind your laptop. Don’t. Stay present.

Overpromising. Underpromise and overdeliver. “Done by Wednesday” when you finish Tuesday beats “Done by Monday” when you finish Wednesday. Solving the wrong problem. “What are you trying to accomplish?” is the most important question in an FDE’s toolkit.

The Career Path

Year 1: Learn. Absorbing context, building relationships, developing judgment. Handling individual customer problems. Success: customers trust you, team relies on you for insights, you’ve shipped things that made a real difference.

Year 2-3: Lead. Owning larger customer relationships or leading small teams. Making prioritization decisions. Mentoring newer FDEs. Success: turned around a struggling customer relationship, shaped the product roadmap, people come to you when they don’t know what to do.

Year 4+: Multiply. Going deep (domain expert) or going broad (leading an FDE org, product leadership, starting a company). Many FDEs become founders because they have both technical skills and customer understanding. Others become product leaders. Some stay as senior FDEs handling the most complex deployments.

Frontline alumni have become founders of AI companies, heads of product at public companies, and engineering leaders at defense contractors. The FDE skill set is rare and valuable in almost any technical leadership role.

Resources

Companies hiring FDEs: Palantir, Anduril, OpenAI, Anthropic, Scale AI, Helsing, Hex, Databricks, Weights & Biases

FDE-adjacent: FDE as a service companies, defense tech startups, vertical SaaS with complex deployments, AI companies with enterprise customers

What to read:

• Palantir’s S-1 filing (deployment methodology section)
• Shyam Sankar’s blog - https://www.shyamsankar.com/
• “The Mom Test” by Rob Fitzpatrick - https://www.momtestbook.com/
• Julia Evans’ blog and zines - https://jvns.ca/

The Path Forward

The grey track jacket was just a symbol. What it represented was something deeper: the transformation that happens when you stop building software in a vacuum and start building it for real people, with real problems, in the real world.

That transformation is available to you whether or not you work at Palantir. It requires putting yourself in the field, feeling the pain of deployment, and understanding what customers actually need. It requires going through something that changes you.

The field is where the real education happens. Everything else is just preparation.

Everyone’s talking about context engineering right now, but most of the conversation is focused on the wrong thing. Read the blog posts, the guides, the thought leadership. They’re all asking the same questions: What should I include in the context window? How do I manage tokens efficiently? How do I curate what the model sees?

These are valid questions. They’re also the easy part.

The hard part isn’t deciding what context to include. It’s building systems that deliver that context reliably, with provenance, at scale, every single time. That’s not a context problem. That’s an engineering problem. And engineering means something specific.

I spent fifteen years building data infrastructure at Palantir and Citadel, working on defense and intelligence systems where wrong answers weren’t acceptable. Here’s what I learned: knowing what data you need is trivial. Everyone knows what data they need. The hard part is building systems that deliver it validated, traceable, reproducible, and on demand.

The data world spent two decades learning this lesson. You don’t solve data problems by choosing the right data. You solve them by building the infrastructure that makes data trustworthy. Context is the same problem wearing different clothes.

Watch how people approach context engineering today. They’re selecting documents. Tuning retrieval parameters. Adjusting chunk sizes. Experimenting with what to include and what to leave out. Managing memory hierarchies. Optimizing token usage. This is prompt engineering with more inputs. It’s not engineering.

Engineering means provenance: where did this context come from, and can I trace every piece back to its source? If the output is wrong, can I figure out which input caused it? Engineering means versioning: the context I retrieved yesterday and the context I retrieve today, are they the same, and if not, why? Can I reproduce last week’s answer? Engineering means validation: how do I know the context is accurate before it reaches the model, what checks exist, and what happens when something fails? Engineering means determinism: same query, same context, every time, not mostly the same, but exactly the same. Engineering means observability: when context retrieval breaks, how do I know, how do I debug it, and how do I fix it without guessing?

Most context engineering today has none of this. It’s artisanal, manual, and fragile. It works in demos and falls apart in production.

The failure mode is predictable. Someone builds a system that retrieves context and generates answers. It works well enough in testing, so they ship it. Then the context changes. A document gets updated. A source goes stale. A retrieval pipeline silently returns different results. The model keeps generating answers with the same confidence, but now the answers are wrong. Nobody notices until a customer does.

This is the same failure mode the data world lived through. Dashboards that showed numbers nobody could trace. Reports that changed when you refreshed the page. Metrics that meant different things to different teams. We solved it by building infrastructure: pipelines with lineage, transformations with tests, data contracts, observability. The unsexy plumbing that makes data trustworthy. Context needs the same infrastructure, and almost nobody is building it.

The current generation of AI tools is fragile in ways that aren’t obvious. They work until they don’t. They’re right until they’re wrong. And when they break, there’s no way to diagnose why. The teams treating context engineering as an optimization problem will keep hitting this wall. They’ll tune retrieval, adjust chunk sizes, swap embedding models, and still get inconsistent results they can’t explain. The teams treating it as an engineering problem will build systems that are auditable, reproducible, and debuggable. They’ll know where their context comes from. They’ll know when it changes. They’ll be able to defend their outputs. One approach scales. The other doesn’t.

The term context engineering is new. The problem isn’t. It’s the same problem the data world faced: how do you build systems that deliver the right information, reliably, at scale? The answer was never “choose the right data.” The answer was always “build the infrastructure that makes data trustworthy.”

Context is data by another name. The lessons are the same. The question is whether we’re going to spend another decade relearning them.

AI produces numbers that are incorrect, untraceable, and unverifiable. Asking the same question twice yields different answers, and there’s no way to distinguish right from wrong outputs. This unreliability poses serious risks across industries like healthcare, legal, insurance, government, and finance where numerical accuracy directly impacts outcomes.

Through conversations with 137 financial firms - including private equity, hedge funds, and investment banks - a consistent pattern emerged: everyone wants to use AI, but nobody trusts it. As one managing director stated, “I can’t put a number in front of a client if I can’t show where it came from.”

The problem is fundamental: AI hallucination can’t be solved by making AI smarter alone. Training data improvements and guardrails reduce errors but don’t eliminate them.

Kepler’s solution is distinct: prevent AI from producing numbers altogether. When users ask Kepler questions, AI interprets intent while deterministic code retrieves data and performs calculations. AI and code operate in separate lanes, eliminating the possibility of AI-generated numerical errors. Every figure traces to its source; every answer is reproducible.

The team draws expertise from Palantir, where I spent seven years building data infrastructure, and Citadel, where I was Head of Business Engineering for Ashler. Team members come from Meta, Bloomberg, and Stanford, backed by founders of MotherDuck, dbt, and figures from Facebook AI Research and OpenAI.

The company invokes Johannes Kepler’s historical use of trusted astronomical data to make groundbreaking discoveries, positioning trustworthy data infrastructure as foundational for AI advancement.

The world of AI engineering is moving incredibly fast. Every week brings new models, techniques, and breakthroughs. But beneath all that chaos, there are sophisticated patterns and architectural principles that remain consistent across implementations.

I recently published a new LinkedIn Learning course: Fundamentals of AI Engineering: Principles and Practical Applications.

Why This Course Matters

After building mission-critical systems at companies like Palantir and Citadel, I’ve learned that the gap between AI research and production-ready systems is often wider than expected. This course bridges that gap by focusing on the engineering fundamentals that actually matter in production environments.

This isn’t another theoretical AI course. It’s designed for software engineers who want to build AI systems that scale, perform reliably, and solve real business problems.

Course Approach

Hands-On Implementation: Everything is built using open-source tools like LlamaIndex and Hugging Face. The course uses real code, real data, and real challenges rather than theoretical examples.

Production-First Mindset: The focus is on systems that can handle real-world loads, not just demo scenarios.

GitHub Codespaces Integration: Students can start coding immediately without environment setup complexity.

Course Deep Dive

Foundation: Local LLM Operations

We start by running large language models locally, understanding the complete pipeline from tokenization to inference. You’ll learn to move beyond the API-driven approach and understand what’s actually happening under the hood.

Document Processing at Scale

Real-world AI applications need to handle messy, unstructured data. We cover:

• Advanced text extraction techniques
• Structure recognition and metadata enrichment
• Optimal chunking strategies for different document types
• Performance considerations for large document corpuses

The Embedding Ecosystem

Embeddings are the foundation of modern AI retrieval systems. You’ll master:

• Comparing and selecting embedding models for your use case
• Efficient embedding generation and batch processing
• Understanding the trade-offs between speed, accuracy, and cost

Vector Database Mastery

Moving beyond simple similarity search to production-grade vector operations:

• Database selection and optimization
• Approximate Nearest Neighbor (ANN) algorithms
• Caching strategies for performance
• Scaling considerations and cost management

Advanced Retrieval Engineering

This is where the magic happens. We build sophisticated retrieval systems that combine:

• BM25 and vector search for comprehensive coverage
• Hybrid retrieval that leverages the strengths of both approaches
• Cross-encoder reranking for precision improvements
• Complete pipeline integration with monitoring and observability

What You’ll Learn

This 4+ hour course covers:

• Building production-ready RAG systems using embeddings and vector database pipelines
• Implementing monitoring and observability for AI applications using telemetry tools
• Creating efficient document processing pipelines with hybrid search capabilities
• Designing CI/CD workflows for deploying and testing AI applications
• Optimizing AI system performance and costs through caching and resource management

Who Should Take This Course

This course is perfect for:

• Software engineers looking to add AI capabilities to their toolkit
• Backend developers who want to understand AI system architecture
• Technical leaders planning AI implementations
• Anyone building production AI applications who needs to go beyond simple API calls

The course assumes intermediate programming knowledge but doesn’t require prior AI experience.

Real-World Applications

Throughout the course, we build systems that mirror real production challenges:

• Enterprise document search and retrieval
• Customer support automation
• Knowledge base augmentation
• Multi-modal content processing

Key Takeaways

AI engineering isn’t just about calling APIs or fine-tuning models. It’s about building reliable, scalable systems that solve real problems. The course focuses on developing the engineering judgment needed to build AI systems that actually work in production.

The field is moving fast, but the fundamentals remain constant. Understanding these patterns provides a solid foundation for whatever comes next in AI development.

You can find the course on LinkedIn Learning.

Questions about the course content or AI engineering in general? Feel free to reach out – I love talking about this stuff.

The conversation also covers LLMs and AI, and where they do and don’t change the fundamentals of data engineering work.

Link

• YouTube
• Spotify
• Apple Podcasts

Databricks’ DLT offering showcases a substantial improvement in the data engineer lifecycle and workflow. By offering a pre-baked, and opinionated pipeline construction ecosystem, Databricks has finally started offering a holistic end-to-end data engineering experience from inside of its own product, which provides superior solutions for raw data workflow, live batching and a host of other benefits detailed below.

Since its release in 2022, Databricks’ Delta Live Tables have quickly become a go-to end-to-end resource for data engineers looking to build opinionated ETL pipelines for streaming data and big data. The pipeline management framework is considered one of the most valuable offerings on the databricks platform, and is used by over 1,000 companies including Shell and H&R block.

What Are Delta Live Tables?

Delta Live Tables, or DLT, is a declarative ETL framework that dramatically simplifies the development of both batch and streaming pipelines. Concretely though, DLT is just another way of authoring and managing pipelines in databricks. Tables are created using the @dlt.table() annotation on top of functions (which return queries defining the table) in notebooks.

Delta Live Tables are built using Databricks foundational technology such as the Delta Lake and Delta File format. As such, they operate in conjunction with these two. However, whereas these two focus on the more “stagnant” portions of the data process, DLT focuses on the transformation piece. Specifically, the DLT framework allows data engineers to describe how data should be transformed between tables in the DAG.

The magic of DLT is most apparent when it comes to datasets that both involve streaming data and batch processing data. Whereas, in the past, users had to be keenly aware of and design pipelines for the type of the “velocity” (batch vs. streaming) of data transformed, DLT allows users to push this problem to the system itself. Meaning, users can write declarative transformations and let the system figure out how to handle the streaming or batch components.

How are Delta Live Tables, Delta Tables, and Delta Lake related?

The word “Delta” appears a lot in the Databricks ecosystem, and to understand why, it’s important to look back at history. In 2019, Databricks publicly announced the Delta Lake, a foundational element for storing data (tables) into the Databricks Lakehouse. Delta Lake popularized the idea of a Table Format on top of files, with the goal of bringing reliability to data lakes.

Tables that live inside of this Delta Lake are written using the Delta Table format and, as such, are called Delta Tables. Delta Live Tables focus on the “live” part of data flow between Delta tables – usually called the “transformation” step in the ETL paradigm. Delta Live Tables (DLTs) offer declarative pipeline development and visualization.

Breaking Down The Components of Delta Live Tables

There are two main ways to create tables within a Delta Live Tables pipeline:

Tables

Tables in DLT are materialized views that are stored in the lakehouse. They represent the physical datasets that will be persisted and can be queried directly. These tables are created using the @dlt.table() decorator and contain the actual transformed data.

Views

Views in DLT are temporary datasets that exist only during the pipeline execution. They’re useful for intermediate transformations and don’t consume storage since they’re computed on-demand. Views are created using the @dlt.view() decorator.

You can declare your datasets in DLT using either SQL or Python. These declarations can then trigger an update to calculate results for each dataset in the pipeline.

When to Use Views or Materialized Views in Delta Live Tables

The choice of View or Materialized View primarily depends on your use case. The biggest difference between the two is that Views are computed at query time, whereas Materialized Views are precomputed. Views also have the added benefit that they don’t actually require any additional storage, as they are computed on the fly.

The general rule of thumb when choosing between the two has to do with the performance requirements and downstream access patterns of the table in question. When performance is critical, having to compute a view on the fly may be an unnecessary slowdown, in which case, Materialized Views may be preferred. The same is true when there are multiple downstream consumers of a particular View.

However, there are multiple situations where users just need a quick view, computed in memory, to reference a particular state of a transferred table. Rather than materializing this table, creating a View is more straightforward and efficient.

What Are the Advantages of Delta Live Tables?

There are many benefits to using Delta Live Tables:

Unified Streaming/Batch Experience

By removing the need for data engineers to build distinct streaming/batch data pipelines, DLT simplifies one of the most difficult pain points of working with data, thereby offering a truly unified experience.

Opinionated Pipeline Management

The modern data stack is filled with orchestration players, observability players, data quality players, and many others. DLT offers an opinionated way to orchestrate and assert data quality.

Performance Optimization

DLTs offer the full advantages of Delta Tables, which are designed to handle large volumes of data and support fast querying. Their vectorized query execution allows them to process data in batches rather than one row at a time.

Built-in Quality Assertions

Delta Live Tables provide data quality features, such as data cleansing and data deduplication, out of the box. Users can specify rules to remove duplicates or cleanse data as data is ingested, ensuring data accuracy.

ACID Transactions

Because DLTs use Delta format they support ACID transactions (Atomicity, Consistency, Isolation and Durability) which has become the standard for data quality and exactness.

Pipeline Visibility

DLT provides a Directed Acyclic Graph of your data pipeline workloads, giving you a clear, visually compelling way to both see and introspect your pipeline at various points.

Change Data Capture (CDC) in Delta Live Tables

One of the large benefits of Delta Live Tables is the ability to use Change Data Capture while streaming data. Change Data Capture refers to the tracking of all changes in a data source so they can be captured across all destination systems.

With Delta Live Tables, data engineers can easily implement CDC with the Apply Changes API (either with Python or SQL). The capability lets ETL pipelines easily detect source data changes and apply them to data sets throughout the lakehouse.

Delta Live Tables support Slowly Changing Dimensions (SCD) both type 1 and type 2. This is important because SCD type 2 retains a full history of values, which means you can retain a history of records in your data lakehouse.

What is the Cost of Delta Live Tables?

The cost of Delta Live Tables depends on the compute function itself. On AWS, DLT compute can range from $0.20/DBU for DLT Core Compute Photon up to $0.36/DBU for DLT Advanced Compute. However, these prices can be up to twice as high when applying expectations and CDC.

From an efficiency perspective, DLT results in a reduction in total cost of ownership. Automatic orchestration tests by Databricks have shown total compute time to be reduced by as much as half with Delta Live Tables – ingesting up to 1 billion records for under $1.

Conclusion

Delta Live Tables represent a significant advancement in data engineering workflows, offering a unified approach to batch and streaming data processing. By providing built-in data quality checks, automatic orchestration, and comprehensive pipeline visibility, DLT simplifies many of the traditional pain points in data pipeline development.

While there are cost considerations to keep in mind, the efficiency gains and reduced operational overhead often justify the investment, especially for organizations dealing with complex data transformation workflows.

This post was originally published on Sync Computing’s blog on March 8, 2024.

Throughout my career, I have dedicated myself to creating tools, products, and technologies that help people effectively utilize their data. My passion lies in developing products that enable users to efficiently and scalably gain maximum value from their data.

My journey in understanding the intricacies of data and its potential began at Palantir Technologies, where I began working on search and indexing products. As data volumes grew, I focused my efforts on solving some of Palantir customers’ core problems across the financial and defense verticals before leading customer focused compute teams. After Palantir, I served as CTO at Veraset, a cloud-based data-as-a-service company. Veraset delivered high-quality, large scale data to a number of enterprises and grew to 15M ARR before being acquired. Following this, I joined Citadel Investment Group as the Head of Business Engineering of Ashler. In that role, I managed crucial data operations, including overseeing data pipelines, investment platforms, data lakes, and the software and data engineering teams responsible for them.

Currently, I’m working on building a new company in the data platform + AI space.

What are you building?

I am developing a prototype to showcase how Large Language Models (LLMs) can revolutionize a major public institution in New York City. This institution’s data is currently scattered across various systems (not all of which are organized). The solution I’m creating will integrate external APIs, fundamental coding elements, user-generated commands, and the intuitive interaction with a large language model. This will allow individuals to pose complex inquiries in everyday language. My objective is straightforward: by highlighting the recent progress and sophisticated tools available in our field, I aim to initiate this enterprise’s digital transformation.

What technology have you been using so far?

Before adopting CALM, I utilized a blend of LangChain/LlamaIndex for “routing,” Chroma for my Vector Database, and OpenAI’s models for my cloud-based LLM tasks. For on-prem LLMs, I used Databricks Dolly/MPT models. Each of these technologies excelled in their specific functions, but there was a notable lack of cohesive integration among them. As such, my experience largely involved playing a guessing game with temperatures and prompts to steer a conversation into the fixed pattern that I wanted. Additionally, the unpredictable nature of the models meant that maintaining a natural, fluid conversation often had to be sacrificed in favor of rigid rules. This constraint resulted in a user experience that was less enchanting, spontaneous, and as early customers described…less magical.

Describe CALM in 3 words:

“Transformative, Extensible, Simple.”

What are your biggest challenges in building and maintaining an AI assistant?

The majority of tools in the industry are developed with a primary emphasis on the tool itself, rather than creating a holistic user experience. This focus can make the development process challenging and unwieldy for developers who have to navigate and integrate various backend elements, models, and systems in an effort to achieve a smooth user experience.

Furthermore, there is a prevalent ‘all-or-nothing’ mentality in the utilization of Large Language Models (LLMs). This approach suggests that LLMs should either be responsible for the entire workflow, from data analysis and code generation to producing readable output, or not be used at all. Finding products that leverage the strengths of LLMs while also providing developers with flexibility and choice, especially in scenarios where LLMs may not be the most suitable option, remains a challenge.

Finally, AI assistants are fundamentally task oriented. These tasks generally have multiple steps and workflows associated with them. Actually visualizing these steps and ensuring they work under a variety of conditions is a challenging part of the AI assistant development process.

Which functionality of CALM helps the most to resolve the challenges of building AI assistants?

I don’t think I have ever seen a product that has increased my developer velocity more than Rasa Pro with CALM. CALM had 2 features that, in particular, stood out to me.

Declarative YAML flows - CALM’s declarative user based flows puts developers right in the seat of the user. By describing the exact user flow, coupled with validation and data gathered, developers not only can clearly and deliberately articulate the ideal conversation flow, but they can also visualize and extend it. It allowed me to start my development work with an articulation of the happy paths, so I was able to build quickly and with guardrails.

Conversation Repair - In the ideal world, users would follow our predefined conversation flows to the letter. However, real-world interactions are rarely so linear. Users often diverge from the set path, whether to ask questions, explore tangents, or jump into different flows entirely. This is where the magic of conversation repair comes into play. It’s a foundational element that gracefully handles these deviations, allowing users the freedom to stray from the main path without getting lost. They can ask something off-topic, follow a different thread, and then seamlessly return to the original flow. I’ve found that integrating conversation repair into chatbots is crucial for mimicking human interaction. It’s not just about guiding users through a set journey; it’s about creating an experience that acknowledges and adapts to their natural conversational behavior. This flexibility is key to making chatbots more relatable and engaging, transforming them from mere tools into conversational partners.

What prompted you to explore CALM?

I quickly realized the power of a structured approach to chatbot creation while playing with Rasa’s open source tool. This platform enabled me to set specific rules for extracting entities and establish policies for guiding the flow of conversations. Working with Rasa’s open source solution was a game-changer for me as a developer. However, during this process, I began to recognize a limitation: while the Natural Language Understanding (NLU) models were powerful, they didn’t quite capture the more dynamic, almost magical interaction quality offered by Large Language Models (LLMs). This realization led me to explore how LLMs could be integrated with Rasa. This is when I discovered CALM, which combines the power of LLMs while maintaining the controls of an NLU based-approach.

The Rasa Pro Developer Edition from Rasa gave me the chance to explore the full scope of the product functionality and build a prototype.

Link

https://rasa.com/blog/navigating-calm-s-benefits-technical-insights-from-vinoo-ganesh/

Serverless functions have had their 15 minutes of fame (and runtime).

As is the case with many concepts in technology, the term Serverless is abusively vague. As such, discussing the idea of “serverless” usually invokes one of two feelings in developers. Either, it’s thought of as the catalyst for this potential incredible future, finally freeing developers from having to worry about resources or scaling concerns, or it’s thought of as the harbinger of yet another “we don’t need DevOps anymore” trend.

The root cause of this confusion has to do with the fact that the catch-all term “Serverless” actually comprises two large operating models: functions and jobs. At Sync – we’re intimately familiar with optimizing jobs, so when our customers gave us feedback to focus a portion of our attention on serverless functions, we were more than intrigued.

The hypothesis was simple. Could we extend our expertise and background in optimizing Databricks large scale/batch compute workloads to optimizing many smaller batch compute workloads.

Serverless Functions: How We Got Here

One of the most painful parts of the developer workflow is “real world deployment.” In the real world, deploying code that was written locally to the right environment and to work in the same way was extraordinarily painful. Library issues, scaling issues, infrastructure management issues, provisioning issues, resource selection issues, and a number of other issues plagued developers. The cloud just didn’t mimic the ease and simplicity of local developer environments.

Then Serverless functions emerged. All of a sudden, developers could write and deploy code in a function with the same level of simplicity as writing it locally. They never had to worry about spinning up an EC2 instance or figuring out what the material differences between AMI and Ubuntu are. They didn’t have to play with docker files or even have to do scale testing. They wrote the exact same Python or NodeJS code that they wrote locally in a Cloud IDE and it just worked. It seemed perfect.

Soon, mission critical pieces of infrastructure were supported by double digit line python functions deployed in the cloud. Enter: Serverless frameworks. All of a sudden, it became even easier to adopt and deploy serverless functions. Enterprises adopted these functions like hotcakes. Many deployed in the hundreds or even thousands of these functions.

Industry Adoption and Trends

In 2022, an IBM blog post titled “The Future Is Serverless” was published, which cited the “energy-efficient and cost-efficient” nature of serverless applications as a primary reason that the future will be serverless. They make the – valid – case that reserving cloud capacity is challenging and consumers of cloud serverless functions are better served by allowing technologies such as KNative to streamline the “serverless-ification” processes.

In 2023, Datadog released their annual “State of Serverless” post, showing the continued adoption of Serverless technologies across all 3 major cloud vendors. The leader of the pack is AWS Lambda, which has traditionally been the entry point for developers to deploy their Serverless workloads.

Interestingly, 40%+ of Lambda invocations happen in NodeJS – which is not traditionally thought of as a distributed computing framework, nor is it generally used for large scale orchestration of compute tasks. But it seems to be dominating the Lambda serverless world.

What Serverless Actually Solved

Before diving into the problems, let’s acknowledge where serverless functions truly excel:

Uptime Guarantees

One of the critical, but most frustrating pieces of the developer lifecycle is uptime requirements. Many developers hear the term “five-nines” and shudder. Building applications that have specific uptime guarantees is not only challenging, it’s also time-intensive. When large scale systems are made up of small, discrete pieces of computation, the problem can become all the more complex.

Lambda SLAs guarantee a fairly reasonable amount of uptime, right out of the box. This can save otherwise substantial developer efforts of scoping, building, and testing highly available systems.

Concurrency + Auto Scaling

Introspecting a large scale system isn’t easy. In an environment where requests can burst unexpectedly, creating and designing systems that scale based on spot user demand is also difficult.

One of the most powerful aspects of a serverless or hosted model is the demand-based auto-scaling capabilities offered by the infrastructure. These effects are compounded, especially when the functions themselves are stateless. This effectively eliminates developers having to care about the operational concerns of autoscaling.

The Problems with Serverless Functions

Despite these benefits, our analysis revealed four critical problems that are undermining the serverless promise:

Problem 1: Developer Bandwidth

In a typical Serverless Function deployment, the initial choice of configuration tends to be the perpetual choice of configuration.

Wait, “initial choice of configuration”? It turns out, yes, users still need to manually pick a particular configuration for each serverless function they deploy. It’s actually a bit ironic – with the promise of true 0-management jobs, users are still required to intelligently select resource configuration.

If an engineer deploys and accidentally overspecs a serverless function initially, it’s fairly unlikely that they will ever revisit the function to optimize it. This happens for several reasons:

1. Time – Most engineers don’t have the time to go back and ensure that functions they have written weeks, months, or even years ago are operating under the ideal resources.

2. Incentives – Engineers are not incentivized by picking the optimal resource configuration for their jobs. They’d rather have the job be guaranteed to work, while spending a bit more of their company’s compute budget.

3. Employee Churn – Enterprises have inherent entropy and employees are oftentimes transient. When other engineers inherit previous work, they are significantly more incentivized to just ensure it works, rather than ensure that it works optimally.

Problem 2: Serverless Still Requires Tuning

Lambda is predicated on a simple principle – the resource requirements for workloads that take less than 15 minutes to run can be pretty easily approximated. Lambda makes it easy for developers to set-and-forget, offering only one knob for them to worry about.

That knob is memory. Using Lambda, you can configure the memory allocated to a lambda function as a value between 128 MB and 10,240 MB. Lambda will automatically decide how much vCPU to allocate based on the memory setting.

This sounds great in theory. “I only have to pick one lever and all of a sudden, I get everything else figured out for me?” If that were the end of the story, this would be a much shorter post.

Instead, life is all about tradeoffs – generally correlated tradeoffs. In this case, it’s cost and performance. As an engineer, it’s easy to pick the largest memory setting available just to ensure the Lambda function works, regardless of what its actual resource requirements are. Once it works, why would anyone ever touch it again?

Well, it turns out that picking large, frequently uncorrelated-to-necessary-resources values isn’t the most cost effective approach. So much so that an AWS Solutions Engineer built and open sourced a tool to help users actually find the correct memory levels for their Lambda functions. The tool uses AWS Step Functions to walk users down to the minimum necessary level. It’s been so popular that it has 5K stars on GitHub and 18.8K deployments.

Clearly, the one-knob-rules-all solution isn’t working.

Problem 3: Serverless Is Hard to Introspect

The scale and growth testing that plagued engineers for decades before the rise of Serverless was unfortunately not in vain. Understanding how users will interact with an application, in terms of number of requests or compute load, gives engineers a powerful understanding of what to expect when things go live.

In the Serverless Function architecture, engineers don’t think about these considerations and push the burden onto the infrastructure itself. As long as the infrastructure works – it’s unlikely that an already oversubscribed engineer would spend time digging into the performance or cost characteristics of the Serverless function.

Absent home-rolled solutions, there are few tools that allow for the detailed observability of a single serverless function. Furthermore, there are usually hundreds if not thousands of serverless functions deployed. Observability across a fleet of functions is nearly impossible.

The primary mechanism for per-function observability is AWS CloudWatch. CloudWatch logs events for each lambda invocation and stores a few metrics. The major problem is that just collecting this information in CloudWatch has been observed to be more expensive than Lambda itself. There are full articles, posts, and best practices around just managing the costs associated with Lambda CloudWatch logs.

Problem 4: No Auto-Optimization

The year 2023 brought on a material shift in the mentality of “compute” consumers. Enterprises that were previously focused on growth at all costs shifted their focus to efficiency. Vendors in the generic Cloud, Snowflake, and Databricks ecosystem popped up at increasing rates. Most had a simple goal – provide high level visibility into workloads.

They provided interactive charts and diagrams to show ongoing cost changes… But they didn’t provide the fundamental “healing” mechanisms. It would be like going to the doctor, having them diagnose a problem, but provide no recourse.

Consistent with their focus on efficiency, enterprises had a few options. Larger ones deployed full teams to focus on this effort. Smaller ones that didn’t have the budget or manpower turned to observability tools… nearly all of which fell short, as they missed the fundamental optimization component.

Providing detailed visibility across a few, large scale jobs is considered table stakes for many observability providers, but for some reason providing that same level of visibility across many, small scale jobs, in an efficient and easy to optimize way hasn’t become standard.

Conclusion

We’re in a fairly unique period as an industry. Job visibility, tuning, introspection, and optimization have reemerged as key pieces of the modern tech stack. But most focus on the whales, when they should be focusing on the barracudas.

Serverless functions promised to eliminate the complexity of infrastructure management, but they’ve simply shifted that complexity elsewhere. While they excel in certain areas like uptime guarantees and auto-scaling, the hidden costs of poor optimization, lack of visibility, and ongoing manual tuning requirements suggest that the serverless revolution still has room for improvement.

The future of serverless likely lies not in the elimination of optimization concerns, but in the automation of those concerns – turning the promise of “set-and-forget” into reality through intelligent, automated resource management and optimization.

This post was originally published on Sync Computing’s blog on February 9, 2024.

Managing SQL code at scale is one of the biggest challenges in data engineering. As data teams grow and pipelines become more complex, traditional approaches to SQL development quickly become unwieldy.

This LinkedIn Learning course explores how dbt (data build tool) transforms the way we think about SQL development, bringing software engineering best practices to analytics engineering.

Course Approach

Real-World Problem Solving: Each chapter presents actual situations and challenges that data engineers face, with focused code examples showing practical solutions.

Hands-On Implementation: The course covers both basic and advanced dbt concepts through working examples rather than theoretical explanations.

Production-Ready Techniques: Learn to build maintainable, testable SQL transformations that scale with your organization.

What You’ll Learn

The course covers essential dbt concepts including:

• Schema design fundamentals for maintainable data models
• Generating SQL model files efficiently and consistently
• Table materializations and when to use different strategies
• Implementing CTEs (Common Table Expressions) within dbt models
• SQL unit tests to ensure data quality and catch regressions
• Code organization patterns for large dbt projects

Why dbt Matters

Traditional SQL development often involves:

• Copy-pasting code across multiple files
• Manual dependency management
• No testing framework
• Difficult collaboration and code review processes

dbt addresses these challenges by providing:

• Modularity: Break complex transformations into manageable pieces
• Dependencies: Automatic resolution of table and view dependencies
• Testing: Built-in data quality testing framework
• Documentation: Generate and maintain data documentation automatically
• Version Control: Treat analytics code like software with proper CI/CD

Who This Course Is For

This course is designed for:

• Data engineers working with SQL transformations
• Analytics engineers building data models
• Data analysts who want to improve their SQL workflow
• Anyone managing complex SQL codebases looking for better organization

The course assumes familiarity with SQL but doesn’t require prior dbt experience.

Real-World Applications

Throughout the course, we tackle common data engineering challenges:

• Building dimensional models for analytics
• Handling slowly changing dimensions
• Creating reusable macros for complex logic
• Implementing data quality checks
• Managing environments (dev, staging, production)

Key Takeaways

dbt brings software engineering discipline to analytics engineering. By treating SQL transformations as code, teams can build more reliable, maintainable data pipelines.

The tool has fundamentally changed how many organizations approach data transformation, moving from ad-hoc SQL scripts to well-structured, tested, and documented data models.

You can find the course on LinkedIn Learning.

Questions about dbt or data engineering practices? Feel free to reach out.

Data engineering is the unsung hero fueling the rapid growth and consumption of artificial intelligence. It transforms AI’s potential into reality, driving digital innovation and reshaping the world. In this comprehensive discussion, we explore how data engineering unlocks and enables democratized use of Artificial Intelligence.

Video: The Future in Tech - Data Engineering and AI Discussion (1,668 views)

About the Discussion

This LinkedIn Learning session features an in-depth conversation about the critical role of data engineering in the AI revolution. The discussion covers everything from fundamental data engineering principles to the future of AI implementation in organizations of all sizes.

Key Topics Covered:

The Foundation: Data as Infrastructure

“Data as the Ultimate Disinfectant” - The conversation begins with exploring how transparent, well-structured data serves as the foundation for reliable AI systems. Just as sunlight disinfects, proper data engineering practices ensure AI models are built on clean, trustworthy foundations.

From Philosophy to Engineering

The discussion explores an interesting career transition from philosophy to computer engineering, highlighting how diverse educational backgrounds can provide unique perspectives in the data engineering field. This philosophical approach brings valuable analytical thinking to technical problem-solving.

AI Readiness in Organizations

Assessing Company Preparedness

A critical insight emerges: AI readiness mirrors data strategy readiness. Organizations that have invested in robust data infrastructure find themselves better positioned to implement AI solutions effectively. The conversation covers:

• How to evaluate an organization’s AI readiness
• The relationship between data maturity and AI success
• Long-term AI implementation strategies vs. quick wins

The Generative AI Revolution

The discussion delves deep into generative AI, covering:

• Trust in Generative AI: How organizations can build confidence in AI-generated outputs
• Creative Potential: The unprecedented possibilities that generative AI unlocks
• Model Size Advancements: How larger models are changing capabilities
• Context Window Challenges: Technical limitations and their implications

What is Data Engineering?

The session provides a comprehensive definition of data engineering, breaking down:

• Core responsibilities and functions
• How data engineering differs from data science
• The infrastructure challenges unique to data engineering
• Career paths and specializations in the field

Getting Started in Data Engineering

Practical advice for aspiring data engineers includes:

• Educational Paths: Various routes into the field
• Specializations: Different areas of focus within data engineering
• Unstructured Data Engineering: Emerging opportunities in handling complex data types
• Essential Skills: Technical and soft skills needed for success

The Changing Landscape

AI’s Impact on Data Engineering Roles

The conversation explores how AI is transforming data engineering work:

• Operationalizing Dark Data: Making previously unusable data valuable
• Contextualizing AI Models: The critical work of preparing data for AI consumption
• Future Role Evolution: How data engineering positions will adapt and grow

Opportunities for Organizations

Small Companies’ AI Advantages: Surprisingly, smaller organizations may have unique opportunities in the AI space:

• Agility Benefits: Faster implementation and iteration
• Differentiation Strategies: Using unique data as competitive advantage
• Building Around AI Capabilities: Creating AI-native solutions from the ground up

Technical Deep Dives

The discussion covers specific tools and technologies:

• Apache Airflow: Workflow orchestration and management
• Vector Databases: Including Pinecone and Chroma for AI applications
• Data Storage Solutions: From Apache Cassandra to modern cloud platforms
• Unstructured Data Solutions: Handling the growing volume of complex data types

Key Insights and Takeaways

1. Data Strategy First

Organizations must establish solid data foundations before attempting AI implementation. The quality of AI outputs directly correlates with the quality of underlying data infrastructure.

2. The Open Source Advantage

The rapidly evolving open-source ecosystem provides unprecedented opportunities for innovation, especially for smaller organizations that can move quickly.

3. Standardization Challenges

The lack of standards in the AI space creates both challenges and opportunities for differentiation.

4. Future-Proofing Careers

Data engineers who understand both traditional data infrastructure and emerging AI needs will be best positioned for future success.

Episode Resources

The discussion references numerous valuable resources:

• Training Courses: Hands-on data engineering education
• AI Tools: ChatGPT, Claude AI, and other platforms
• Technical Documentation: Apache Airflow, Cassandra, and more
• Industry Analysis: Competitive edge through AI implementation

The Road Ahead

As AI continues its rapid advancement (over 50 minutes of detailed discussion!), data engineering remains the critical enabler. The conversation emphasizes that while AI captures headlines, it’s the underlying data engineering work that makes AI applications possible and reliable.

For Practitioners

Whether you’re starting your data engineering journey or looking to adapt to AI-driven changes, this discussion provides valuable insights into:

• Career development strategies
• Technical skill priorities
• Industry trends and opportunities
• Practical implementation advice

For Organizations

Companies at any stage of AI adoption can benefit from understanding:

• How to assess AI readiness
• The importance of data strategy
• Opportunities for competitive differentiation
• Building sustainable AI capabilities

Conclusion

Data engineering truly is the unsung hero of the AI revolution. As organizations continue to explore AI’s potential, those with strong data engineering foundations will be best positioned to turn that potential into reality.

The future belongs to organizations that understand this fundamental truth: great AI starts with great data engineering.

Watch the full discussion on YouTube - Originally streamed live on LinkedIn Learning’s “The Future in Tech” series.

This course is integrated with GitHub Codespaces, an instant cloud developer environment that offers all the functionality of your favorite IDE without the need for any local machine setup. With GitHub Codespaces, you can get hands-on practice from any machine, at any time—all while using a tool that you’ll likely encounter in the workplace. Check out the “Using GitHub Codespaces with this course” video to learn how to get started.

Link

https://www.linkedin.com/learning/hands-on-introduction-data-engineering

The majority of my career has been focused on making data systems more efficient — whether that means performance, scalability, or cost. This series aims to democratize knowledge about how to Efficiently operationalize data.

TLDR

4 changes you can make right now to run Snowflake more Efficiently:

1. File a Snowflake support ticket and request access to the GET_QUERY_STATS function
2. ALTER WAREHOUSE <warehouseName> SET AUTO_SUSPEND = 60;
3. For multi-cluster warehouses:
   • ALTER WAREHOUSE <warehouseName> SET MIN_CLUSTER_COUNT = 1;
   • ALTER WAREHOUSE <warehouseName> SET SCALING_POLICY = ECONOMY;
4. ALTER WAREHOUSE <warehouseName> SET STATEMENT_TIMEOUT_IN_SECONDS=36000

Snowflake + Driving

Snowflake optimization resembles efficient driving. There are four parallel constraints:

• Car efficiency = warehouse size selection
• Driver skill = query authorship ability
• Road congestion = warehouse saturation
• Route optimization = query construction, schema design, and partitioning

Top Down vs. Bottom Up

This framework separates optimization into two approaches: Top Down (optimizing the environment/infrastructure without affecting users) and Bottom Up (optimizing operations — the “driver”).

This post focuses on the Top Down approach.

Insight #1: Get Data

Solving problems requires data. Two critical functions provide the visibility needed for optimization:

• GET_QUERY_STATS
• GET_QUERY_OPERATOR_STATS (preview feature, available on all accounts)

These functions are not available by default — you’ll need to file a support ticket with Snowflake to request access.

Insight #2: Turn Your Car Off

Your car (warehouse) should only be on when you need it. Minimize the time that it is on and doing nothing.

Warehouses consume credits continuously while active, even during idle periods. The AUTO_SUSPEND parameter automatically stops idle warehouses after a specified duration.

There are four warehouse states to understand:

• Suspended — Off, no charges
• Running/Idle (Will Suspend) — On but no queries executing, within the auto-suspend window. Charging unnecessarily.
• Running/Idle (Won’t Suspend) — On but no queries, yet a query is incoming before suspension triggers. This requires investigation.
• Running/Active — Queries are executing. This is the desired state.

The goal: minimize the Red “No Queries Running” while looking for ways to also minimize the Pink “No Queries Running.”

Snowflake’s minimum auto-suspend is 30 seconds, but charges are incurred for full-minute increments regardless. Setting to 60 seconds is the optimal minimum:

1

ALTER WAREHOUSE <warehouseName> SET AUTO_SUSPEND = 60;

Insight #3: Car Engines + Cylinders

Clusters function like engine cylinders — more clusters mean more processing power but higher consumption.

Snowflake bills at # Warehouse * # Clusters. More clusters enable parallel processing but increase costs proportionally.

Two configuration changes to make right now:

Start with a single cluster minimum:

1

ALTER WAREHOUSE <warehouseName> SET MIN_CLUSTER_COUNT = 1;

Use economy scaling policy (scales only when strictly necessary):

1

ALTER WAREHOUSE <warehouseName> SET SCALING_POLICY = ECONOMY;

The economy policy prioritizes cost efficiency over performance, scaling up only when queries are being queued.

Insight #4: Restrict Trip Distance

Like Federal Motor Carrier Safety Administration regulations limiting how long drivers can be on the road, queries should have maximum execution times.

Snowflake’s STATEMENT_TIMEOUT_IN_SECONDS defaults to 172,800 seconds — that’s 2 days. This permits excessively long, potentially runaway queries to consume credits indefinitely.

Set a reasonable timeout of 36,000 seconds (10 hours):

1

ALTER WAREHOUSE <warehouseName> SET STATEMENT_TIMEOUT_IN_SECONDS=36000

Conclusion

There is a prevalent idea floating around that Snowflake is expensive. That can be true, but as is the case in most of these systems, it really comes down to how effectively and Efficiently you use Snowflake.

We’ve spoken a lot about on-disk and distributed storage, as well as blocks. All of this theory is great, let’s talk about this in practice.

In this post, I’m going to:

1. Read a CSV dataset into Spark
2. Write the dataset into 5 Parquet files (treating each file as a block)
3. Introspect metadata existing on the files
4. Run queries demonstrating predicate pushdown power

Hands-On: Setup

The tutorial uses an airports dataset. Download it via:

1

wget https://raw.githubusercontent.com/curran/data/gh-pages/vegaExamples/airports.csv -O dataset.csv

The CSV contains columns: iata, name, city, state, country, latitude, longitude.

Loading Data into Spark

1

val dataset = spark.read.option("header","true").option("inferSchema","true").csv("dataset.csv")

The inferred schema shows latitude and longitude as double types.

Writing Parquet Files

1

dataset.repartition(5).write.parquet("/root/parquet_dataset")

This creates 5 Parquet files plus a _SUCCESS flag file.

Inspecting Parquet Metadata

Install the inspection tools:

1
2

pip3 install parquet-tools
pip3 install parquet-metadata

View file contents:

1

parquet-tools show part-00000-53b27d15-b049-41db-a8aa-fa3033763836-c000.snappy.parquet

Hands-On: Query Plans

Simple Filter Query

1
2

val simpleFilter = dataset.filter($"latitude" > 30)
simpleFilter.show()

The result shows all rows where latitude exceeds 30.

The query plan analysis reveals three optimization stages: parsed logical plan, analyzed logical plan, and optimized logical plan. The optimized logical plan has added some null checking — which also matches our predicate.

Complex Filter Query

1

val complexFilter = dataset.filter($"latitude" > 30).filter($"latitude" < 40)

As you can see, the plan has combined both of our predicates into one step as part of the query process (meaning that what would previously take two passes over the data now only requires one).

The optimized plan consolidates the filters: Filter ((isnotnull(latitude#21) AND (latitude#21 > 30.0)) AND (latitude#21 < 40.0))

Hands-On: Querying with Parquet

Row Group Metadata Analysis

1

parquet-metadata /root/parquet_dataset/part-00000-...

Critical metadata fields include:

• stats:min — smallest value in the column
• stats:max — largest value in the column

Example statistics from one file:

1
2

row_group 0 latitude stats:min 14.1743075
row_group 0 latitude stats:max 70.46727611

Predicate Pushdown Mechanism

For a second file with stats:min=44.4430157 and stats:max=74.46727611, a query filtering for latitude between 30 and 40 would exclude this entire file — because we know from the metadata that no values in this file fall within our filter range.

In practice, this is called predicate pushdown. The requirements of the predicate (the query) have been pushed down, allowing the optimizers to look at the metadata on the row groups themselves to decide which row groups to read, and when they can be ignored.

Conclusion

There is a lot of magic that goes into our ability to query data quickly and Efficiently. Query optimizers do a lot for us — and understanding how they work under the hood helps us write better queries and design better data layouts.

This is a continuation of a previous blog post about efficient data partitioning.

In the previous post, I discussed how data layout on disk impacts analytics performance. This post focuses on tactical implementation using open source technologies.

Topics I’ll cover:

1. HDFS
2. Blocks + Block Size
3. Block sizes + tradeoffs

Background

Data organization on disk dramatically affects analytics performance. I previously explored row-oriented, columnar, and hybrid storage models — now let’s connect these concepts to modern data infrastructure.

Hadoop

Hadoop provides an ecosystem enabling:

1. Distributed data storage (HDFS — Hadoop Distributed File System)
2. Data querying (MapReduce)
3. Compute resource management (YARN)
4. Additional commons library and object store (Ozone)

This discussion focuses on HDFS as the most relevant storage-side component for understanding data partitioning.

HDFS

Let’s say you have an invaluable file that contains the names, addresses, and phone numbers of employees at your company.

That file lives on a single server. What happens if the server goes down?

Nobody can access the file. That’s a problem. So, you decide to store a copy of the file on two servers.

Great — now if one server goes down, the file is still available. But what happens when someone updates the file? You now have two copies that need to stay in sync.

This orchestration complexity is the central problem that HDFS solves.

Blocks

HDFS stores data in blocks. These are the same Blocks from the last post. They are indivisible segments of data.

A block represents the minimum amount of data readable in a single operation — any read requires reading at least one complete block.

Files are divided into blocks during storage.

These blocks are then distributed across multiple cluster nodes.

And then replicated for fault tolerance.

Architecture Note: These diagrams intentionally simplify HDFS architecture, omitting NameNode/DataNode separation, clients, and other components. The focus remains on partitioning concepts.

Block Size

The default Hadoop block size is 128MB (previously 64MB). While initially seeming massive when working with small local files, this size becomes relevant when storing files of hundreds of gigabytes.

The block size is configurable on a per-client basis.

Tradeoffs

Block size selection determines the number of chunks and corresponding I/O operations for reading and writing.

Larger blocks:

• Fewer blocks created
• Fewer I/O operations needed
• Increased memory requirements during processing

Smaller blocks:

• More blocks created
• More I/O operations required
• Lower memory consumption per block
• Benefits for small files and random access patterns
• Increases NameNode metadata overhead
• Creates scalability risks

The small files problem is a well-documented concern in distributed systems — too many small files create operational challenges.

Block size also impacts HDFS fault tolerance, though that falls outside this article’s scope.

Tuning for Efficiency

Reducing I/O through partition pruning provides substantial performance gains. Optimal block size selection yields strong results.

However, no universal solution exists. Selection depends on dataset characteristics, usage patterns, data type, use case, and additional factors.

High-level recommendations:

Files under a few hundred megabytes: Use smaller block sizes (64MB or 128MB) to minimize wasted block space.

Files several gigabytes or larger: Use larger block sizes (256MB or 512MB) to minimize generated blocks and unnecessary I/O.

These are just guidelines and selecting your optimal block size will likely require some testing on your side.

A few years ago, I gave a talk at Spark Summit 2020 about File Formats covering Avro, ORC, and Parquet. I received numerous questions about that topic, responding point-to-point, leaving the knowledge confined to those forums alone.

That isn’t helpful for most people. This post aims to fix that.

In this series, I’ll outline the primitives of this topic and then explore the hands-on details.

Problem

In the efficiency space, minimizing “work” is key. Whether work requires compute, network, or storage, “the goal of efficient data usage is to get the most accurate answer in the fastest and cheapest way possible.”

File Formats help data practitioners store their data in ways that minimize work. When you think of a file format, you may think of extensions like .xlsx, .pdf, .pptx. Similarly, technologies like Parquet, Avro, and ORC serve this purpose.

Background / Example Data

A partition is a logical segment of data. In the big data world, this usually means a piece of a larger dataset. For our purposes, I’m going to use an example dataset below.

This dataset has 3 columns (Column A, Column B, and Column C) and 4 rows (Row 0, Row 1, Row 2, and Row 3).

This table should look familiar — something you’ve seen in Excel, Pandas, etc. Let’s take this example further and split the individual elements into their own logical “pieces.”

We can refer to each “cell” by its “<column><row>.” For example, the second row in Column B is called B1.

Storage

Background

Data is stored on hard disks in what is called a block. A block is the minimum amount of data read during any read operation.

Blocks function like a suitcase. When checking a bag on a trip, you pay the same price regardless of how full or empty your suitcase is. It’s optimal to fill your suitcase with as many relevant objects as possible, in as easy a way to find as possible.

Extending this analogy: packing unnecessary stuff isn’t great. Bringing too many suitcases (unless strictly necessary) also isn’t great. Inside the suitcase, you want to “group” similar things together — each pair of socks should be next to each other in the same suitcase, rather than split across different ones.

In hard drives, these insights apply. Reading unnecessary data is expensive. Reading fragmented data is expensive. Random seeks are expensive as well.

Our goal is to lay data out in a manner optimized for our workflows.

Row-wise Storage

In database land, the common way to store data used to be row-wise. It’s pretty easy to understand why. Most people think about datasets as a list of rows.

Taking our dataset above, let’s store this in a row-wise method.

I have taken each row in order and packed as much of the rows as I can into a block, before moving to the next block.

This method works great when the goal is to read the data sequentially. All that’s required is a simple linear scan of the block in order. It doesn’t work as well if, for example, you want to only look at Column C. In that case, you’re required to read all of the block (i.e., read all of the data) and filter down to Column C.

This is row-wise storage methodology.

Columnar (Column-wise) Storage

Column-wise storage takes the opposite approach and orients around columns.

As you can see, we first take the entire column, pack it into a block, and then move onto the next column.

This method works great when the data is read in a columnar way (i.e., one column at a time). It doesn’t work well if, for example, you want to reconstruct Row 0. In that situation, you’d need to read all of the data and filter down to the elements that make up Row 0.

Now, we’re in a dilemma — one approach seems to favor a row-oriented workflow, one approach seems to favor a column-oriented workflow. Luckily for us (and Goldilocks), there’s a middle ground.

Hybrid Storage

A hybrid storage model gives us the best of both worlds. First, we group a fixed number of Rows together and then further group that by columns. We segment these and call these “Row Groups” (at least in the Parquet terminology).

In this example, we first selected two rows — Row 0 and Row 1. We then grouped those rows by column, and inserted them into our first Row Group.

I called these logical Row Groups because this is more of how we should be thinking about them, rather than how they may necessarily end up on disk.

This representation of data is actually immensely powerful. It allows us to optimize our workflows for both row-oriented and column-oriented operations.

Let’s talk about how this works.

In the case of a row-oriented workflow, let’s say you want to recreate Row 2. To do this, you would simply need to look at Block 1 and Block 2. If you were operating in a Columnar storage model, you would need to look at Block 1, Block 2, and Block 3. You’ve saved a whole Block!

In the case of a column-oriented workflow, let’s say you want to recreate Column B. In this case, you would simply need to look at Block 1 and Block 2. If you were operating in a Row-wise storage model, you would need to look at Block 1, Block 2, and Block 3. You’ve once again saved a whole Block!

Our examples used very small data, you can imagine how this extrapolates further with larger datasets.

Data Workflows

Throughout this post, I’ve referred to my data workflows as “row oriented” or “column oriented.” Luckily for us, the big data community has come up with some terminology that should help bring these two workflows to life.

OLTP

Online Transaction Processing (OLTP) workloads generally involve larger amounts of short queries/transactions. These tend to be more focused on processing than analytics and as such have more data updates and deletions. Roughly — we can consider OLTP workflows as “row oriented” workflows.

OLAP

Online Analytical Processing (OLAP) workloads are more analysis than processing focused. As such, there tends to be more analytical complexity per query and fewer CRUD transactions. Roughly — we can consider OLAP workflows as “column oriented” workflows.

Conclusion

Using data efficiently relies on using all levels of the “data stack” (storage, network, compute) efficiently. Reducing the amount of unnecessary data read during a query process can have compounding effects on the speed and efficiency of your analytics process.

In subsequent parts of this series, I’ll be digging more into the details of how everything we have covered thus far can be applied in analytics workloads.

My name is Vinoo and I’ll be your guide, writer, and (likely) ranter throughout this series.

0. What this is

Recently, I had a friend recommend Austin Kleon’s Show Your Work. I read it over the course of a short plane ride and realized… it’s time to start showing my work. Publicly.

This blog series, at its core, is a collection of things learned while building analytical tools, creating data products, and consuming data products. Mostly, it’s intended as a technical series for technical audiences, but the direction remains uncertain.

My goal is to extract meaningful and applicable lessons from the challenges I’ve faced working in the data ecosystem and democratize the best practices I’ve learned along the way.

1. Why this, why now

The data industry experienced a golden age. Cloud migration accelerated. Organizations became data-driven. The modern data stack seemed powerful and limitless. Capital flowed freely, making expensive data platform investments seem justified.

Economic conditions shifted dramatically. Layoffs multiplied. Companies cut visible expenses — office space, travel, contractor fees, and staff. However, their compute bills are increasing by an order of magnitude year over year.

The compounding cost of clouds is an existential threat for enterprises without capital assessment mechanisms. Data practitioners who understand cost optimization methodology could literally save their companies from unexpected bills — or worse.

2. Who may benefit from this

Y Combinator recently distributed guidance to founders emphasizing cost management. This series targets anyone seeking to optimize data and technology spending during constrained periods.

3. Why me

I’ve spent over a decade working across tech and finance, building analytical tools, brokering data, and optimizing internal tech stacks. I’ve made numerous mistakes along the way, and I’ve learned from them.

4. Where I’ll start

The series will cover data storage, management, analysis, visualization, case studies, and best practices based on personal experience and industry examples.

I’m thrilled to have you join me on this journey, and I hope you’ll find value in the content we’ll be sharing.

Video

Link

https://datastackshow.com/podcast/optimizing-query-workloads-and-your-snowflake-bill-with-vinoo-ganesh-of-bluesky-data/

About the Data Superstream Series: This three-part Superstream series is designed to help your organization maximize the business impact of your data. Each day covers different topics, with unique sessions lasting no more than four hours. And they’re packed with insights from key innovators and the latest tools and technologies to help you stay ahead of it all.

Vinoo Ganesh: Zero to Pipeline (30 minutes) - 9:20am PT | 12:20pm ET | 4:20pm UTC/GMT

There are few moments more daunting to data practitioners than deploying your first data pipeline. The flexibility, freedom, and development speed of the data pipeline ecosystem allows for endless tuning, customization, and configuration. . .but makes getting started overwhelming and difficult. In this live coding session, Vinoo Ganesh takes you through scoping, building, deploying, and running a fully functioning ETL pipeline in Airflow in just 30 minutes—all in a local developer environment. You’ll also learn how to simplify each step of the ETL process into a task in a job execution DAG. Join in to get the tools and knowledge to stand up your own pipeline developer environment at home.

Link

https://learning.oreilly.com/live-events/data-superstream-building-data-pipelines-and-connectivity/0636920064968/0636920064967/

Code

https://github.com/apache/parquet-site/commit/3563721676b364b767058a953f2bcc3e2c0c4b09

Link

https://parquet.apache.org

Link

https://www.horangi.com/blog/s3-buckets-permissions-and-iam-audits

This session will cover the core components that each data pipeline needs from an operational and functional perspective. We’ll discuss a framework that will allow practitioners to set their pipelines up for success. We’ll also discuss how to leverage data pipelines for metrics gathering and how pipelines can be architected to alert on potential data problems before the fact.

Sessions

• March 10, 2022
• June 17, 2022
• September 19, 2022

Link

• https://www.oreilly.com/videos/oreilly-radar-data/0636920654667/
• https://www.businesswire.com/news/home/20210909005792/en/O%E2%80%99Reilly-Announces-O%E2%80%99Reilly-Radar-Data-AI-to-Help-Tech-Leaders-Drive-Innovation-and-Successful-Implementation

Link

https://web.archive.org/web/20221007160344/https://databand.ai/mad-data-podcast/defining-data-quality-data-sla-nightmares-lessons-learned/

As data products become central to companies’ bottom line, data engineering teams need to create higher standards for the availability, completeness, and fidelity of their data.

In this session we’ll demonstrate how Databand helps organizations guarantee the health of their Airflow pipelines. Databand is a data pipeline observability system that monitors SLAs and data quality issues, and proactively alerts users on problems to avoid data downtime.

The session will be led by Josh Benamram, CEO and Cofounder of Databand.ai. Josh will be joined by Vinoo Ganesh, an experienced software engineer, system architect, and current CTO of Veraset, a data-as-a-service startup focused on understanding the world from a geospatial perspective.

Join to see how Databand.ai can help you create stable, reliable pipelines that your business can depend on!

Link

https://airflowsummit.org/sessions/2021/data-quality-standards-databand/

Video

Link

https://www.dremio.com/subsurface/migrating-to-parquet-the-veraset-story/

This breakout session will describe Veraset’s partnership with Databricks, and how we have white labeled Databricks to showcase and accelerate the value of our data. We’ll discuss the challenges that data brokers have faced to date and some of the primitives of our businesses that have guided our direction thus far. We will also actively demo our white label instance and notebook to show how we’ve been able to provide key insights to our customers and reduce the TTFB of data onboarding.

Link

https://databricks.com/session_na21/brokering-data-accelerating-data-evaluation-with-databricks-white-label

Video

In this event, you’ll gain insight into design and engineering best practices through interactive sessions and live coding demos. Join us to learn how to make the right decisions for your applications.

About the Strata Data Superstream Series: This four-part series of half-day online events gives attendees an overarching perspective of key topics that will help your organization maximize the business impact of your data.

Link

https://www.oreilly.com/videos/strata-data-superstream/0636920551973/

He then goes over the techniques used to extract useful data. The distribution of data and data types have the most significant impact on data quality. Vinoo talks about the challenges of working with data, where a simple data movement can present a massive problem. Constant profiling is needed to help scale the data and make sure that the computing power can cope.

Finally, the guest talks about handling messy data that doesn’t have the required quality. He talks about the multiple problems data scientists have to consider to sort messy data to make it more useful.

Link

https://datastandard.io/podcast/large-scale-data-analytics-with-vinoo-ganesh-at-veraset/

After this presentation, attendees should understand the core concepts behind the prevalent file formats, the relevant file-format specific settings, and finally how to select the correct file format for their jobs. This presentation is relevant to Spark+AI summit because as more AI/ML workflows move into the Spark ecosystem (especially IO intensive deep learning) leveraging the correct file format is paramount in performant model training.

Link

https://databricks.com/session_na20/the-apache-spark-file-format-ecosystem

Video

An engineering undergraduate student at Washington University in St. Louis helped create and launch a mobile app that helps students track the campus circulator shuttle. Vinoo Ganesh, a senior majoring in computer science in the School of Engineering & Applied Science, developed the real-time tracking app, titled “WUSTL Circulator.” The app also shows the circulator’s route and stopping points along with a full schedule that students can browse.

Ganesh developed the application in collaboration with staff from WUSTL’s Parking and Transportation Services and Information Services & Technology.

Ganesh says he took a class on building apps for iPhones and was looking for a way to translate that knowledge into a real-world project. Several friends on campus suggested that knowing when the circulator shuttle will arrive would be a big help.

“The reason I did this is a service to the Wash U students,” he says. “All the administrators got completely behind the effort.”

John Bailey, assistant director of Student Technology Services in IS&T, coordinated the effort and worked closely with Ganesh on the project.

“Vinoo’s idea and prototype app were fantastic,” Bailey says, adding that some additional work was necessary.

The university needed to purchase and install GPS units on the shuttles, set up a server application to feed location data to the app and extensively test it, he says.

Parking and Transportation provided funding for the project, which allowed the university to purchase the necessary GPS units and associated data plans. IS&T provided the server infrastructure to drive the app.

Oliver Jones, a graduate engineering student and a student staff member in Student Technology Services, was instrumental in developing the server mechanisms that collect GPS data and feed it to the app.

As with other student-developed university mobile apps, Ganesh partnered with IS&T during the publication process. IS&T has established guidelines and procedures for reviewing and publishing student-developed apps and already has helped publish 11 apps that were developed in part, or in whole, by students, Bailey says.

The circulator app for iPhones, iPads and iPod touch devices went live on Jan. 23 and is available for free through the iOS app store. An Android version also is in the works and could be ready in coming weeks. Jones is working on that project as well.

As an indication of the app’s demand and popularity, Apple’s stat tracking tool indicated the WUSTL Circulator app had 1,102 downloads on its first day of availability – three to four times the typical number for the first day of a new WUSTL app, Bailey says. By now, it has been downloaded more than 2,300 times, he says.

Ganesh says it’s great to add publishing an app to his résumé, but he also hopes to encourage other student developers to pursue their ideas, saying the university’s support was instrumental in helping his project succeed.

Parking and Transportation hopes the WUSTL Circulator app encourages more people to use the shuttle and improves the experience for regular student riders.

Link

https://source.wustl.edu/2013/02/new-mobile-app-helps-students-track-campus-shuttle/

Before Kepler, I was Head of Business Engineering for Ashler at Citadel, CTO of Veraset (a geospatial data-as-a-service startup processing 2+ TB daily for 80+ customers), and started my career at Palantir, where I worked on its search and indexing backend, managed its Spark/YARN compute platform, and designed Project Frontline, the program for scaling Palantir’s Forward Deployed Engineering team.

I teach data engineering, dbt, and AI engineering on LinkedIn Learning, advise a handful of startups, and serve on Washington University’s McKelvey Engineering National Council and Alumni Board of Governors.

I write about data engineering, AI, and building systems in my writing, and share talks & courses and projects I’ve built. See what I’m up to now.

Drop me a note. Would love to chat.

Email me • LinkedIn • GitHub

We’re a team of engineers who’ve built mission-critical systems at Palantir, Citadel, and PhysicsX. We’re backed by some of the architects of modern data and AI platforms, including founders of OpenAI, Facebook AI, MotherDuck, dbt, Outerbounds, and others.

Email me about Kepler. • Follow along on the Kepler website or LinkedIn page.

Where my time goes

• Kepler. Most of it. Recruiting, customer conversations, and shipping the platform with the team.
• Writing. When something is hard enough to explain that I want to write it down, it ends up on this site.
• Teaching. Courses on LinkedIn Learning. My latest is Fundamentals of AI Engineering.
• Advising. A small number of founders building data and AI infrastructure.

What I’m thinking about

• Why verifiable AI is the only kind of AI that will work in regulated industries
• How Forward Deployed Engineering scales, and where it breaks
• Context engineering as a data engineering problem in disguise
• The tooling gap between AI demos and AI in production

Recently

• How Kepler built verifiable AI for financial services with Claude: published on the Anthropic customer blog
• Build Products Like a Forward Deployed Engineer: talk on the FDE mindset for product builders
• The Definitive Guide to Forward Deployed Engineering: the role, the skills, and how Palantir’s Project Frontline scaled it
• Context Is The Easy Part: why context engineering is really an engineering problem

Read more about me or check out my writing, talks & courses, and projects.

Email me directly.