Data Engineering

Originally published on Next Play If you work in tech, you should learn about Forward Deployed Engineering (FDE). It’s quickly become one of the most popular roles that nearly all fast-growing AI companies are looking for…at the same time, it’s the type of role very few people and companies seem to actually understand. I designed Project Frontline at Palantir, a program that sent over 250 engineers into live customer deployments. Those engineers are now at OpenAI, xAI, Anduril, and dozens of leading companies. ...

Originally published on Kepler Everyone’s talking about context engineering right now, but most of the conversation is focused on the wrong thing. Read the blog posts, the guides, the thought leadership. They’re all asking the same questions: What should I include in the context window? How do I manage tokens efficiently? How do I curate what the model sees? These are valid questions. They’re also the easy part. The hard part isn’t deciding what context to include. It’s building systems that deliver that context reliably, with provenance, at scale, every single time. That’s not a context problem. That’s an engineering problem. And engineering means something specific. ...

Originally published on Kepler AI produces numbers that are incorrect, untraceable, and unverifiable. Asking the same question twice yields different answers, and there’s no way to distinguish right from wrong outputs. This unreliability poses serious risks across industries like healthcare, legal, insurance, government, and finance where numerical accuracy directly impacts outcomes. Through conversations with 137 financial firms - including private equity, hedge funds, and investment banks - a consistent pattern emerged: everyone wants to use AI, but nobody trusts it. As one managing director stated, “I can’t put a number in front of a client if I can’t show where it came from.” ...

I joined Joe Reis on The Joe Reis Show to talk about open source communities — what makes them strong, what causes them to stall, and what contributing to a project like Apache Parquet actually looks like from the inside. The conversation also covers LLMs and AI, and where they do and don’t change the fundamentals of data engineering work. Link YouTube Spotify Apple Podcasts

Originally published on Sync Computing Databricks’ DLT offering showcases a substantial improvement in the data engineer lifecycle and workflow. By offering a pre-baked, and opinionated pipeline construction ecosystem, Databricks has finally started offering a holistic end-to-end data engineering experience from inside of its own product, which provides superior solutions for raw data workflow, live batching and a host of other benefits detailed below. Since its release in 2022, Databricks’ Delta Live Tables have quickly become a go-to end-to-end resource for data engineers looking to build opinionated ETL pipelines for streaming data and big data. The pipeline management framework is considered one of the most valuable offerings on the databricks platform, and is used by over 1,000 companies including Shell and H&R block. ...