Data Engineering

Originally published on Next Play If you work in tech, you should learn about Forward Deployed Engineering (FDE). It’s quickly become one of the most popular roles that nearly all fast-growing AI companies are looking for…at the same time, it’s the type of role very few people and companies seem to actually understand. I designed Project Frontline at Palantir, a program that sent over 250 engineers into live customer deployments. Those engineers are now at OpenAI, xAI, Anduril, and dozens of leading companies. ...

Originally published on Kepler Everyone’s talking about context engineering right now, but most of the conversation is focused on the wrong thing. Read the blog posts, the guides, the thought leadership. They’re all asking the same questions: What should I include in the context window? How do I manage tokens efficiently? How do I curate what the model sees? These are valid questions. They’re also the easy part. The hard part isn’t deciding what context to include. It’s building systems that deliver that context reliably, with provenance, at scale, every single time. That’s not a context problem. That’s an engineering problem. And engineering means something specific. ...

Originally published on Sync Computing Databricks’ DLT offering showcases a substantial improvement in the data engineer lifecycle and workflow. By offering a pre-baked, and opinionated pipeline construction ecosystem, Databricks has finally started offering a holistic end-to-end data engineering experience from inside of its own product, which provides superior solutions for raw data workflow, live batching and a host of other benefits detailed below. Since its release in 2022, Databricks’ Delta Live Tables have quickly become a go-to end-to-end resource for data engineers looking to build opinionated ETL pipelines for streaming data and big data. The pipeline management framework is considered one of the most valuable offerings on the databricks platform, and is used by over 1,000 companies including Shell and H&R block. ...

Originally streamed live on August 3, 2023 - LinkedIn Learning’s “The Future in Tech” series Data engineering is the unsung hero fueling the rapid growth and consumption of artificial intelligence. It transforms AI’s potential into reality, driving digital innovation and reshaping the world. In this comprehensive discussion, we explore how data engineering unlocks and enables democratized use of Artificial Intelligence. Video: The Future in Tech - Data Engineering and AI Discussion (1,668 views) ...

Originally published on Efficiently (Substack) The majority of my career has been focused on making data systems more efficient — whether that means performance, scalability, or cost. This series aims to democratize knowledge about how to Efficiently operationalize data. TLDR 4 changes you can make right now to run Snowflake more Efficiently: File a Snowflake support ticket and request access to the GET_QUERY_STATS function ALTER WAREHOUSE <warehouseName> SET AUTO_SUSPEND = 60; For multi-cluster warehouses: ALTER WAREHOUSE <warehouseName> SET MIN_CLUSTER_COUNT = 1; ALTER WAREHOUSE <warehouseName> SET SCALING_POLICY = ECONOMY; ALTER WAREHOUSE <warehouseName> SET STATEMENT_TIMEOUT_IN_SECONDS=36000 Snowflake + Driving Snowflake optimization resembles efficient driving. There are four parallel constraints: ...