Most products fail not because of bad engineering, but because teams build the wrong thing. They either overbuild features nobody asked for or underbuild solutions that miss the real problem. Forward Deployed Engineering (FDE) is the antidote.

What is FDE?

Forward Deployed Engineering is a methodology where technical teams embed directly with customers to witness how software meets reality. Instead of building from assumptions, FDEs develop customer instinct by being in the field — watching real users interact with real products under real conditions.

Companies like Palantir, OpenAI, and Anduril use this approach to achieve dramatically higher product adoption rates.

What You’ll Learn

This 30-minute lesson covers the core FDE playbook:

• Detecting real customer problems — how to separate signal from noise when users tell you what they want vs. what they actually need
• The four core FDE moves — detect problems, demonstrate through action, control the narrative, and ship fast while maintaining production quality
• A structured week-by-week playbook for developing field-based customer instinct
• How AI is changing the FDE role and what that means for engineers building products today

Background

I built and led Project Frontline at Palantir, training 250+ engineers in the FDE methodology. Alumni now work at OpenAI, xAI, Anduril, and other AI-focused companies. I also led FDE practices at Citadel before co-founding Kepler.

Who This Is For

Whether you’re a product engineer, technical founder, or engineering leader — if you ship software to customers, the FDE mindset will change how you think about building products. The lesson also covers how FDE differs from professional services and how to balance customer wants vs. actual needs.

The lesson is free and includes Q&A. You can watch it on Maven.

Questions about FDE or product engineering? Feel free to reach out.

The world of AI engineering is moving incredibly fast. Every week brings new models, techniques, and breakthroughs. But beneath all that chaos, there are sophisticated patterns and architectural principles that remain consistent across implementations.

I recently published a new LinkedIn Learning course: Fundamentals of AI Engineering: Principles and Practical Applications.

Why This Course Matters

After building mission-critical systems at companies like Palantir and Citadel, I’ve learned that the gap between AI research and production-ready systems is often wider than expected. This course bridges that gap by focusing on the engineering fundamentals that actually matter in production environments.

This isn’t another theoretical AI course. It’s designed for software engineers who want to build AI systems that scale, perform reliably, and solve real business problems.

Course Approach

Hands-On Implementation: Everything is built using open-source tools like LlamaIndex and Hugging Face. The course uses real code, real data, and real challenges rather than theoretical examples.

Production-First Mindset: The focus is on systems that can handle real-world loads, not just demo scenarios.

GitHub Codespaces Integration: Students can start coding immediately without environment setup complexity.

Course Deep Dive

Foundation: Local LLM Operations

We start by running large language models locally, understanding the complete pipeline from tokenization to inference. You’ll learn to move beyond the API-driven approach and understand what’s actually happening under the hood.

Document Processing at Scale

Real-world AI applications need to handle messy, unstructured data. We cover:

• Advanced text extraction techniques
• Structure recognition and metadata enrichment
• Optimal chunking strategies for different document types
• Performance considerations for large document corpuses

The Embedding Ecosystem

Embeddings are the foundation of modern AI retrieval systems. You’ll master:

• Comparing and selecting embedding models for your use case
• Efficient embedding generation and batch processing
• Understanding the trade-offs between speed, accuracy, and cost

Vector Database Mastery

Moving beyond simple similarity search to production-grade vector operations:

• Database selection and optimization
• Approximate Nearest Neighbor (ANN) algorithms
• Caching strategies for performance
• Scaling considerations and cost management

Advanced Retrieval Engineering

This is where the magic happens. We build sophisticated retrieval systems that combine:

• BM25 and vector search for comprehensive coverage
• Hybrid retrieval that leverages the strengths of both approaches
• Cross-encoder reranking for precision improvements
• Complete pipeline integration with monitoring and observability

What You’ll Learn

This 4+ hour course covers:

• Building production-ready RAG systems using embeddings and vector database pipelines
• Implementing monitoring and observability for AI applications using telemetry tools
• Creating efficient document processing pipelines with hybrid search capabilities
• Designing CI/CD workflows for deploying and testing AI applications
• Optimizing AI system performance and costs through caching and resource management

Who Should Take This Course

This course is perfect for:

• Software engineers looking to add AI capabilities to their toolkit
• Backend developers who want to understand AI system architecture
• Technical leaders planning AI implementations
• Anyone building production AI applications who needs to go beyond simple API calls

The course assumes intermediate programming knowledge but doesn’t require prior AI experience.

Real-World Applications

Throughout the course, we build systems that mirror real production challenges:

• Enterprise document search and retrieval
• Customer support automation
• Knowledge base augmentation
• Multi-modal content processing

Key Takeaways

AI engineering isn’t just about calling APIs or fine-tuning models. It’s about building reliable, scalable systems that solve real problems. The course focuses on developing the engineering judgment needed to build AI systems that actually work in production.

The field is moving fast, but the fundamentals remain constant. Understanding these patterns provides a solid foundation for whatever comes next in AI development.

You can find the course on LinkedIn Learning.

Questions about the course content or AI engineering in general? Feel free to reach out – I love talking about this stuff.

Managing SQL code at scale is one of the biggest challenges in data engineering. As data teams grow and pipelines become more complex, traditional approaches to SQL development quickly become unwieldy.

This LinkedIn Learning course explores how dbt (data build tool) transforms the way we think about SQL development, bringing software engineering best practices to analytics engineering.

Course Approach

Real-World Problem Solving: Each chapter presents actual situations and challenges that data engineers face, with focused code examples showing practical solutions.

Hands-On Implementation: The course covers both basic and advanced dbt concepts through working examples rather than theoretical explanations.

Production-Ready Techniques: Learn to build maintainable, testable SQL transformations that scale with your organization.

What You’ll Learn

The course covers essential dbt concepts including:

• Schema design fundamentals for maintainable data models
• Generating SQL model files efficiently and consistently
• Table materializations and when to use different strategies
• Implementing CTEs (Common Table Expressions) within dbt models
• SQL unit tests to ensure data quality and catch regressions
• Code organization patterns for large dbt projects

Why dbt Matters

Traditional SQL development often involves:

• Copy-pasting code across multiple files
• Manual dependency management
• No testing framework
• Difficult collaboration and code review processes

dbt addresses these challenges by providing:

• Modularity: Break complex transformations into manageable pieces
• Dependencies: Automatic resolution of table and view dependencies
• Testing: Built-in data quality testing framework
• Documentation: Generate and maintain data documentation automatically
• Version Control: Treat analytics code like software with proper CI/CD

Who This Course Is For

This course is designed for:

• Data engineers working with SQL transformations
• Analytics engineers building data models
• Data analysts who want to improve their SQL workflow
• Anyone managing complex SQL codebases looking for better organization

The course assumes familiarity with SQL but doesn’t require prior dbt experience.

Real-World Applications

Throughout the course, we tackle common data engineering challenges:

• Building dimensional models for analytics
• Handling slowly changing dimensions
• Creating reusable macros for complex logic
• Implementing data quality checks
• Managing environments (dev, staging, production)

Key Takeaways

dbt brings software engineering discipline to analytics engineering. By treating SQL transformations as code, teams can build more reliable, maintainable data pipelines.

The tool has fundamentally changed how many organizations approach data transformation, moving from ad-hoc SQL scripts to well-structured, tested, and documented data models.

You can find the course on LinkedIn Learning.

Questions about dbt or data engineering practices? Feel free to reach out.

Data engineering is the unsung hero fueling the rapid growth and consumption of artificial intelligence. It transforms AI’s potential into reality, driving digital innovation and reshaping the world. In this comprehensive discussion, we explore how data engineering unlocks and enables democratized use of Artificial Intelligence.

Video: The Future in Tech - Data Engineering and AI Discussion (1,668 views)

About the Discussion

This LinkedIn Learning session features an in-depth conversation about the critical role of data engineering in the AI revolution. The discussion covers everything from fundamental data engineering principles to the future of AI implementation in organizations of all sizes.

Key Topics Covered:

The Foundation: Data as Infrastructure

“Data as the Ultimate Disinfectant” - The conversation begins with exploring how transparent, well-structured data serves as the foundation for reliable AI systems. Just as sunlight disinfects, proper data engineering practices ensure AI models are built on clean, trustworthy foundations.

From Philosophy to Engineering

The discussion explores an interesting career transition from philosophy to computer engineering, highlighting how diverse educational backgrounds can provide unique perspectives in the data engineering field. This philosophical approach brings valuable analytical thinking to technical problem-solving.

AI Readiness in Organizations

Assessing Company Preparedness

A critical insight emerges: AI readiness mirrors data strategy readiness. Organizations that have invested in robust data infrastructure find themselves better positioned to implement AI solutions effectively. The conversation covers:

• How to evaluate an organization’s AI readiness
• The relationship between data maturity and AI success
• Long-term AI implementation strategies vs. quick wins

The Generative AI Revolution

The discussion delves deep into generative AI, covering:

• Trust in Generative AI: How organizations can build confidence in AI-generated outputs
• Creative Potential: The unprecedented possibilities that generative AI unlocks
• Model Size Advancements: How larger models are changing capabilities
• Context Window Challenges: Technical limitations and their implications

What is Data Engineering?

The session provides a comprehensive definition of data engineering, breaking down:

• Core responsibilities and functions
• How data engineering differs from data science
• The infrastructure challenges unique to data engineering
• Career paths and specializations in the field

Getting Started in Data Engineering

Practical advice for aspiring data engineers includes:

• Educational Paths: Various routes into the field
• Specializations: Different areas of focus within data engineering
• Unstructured Data Engineering: Emerging opportunities in handling complex data types
• Essential Skills: Technical and soft skills needed for success

The Changing Landscape

AI’s Impact on Data Engineering Roles

The conversation explores how AI is transforming data engineering work:

• Operationalizing Dark Data: Making previously unusable data valuable
• Contextualizing AI Models: The critical work of preparing data for AI consumption
• Future Role Evolution: How data engineering positions will adapt and grow

Opportunities for Organizations

Small Companies’ AI Advantages: Surprisingly, smaller organizations may have unique opportunities in the AI space:

• Agility Benefits: Faster implementation and iteration
• Differentiation Strategies: Using unique data as competitive advantage
• Building Around AI Capabilities: Creating AI-native solutions from the ground up

Technical Deep Dives

The discussion covers specific tools and technologies:

• Apache Airflow: Workflow orchestration and management
• Vector Databases: Including Pinecone and Chroma for AI applications
• Data Storage Solutions: From Apache Cassandra to modern cloud platforms
• Unstructured Data Solutions: Handling the growing volume of complex data types

Key Insights and Takeaways

1. Data Strategy First

Organizations must establish solid data foundations before attempting AI implementation. The quality of AI outputs directly correlates with the quality of underlying data infrastructure.

2. The Open Source Advantage

The rapidly evolving open-source ecosystem provides unprecedented opportunities for innovation, especially for smaller organizations that can move quickly.

3. Standardization Challenges

The lack of standards in the AI space creates both challenges and opportunities for differentiation.

4. Future-Proofing Careers

Data engineers who understand both traditional data infrastructure and emerging AI needs will be best positioned for future success.

Episode Resources

The discussion references numerous valuable resources:

• Training Courses: Hands-on data engineering education
• AI Tools: ChatGPT, Claude AI, and other platforms
• Technical Documentation: Apache Airflow, Cassandra, and more
• Industry Analysis: Competitive edge through AI implementation

The Road Ahead

As AI continues its rapid advancement (over 50 minutes of detailed discussion!), data engineering remains the critical enabler. The conversation emphasizes that while AI captures headlines, it’s the underlying data engineering work that makes AI applications possible and reliable.

For Practitioners

Whether you’re starting your data engineering journey or looking to adapt to AI-driven changes, this discussion provides valuable insights into:

• Career development strategies
• Technical skill priorities
• Industry trends and opportunities
• Practical implementation advice

For Organizations

Companies at any stage of AI adoption can benefit from understanding:

• How to assess AI readiness
• The importance of data strategy
• Opportunities for competitive differentiation
• Building sustainable AI capabilities

Conclusion

Data engineering truly is the unsung hero of the AI revolution. As organizations continue to explore AI’s potential, those with strong data engineering foundations will be best positioned to turn that potential into reality.

The future belongs to organizations that understand this fundamental truth: great AI starts with great data engineering.

Watch the full discussion on YouTube - Originally streamed live on LinkedIn Learning’s “The Future in Tech” series.

This course is integrated with GitHub Codespaces, an instant cloud developer environment that offers all the functionality of your favorite IDE without the need for any local machine setup. With GitHub Codespaces, you can get hands-on practice from any machine, at any time—all while using a tool that you’ll likely encounter in the workplace. Check out the “Using GitHub Codespaces with this course” video to learn how to get started.

Link

https://www.linkedin.com/learning/hands-on-introduction-data-engineering

Video

Link

https://datastackshow.com/podcast/optimizing-query-workloads-and-your-snowflake-bill-with-vinoo-ganesh-of-bluesky-data/

About the Data Superstream Series: This three-part Superstream series is designed to help your organization maximize the business impact of your data. Each day covers different topics, with unique sessions lasting no more than four hours. And they’re packed with insights from key innovators and the latest tools and technologies to help you stay ahead of it all.

Vinoo Ganesh: Zero to Pipeline (30 minutes) - 9:20am PT | 12:20pm ET | 4:20pm UTC/GMT

There are few moments more daunting to data practitioners than deploying your first data pipeline. The flexibility, freedom, and development speed of the data pipeline ecosystem allows for endless tuning, customization, and configuration. . .but makes getting started overwhelming and difficult. In this live coding session, Vinoo Ganesh takes you through scoping, building, deploying, and running a fully functioning ETL pipeline in Airflow in just 30 minutes—all in a local developer environment. You’ll also learn how to simplify each step of the ETL process into a task in a job execution DAG. Join in to get the tools and knowledge to stand up your own pipeline developer environment at home.

Link

https://learning.oreilly.com/live-events/data-superstream-building-data-pipelines-and-connectivity/0636920064968/0636920064967/

Link

https://www.horangi.com/blog/s3-buckets-permissions-and-iam-audits

This session will cover the core components that each data pipeline needs from an operational and functional perspective. We’ll discuss a framework that will allow practitioners to set their pipelines up for success. We’ll also discuss how to leverage data pipelines for metrics gathering and how pipelines can be architected to alert on potential data problems before the fact.

Sessions

• March 10, 2022
• June 17, 2022
• September 19, 2022

Link

• https://www.oreilly.com/videos/oreilly-radar-data/0636920654667/
• https://www.businesswire.com/news/home/20210909005792/en/O%E2%80%99Reilly-Announces-O%E2%80%99Reilly-Radar-Data-AI-to-Help-Tech-Leaders-Drive-Innovation-and-Successful-Implementation

Link

https://databand.ai/mad-data-podcast/defining-data-quality-data-sla-nightmares-lessons-learned/

As data products become central to companies’ bottom line, data engineering teams need to create higher standards for the availability, completeness, and fidelity of their data.

In this session we’ll demonstrate how Databand helps organizations guarantee the health of their Airflow pipelines. Databand is a data pipeline observability system that monitors SLAs and data quality issues, and proactively alerts users on problems to avoid data downtime.

The session will be led by Josh Benamram, CEO and Cofounder of Databand.ai. Josh will be joined by Vinoo Ganesh, an experienced software engineer, system architect, and current CTO of Veraset, a data-as-a-service startup focused on understanding the world from a geospatial perspective.

Join to see how Databand.ai can help you create stable, reliable pipelines that your business can depend on!

Link

https://airflowsummit.org/sessions/2021/data-quality-standards-databand/

Video

Link

https://www.dremio.com/subsurface/migrating-to-parquet-the-veraset-story/

This breakout session will describe Veraset’s partnership with Databricks, and how we have white labeled Databricks to showcase and accelerate the value of our data. We’ll discuss the challenges that data brokers have faced to date and some of the primitives of our businesses that have guided our direction thus far. We will also actively demo our white label instance and notebook to show how we’ve been able to provide key insights to our customers and reduce the TTFB of data onboarding.

Link

https://databricks.com/session_na21/brokering-data-accelerating-data-evaluation-with-databricks-white-label

Video

In this event, you’ll gain insight into design and engineering best practices through interactive sessions and live coding demos. Join us to learn how to make the right decisions for your applications.

About the Strata Data Superstream Series: This four-part series of half-day online events gives attendees an overarching perspective of key topics that will help your organization maximize the business impact of your data.

Link

https://www.oreilly.com/videos/strata-data-superstream/0636920551973/

He then goes over the techniques used to extract useful data. The distribution of data and data types have the most significant impact on data quality. Vinoo talks about the challenges of working with data, where a simple data movement can present a massive problem. Constant profiling is needed to help scale the data and make sure that the computing power can cope.

Finally, the guest talks about handling messy data that doesn’t have the required quality. He talks about the multiple problems data scientists have to consider to sort messy data to make it more useful.

Link

https://datastandard.io/podcast/large-scale-data-analytics-with-vinoo-ganesh-at-veraset/

After this presentation, attendees should understand the core concepts behind the prevalent file formats, the relevant file-format specific settings, and finally how to select the correct file format for their jobs. This presentation is relevant to Spark+AI summit because as more AI/ML workflows move into the Spark ecosystem (especially IO intensive deep learning) leveraging the correct file format is paramount in performant model training.

Link

https://databricks.com/session_na20/the-apache-spark-file-format-ecosystem

Video