Effective document management and retrieval for generative AI

Building production RAG systems requires far more than embedding documents. This 90-minute deep dive covers the complete technical pipeline: document structuring principles, semantic vs. fixed-size chunking tradeoffs, contextual and multi-modal embeddings, and advanced retrieval optimization, accompanied by practical implementation on Databricks. We'll demonstrate a real-world Databricks RAG pipeline covering pre-processing at scale, embedding optimization and advanced retrieval techniques—including re-ranking, contextual filtering and real-time quality evaluation. Beyond retrieval, we showcase how Agent Bricks transforms RAG systems into autonomous, agentic applications capable of multi-step reasoning and complex tasks. The session also explores emerging trends—vision-space retrieval, contextual clustering and real-time quality assurance—critical for production systems where accuracy directly impacts outcomes.