Building a RAG Chatbot with Knowledge Assistant

In this demo, you will learn how to use the "Knowledge Assistant" feature within Databricks (referred to as "Agent Bricks") for unstructured data, including how to:

• Ingest Data: Documents are ingested via a pipeline, starting with Unity Catalog (UC) files or a vector index. The demonstration uses volumes with PDF documents.
• Create and Configure an Agent: Create an agent, provide a description, and connect it to various sources, such as staff recommendations and new releases. The agent's goal is to look up information and provide answers with source links.
• Test and Evaluate: Test the agent with a query and review the suggestions, source links, and reasoning.
• Improve Quality: Improve the agent's performance using three methods:
  1. General Instructions: Use natural language feedback to guide the agent's behavior, such as instructing it to use general knowledge when no relevant documents are found.
  2. Labeled Data Feedback: Directly add specific information to enhance the agent's knowledge base.
  3. Guidelines with Specific Information: Provide a question and include guidelines with new information (like an upcoming release) and specifying the agent's logic (e.g., check inventory, then recommend the new release).