Hands-on Workshop: RAG pipeline With Data Prep Kit + Milvus + Llama

RAG (Retrieval-Augmented Generation) or fine-tuning a model, a significant portion of your time will be dedicated to data wrangling (cleaning, de-duping, removing markups, etc.). Data Prep Kit ([https://github.com/IBM/data-prep-kit](https://github.com/IBM/data-prep-kit)) can help you with data wrangling.

Noteworthy features of DPK include: de-duping documents (exact dedupe and fuzzy dedupe), handling documents and code, language detection (spoken languages and programming languages), malware detection and creating embeddings.

In this hands-on workshop, we will demonstrate implementing an end-to-end RAG pipeline using all opensource technologies.

1. Data Prep Kit for processing documents

2. Milvus as vector database

3. Llama 3 as the LLM