Using your repository for RAG: Learnings from GitHub Copilot Chat

Intro (0s)

• This session is about using a repository for Retrieval Augmented Generation (RAG) and how it looks in Copilot Chat, as well as the learnings from the experience (10s).
• RAG translates into project awareness in Copilot Chat, which can be achieved through different methods in various platforms, such as VS Code, Visual Studio, and JetBrains (34s).
• The methods include workspace or intent detection in VS Code, hashtags in Visual Studio, and implicit intent detection in JetBrains (40s).
• The feature is currently being rolled out in JetBrains, and users of the JetBrains extension version 1.5.26 or later may have seen it (56s).
• The presentation will focus on the implementation details for JetBrains, but the general concepts are applicable to all implementations (1m2s).
• The speaker, Kimin Miguel, is a senior software engineer working on Copilot for IDE, focusing on project context for JetBrain Chat and prompt craft for code completions (1m22s).
• Kimin Miguel is based in France and enjoys taking care of plants and playing Final Fantasy 14 in their free time (1m32s).

Agenda (1m41s)

• The agenda includes a breakdown of how project context works in JetBrains, covering local indexing, local search, and the final reranking step (1m44s).
• Key steps behind the process will be explained, including local indexing, local search, and the final reranking step (1m48s).
• The explanation of the key steps will be provided before moving on to the next topic (1m57s).

Project Context in JB Chat (2m0s)

• Project context is a feature that helps large language models provide more accurate and relevant responses by enriching the prompt with code snippets from the user's codebase, allowing the model to reference symbols and constructs defined in the codebase and provide more helpful answers (2m29s).
• The goal of project context is to ground the model with factual data, reducing the likelihood of generic or misleading responses (2m43s).
• Project context can be used through various methods, including the "at workspace" action, implicit intent detection in Visual Studio, and implicit intent detection in JetBrains (3m1s).
• Initially, JetBrains planned to implement project context using the "at project" method, but due to low usage numbers, they shifted to implicit intent detection (3m11s).
• As of the current numbers, about 11.8% of answers in JetBrains include context using project context (3m36s).
• Project context can be enabled or disabled through a checkbox in the chat panel, and users who do not have it enabled yet will receive it after the Universe update (4m17s).
• When project context is enabled, the model takes a little longer to respond, and users can see an info message indicating that the model is collecting relevant project context (5m18s).
• The feature provides references to relevant code snippets from the user's codebase, which can be clicked to open the corresponding code snippet in the code editor (4m50s).
• The demo showcases how project context works in a live scenario, where the user asks a question and the model provides a response with references to relevant code snippets (4m11s).

How does project context work (5m35s)

• Local project context is built using three main building blocks: local indexing, the first ranking pass, and the second ranking pass, which work together to provide relevant snippets from a project to answer a user's question (5m37s).
• The local indexing step starts when a project is opened in an IDE and the extension is activated, and it indexes the project in the background in a non-blocking way by tokenizing files using the Microsoft White Pair encoding tokenizer and splitting each file into chunks of 500 tokens (6m48s).
• When a user asks a question, the query is processed, and the result is used to find a wide area of snippets from the local index built earlier through the first ranking pass, which uses the BM25 scoring function to rank the snippets (6m5s).
• The top 50 snippets from the first ranking pass are then refined through a second ranking pass, which uses a text embedding 3 model to return embeddings, or vector representations, of the snippets and the user query, and compares them using cosine similarity (7m52s).
• The five topmost snippets from the second ranking pass are extracted, included in the prompt, and used to generate an answer, along with references to the original files and selection ranges (8m26s).
• The local indexing process is done on startup and is a non-blocking process that runs in the background, allowing the user to interact with the project while the indexing is being completed (7m11s).
• The keywords related to the user's question are used to find the most similar snippets in the local index, which are then ranked and selected for inclusion in the prompt (7m32s).
• The entire process is designed to provide relevant project context to the user's question, using a combination of natural language processing and machine learning techniques to generate accurate and helpful responses (6m39s).

What constraints are we operating under? (8m41s)

• The key takeaway from the session is the need to balance engineering and science when implementing a solution, with the goal of achieving a snappy experience that provides relevant information backed by data (8m51s).
• To achieve this balance, it is essential to iterate, experiment, and ultimately compromise between engineering and science (9m8s).
• The process involves considering constraints and key points at each step, starting with local considerations (9m19s).

Local Indexing (9m19s)

• Local indexing does not persist the index, meaning that when a project is closed, everything gets released, and when the project is reopened, everything is reindexed (9m19s).
• Indexing needs to be done in the background to avoid blocking operations, which can be achieved through multi-threading (9m50s).
• A bug was fixed that caused issues with indexing a file with 10,000 nested open arrays, which was taking a minute to index (10m3s).
• Files specified in .gitignore and excluded in the IDE are not indexed, and node modules and Python environments are also excluded from indexing (10m29s).
• A cap is placed on the number of files and code chunks stored in memory to prevent indexing and ranking from taking too long (10m47s).
• If a project has more than 10,000 files, only partial project context can be offered, and an info message is displayed when project context is invoked on large projects (11m9s).
• The time it takes to index a project depends on the number of files, with smaller repos taking around 5 seconds, medium-sized repos taking around 50 seconds, and large repos taking over 2 minutes (11m34s).
• Having everything in memory allows for listening to file system notifications and reindexing files quickly to keep the freshest information in the codebase (12m7s).
• The indexing algorithm used is fixed-size chunking with 500 tokens, but alternative methods like semantic chunking could be explored for better results, although they may come with a performance hit (12m47s).

First ranking pass (13m31s)

• The first ranking pass involves extracting keywords from the user question and using the BM25 scoring function to score documents, in this case code snippets, against those keywords (13m36s).
• BM25 is a scoring function that depends on the frequency of the keyword, relative to the document length, and the number of documents that contain the keyword (13m55s).
• Two key questions were how to extract keywords from a natural language question and how to make it fast, with the solution being to leverage an LLM (Large Language Model) to do the work (14m14s).
• The LLM used was GPT 3.5 Turbo or GPT 40 Mini, hosted on Azure OpenAI, to extract the most relevant keywords, synonyms, and variations (14m29s).
• Testing showed that GPT 40 Mini returned more generic results than GPT 3.5 Turbo, highlighting the need for AB experimentation when swapping models (14m52s).
• The keyword and synonym request is independent of the repository size and takes between 600 milliseconds and 1.5 seconds, depending on the query complexity (15m14s).
• The processing time for the scoring logic using BM25 scales with the number of index chunks, with search times ranging from 900 milliseconds for a small repository to 10 seconds for a large one (15m36s).
• To optimize performance, only a subset of snippets is passed to the second ranking step, with the current approach selecting the top 47 highest-scored snippets (16m29s).
• The fixed snippet count may not work for larger repositories, and experimenting with scaling the number of snippets to pass to the second ranking pass is being considered (16m48s).

Second ranking pass and final prompt (17m2s)

• The process involves taking the 47 snippets in the user query and sending them to the Text Embedding 3 model hosted on Azure OpenAI, which returns a vector for each string (17m3s).
• The vectorized snippets are then compared to the vectorized user query using cosine similarity, and the top five results are selected (17m19s).
• There are two levers that can be adjusted in this process: the amount of data sent to the embedding model and the vector size (17m23s).
• Increasing the amount of data sent to the model increases the time it takes to process, with 48 strings taking around 4.5 seconds (17m38s).
• The vector size can also be adjusted, with the default size being 1,536 dimensions, but reducing the dimensions can result in an accuracy cost (18m3s).
• A compromise was reached by using 1,024 dimensions, which speeds up the process to around 4-4.5 seconds without significantly impacting the ranking output (19m1s).
• The top five vectors are stored and included in the prompt, with some exceptions, such as files under content exclusion policies (19m18s).
• References to the files and code snippets included in the prompt are displayed to the user (19m51s).
• The process involves some waiting time, and an info message is displayed to inform users that responses may take a little longer (20m19s).
• Users can opt out of project context or add files references directly to the prompt if they know which files are useful (20m27s).
• A file picker can be used to select files and add them directly to the context that will be sent as part of the prompt (20m54s).

Summary (21m43s)

• Building blocks for project context include local indexing, local search, and reranking using embeddings, which are essential components for creating a functional system (21m46s).
• A balance between engineering and science is necessary, as they have a healthy tension that feeds into each other, requiring consideration of the impact of changes on both sides (21m54s).
• Having a baseline to compare against is crucial, as any change is not necessarily a good change, allowing for evaluation and improvement (22m8s).
• The needs and goals of the users should not be forgotten, as the ultimate objective is to create a helpful and successful product that enables users to accomplish more during their day (22m16s).