Building agentic systems with VS Code extensibility and GitHub Copilot at Uber

Introduction (0s)

• The presentation is given by Sorab Sherti, a product manager at Uber, and his colleague, Mat Ranis, a senior software engineer at Uber, who will discuss their experience in building agentic systems with Visual Studio Code extensibility and GitHub Copilot at Uber (11s).
• Uber is a well-known company that operates in over 10,000 cities, does 30 million trips a day, and has a massive code base of tens of millions of lines of code (42s).
• The company faces challenges with its large code base and the constant addition of new code, which led to the development of a tool to solve a specific problem, particularly around authoring tests (51s).
• The presentation will cover a packed agenda, including a live demo, and will also allow time for questions (59s).
• The speakers are excited about the announcements around Co-Pilot extensibility, including the Co-Pilot multiple file edit feature, and believe that their product has the right abstractions in place to take advantage of these features (1m38s).
• Although the product has not been updated to use the new features announced in the past 24 hours, the speakers are confident that the audience can extrapolate from what they will be showing (1m45s).

Problem Statement (1m50s)

• Writing unit tests is extremely important while developing code, but it is hard on engineers and difficult to make them complete enough and adhere to company-specific conventions (1m54s).
• Maintenance after writing the test is also often a challenge, and having a super complete suite requires a significant maintenance burden (2m15s).
• The process of writing a unit test at Uber involves 29 steps, which can be time-consuming and take away from the productivity of the engineer (2m30s).
• Most engineers do not enjoy writing unit tests every day, and the goal is to provide a tool to speed up the process and help arrive at better unit tests (2m45s).
• The initial tooling for writing unit tests consisted of prompt presets that could generate unit tests with Uber conventions, but they were clunky and not very usable in practice (3m28s).
• The initial tooling required engineers to read the documentation, get the prompts, put them in, generate the test, splice them in, test them, and iterate, which led to people not using them very often (3m43s).
• The goal is to provide a better tool to solve the problem of writing unit tests and make the process more efficient and productive (2m55s).

Introducing AutoCover (3m55s)

• AutoCover is a product that is being iterated on, and it involves using GitHub Copilot to generate tests through a chat participant, resulting in a table test with edge cases and everything after some internal looping (4m22s).
• An agentic system is a system comprised of multiple steps, each performing a specific task, and these steps are chained together to allow agents to collaborate and specialize in solving problems (4m47s).
• By adding guardrails from deterministic tools like compilers and linters to the output of a probabilistic model, the overall precision and quality of the output can be greatly increased (5m24s).
• The approach involves breaking down a large problem like writing a test into smaller, solvable problems, similar to what a human would do when writing a unit test, and then feeding that into a model to get better results (6m1s).
• The process of writing a unit test involves diving into the code base, understanding how it works, figuring out the state of coverage, trying to write a test, compiling it, and feeding all that back into the system (6m22s).
• The presentation will walk through each of the systems involved in the process, starting with an example of a complex code snippet that is indicative of something that may be seen in a code base (6m43s).

How does it work? (6m59s)

• The example code used for demonstration is a piece of source code that contains functionality to parse resource identifiers, such as URIs, and can be found in a utel folder in prod, with one Go file and no existing tests. (7m0s)
• The code's coverage status is unknown due to the lack of test files, but as soon as at least one test case is added, the coverage status can no longer be inferred and must be calculated. (7m45s)
• Autocover is introduced as a tool that can be dropped into a code base to calculate coverage, starting with the preparer agent, which sets up the environment for the rest of the agents to operate properly. (8m2s)
• The preparer agent performs a setup step, including setting up the file, creating the file if needed, scaffolding out the file for some languages, and collecting context such as surrounding files and summaries of imports. (8m10s)
• The preparer agent also performs other setup tasks, such as regenerating mocks, to prevent issues in subsequent steps. (8m48s)
• The preparer agent's tasks can be done deterministically, and this is a key aspect of the system that will be referred to later. (9m3s)
• The generator is the first probabilistic agent, which uses the collected intelligence to generate test cases for the uncovered lines of code, producing candidate test cases that can be refactored later. (9m21s)
• The generator takes the target area and generates test cases, which are individual and atomic units that can be deterministically checked against build tooling and coverage tooling. (9m33s)
• The initial phase of the process involves generating test cases and setting up and tearing down sequences for each specific test case, which are then packaged and put in shared memory for other agents to use (10m32s).
• The test generation is similar to the "/test" command in Copilot, which generates a test and everything else around it is machinery to aid and produce better results (10m41s).
• The executor is a deterministic agent that does a lot of work to ensure that the test cases are initially good or not, and it runs tooling to make sure dependencies are imported and the file is syntactically correct (11m7s).
• The executor tries out the test cases one by one, and if a test case fails, the feedback is included in the shared memory for future agents to use (11m39s).
• Building the executor requires knowledge of all languages and frameworks used in the organization, and it can be challenging to consolidate and prepare the organization for this (12m16s).
• The results of the executor are then passed to the fixer, a probabilistic step that has a specific purpose of fixing failed test cases (13m4s).
• The fixer sends parallel prompts or calls to the language model (LM) for each test case, giving it a larger context window to elaborate and produce better results (13m26s).
• The process is built in Langra, which provides a nice abstraction for doing these types of things (13m38s).
• Fixer is a tool that addresses issues in test cases by analyzing compilation errors and making necessary changes, such as removing or adding versions, to make the test cases work properly (13m58s).
• Fixer can collaborate with the executor to mark candidate test cases for re-execution, as the initial fix may not be valid, and it may take multiple iterations to get it right (14m32s).
• The system is a complex state machine that manages the flow of test cases, and it's not just a linear flow through the system (14m51s).
• After fixing test cases, they are passed to the refactor agent, which ensures that the test cases adhere to Uber's best conventions and combines them into larger table tests (15m9s).
• The refactor agent uses a directive to look at Uber's best conventions, which are split up by language and are modular and nice (15m13s).
• The abstraction used for an agent must support composability of multiple sub-agents, allowing for the integration of different types of tooling, such as syntax tree-based linters (15m42s).
• The refactor agent can integrate multiple different types of tooling, such as go format, ra, or black, to reduce verbosity and improve test cases (16m9s).
• The validator is the QA component that checks if the generated test case asserts anything and if it asserts the right things (16m31s).
• The validator ensures that the test cases are not just blindly executing code and not asserting anything, which is a big worry for engineers (16m41s).
• The validator has multiple sub-agents, and it's the first one that is used to validate the test cases (16m56s).
• A system has been developed with multiple sub-agents to improve test case generation, including a simple deterministic sub-agent that checks for assertions, a probabilistic sub-agent (LLM Critic) that validates tests against a list of conventions, and a mutation tester. (17m0s)
• The LLM Critic sub-agent goes through conventions and maps them back to the test, flagging inconsistencies and providing feedback, such as suggesting additional test cases for concurrency functionality. (17m14s)
• The system uses a multi-shot approach to adherence, where a model is asked to generate or refactor a test, and then another model validates whether the first model complied with the style guide. (18m7s)
• The agent abstraction needs to account for situations where the model cannot infer what's going on and needs to return control to the human, such as when there are insufficient doc comments or the code is written in an obtuse way. (18m43s)
• The system is being developed to add more interaction paradigms and UI elements in the IDE, allowing the model to ask for more context when it doesn't understand the code. (19m5s)
• An example of the validator's feedback is shown, where it suggests adding edge cases to a table test, such as checking for Unicode symbols or empty identifiers. (19m25s)
• The system iterates on the test cases based on the feedback, passing it back to the fixer and executor to arrive at a new iteration of the test case with more comprehensive coverage. (19m50s)
• The mutation testing subcomponent is being developed to further improve the test cases. (20m10s)
• The concept of a mutant is introduced, which refers to making an unintended change to the source code and checking if the test suite catches it, with the goal of identifying potential bugs or shortcomings in the test suite (20m31s).
• An example of a mutant is provided, where the code is changed to parse a string as V1 instead of V2, which is a breaking change that the test suite should catch (21m5s).
• Another example of a mutant is given, where an error is swallowed instead of being returned, which is a contract-breaking change that the test suite should detect (21m31s).
• The mutants that pass the test suite are bundled up into feedback and used to add more test cases to address the shortcomings found (21m55s).
• The generation of mutants can be done using both deterministic and model-based approaches, such as parsing the syntax tree and inverting if conditions, or using a large language model (LLM) to introduce subtle bugs (22m7s).
• The system uses a complex state machine to generate and test code, and the candidate code travels through each state without the user's knowledge (22m37s).
• A live demo is mentioned, where the execution of Auto Cover will be stopped at pre-selected states to show what the model or system was thinking and what the feedback is before the next iteration (23m2s).

Demo (23m21s)

• A sample project is demonstrated, which includes the "bars" and "bars V2" projects, as well as a stringification tool, but lacks tests and has 0% coverage (23m22s).
• The project's source code is highlighted in red, indicating that it hasn't been tested, but some lines are not marked as uncovered because they are just declarations (23m50s).
• A more targeted approach is taken by scoping in on specific functions that need testing, rather than selecting the entire file and generating tests for it (24m22s).
• The process of generating tests is started by invoking the "autocover" command in the chat panel, which opens up a new test file automatically (24m48s).
• The "autocover" command is a fire-and-forget process, but in the future, it is planned to allow users to chat again with the copilot to ask questions, make corrections, or add more scenarios (25m8s).
• A debug viewer is used to show the intermediate state of the test generation process, which is not intended for human consumption (25m30s).
• The first state in the graph is shown, which is the initial state after the test cases have been generated, and the test configuration panel is introduced, which shows the current node, the number of tests generated, and the number of tests needed to cover the code (25m49s).
• The current state of the code is unknown, as it hasn't been run yet, and the test case state is also unknown, with the system having generated a table test or future with a good format to make refactoring easier (26m36s).
• The feedback section is empty because the test case was just generated, and there's no information on whether it passes or builds (27m5s).
• The imports were split up because sometimes interaction with the build system is necessary, such as running the package manager to pull in new dependencies (27m14s).
• After the initial run of the executor, the test functions remain the same, but a new test case state is added, showing that the generated code builds, the test passes, and coverage is raised (27m36s).
• The system currently uses coverage as a metric but plans to expand to other metrics like survival or mutation testing mutant survival rate and branch coverage (27m55s).
• If a test fails, the system would show a different state with a build or test failure, and provide feedback in the form of output from the test runner or build system (28m28s).
• The system has simplified tech feedback to a simple textual field, but can also handle complex structure data that can be passed back to the model for subsequent generation phases (29m0s).
• After the initial validation step, the system passes through a few steps and reaches validation, where it determines that the new test state is validation failed, indicating something wrong with the test case (29m26s).
• The system generates test cases and provides feedback on how to improve them, including suggestions for adding edge cases and improving readability (29m34s).
• The feedback is based on various heuristics, such as catching edge cases, improving readability, and using named parameters, which are continuously evolving (30m17s).
• The system also incorporates mutation testing results, which can show up in the feedback if a mutation testing surviving bug is found (30m42s).
• The final state of the system is that it outputs a test case that has been iterated upon and improved based on the feedback, with the exact examples suggested in the feedback added to the test case (31m3s).
• The system can be adjusted to iterate a certain number of times, and the feedback is flagged as addressed once it has been incorporated into the test case (31m33s).
• The user of the system, Autocover, would see a populated test file with multiple tests and expansions, including new edge cases that have been added (31m44s).
• The system is able to achieve 100% coverage of the file, with highlighted parts that are covered in green and potential indicators for tested branches and covered code (32m20s).
• The system also provides metrics on the number of test cases that cover a particular piece of code, which is a new metric being experimented with (32m41s).
• The Auto Cover feature can generate around 110 lines of code with just 15 characters typed to invoke it, making it a keystroke-safe and effective tool, with many engineers at Uber using it (33m0s).
• If there is missing coverage, it is likely that the code is too complex or there is a bug in the source that should not be covered (33m17s).
• The goal is to expand the feature to support other languages, with Java and Python support currently in development and expected to be productionized at Uber (33m39s).
• A standalone mode is being developed to allow users to invoke features like validation logic and mutation testing without having to fully adopt AI-generated code (33m55s).
• The standalone mode aims to provide value to developers who are not ready to fully trust AI-generated code, but still want to use specific features (34m8s).
• The feature will allow users to get feedback earlier in the development cycle, rather than waiting until submitting a PR (34m21s).
• Mutation testing will be made easier by allowing users to invoke a tool to do it for them, making it more accessible to developers (34m30s).
• A consulter mechanism is being added to integrate with native UI or UX in the IDE, allowing agents to seed control back to humans when needed (34m47s).
• The consulter mechanism will integrate with features like chat boxes, doc comments, and other paradigms to prompt users for input (35m0s).
• The feature will also integrate with other entry points in the IDE, such as gutter indicators, code lens, and highlighting, to make it easier for users to discover and use (35m25s).