How to measure the effectiveness of GitHub Copilot

Introduction (0s)

• This session is led by Kitty Trueu, a Senior Developer Architect, and Mickey, a Staff Developer Architect, both from the Customer Success Strategy team at GitHub, who work with customers to solve complex problems on a daily basis (12s).
• The session aims to provide three key takeaways: productive insights from the Copilot platform, insights from the Copilot Matrix, and suggestions on how to survey developers without causing fatigue (53s).
• The session is focused on providing numbers and insights to help make a business case for using GitHub Copilot, rather than just relying on the hype surrounding AI tools (2m49s).
• The presenters will provide a demo and offer suggestions on how to measure the effectiveness of GitHub Copilot, with the goal of helping attendees to show the value of the tool to their business and make a case for its adoption (3m10s).
• The session is motivated by a common scenario where a developer requests approval for a GitHub Copilot subscription, but the boss wants to know what value it will bring to the business and what numbers can be used to justify the expense (1m33s).
• The presenters will help attendees to go beyond the hype surrounding AI tools and to understand exactly what they are measuring, with the goal of providing a clear business case for using GitHub Copilot (2m58s).

Why we measure (3m17s)

• When modernizing Engineering Systems, it's essential to build a Competitive Edge for the business by constantly building competency around people, process, and technology (3m33s).
• The "people" aspect involves investing in the team to increase productivity, ensuring they have the right tools, skills, and communication, as well as adequate feedback and mental space to work effectively (3m57s).
• The "process" aspect refers to the orchestration of activities to achieve a goal, and leaders should constantly challenge the status quo and remove red tape to improve efficiency (4m26s).
• The quote from James Clear's "Atomic Habits" emphasizes that systems need to evolve to achieve goals, and leaders should focus on building systems that support their objectives (4m42s).
• The "technology" aspect involves using tools to enable productivity, and investing in GitHub Copilot is not just about individual productivity but also about increasing system efficiency (5m9s).
• The ultimate goal of investing in GitHub Copilot is to ship faster and reduce rework (5m27s).
• To measure the effectiveness of GitHub Copilot, it's essential to establish a baseline of the engineering system and start measuring now, as you can't manage what you don't measure (6m25s).
• GitHub users have access to metrics and statistics that can provide insights into their platform's productivity, and it's crucial to have a holistic set of signals from people, process, and technology to measure performance (6m50s).
• Engineering systems may have performance signals from various dimensions, including high-quality code, which is a delight to both developers and the business (7m39s).
• A developer's achievement is to write quality code, while a business's goal is to avoid the cost of delay and rework, and adapt to the market dynamics (7m45s).
• High-quality code is characterized by being readable, reusable, maintainable, concise, resilient, and secure, which are the quality attributes that developers aim to write into their code (8m0s).
• Developer happiness is about how satisfied developers are with their tasks, whether they can focus, feel challenged, and are engaged, as well as their mental energy when working from home (8m21s).
• A philosophy in engineering systems is not just about speed at a single point in time, but about delivery throughput, pace, friction, waste, delays, and manual validation (8m46s).
• The goal is to build things right and build the right things, ensuring that modernizing work and effort enable and accelerate engineering system success, making a positive business impact and return on investment (9m23s).
• When defining the purpose of using a tool like GitHub Copilot, considerations should include building quality code, developer happiness, philosophy in engineering systems, and making a positive business impact (9m49s).

What we measure (9m52s)

• When implementing GitHub Copilot, organizations typically go through four phases: adoption, activity, satisfaction, and impact, with the adoption phase focusing on ensuring users are actually using the product (10m21s).
• To measure adoption, an API can be used to track basic usage information, such as the number of people using Copilot every day (10m54s).
• The activity phase involves analyzing how users are utilizing Copilot's features, such as chat suggestions and auto-completion, which can be pulled from the API (11m26s).
• Acceptance rate is a key metric, calculated by dividing the number of accepted suggestions by the total number of suggestions made, with a 30% acceptance rate considered good (11m45s).
• Low acceptance rates often indicate that users need training or guidance on how to effectively use Copilot (12m21s).
• The satisfaction phase involves assessing whether developers are happy using the tool, feel satisfied, and believe it helps them solve problems and get into a flow state (13m18s).
• The length of these phases can vary, lasting from weeks to months, and cannot be rushed (13m2s).
• Measuring the effectiveness of GitHub Copilot ties into the developer experience, but there isn't an API available for this, making surveys a viable solution (13m29s).
• When creating surveys, it's essential to keep them short, as respondents are more likely to complete a four-question survey than a 20-question one (13m46s).
• The ultimate goal is to determine the downstream impact of using GitHub Copilot, such as whether it improves software development speed and quality (14m0s).
• Various metrics can be used to calculate the downstream impact, some of which can be obtained through the GitHub API, while others may require data from other systems (14m11s).
• GitHub-specific metrics that can be used include average merge request frequency, which can help determine if GitHub Copilot is leading to shorter, smaller commits and faster development (14m41s).
• When working with customers to adopt GitHub Copilot, it's essential to take a phased approach and not rush the process, as changing people's behavior takes time (15m0s).
• It's crucial to understand that people's behavior, not just machine performance, is a critical factor in measuring the effectiveness of GitHub Copilot (15m19s).
• A reasonable timeframe for measuring the downstream impact of GitHub Copilot is months, not days or hours (15m28s).

How we measure (15m31s)

• To measure the effectiveness of GitHub Copilot, the first step is to pull information from the API, which can be done using the GitHub CLI, a tool that encapsulates authentication and allows for easy interaction with GitHub and API calls (15m33s).
• The API call returns the last 28 days of information, which is a rolling 28-day period, and the data is in JSON format, including metrics such as total suggestion count, total acceptance count, and active user count (16m15s).
• The data can be used to calculate the acceptance rate, which is a key metric for measuring the effectiveness of GitHub Copilot, and can also be broken down by language and IDE (16m52s).
• A Power BI dashboard created by GitHub can be used to visualize the data and provide a nicer look at metrics such as engaged users, suggestions made, and lines accepted (17m42s).
• The dashboard provides a better understanding of what's going on in the environment, including acceptance rate, acceptance by language and IDE, and engaged users (18m14s).
• The data can also be used to identify trends and areas for improvement, such as a decline in acceptance rate over time, and can be used to inform decisions about how to improve the use of GitHub Copilot (18m20s).
• As GitHub adds more features to Copilot, more data will be added to the API, allowing for more detailed analysis and visualization (18m42s).
• The data and visualization can be used to provide insights and recommendations to managers and stakeholders, who often prefer to see data presented in a visual format (17m24s).
• To measure the effectiveness of GitHub Copilot, surveys can be used to gather information from developers, and there are various options available, including third-party survey engines or a simple email to developers (19m2s).
• A free, open-source project called the Co-Pilot Survey Engine is available, which can be set up with a database backend and uses a GitHub app to open an issue with four questions every time a pull request is closed (19m21s).
• The four questions in the Co-Pilot Survey Engine ask if Copilot helped with the task, how the developer felt about using it, if it saved time, and what was done with the time saved (19m52s).
• Another example of a Co-Pilot developer survey is available, which has 10 questions with more detail, and the choice of survey depends on the specific needs and understanding of the users (20m11s).
• It is recommended to start with fewer questions, at least initially, to get developers on board and used to answering the questions seriously (20m32s).
• Downstream metrics, such as deployment frequency and faster merge rates, can be used to measure the effectiveness of GitHub Copilot, and some of this data can be obtained from GitHub using the GitHub API (20m51s).
• A shell script can be used to pull data from GitHub and calculate downstream metrics, such as deployment frequency, and this data can be used to determine if Copilot is helping to solve problems (21m15s).
• The discussion revolves around measuring the effectiveness of GitHub Copilot, focusing on key metrics that matter in different phases (22m29s).

Key takeaways (22m51s)

• The key takeaways from the presentation include recalling why measuring the effectiveness of GitHub Copilot is important and going beyond the hype to understand the value it brings to the engineering system and the business as a whole (22m51s).
• Measuring the effectiveness of GitHub Copilot is crucial to understand the return on investment and to build a competitive edge for the business, and it's essential to start measuring now if not already done (23m30s).
• To measure the effectiveness, consider defining signals from the existing Engineering Excellence Matrix or use the four dimensions presented, and integrate third-party system telemetry for a holistic view of the value of the AI investment (23m44s).
• The presentation demonstrated how to measure from the first-party GitHub platform and the Matrix API, and the importance of integrating third-party system telemetry to show the value of the AI investment (23m50s).
• A dashboard was demoed, which will be open-sourced, and the Copa-generated script and the dashboard link will be posted on the community site for further discussion and questions (24m20s).
• Measuring value is a subjective matter, and the community's input and discussions are encouraged to help define the right metrics and numbers to measure value (25m0s).
• The presentation concluded with a thank you note to the attendees and an invitation to provide feedback through a survey to improve future experiences (25m17s).