How GitHub Copilot helped SAP reimagine the developer experience

Introduction by Sumeet Shetty and Tobias Schimmer, outlining the session and its goals. (0s)

• Sumeet Shetty and Tobias Schimmer are presenting at GitHub Universe, discussing how GitHub Copilot is helping SAP reimagine the developer experience (11s).
• Sumeet Shetty is the head of tools India at SAP, and Tobias Schimmer is the head of developer experience (28s).
• The presentation will cover SAP's point of view on developer experience, their AI programming journey, key learnings, and the impact of GitHub Copilot on SAP (47s).
• The AI programming journey at SAP has lasted over a year and a half (59s).
• The presentation will also share key takeaways on the impact of GitHub Copilot at SAP (1m14s).
• Sumeet Shetty and Tobias Schimmer have known each other for over a decade, and Sumeet initially reached out to Tobias for research support in evaluating GitHub Copilot (1m29s).
• Tobias was asked to help understand how GitHub Copilot resonates with people and its impact to make a better decision for SAP (1m49s).

Developer Experience at SAP (1m56s)

• Developer experience at SAP is focused on three key aspects: efficiency, effectiveness, and creating a motivating and retaining team, with the foundation being good practices such as DevOps, Agile, and Lean (2m17s).
• The productivity equation consists of efficiency, effectiveness, and creating a motivating team, with effectiveness meaning addressing the right markets, talking to the right customers, and solving problems for the right end-users (2m20s).
• The mission for developer experience is to deliberately put humans at the center, ensuring a perfect flow for individuals and teams, and guaranteeing the value flow of entire product units (3m11s).
• The developer experience function is human-centered, taking the perspective of one developer with a certain problem to solve, and is data-driven, with two aspects: data about products and data about the process (5m5s).
• Data about products includes understanding how a product is being adopted, customer satisfaction, and usage patterns, while data about the process includes measuring Dora 4 and flow metrics (5m40s).
• The goal is to provide teams with a way of data-driven continuous process improvement and to leverage the power of AI along the software development life cycle where it makes sense (6m2s).
• Generative AI was the starting point for reimagining what developer experience is and will be at SAP, with the aim of creating a better experience for developers (6m17s).
• SAP has over 100,000 employees and rarely builds one product with one team, so ensuring the value flow of entire product units is crucial (3m44s).
• Cognitive load is also an important consideration, with technical cognitive flow and domain knowledge being key factors, as developers need to have specific knowledge in their minds to build software for specific industries (4m27s).
• The developer experience team aims to stay connected to customers and end-users, with customer feedback being essential, not just from a technical perspective but also from an actual customer feedback perspective (4m4s).

AI Pair Programming at SAP (6m30s)

• Generative artificial intelligence is central to developer experience at SAP, with AI pair programming being the number one use case for enhancing developer experience using generative artificial intelligence (6m36s).
• AI pair programming involves programming with the support of an assistant powered by artificial intelligence (6m56s).
• SAP evaluated various AI pair programming tools, including GitHub Copilot, Safai, Tab9, Codium, Ghost Writer, Gemini Code Assist, Q Developer, Code GPT, Cod, and Cursor, before selecting GitHub Copilot as their AI pair programming tool of choice (7m43s).
• The evaluation was based on a list of desired capabilities for an ideal AI pair programming tool, which included intelligent autocomplete suggestions, generating code from scratch using prompts, and generating unit tests (8m1s).
• GitHub Copilot provides additional capabilities, such as finding and fixing bugs, reviewing code, refactoring code, translating code from one programming language to another, generating developer documentation, and answering questions across multiple programming languages (9m7s).
• GitHub Copilot brings all these capabilities to SAP developers, enhancing their experience and providing a huge value proposition (9m53s).
• The journey of AI pair programming at SAP began with an evaluation of tools in March 2023 and involved working with various corporate functions, including AI ethics, legal, data protection, and procurement (10m25s).
• A group of 500 pilot users, called early adopters, tested GitHub Copilot for three months, and the collected data led to a decision to roll out the tool across the organization (11m20s).
• The rollout of GitHub Copilot started in November 2023, and the journey involved gathering data through systems and people, with a focus on human centricity (12m9s).

Research strategy & using survey to gather data (12m33s)

• The research strategy involved using Telemetry data from GitHub Copilot and enriching it with a comprehensive set of questions in a survey to gather experience data from developers and understand their perceptions of productivity, ease of use, and willingness to promote the tool (13m12s).
• The survey aimed to gather data on how developers perceive GitHub Copilot, including its effectiveness, ease of use, and willingness to promote it to colleagues, as well as understanding the challenges they face while using the tool (13m25s).
• A welcome survey was conducted to understand the demographics of the 500+ participants, including their tenure, location, tools used, and technology stack, followed by regular two-weekly check-ins to understand specific use cases and challenges (13m47s).
• The data gathered from the surveys and Telemetry data was shared with GitHub to provide feedback and improve the tool (14m20s).
• A comprehensive survey was conducted after three months, and a closing survey was conducted a couple of weeks later to gather additional information and address any remaining questions (14m23s).
• The research was not conducted alone, but in collaboration with research partners, including the Hasso Plattner Institute in Germany, the University of California, Irvine, and the University of Mannheim, among others (14m51s).
• The research team also included a staff researcher hired specifically to study GitHub Copilot and other development tools, and collaborations with other universities, including the Technical University of Munich, to study AI use cases in software development (15m32s).

Results and feedback from the first phase of GitHub Copilot implementation. (15m59s)

• The early adopters of GitHub Copilot, consisting of 500 developers, found the tool to be easy to use, successfully drawing context from their prompts and code, making them feel more productive, and increasing the time for value-adding tasks (16m7s).
• The use of GitHub Copilot also made software development easier and faster, allowing developers to write more code, deliver more product features, and reduce the effort for repetitive tasks (16m29s).
• A survey of the 500 early adopters showed that 87% wanted to continue using GitHub Copilot at the end of the early adoption period (16m47s).
• A key summary of the early adoption phase revealed that 93% of developers found GitHub Copilot easy to use, 84% said it made software development faster, 81% felt more productive, 88% said time for repetitive tasks decreased, and 87% wanted to continue using it (17m13s).
• The introduction of Copilot Chat led to an increase in most of these numbers, with 84% of developers saying they were able to learn from the suggestions provided by GitHub Copilot, 78% saying they could write more code, 80% feeling more productive, and 71% saying they could write code with better quality (17m34s).
• The percentage of people who wanted to continue using GitHub Copilot increased from 87% to 93% after the introduction of Copilot Chat (18m0s).
• A larger survey of around 10,000 developers confirmed the results of the early adoption phase, with 93% saying GitHub Copilot was easy to use, 81% agreeing that it made software development faster, 79% feeling more productive, and 85% saying repetitive tasks got faster (18m28s).
• The larger survey also showed that 88% of developers, who use multiple programming languages, technology stacks, and code editors, would love to continue using GitHub Copilot (19m1s).

Learnings from our journey (19m10s)

• Research with Microsoft found that GitHub Copilot's value proposition was challenged, but SAP's own research yielded similar or better numbers, which was the first surprise (19m27s).
• The introduction of GitHub Copilot Chat led to a step change, with numbers showing increased learning and willingness to use GitHub Copilot, translating to a net promoter score of 77 (19m50s).
• Scaling out to 10,000 people still showed the same numbers, with 88% willing to continue using GitHub Copilot, translating to a net promoter score of 66 (20m15s).
• A closing survey found that developers were using their time to tackle technical debt, with 40% of responses related to this, which is a top productivity blocker in annual developer experience surveys (21m0s).
• The survey also found that introducing AI programming with GitHub Copilot led to cultural changes, including enhanced developer experience, productivity, code quality, and learning, as well as the ability to focus on creative and value-adding work (21m40s).
• Potential negative consequences of relying on AI-generated code include over-reliance, decreased programming skills, code review skills, and motivation to learn new languages and technologies, as well as a decrease in human-centered programming (22m18s).
• A quote from Fabian Hala, a developer with a strong data science background, stated that GitHub Copilot makes it easier to be a full-stack developer proficient in multiple languages (23m11s).
• A developer at SAP, who prefers the GitHub Copilot metaphor, uses it for smooth and fast coding with Java, and was able to provide a quote and picture within five minutes of being asked, showcasing his satisfaction with the tool (23m34s).
• Surui Brahma, a Java developer at SAP in Bangalore, uses GitHub Copilot for its ability to intelligently complete her code, generate code from scratch, generate unit tests, and find and fix bugs, and she loves using it with Java, a language she is an expert in (23m51s).
• Surui Brahma also used GitHub Copilot to work with Go, a new programming language to her, and was astonished by how easily she was able to get started with it, describing the experience as "magical" (24m21s).
• Max Milon Teitz, who works with Kubernetes and Go in Berlin, uses GitHub Copilot to develop cloud-native applications better and faster, and it significantly lowered his entry barrier to getting started with Go, a programming language he was unfamiliar with (24m57s).
• As of October last year, SAP had 500 early adopters of GitHub Copilot, but today they have over 18,000 developers onboarded to the platform, which is a significant milestone worth celebrating (25m33s).
• SAP took a phased approach to the global rollout of GitHub Copilot, starting with 500 developers in October, then 3,000 by the end of the year, and scaling rapidly since January 2024, reaching their target of 18,000 developers in September 2024 (25m55s).
• Scaling in phases allowed SAP to learn from each phase, apply those learnings to the next phase, and regroup and react to feedback from previous rounds, giving them options and time to reflect on what went well and what did not (26m40s).

Impact of GitHub Copilot (27m4s)

• GitHub Copilot has provided 23.6 million AI-powered suggestions to developers at SAP between January and September 2024, with 6 million suggestions accepted and 9.3 million lines of code incorporated into SAP products and solutions (27m11s).
• The average code acceptance rate is 25.6%, calculated as the ratio of suggestions accepted to suggestions provided, with a significant increase in suggestions provided and accepted over time (27m39s).
• The number of suggestions provided increased consistently as more developers were added, with a significant increase in suggestions accepted by developers, from 1 million in January to 3.5 million in August (28m6s).
• The number of lines of code accepted also increased, from 300,000 to 350,000 in January to 1 to 1.25 million in August and September (28m35s).
• The code acceptance rate initially hovered between 20% to 22% but increased to around 24% after the introduction of chat and has since averaged around 30% for the last three months (29m35s).
• SAP provides multiple resources to onboard developers to GitHub Copilot, including a SharePoint site with an overview, FAQs, setup instructions, and dos and don'ts for responsible use (30m6s).
• A GitHub Copilot Community space allows users to share and collaborate, and a Q&A Channel provides support from experts within SAP and GitHub (30m40s).
• A series of expert talks has been added as an onboarding resource, featuring experts in AI programming and GitHub Copilot discussing various topics related to AI programming (31m10s).
• These onboarding resources have been successful, with the SharePoint site viewed over 115,000 times and the channels being actively used (31m30s).
• The SAP community and Q&A Channel have over 18,000 members each, with expert talks having more than 10,000 participants in just a few months (31m37s).
• SAP's adoption of GitHub Copilot began with an unusual start when a colleague from Buer was contacted on LinkedIn, leading to connections with other colleagues, including Tim Mina Zagen, who is at Universe (32m2s).
• SAP is now sharing their learnings from the 1.5-year journey with customers, partners, and friends, including companies like Daimler Truck, Bosch, Raj da Telecom, and BASF (32m29s).
• An important moment in SAP's adoption of GitHub Copilot was when they reached 18,000 developers on boarded (32m54s).
• Another significant moment was when Gartner released their first-ever Magic Quadrant for AI assistance, rating GitHub as a leader, which validated SAP's research and decision to roll out Copilot at scale (33m0s).

Key Takeaways (33m31s)

• SAP, the world's largest provider of Enterprise application software, rolled out AI pair programming with GitHub Copilot to over 18,000 developers, focusing on three key qualities: speed, efficiency, and security (33m31s).
• The adoption of GitHub Copilot at SAP helped create trust in the ease of use and output of generative artificial intelligence, and also allowed the company to gain empathy for corporate users of their software (33m56s).
• Research indicates that GitHub Copilot significantly enhances developer experience at SAP, which was a highlight of the last year and helped the company start reimagining the developer experience (34m32s).
• GitHub Copilot brings fun, creativity, and flow back into programming, as stated by GitHub CEO Thomas Dohmke, which aligns with SAP's experience and research findings (34m57s).
• SAP successfully scaled GitHub Copilot adoption from 500 to over 18,000 developers, with support from the GitHub Copilot adoption team and GitHub leadership (35m32s).
• SAP is proud to share their story of AI pair programming with GitHub Copilot at the 10th anniversary of GitHub Universe, and is open to sharing additional details and receiving feedback (35m49s).