Securely building GitHub on GitHub

Introduction and the Impact of AI

• Mike Hanley, the Chief Technology Officer at GitHub, discusses the transformative impact of AI on software development, highlighting tools like GitHub Copilot that enhance developer productivity and happiness while making coding more secure. (47s)
• Security has become integral to software development, and GitHub aims to enhance productivity without the friction associated with traditional security approaches. (1m32s)
• GitHub faces unique security challenges due to its scale, with 518 million repositories and 1.5 billion pull requests merged, reflecting significant growth and activity on the platform. (2m10s)
• GitHub processes about 7.1 billion API requests daily, with peaks reaching two to three times higher, underscoring the platform's extensive use. (2m51s)
• Security is often treated as an afterthought, but effective security should be integrated from the start. At GitHub, security and engineering are closely aligned to ensure security is embedded throughout the software development lifecycle. (3m17s)
• GitHub frequently receives questions about how it secures its platform and shares best practices, emphasizing that security is a collaborative effort rather than a trade secret. A strong security culture is crucial, with all employees, not just those in engineering, playing a role in protecting GitHub. (4m45s)
• The talk aims to demystify GitHub's security processes and demonstrate how security is scaled across the organization using the GitHub flow, specifically the deploy and merge model. The goal is to provide insights into tools and processes that can enhance security posture when building on GitHub. (5m36s)

AI's Role in Security

• Artificial intelligence (AI) is a significant component in reshaping software development experiences, helping to eradicate software vulnerabilities at scale and prevent them from being written. AI is also used to make security expertise accessible to every developer in an organization. (6m7s)

GitHub's Development Process and Security

• The development process at GitHub typically begins in the editor, with most developers working on individual branches and repositories associated with their team's services. Development primarily occurs in GitHub Codespaces, a cloud-based environment that mitigates traditional endpoint security concerns. (6m42s)
• Multiple security checkpoints and controls are in place throughout the development process to ensure that security standards and customer expectations are consistently met, providing a high level of security and compliance while maintaining a positive developer experience. (7m22s)

Managing Code Quality and Security at Scale

• Whitney Imura, a Senior Director of Engineering at GitHub, is introduced to further discuss how GitHub manages code quality and security at scale. (7m53s)
• The discussion covers the management of various GitHub products, including git, Enterprise server instances, and migration tools, highlighting the speaker's involvement in these areas. (8m15s)

A Developer's Experience: Working with the Monolith

• Recently, the speaker began managing the pull request teams and decided to work on a backlog ticket as part of their onboarding process to understand the developer experience. (8m30s)
• The workflow begins with using Visual Studio Code in a codespace to access the GitHub repository, referred to internally as the "monolith," which powers most of GitHub's visible features. (9m6s)
• The task involves sanitizing user-generated content in review comments to prevent security risks like cross-site scripting attacks, using GitHub Copilot to assist in the process. (9m28s)
• GitHub Copilot helps generate code for sanitization, saving time and ensuring security best practices, which the speaker reviews and accepts before attempting to push the changes. (10m4s)
• An error occurs due to GitHub Branch protection, which prevents direct pushes to the default branch, ensuring code stability and quality through checks and peer reviews. (10m36s)
• The speaker creates a new branch to push the changes successfully, highlighting the extensive activity within the main repository, including numerous pull requests, branches, and commits. (11m0s)

Branch Protection and Repo Rules

• To maintain fast and confident development, GitHub uses controls like Branch protection and a merge queue to adhere to engineering and security standards. (11m35s)
• Repo rules, an evolution of Branch protection, are introduced as a feature to keep code safe and secure at scale, showcasing the rule sets that help manage this process. (11m50s)
• GitHub Enterprise plans include organization-level rules that help ensure the security of critical branches, which can be customized by repository admins or sourced from others. (12m21s)
• A rule set called "test friction" is being rolled out to internal developers in a dry run mode to evaluate its impact and understand potential developer friction before full enforcement. (12m51s)
• GitHub allows teams or GitHub apps to bypass certain rules without needing elevated permissions, with these manual overrides being logged for auditing purposes. (13m24s)
• Rule insights provide visibility into rule enforcement, helping managers and teams ensure accountability and understand where rules might cause friction before full enforcement. (13m55s)
• GitHub publishes baseline rules in an open-source rule set recipes repository, making it easy to implement and share security protocols across projects. (14m30s)
• Features like repo rules, insights, and open-source rule set recipes help enforce best security practices, making complex security protocols accessible and ensuring projects follow best practices. (15m6s)
• GitHub's secure by design approach includes internal safeguards that prevent teams from bypassing rules, allowing compliance teams to evaluate and enforce safety standards across repositories at scale. (15m32s)

Pull Requests and Security Checks

• GitHub has built-in tools such as Branch Protection, Repository Rules, and Co-pilot to securely manage access to important repositories, helping teams write secure code by default. (16m11s)
• Additional security controls are enforced through pull requests, including code reviews and sign-offs, to detect and remediate issues. (16m46s)
• Security checks include secret scanning, code scanning, and Dependabot, with GitHub code scanning used across all repositories to find vulnerabilities. (16m59s)
• The use of AI, particularly Co-pilot Autofix, is emphasized for improving the remediation experience by analyzing code vulnerabilities, explaining them in natural language, and offering solutions directly in pull requests. (17m22s)
• The mantra "found means fixed" highlights the importance of not just identifying but also immediately fixing vulnerabilities, with AI enabling quick remediation. (18m2s)

Leveraging Security Expertise and Code Scanning

• Zavier Rene Carel, Senior Director of Security Research at GitHub Security Lab, emphasizes the role of pull requests in leveraging security expertise from peers, security teams, and the community. (18m28s)
• GitHub developers use code scanning and Co-pilot Autofix to detect and remediate common vulnerabilities, bringing community expertise to their fingertips. (19m22s)

Vulnerability Detection and Remediation with Copilot

• A vulnerability was detected in the pull request creation code using CodeQL, which involved uncontrolled data used in a path expression. This vulnerability could allow an attacker to manipulate input to access unauthorized files or directories on the server. (19m54s)
• GitHub Copilot Chat was used to explain the vulnerability, providing a detailed answer with code examples. The explanation revealed that an attacker could exploit the vulnerability by using path traversal techniques. (20m11s)
• Copilot Autofix suggested a fix that adds a check to ensure files are uploaded only to the intended directory, allowing the vulnerability to be fixed quickly. (21m2s)
• Copilot Autofix can also help address technical debt by accelerating the process of fixing existing code vulnerabilities. (21m42s)
• A GitHub workflow was demonstrated, where an AI chatbot interacts with new issues to provide potential solutions. However, this workflow was vulnerable to code injection, which could lead to the execution of malicious code. (22m20s)
• Copilot Autofix was used to generate a fix for the code injection vulnerability, demonstrating its effectiveness in learning about insecure patterns and correcting them. (23m19s)

Copilot Autofix: Impact and Availability

• In two weeks, over 600 vulnerabilities were fixed across more than 10,000 repositories using Copilot Autofix, highlighting its efficiency in improving security. (24m35s)
• Internally, Copilot Autofix has been helpful for developers, with half of the suggested fixes being used as-is or with minor tweaks. However, only 47% of security alerts are remediated. (24m51s)
• When developers use the suggestion feature, the remediation time for issues drops significantly from 1 hour and 13 minutes to 11 minutes. Even without using the suggestion, the presence of a suggestion reduces remediation time to 22 minutes. (25m20s)
• Open source code scanning and Copilot Autofix source their security knowledge from the open source community, which is considered invaluable. GitHub provides many security products at no cost to open source projects, including Copilot Autofix, to help maintainers address security issues more efficiently. (25m55s)
• In one month, open source developers used the Autofix suggestion 50% of the time, increasing the fix rate from 47% to 98% and reducing the median remediation time from 70 to 20 minutes. (26m41s)
• Copilot Autofix is available for every GitHub Advanced Security customer and for free to maintainers on any public repository, helping to eradicate software vulnerabilities at scale. (28m10s)

GitHub Fundamentals Program and Security at Scale

• GitHub's internal operations include the GitHub Fundamentals program, which governs security, availability, and accessibility across thousands of repositories and services. This program is integrated into GitHub, allowing developers to easily interact with security requirements. (28m30s)
• Michael Reinas, a staff engineer in product security, leads the development of tooling and static analysis to ensure services adhere to security standards at scale, addressing the challenges of manual processes. (29m21s)
• Over 600 GitHub Actions CodeQL alerts were remediated, showcasing the effectiveness of the fundamentals program in ensuring secure, high-quality code through collaboration between engineering and security teams. (29m40s)
• The fundamentals program is built around three pillars: accessibility, availability, and security, with the security pillar enforced using scorecards managed through an internal tool called Service Catalog. (29m51s)
• Service Catalog provides visibility into infrastructure health and helps enforce security best practices, offering detailed insights for engineers and a holistic view for engineering leadership. (30m27s)
• The code scanning scorecard shows that the organization consistently meets 98-99% of its security requirements, with significant progress made through prioritizing security improvements and automating remediation efforts. (31m10s)
• The use of autofix features allows for quick resolution of CodeQL alerts, exemplifying a strong security culture where security is integrated into all aspects of development and deployment. (32m30s)

Security Campaigns and Automation

• Inspired by internal success, security campaigns were created to automate security improvements across the codebase, combining detection and remediation capabilities with governance and collaboration features. (33m1s)
• Security campaigns enable proactive remediation at scale while maintaining the GitHub workflow, with features like Copilot suggesting fixes for alerts. (33m37s)
• Security campaigns in GitHub Advanced Security provide visibility and foster collaboration between security and engineering teams, automating processes that were previously manual. This feature is now in public preview. (34m21s)

Access Control and Entitlements

• Automation through service catalog and security campaigns ensures code security, with a focus on managing access control and privileges. GitHub has developed an open-source system called Entitlements for managing identity and access efficiently. (35m0s)
• Entitlements allows users to request access via pull requests on GitHub, ensuring compliance and proper access management. The system uses the concept of virtual teams, exemplified by a "pizza team" setup for a dinner event. (35m56s)
• Entitlements has been open-sourced for public use, and users can explore it at g. entitlements. (37m10s)

GitHub's Security Philosophy and Best Practices

• GitHub emphasizes being its own best customer by integrating its tools and practices to ensure a high standard of security and a positive developer experience. Security is foundational to GitHub's operations, and the company encourages others to adopt similar practices. (37m32s)
• Decisions that benefit both developers and security are not mutually exclusive, and enhancing the developer experience can simultaneously strengthen security. (38m49s)
• AI is transforming the concept of "Shifting security left" by integrating security expertise into every step of the software development lifecycle, making it accessible to all developers. (39m3s)
• AI-driven tools like Copilot Autofix and security campaigns are helping developers significantly reduce software vulnerabilities in both new and existing code. (39m22s)
• Building a supportive security culture within an organization, characterized by transparency and collaboration, is crucial. Security teams should be seen as enablers rather than obstacles. (39m47s)
• GitHub is committed to sharing best practices and openly discussing effective strategies to help customers and the open-source community improve their security. (40m7s)