Monorepos: beyond the Technicalities

Introduction to Monorepos and Polyrepos

• Monorepos and polyrepos are two approaches to organizing code, with monorepos being a single repository that produces multiple artifacts, and polyrepos being multiple repositories that produce a single artifact or multiple artifacts with complex dependency relationships (10s).
• A polyrepo setting can be identified by multiple repositories with clearly defined dependency relationships that work together to produce a single artifact, or multiple repositories with complex dependency relationships and code sharing that produce multiple artifacts (1m18s).
• A monorepo setting can be identified by a single repository that produces multiple artifacts, with code sharing between internal modules (2m18s).
• The key factor in determining whether a setting is a polyrepo or monorepo is the way code is shared between repositories, with polyrepos having code sharing between multiple repositories and monorepos having code sharing within a single repository (3m36s).

Characteristics of Monorepos and Polyrepos

• Monolithic applications are not always the result of monorepos, and simply putting code together in the same repository does not necessarily make it a monorepo (4m20s).
• A monorepo requires well-defined relationships between internal modules, allowing for a clear diagram of dependency relationships to be drawn (4m45s).
• There are also cases where it is not clear whether a setting is a polyrepo or monorepo, such as when multiple repositories produce a single artifact without code sharing, or when a single repository has internal modules that do not share code (2m33s).
• Most real-world setups are a mix of polyrepo and monorepo characteristics, making it difficult to categorize them as one or the other (3m11s).
• Modules in a monorepo are part of the same build system, similar to Maven builds in Java or Go, producing a single artifact or multiple smaller artifacts that may be important for publication or consumption (5m4s).
• A monorepo is not necessarily big or messy, but rather a repository that contains code producing more than one interesting artifact after the build, such as publishing multiple Docker images from one single repo (5m57s).
• Code sharing is also a key aspect of monorepos, allowing a single module to be reused by final artifacts (6m18s).
• It's rare to see a company with everything inside the same repository, but many companies have monorepos in the sense that they have a single repository publishing multiple artifacts (6m44s).
• Using the given definition, it can be said that everyone has a monorepo, as many repositories have multiple libraries that get published (7m21s).

Discussion on Monorepo vs Polyrepo

• This session aims to answer whether one should have a monorepo by discussing codebase structure, team operations, software development, and code reuse (7m52s).
• The discussion will be based on general terms and oversimplified examples, as it's hard to capture the uniqueness of each company or organization (8m39s).
• The conversation will also touch on how people interact to produce software, which can easily become about management and separation (9m31s).
• The structure of a company's codebase may closely mirror the structure of its teams and management, so it's essential to consider this when discussing monorepos or polyrepos (9m42s).
• A typical example of a codebase consists of multiple microservices, backend-for-frontend repos, frontend applications, common code repos, and an end-to-end tests repository, which many people can relate to, although it's oversimplified (9m53s).
• When presenting a simple example, people often have different opinions on what should be done, which can hinder the conversation about whether to pursue a monorepo or polyrepo setting (10m41s).
• To have a productive conversation, it's crucial to get everyone moving in the same direction, exposing them to the same problems and understanding the implications of library building, Ops Team deployment, and other factors (11m20s).
• The conversation about monorepos or polyrepos is slow, lengthy, and complex, and cannot be decided in a single meeting with stakeholders or architects (11m53s).
• The uniqueness of a company's operating dynamics and team alignment come before choosing a monorepo or polyrepo approach, and it's essential to document the decision-making process and ensure everyone understands why a particular approach is chosen (12m26s).

Team Dynamics and Codebase Structure

• Breaking down a codebase into teams can reveal affinities between repos and help understand how people should work together, but it's also important to consider the unique operating dynamics of each organization (13m7s).
• Different teams may have different names, operating dynamics, and collaboration styles, which can affect how they work on repos and tests (13m34s).
• It's common for multiple teams to collaborate on the same repo or end-to-end tests, and everyone may own the tests, but it's essential to consider the unique aspects of each organization's operations (14m3s).
• Different teams within an organization may have varying needs and priorities, such as a focused team for security and another for tests, which can inform decisions about monorepos and polyrepos (14m35s).
• As decision-makers, individuals have a better understanding of their team's operations and can evaluate the benefits of monorepos and polyrepos for specific parts of their organization (14m50s).
• Teams can choose to use monorepos for certain parts of their organization and polyrepos for others, rather than having to choose one approach exclusively (15m20s).

Dependency Management and Complexity

• The dependency relationships between repositories can be complex and may involve multiple layers of dependencies, including common libraries and microservices (15m39s).
• There are different ways to understand and visualize these dependency relationships, and teams may have varying approaches to managing them (15m51s).
• In addition to internal dependencies, teams must also consider third-party dependencies and versioning, which can add complexity to the development process (16m42s).
• Some teams may choose to version and publish their own libraries, which can be consumed by their services, but this can also create challenges in terms of dependency management (16m55s).
• To address these challenges, teams may adopt strategies such as updating all services to use the same version of a library whenever a new release is published, or automating the process of publishing changes to services (17m22s).
• Ultimately, the complexities of software development and dependency management can be addressed in various ways, and teams must find the approach that works best for their specific needs (17m55s).

Managing Dependencies in Polyrepos

• Managing dependencies in a polyrepo setting can be difficult, especially when dealing with third-party dependencies, as it requires managing complexity, upgrading, tracing, and ensuring reliability (18m58s).
• Making software reusable by others adds another layer of complexity, as it requires considering who is using the software and managing the impact of changes (19m7s).
• A polyrepo is typically understood as a small repository containing software for a specific purpose, producing a single artifact or a few, and the responsibility of upgrading dependencies lies with the users (19m37s).

Advantages of Monorepos

• A monorepo, on the other hand, contains multiple modules that can or cannot be related, and code is reused internally by pointing to local artifacts (20m3s).
• Monorepos can have fast builds if set up correctly, as only the necessary parts need to be built, without requiring a full rebuild of the entire repository (20m22s).
• The concepts of upstream and downstream are important in understanding the relationships between repositories and artifacts, with upstream referring to libraries and services used, and downstream referring to libraries, Docker images, and services that depend on the repository (21m11s).

Upstream and Downstream Dependencies

• In a polyrepo setting, changes to a repository can have implications for multiple downstream artifacts, making it essential to manage these dependencies carefully (21m57s).
• Publishing libraries using semantic versioning is a common setup for sharing code, but it can lead to a chain of updates and version publications, which can be wasteful if not managed efficiently (22m11s).
• Using a monorepo can potentially simplify this process by eliminating the need for multiple version publications and allowing for more efficient management of dependencies (23m16s).

Making Changes in a Monorepo

• In a monorepo, every component is in a module inside the same repository under the same build system, allowing for changes to be made and released together (23m19s).
• This approach can be beneficial, but it also means that making big changes can take weeks, especially for major version bumps, as all changes must be made at once (23m52s).
• The responsibility for updating downstream code in a monorepo depends on the team's dynamics and preferences, and this decision has an impact on whether a monorepo should be used (24m26s).
• In a monorepo, introducing changes can be more complex, as the team making the changes may need to ask for help updating downstream code, but once done, the new artifacts are deployed and aligned for everyone (24m57s).

Monorepos and Polyrepos as Complementary Techniques

• Monorepos and polyrepos are not mutually exclusive, but rather complementary techniques that can coexist in the same codebase, depending on the organization's needs and structure (25m21s).
• The decision to use a monorepo or polyrepo depends on the specific needs of each part of the organization, with some teams requiring independence and others needing alignment and synchronicity (25m58s).
• Some parts of an organization may be better suited to a monorepo, where alignment and synchronicity are crucial, while others may be better suited to a polyrepo, where independence is necessary (26m28s).
• The choice between monorepo and polyrepo is not a one-size-fits-all solution, but rather a decision that depends on the specific context and needs of each organization (26m43s).

Example: Apache Key Tools Project

• The Apache Key Tools project is an example of a large monorepo, with 200 packages, a custom build system, and a package.json file for each package, which defines how to build it and its relationships with other modules (27m11s).
• A monorepo is used to build a section, allowing the selection of the exact part of the tree to be built, with almost 50 artifacts coming from the monorepo, including Docker images, VS Code extensions, and Maven modules and applications (27m56s).
• Standardized script names are used, with each package having "build Dev" and "build prod" commands, and standalone-developed packages having a "start" command, to put everyone under the same build system (28m22s).
• Configuration is done through environment variables, borrowing from the 12 Factor App Manifesto, and an internal tool manages a large amount of environment variables to configure things like logo pass, optimizer, minifier, and tests (28m40s).
• Environment variables are used to make references to other packages, and symbolic links and definitions in package.json are used to safely reference dependencies (29m7s).
• A system is in place to prevent mistakes during builds by only referencing declared dependencies, and partial builds of the monorepo are possible in PR checks, depending on the files changed (29m34s).
• The ability to partially build the monorepo in PR checks is helpful, with a script figuring out what packages need to be rebuilt and retested, and the slowest part of a run taking 16 minutes (29m59s).
• The monorepo can scale well, with the ability to split builds into partitions and sections of the tree, and it supports multiple languages, including Java, TypeScript, Go, and container images (30m36s).
• Sparse checkout ability is available, allowing users to clone the repo and select only a portion of it, even if the repo gets very big (31m20s).

Challenges and Improvements in the Apache Key Tools Project

• Challenges exist, including the lack of a user manual, with knowledge currently residing in people's heads and private messages, but a user manual is being written (31m46s).
• Improvements are being made to the development experience for Maven-based packages, including better importation in IDEs and accurate reference picking, with issues being highlighted when they are incorrect (32m12s).
• A problem exists where changes to the top-level lock file are not understood by the partitioning system, affecting which model modules are impacted, but a solution is being researched and implemented (32m30s).
• A full build will not be required for every code change, unless it's a root-level file, and a merge queue is being considered to simulate merges and prevent semantic conflicts (32m50s).
• The merge queue will allow code to be merged automatically after passing checks, preventing breaks to the main branch and reducing conflicts (33m21s).
• Multiple cores will be made available for each package to build, enabling parallel builds, with environment variables likely being used to control core allocation (33m42s).
• Research is being conducted on using environment variables to configure parameters that distinguish between production and development builds, reducing duplication in build commands (34m10s).
• The use of Turbo Repo is being explored, which includes a test runner that understands packages and files, and has caching capabilities that can speed up development and onboarding (34m26s).

Recommendations for Building a Monorepo

• When building a monorepo, it's recommended to start small, choosing a few languages or one, and a single build tool, rather than trying to incorporate the entire codebase at once (35m15s).
• It's also recommended to establish defaults and conventions from the beginning, even if they may not be perfect, to ensure everyone is working in the same environment and can provide feedback (35m52s).
• When implementing a monorepo, it's essential to make the relationship between modules easy to visualize to avoid confusion and dependencies, as it can be challenging to navigate with many small modules (36m20s).
• Be prepared to write custom tools for unique build necessities, such as dealing with network issues, old platforms, or special build requirements (36m42s).
• Be prepared to discuss the monorepo extensively, as it's a controversial topic that may require explaining the reasoning behind it multiple times (37m18s).
• Optimize the monorepo for development by making it easy for people to clone and start working immediately, with minimal configuration steps (37m35s).
• A monorepo should have everything turned off by default, targeting development, local host, and no production references or dependencies (37m52s).
• When organizing a monorepo, do not group by technology, but instead group by operating dynamics, team affinity, and how teams interact with each other (38m12s).
• Do not compromise on quality, be thorough about decision-making, and avoid adding low-quality code to the monorepo (39m3s).
• Avoid doing too much at once when implementing a monorepo, and prioritize the most essential features (39m43s).

Re-evaluating and Adjusting the Monorepo

• Be open to reevaluating and adjusting the monorepo if it's not working out, incorporating feedback, learning from mistakes, and prioritizing the well-being of the development team (40m14s).

Q&A and Discussion

• The conversation has come to a close, and the speaker is open to answering questions from the audience, with a preference for discussing topics related to build tools and coding styles in monorepos (40m50s).
• The choice of build tool, such as Maven or Gradle, depends on the team's preference and skill set, and there is no one-size-fits-all solution (41m16s).
• The speaker's team has struggled to transition from a Maven-based approach to a less structured one using tools like npm and JavaScript (41m42s).
• In a polyglot monorepo, the speaker recommends using a flat structure to organize code, with a prefix for package names and minimal nesting to facilitate visualization of relationships between internal modules (42m44s).
• The speaker suggests that coding styles should be specific to each language and can be found in the user manual for each language (43m38s).
• A member of the audience shares their experience with using a monorepo for their entire codebase and warns about the potential drawbacks of polyglot repos, including dependency hell (43m50s).
• The speaker acknowledges the concerns about polyglot monorepos but notes that their use case requires cross-language dependencies, and their structure allows them to navigate complex dependency trees (44m40s).
• The speaker clarifies that their previous statement about polyglot monorepos was referring to individual repositories having multiple languages, rather than a single monorepo with multiple languages (45m31s).
• The company has a coarse-grained dependency structure, with a large codebase, and this structure is not working out for the organization, leading to a decision to move out of it, but some modules will remain in the same repository (45m38s).
• The company's problem was that changes to the package lock file or pnpm lock file would cause downstream things to build, and this was an issue due to the scripting system's behavior when the lock file in the root folder changed (46m15s).
• The company's goal is to build only downstream things when a dependency changes, but the previous issue was that everything would get built when the lock file changed, due to the scripting system's understanding that a root file changed (46m35s).
• The company has implemented a solution using the turbo repo diffing algorithm to understand which packages are affected by the dependencies that changed inside the lock file, allowing for more targeted building (46m54s).
• The new solution enables building only downstream things when a dependency change occurs, which is a desirable outcome (47m4s).