0 → 1, Shipping Threads in 5 Months

The Launch of Threads

• Zahan, an engineer at Meta, discusses the launch of the Threads app, which was developed in response to Elon Musk's acquisition of Twitter and the subsequent changes made to the platform (17s).
• In January of the previous year, Zahan was looking for a new project to work on, and the tech scene was buzzing about Elon Musk's business decisions since taking Twitter private in November (38s).
• Mark Zuckerberg had previously pivoted Facebook and Instagram to focus on private communication between friends, leaving Twitter as the de facto forum for public debate and discussion (54s).
• After Elon Musk took over Twitter, he made several controversial decisions, leading to concerns that the site would experience an extended outage, depriving millions of users of their favorite online platform (1m22s).
• Meta saw an opportunity to create a competing product, but initially thought that making text posts easier on Instagram would be enough, which proved to be unsuccessful (1m53s).
• The team realized that they needed to build a new product with its own culture and norms, rather than trying to adapt an existing product like Instagram (2m21s).
• The main concern was the time it would take to build a new product, as the window of opportunity to capitalize on Twitter's changes was fleeting (2m42s).
• Time to market was prioritized, and every shortcut was considered to ensure that the product could be shipped quickly (3m0s).
• The team defined the basic values of the product, including a text-first format, carrying over Instagram's design language and ethos, and openness, with a community that could craft its own experience with an API (3m22s).
• The team considered the current social media landscape, where public content is widely available through embeds, and people share their Twitter feeds across various platforms, and decided that a new walled-garden product wouldn't be successful (4m0s).
• They took inspiration from Mastodon, the fediverse, and interoperable social networks, and prioritized the needs of creators, who produce the majority of content on social networks, often following a zipf distribution (4m15s).

Development Milestones and Strategy

• The team outlined four milestones to achieve their goal, with each milestone designed to be a possible end-state where they could ship the product if needed, gradually adding essential functionality (4m59s).
• Milestone one focused on standing up the app with basic functionality, such as logging in, making text posts, and associating them with an account (5m23s).
• Milestone two added familiar features like tabs for feed, notifications, and profile, as well as basic integrity features like blocking and reporting profiles (5m35s).
• Milestone three, the lean launch candidate, fleshed out neglected services like people search, full-screen media viewer, and conversation ranking, and added features like copying photographs from Instagram and muting profiles (5m48s).
• Milestone four aimed to enable interoperability with the fediverse, but the team ultimately shipped a product similar to milestone 3.5, with a much-needed iteration on top of milestone three (6m17s).
• The team took a "Polish for launch" mindset as each milestone neared, shifting from building features to refining the product, which was exhausting but helped simplify the product (6m42s).
• The strategy of setting milestones and iterating on the product served as a strong forcing function to simplify the product and focus on essential features (7m10s).

Leveraging Instagram's Infrastructure

• The team started working on the project in earnest in February and aimed to have it ready by the summer, with a "trick up their sleeves" that helped them achieve their goal (7m42s).
• The initial plan was not to build the Threads platform from scratch, but rather to reuse existing features from Instagram, specifically broadcast sharing, which allows users to follow profiles, access a feed of posts, and respond to each other, building communities around interests (7m53s).
• The decision was made to reuse Instagram's back-end wholesale, with some custom functionality for Threads, and to fork the Instagram app code base on each platform, allowing the team to start with a fully featured product and strip it down to the necessary functionality (8m12s).
• The first prototype added a mode to the Instagram feed that surfaced text-only posts, reusing the ranking and layout, but with the caption on top and media on the bottom (8m37s).
• This approach reduced the technical scope of the project, turning the problem of building a new text-based social network into a more specific one of customizing the Instagram feed to display a new text post format (8m58s).
• The approach has major downsides, including accumulating technical debt, using a code base for something it wasn't designed to serve, and requiring a deep understanding of the legacy codebase (9m10s).
• The team focused on simplicity, allowing users to onboard by logging in with Instagram, borrowing Instagram's design language, and surgically reusing what they could while rebuilding only where necessary (9m48s).
• The Threads team is grateful for the foundation laid by the Instagram and product teams over the years, and the focus on simplicity paid off, with much of the praise received at launch being for the Spartan simplicity of the app (10m7s).

Focus on Simplicity and Safety

• To tune the experience quickly, the team used server-driven UI, sending down a full view model that told the client exactly how to render the core interfaces, allowing for easy iteration and experimentation (11m1s).
• Another big focus was making the space feel safe and welcoming for all, having seen public social networks before (11m32s).
• The goal of creating a new social network, Threads, is to build a space where people can have meaningful conversations, but this can be challenging as many online spaces turn into angry environments where people don't listen to each other (11m40s).
• Factors that contribute to achieving this goal include the product culture set by early users, tooling that helps people maintain control over their experience, and moderation (11m51s).
• The product culture of Threads was influenced by its early users, who were experienced in online communities and promoted behaviors such as blocking early and often, not engaging with rage bait, and not using quote posts to dunk on people (11m55s).
• Tooling features that help users maintain control over their experience include the ability to block, restrict, mute, and hide replies, which give users control over the conversations generated by their posts (12m8s).
• Moderation is essential to making a space acceptable to a mainstream audience, and it involves addressing extreme speech and providing a safe environment for users (12m35s).
• Moderation is considered a unique value proposition for a new social network, and it's what people are subscribing to (12m56s).
• The team behind Threads brought a decade of experience from Facebook and Instagram, which helped them start from a good starting point in terms of moderation (13m5s).

Pre-Launch Buzz and Technical Stack

• Creating a buzz for the product before launch is also important, and this can be achieved through features such as the "golden ticket" Easter egg, which sparked interest and curiosity among users (13m25s).
• The technical stack of Threads is based on a monorepo and uses a Python binary called Distillery, which talks to Meta's larger monolith, a massive PHP binary called dubdubdub (14m25s).
• The data for Threads users is stored in a variety of systems, including a graph data model and a sharded MySQL deployment, which stores almost all user data (14m56s).
• The graph data model is natively familiar with links between nodes and has optimized operations to query details of those links, and there are indexing systems on top that let users annotate particular links with how they want to query them (15m8s).
• There are several services that support the decision-making process around restrictions, including a big HLL service, a key-value store called ZDB, a serverless compute platform, and a Kubernetes-like system for managing and scaling services (15m37s).

Launch Day Challenges and Rapid Growth

• The product, Threads, was initially planned for a mid-July launch, but due to Twitter's plan to restrict consumption of tweets for free users, the launch was moved up to July 6th (16m18s).
• To generate buzz, Easter eggs were opened, and an Early Access program was started on July 5th, allowing celebrities to try the product (16m38s).
• On the day before the launch, engineers were preparing for the launch by upsizing systems, sketching out the run of show, and making other necessary preparations (17m4s).
• A data engineer noticed tens of thousands of failed login attempts on the app, which was odd since no one should have had access to the app yet (17m12s).
• The issue was quickly identified as being caused by the app store's pre-order feature, which made the app available in East Asian countries once it was past midnight (17m47s).
• A war room and Zoom call were set up with close to 100 experts to address the issue, and a new target launch time was chosen to give the team time to prepare (18m13s).
• The team spent the time before the new launch time upsizing all the systems that Threads touched, including a Zippy DB cache that needed to be resized to handle 100x the capacity it was provisioned for (18m45s).
• The user growth in the first couple of days was significant, with a million people downloading and onboarding the app in the first hour, and 10 million in the first day (19m20s).
• Threads gained 70 million users in the first two days and 100 million in the first five days after its launch, but the novelty effects started to wear off and the buzziness subsided after that (19m31s).
• Despite the relatively smooth ride with no major visible downtime, the team had to deal with numerous fires, including capacity issues and database query problems (19m50s).

Post-Launch Fires and System Improvements

• One of the fires was related to Mark Zuckerberg's timeline, which was failing to load due to an order of magnitude more interactions with his posts than anyone else in the early network (20m28s).
• The root cause of the issue was a particular database query that needed to render every post, but the runtime scaled with the number of reposts, and it was fixed by adding an index (20m50s).
• Another fire revolved around copying the follower graph from Instagram to Threads, which was a limited operation, but the issue arose when users wanted to follow people who hadn't signed up for Threads yet (21m11s).
• The system originally designed couldn't handle the scale of big celebrities signing up, such as former President Barack Obama, and the team had to redesign the system to horizontally scale and orchestrate workers (22m10s).
• The team manually worked through the backlog to ensure potential engagement wasn't left on the table, and the redesigned system has worked smoothly ever since (22m32s).

Post-Launch Feature Development and Future Plans

• After the launch, the team pivoted to address features users were asking for, including opening to Europe, shipping a following feed, and releasing a web client (23m0s).
• The development team has a limited release of an API that allows users to read and write posts, polished content search, and shows trending conversations on the network (23m24s).
• The team is working on ranking the follow graph and understanding content to match people's revealed preferences, while also catering to the needs of power users from Twitter and new users (23m37s).
• The team is adopting ActivityPub, an open protocol for interoperating between microblogging networks, and is building it piece by piece to ensure it is done correctly (24m7s).
• The reason for not integrating ActivityPub quickly is that it cannot be integrated into the Legacy Instagram broadcast sharing stack, and the team wants to make strong guarantees about how they process imported data (24m37s).
• The team is rewriting their business logic and backend stack, which is an ongoing process (24m50s).

Key Learnings and Reflections

• The biggest takeaway from the experience is the power of keeping things simple, and being clear about the value provided can guide hard decisions on where to cut scope (25m9s).
• Another learning is that cleaner, newer code isn't necessarily always better, and the little learnings encoded into an old, battle-tested codebase add up (25m30s).
• Building a product often answers questions better than abstract analysis, and prototypes can answer questions quicker than slide decks (25m45s).
• The team acknowledges how lucky they are for the opportunity and reception of the product, and none of it was guaranteed (26m4s).

Team Size and Structure

• The engineering team started with 5-10 engineers in January and grew to 30-40 engineers by launch, with a total of around 100 engineers contributing in some way (27m9s).
• The core product team stayed under 50 engineers, and they were able to reuse existing infrastructure and platform teams (27m48s).
• The team ensured the product worked and didn't kill Instagram, despite not having much time to test due to the timeline, and they lived on the product from the time it was a standalone app (28m4s).
• Instagram itself is a big organization with a couple of thousand people, and for at least a couple of months, a couple of thousand people were using the app internally, sharing content and wiping it all before going public (28m31s).
• The team got a fair amount of ad hoc testing, but they were not prepared for the scale, and they had to react quickly and rely on systems built for that scale (28m52s).

Testing and Code Reuse Challenges

• On the back end, the team has a good test-driven development culture, writing a lot of integration tests for APIs, but there's no strict adherence to it (29m13s).
• The team reused the code base from Instagram, which meant reusing it with all its edge case handling, but this made it difficult to know which context they were operating in (29m42s).
• The team has a big project underway to disentangle the code base, which is a trade-off in architecture, and the right trade-off was to take on that tech debt to launch the product sooner (30m1s).
• The team is rewriting the code base and teasing things apart, like changing the engine of a plane while it's in flight (30m28s).
• The team started by forking the Instagram code but is not planning to reunify it, instead, the code bases are drifting apart over time as Threads becomes more unique (30m42s).

Data Model and API Evolution

• The data model and APIs for Threads and Instagram are similar but are being teased apart, with the team moving towards adopting GraphQL, which allows for more granular specification of data and server definition (31m30s).
• The approach used for building Threads involved inverting the data model, where the client requests specific pieces of data, and the server composites data together, allowing for labeling and teasing out specific data for Threads or Instagram (32m14s).
• This approach enabled quick iteration on the UI from mobile applications, similar to how Uber uses a generic card format that the backend returns, but in this case, it was more product-specific (32m41s).
• Frameworks were used to handle generic tasks, such as server-side rendering, but the data model used for Threads was custom, allowing for iteration and experimentation with different rendering ideas (33m2s).
• The custom data model allowed for quick iteration and experimentation with different rendering ideas, such as how to render self-threads and reply CTAs, using a dogfooding community of a few thousand people (33m31s).

Architectural Decisions and Technical Debt

• If given the chance to do it again, some architectural decisions would be made differently, such as spending more time detangling specific data models, which would have saved time in the long run (34m30s).
• The overall ethos of reusing what can be reused was successful and would not be changed, but some technical decisions would be approached differently (35m5s).
• The team uses mature frameworks for data migrations, which were developed from past experiences, such as splitting Messenger from Facebook or Marketplace from Facebook (35m42s).
• These frameworks allow for tailing all mutations that happen on the main data store, MySQL, and enable writing observers that can double-write data and handle data migrations (36m5s).
• The development process involved maintaining two versions of a data model, such as the user model, to keep up to date, and then migrating product logic to read from both places, depending on the context, whether it's Instagram or Threads (36m32s).
• The data model is deep, with a post having many linked things that recursively need to be either Threads data or Instagram data, requiring annotations and updates that can be done statically or dynamically at runtime (37m0s).
• When reusing code, there's a risk of reusing technical debt and mistakes, but this was a trade-off to build the product quickly, with the plan to pay for it later (37m45s).
• The decision to reuse code and take on technical debt was made to build the product in six months, rather than starting from scratch and taking a year or more (38m10s).
• The team is balancing the need to evolve the product with the need to make the code base more maintainable and pay down technical debt, with a separate team focused on migration, but this will eventually impact all product developers (39m25s).
• The process of decoupling code from Instagram and making it more maintainable involves allocating resources and prioritizing tasks, with the goal of making the code easier to read and maintain in the future (38m51s).

Influence, Future Applications, and Organizational Structure

• Development decisions made for Threads have influenced changes that have gone back into other products, although specific examples are not mentioned (40m30s).
• The potential future applications of Threads include interoperating with other networks and utilizing ActivityPub, although this has not happened yet (40m41s).
• The lean nature of Threads has been a valuable lesson for Meta, as it was developed with a small team, contrasting with Meta's typical approach of building large teams around important product goals (41m1s).
• On the technical side, Threads is doing new and interesting work with ranking, particularly in handling the real-time nature of the product and reducing latency (41m32s).
• Unlike other networks like Instagram, Threads requires posts to be eligible for ranking in near real-time, as it is often used for breaking news (42m25s).
• Organizationally, Threads is still an Instagram team and a subsidiary, and while they work with other teams, they are not a straight fork, but rather reuse the same code, which can improve both platforms (43m30s).

Technical Innovation and Short-Term Focus

• Meta has a system called "better engineering" that incentivizes employees to work on technical debt in their spare time, similar to Google's 20% time policy (43m52s).
• The main short-term focus for Threads is making the data models clear between apps (44m17s).
• The decision to use Instagram's architecture for Threads was likely due to Instagram's directed graph nature, as opposed to Facebook's bidirectional friends, which has significant technical implications (44m35s).
• The choice of architecture was also influenced by Conway's Law, which states that the design of a system is often a reflection of the team that built it (45m3s).

Architectural Choices and Team Dynamics

• The team that built Threads was familiar with the Instagram text app, which made it easier to repurpose and create the new app, with the options being Facebook and Instagram, and Instagram seeming like a better benefit (45m11s).
• Initially, around 10 engineers were involved in the project, which eventually grew to around 50 engineers by the end, but the team deliberately kept it small to stay nimble and pivot the product easily (45m46s).
• The team was able to lean on platform teams, which made a big difference, and around 100 engineers touched something that made it into the Threads app (45m57s).
• The team was worried about leak risks and kept the product simple, with the goal of making it easy to pivot and make changes (46m17s).

Combating Bots and Abuse

• Bots were a problem for the team, but they were able to rely on the long history of teams fighting bots on Instagram and Facebook, and they are taking steps to address the issue (46m46s).
• The team is fighting hard against spammy replies and other forms of abuse, and they consider integrity to be a firefighting domain where they have to continually evolve to stay ahead of adversaries (47m17s).
• One of the novel challenges the team faced was dealing with the fact that Threads is more about linking out to the web, which creates challenges around awareness of what's going on and web redirects (47m47s).