Matthew Clark on the BBC’s Migration from LAMP to the Cloud with AWS Lambda, React and CI/CD

02 Oct 2024 (19 days ago)

BBC's Cloud Migration Journey

The BBC initiated its cloud migration around 2012-2013, starting with video transcoding capabilities, and set a goal to move all online services to the cloud by 2020. (3m52s)
The BBC decided to move from its nanoservices architecture to AWS Lambda around 2018 when serverless technology matured, offering improved performance and off-the-shelf functionality. (9m11s)

The original BBC website architecture was a classic LAMP stack (Linux, Apache, MySQL, PHP) with a separate Java API layer, Varnish servers for caching, and other data sources. (5m23s)
The BBC transitioned to a "nanoservices" architecture before adopting AWS Lambda. This approach involved smaller units of change than microservices, focusing on individual web pages, page components, or API endpoints. (8m5s)

The speaker explains that they chose to use AWS Lambda instead of managing their own virtual machines (VMs) because it allows their teams to focus on business logic rather than server management tasks like security, scaling, and patching. (9m52s)
While acknowledging that containerization with Kubernetes or Fargate is a viable option, the speaker states that their organization's early adoption of cloud computing led them to become proficient in building VM images, making serverless a natural progression. (10m47s)

The speaker highlights Lambda's ability to handle rapid traffic fluctuations, going from thousands to tens of thousands of requests per second quickly, as a significant advantage for their workloads. (11m8s)
The BBC found that a 1GB memory profile for their Lambda functions provided the best balance of performance and cost, even though they do not utilize all of the allocated memory. (19m51s)

The HTML rendering process utilizes React and Node.js, employing an isomorphic approach where the same JavaScript code runs on both the server and client-side for dynamic updates. (14m18s)
The presentation logic is decoupled from the data logic, which fetches and processes data through a separate API or "business layer" composed of functions that call each other in a chain. (14m28s)

To maintain consistency, performance, and accessibility, a mono repo approach is used for web development, allowing multiple teams to contribute to a single deployment as a function capable of rendering various pages. (16m33s)
The BBC attempts to limit chains of AWS Lambda functions to three or less to manage cost implications, as they are paying for CPU time regardless of whether it is being fully utilized. (19m11s)

While the BBC has optimized their Lambda functions for fast response times, they acknowledge that cold startup times can still be a challenge, with some functions taking several seconds to return. (20m29s)
The BBC uses Redis as a key-value store for caching data, setting an expiry time for keys to manage data freshness. (23m1s)

The BBC leverages Redis's list and stream data types to implement a queuing system, achieving fast communication between system components. (24m55s)
The BBC's adoption of cloud technologies has led to frequent releases, with updates to user-facing websites and APIs deployed every 10 to 20 minutes during the workday. (26m20s)

Organizations need to find a balance between empowering teams and preventing too much divergence in approaches, especially during large-scale transformations that impact hundreds of people. (27m29s)
Moving from a data center to the cloud provided an opportunity to change approaches, responsibilities, and eliminate duplication across teams. (27m48s)
A "you build it, you run it" DevOps model, where engineers take responsibility for their services in production, including on-call duties, was implemented. (28m33s)