Deepthi Sigireddi on Distributed Database Architecture in the Cloud Native Era

Deepthi Sigireddi and Vitess

• Deepthi Sigireddi is the technical lead for Vitess, a cloud-native, open-source, distributed database project, and also the Vitess engineering lead at PlanetScale, a database-as-a-service company built on Vitess (1m11s).
• Deepthi's career started as an application developer working with databases, including Oracle, DB2, and MySQL, and later transitioned to working on supply chain planning solutions for the retail industry, which required handling massive amounts of data (1m36s).
• To handle large amounts of data, Deepthi and her team developed parallelizable computing solutions to work with monolithic databases, and later worked on cloud security at a startup that was acquired by IBM, where they implemented custom sharding for a multi-tenant system (1m57s).
• Deepthi joined PlanetScale and started working on Vitess, a massively scalable distributed database system built around MySQL, which solves the limitations of monolithic MySQL and provides a distributed data management capability (3m14s).

Vitess Origins and Capabilities

• Vitess was created at YouTube in 2010 to handle the large amount of traffic and video metadata stored in MySQL, which was causing the site to go down daily (4m6s).
• Vitess brings a distributed data management capability to MySQL, allowing it to scale beyond the limits of a single server and providing a high level of performance, availability, scalability, and resilience (3m52s).
• YouTube's MySQL database was unable to handle the increasing load, leading to the development of Vitess, a distributed database architecture, to solve this problem in a fundamental way (4m30s).
• Vitess allows for vertical sharding, where a set of tables is split across multiple MySQL servers, and horizontal sharding, where a single table is distributed across multiple servers, making it transparent to the application (5m22s).
• In the Vitess system, every MySQL instance has a sidecar process called VTTablet, which manages the MySQL instance, and applications interact with the system through a gateway called VTGate, which accepts requests and routes queries to the backing VTTablets (6m1s).
• The VTGate gateway can accept MySQL protocol or gRPC calls, decide how to route queries, and aggregate results if necessary before sending them back to the client application (6m18s).

Choosing a Cloud-Native Database

• When looking for a cloud-native database, developers should consider usability, compatibility with their application needs, and uptime, as these factors are crucial for a cloud database (7m12s).
• Almost every commercially available database has a cloud offering, and developers should evaluate these options based on their specific needs, considering factors such as configuration, tuning, and support for their application (7m30s).
• Compatibility is essential, as there are many cloud databases available, including Amazon RDS, Google Cloud SQL, and Oracle Cloud database offerings, and developers should choose a database that supports their application needs (8m11s).
• Uptime is also critical, as the database is often critical to operations, and developers should evaluate the historical uptime of the database before making a decision (8m36s).
• Cloud providers handle baseline requirements such as data safety, database availability, and data loss prevention, allowing users to focus on other aspects of their applications (8m51s).
• Not every application requires a massively parallel distributed database, and users should consider their specific needs before choosing a database solution (9m21s).
• In general, the only reason to not use a cloud database is due to legal restrictions on storing highly sensitive data with a third-party provider (9m52s).
• Cloud databases are a good choice for most applications, but not all applications require a distributed database, and a standalone database in the cloud can be sufficient (10m27s).

Vitess Sharding

• When choosing a database, users should consider factors such as usability, compatibility with existing applications and frameworks, and reliability (10m44s).
• Distributed databases like Vitess use sharding to provide scalability and replication to provide high availability (11m12s).
• In Vitess, sharding is customizable, allowing users to choose the column and function used for sharding, and also provides a public interface for building custom sharding functions (11m36s).
• Vitess's sharding process involves computing which shard to write data to when inserting a row and computing which shard to read data from when querying a row (12m0s).
• For queries that cannot be computed to a specific shard, Vitess uses a scatter query to send the query to all shards, gather the results, and bring them back (12m44s).
• As a database grows, it eventually reaches a point where everything starts slowing down, and managing things becomes difficult, prompting the need to shard the database (13m9s).
• Horizontal sharding is a method of sharding where data is split across multiple servers, and tools can be used to define the sharding scheme, choose the tables to shard, and copy data onto new shards (13m22s).
• The sharding process can be done in the background while the system is still running, and when ready, the database can be switched to the new sharded configuration, with the original database being stopped for a short period, typically less than 30 seconds (14m2s).

Replication and High Availability in Vitess

• An optional reverse replication can be set up to keep the old databases in sync with the new shards, allowing for rollbacks in case something goes wrong (14m32s).
• Replication is a crucial aspect of modern software applications, as downtime is not acceptable, and services need to be available 24/7 (15m14s).
• Traditionally, high availability is achieved through replication, where a primary database is written to, and replicas follow and apply changes to their local databases, providing an always-available copy of the data (16m17s).
• Vitess uses replication to provide high availability and has tooling to guarantee very high availability, allowing for planned maintenance without downtime by transitioning leadership from the primary to a replica (16m44s).
• This process enables maintenance to be performed on the primary database without affecting the availability of the service (17m6s).
• Vitess uses replication for higher availability, and around that replication, features have been built to handle planned maintenance and unplanned failures, such as a primary MySQL instance going down due to memory issues, disk errors, or network problems (17m56s).
• In Vitess, a monitoring component called VTOR (Vitess Orchestrator) monitors the cluster and elects a new primary instance if the current one is unreachable, ensuring that data is not lost (18m10s).
• VTOR also monitors replication on replica instances and ensures they are replicating correctly from the cluster primary, as well as monitoring for other error conditions and fixing them (18m33s).
• When a node in the cluster is not available or needs to be down for maintenance, a replica node will take over leadership, but the data needs to be properly sharded, partitioned, and replicated (18m59s).
• The process of switching between nodes for planned maintenance is not fully automated by Vitess, but users can automate it using the VTOR and command-line interface, typically during a rolling upgrade of the cluster (19m39s).
• For unplanned failures, VTOR handles the monitoring and failover without human intervention, and the timing for this process can be as quick as 5-10 seconds (20m20s).

Distributed Transactions in Vitess

• During planned failures, applications may see a slight delay in response time, but not errors, due to request buffering, while unplanned failures may result in errors, but typically within a 30-second timeframe (20m38s).
• Distributed databases come with distributed transactions, which involve balancing data consistency and data availability, as stated by the CAP theorem, which says that if you want one, you can't have the other (21m52s).
• In a distributed system, consistency issues arise when reading data from replicas, as they may be caught up to different extents, resulting in an inconsistent view of the data (22m57s).
• Historically, Vitess has dealt with consistency issues by reading from the primary instance if consistency is important, and from replicas if it's not, but people have used tricks like always reading from the primary in user sessions that involve writing (23m24s).
• A desired feature in Vitess is to provide a way to specify read-after-write consistency, which would guarantee up-to-date data whether reading from the primary or a replica (24m13s).
• In a distributed system, there is always a possibility of a distributed transaction, but most transactions go to one shard, in which case MySQL itself provides transactional guarantees (24m49s).
• Distributed transaction problems arise when updating rows in different shards, such as in a bank transaction where both the sender and recipient's balances need to be updated (25m2s).
• To ensure data consistency, distributed transactions are necessary, but true distributed transaction support can be challenging to implement, so creative solutions like writing to a ledger and reconciling after the fact can be used to solve this problem (25m20s).
• Vitess uses a best-effort distributed transaction approach, where writes are executed one at a time, and if a failure occurs, all previous writes are rolled back, reducing the universe of possible failures (25m55s).
• Once all writes are successful, the commits are issued in parallel, and the probability of a commit being rejected is low, making this approach effective for most use cases (26m37s).
• Truly atomic distributed transactions are on the roadmap for Vitess, but the current best-effort approach has proven to be reliable (27m2s).

Schema Management in Vitess

• Schema management and data contracts are gaining attention, and Vitess has a tool called VT Admin, which provides a structured gRPC API and a UI for managing the cluster (27m21s).
• VT Admin allows users to view schemas, but schema management is not yet available through the UI, although the APIs are available for building custom tools (28m58s).
• Best practices for managing schemas and schema versions include using tools and frameworks that provide versioning and change management capabilities, and Vitess plans to improve its schema management capabilities in the future (29m20s).
• Storing schema in a version control system is recommended, as schema changes are a crucial topic in the MySQL community due to historical performance issues with large tables under load (29m38s).
• MySQL has improved its schema change capabilities, making certain changes instant to prevent locking and blocking, but the community had to develop its own tools due to the slow pace of progress (30m20s).
• The principles for online schema changes include being non-blocking, happening in the background, and allowing for reversion without data loss (30m49s).
• Vitess has an online schema change system that follows these principles, allowing for non-blocking changes that can be initiated and cut over manually or automatically (30m56s).
• Vitess's online schema changes are designed to be non-blocking, allowing the system to continue running with minimal additional load (31m17s).

Vitess Roadmap and Future Improvements

• New features on the Vitess roadmap include improved MySQL compatibility, with recent additions such as Common Table Expressions and window functions (32m16s).
• Performance improvements are also ongoing, including a new connection pooling implementation that reduces latency and memory utilization (33m8s).
• The new connection pooling implementation makes the connection pool more efficient, reducing the need to cycle through the entire pool and minimizing the number of open connections required (33m19s).
• Benchmark query performance is run every night and the results are published on a dedicated website called benchmark.us.doio, with an ongoing effort to continuously improve the benchmarks (33m42s).
• Recent functionality improvements include the second iteration of point-in-time recovery in Vitess, with plans to make further improvements based on user feedback (34m4s).

Learning More about Vitess

• To learn more about distributed databases in general or Vitess in particular, online resources include the Vitess website, which has documentation, examples, quick start guides, and links to videos (34m31s).
• Vitess is part of the Cloud Native Computing Foundation, and was donated by Google in 2018, graduating from the foundation in 2019 (34m35s).
• The Vitess website provides resources for running Vitess on a laptop, within Kubernetes, and has links to videos from maintainers, community members, and users (34m49s).
• Searching for "Vitess" on YouTube yields plenty of talks and introductions to Vitess features, architecture, and diagrams (35m26s).
• Working on open-source projects like Vitess has been a positive experience, allowing for new interactions, experiences, and personal growth as a software developer (35m54s).
• Listeners can learn more about data engineering topics by checking out the AIML and data engineering community web pages on infoq.com, and listening to recent podcasts (36m41s).