Mastering Long-Running Processes in Modern Architecture: Real-Life Examples & Tools for Engineers

Introduction to Long-Running Processes

• Long-running processes can be compared to ordering food, such as pizza, where there are different ways to place an order, including phone calls and emails (17s).
• Phone calls represent synchronous blocking communication, where the caller is blocked until the other person answers, and a direct feedback loop is established once the call is answered (42s).
• However, this method has limitations, such as being temporarily coupled to the availability of the other side, and if the person is not available, the caller must try again or wait (1m13s).
• An alternative to phone calls is sending an email, which represents asynchronous non-blocking communication, allowing the sender to send the message even if the recipient is not available (1m40s).
• Emails lack a direct feedback loop, but the recipient can still respond to the email, providing a feedback loop, albeit asynchronously (2m6s).
• The key difference between synchronous and asynchronous communication is not the technology used, but the interaction pattern, which can be decoupled from the technology (2m36s).
• In the case of ordering pizza, the feedback loop is not the same as the result, as the customer is still hungry after receiving confirmation of their order, and the actual result is the pizza being delivered (2m51s).
• The task of making pizza is a long-running process that takes time, involving multiple steps, such as baking and delivery, and this pattern is seen in many other interactions beyond just ordering food (3m10s).
• Synchronous blocking behavior is not suitable for long-running processes, as it would require the customer to wait for an extended period, and asynchronous results are more appropriate (3m39s).
• The process of making coffee at a machine is described as synchronous and blocking, meaning that while waiting for the coffee, no other tasks can be performed, leading to inefficiencies and poor user experience, especially when there is a queue. (4m12s)
• An article by Gor Hooper discusses how Starbucks scales its coffee-making process by separating the ordering and payment from the actual coffee preparation, allowing baristas to work independently and improving scalability and user experience. (4m48s)
• Fast food chains are increasingly using apps for ordering to streamline the initial steps of the process, although the actual preparation, such as coffee making, often still involves human workers like baristas. (5m41s)

Challenges of Long-Running Processes

• Long-running processes are defined as those that involve waiting, which can be due to human tasks such as approvals or decisions, or simply waiting for a response from a customer. These processes can take from hours to weeks. (6m14s)
• An example of a startup is given where they automated a service but intentionally added a delay to simulate human processing time, highlighting the importance of managing waiting times in processes. (7m15s)
• Waiting is challenging because it requires remembering the state of the process over potentially long periods, necessitating persistent state management to ensure continuity when the process resumes. (7m44s)
• Persistent state can be a problem, despite the existence of databases, due to subsequent requirements such as understanding what is being waited for, escalating if waiting for too long, versioning problems, and running at scale (8m9s).
• These technical challenges can be difficult to solve without adding accidental complexity, and homegrown workflow engines are often not a good solution (9m1s).

Workflow Engines as a Solution

• The speaker has experience working on workflow engines, CR engines, and orchestration engines, having co-founded Kamuna, a workflow orchestration company, and worked on open-source workflow engines (9m42s).
• A workflow engine, also known as an orchestration engine or process engine, can solve long-running issues by defining workflows and running instances of them, settling requirements such as versioning and escalation (10m23s).
• A demo of a workflow engine was given to illustrate its capabilities and provide a common understanding of what a workflow engine is (10m45s).
• The demo used an example of an onboarding process, which is a common process in many companies, such as opening a new bank account or mobile phone contract (11m32s).
• The workflow engine used in the demo is available on GitHub, allowing others to run it themselves and experiment with workflow engines (11m2s).
• BPMN (Business Process Model and Notation) is an ISO standard used to define processes graphically, and it's not a proprietary thing, allowing for standardized process modeling (11m51s).
• BPMN models can be used to define manual tasks, such as scoring a customer and approving an order, and can also include automated tasks and escalations (12m9s).
• The BPMN model can be used to define a duration with a period of time, such as 10 seconds, to determine when a task is taking too long and should be escalated (12m48s).
• A Java application, in this case a Spring Boot application, can be used to connect to a workflow engine, deploy the process, and provide a small web UI (13m2s).
• The application can trigger a REST call to start a process instance within the workflow engine, and tools like Operate can be used to look into what's going on and see the versioning and instances running (13m45s).
• The workflow engine can be used to automate tasks, such as sending an email, and can also be integrated with other systems, such as a CRM system, using custom Java code or pre-built connectors (15m1s).
• The workflow engine can also be used to send emails, and the email can be configured using a pre-built connector, such as the one for SendGrid (15m12s).
• The workflow engine is running in the background, and the workflow model has instances running through it, with code or UI attached to connect to systems or humans (15m36s).
• The workflow engine used in this example is Camunda as a Service, and it's integrated with a Spring Boot application (15m49s).
• Workflow isn't just for small-scale processes, but can be run at a huge scale, with thousands of process instances per second, and can be distributed across multiple data centers in different geographic locations, such as the US and UK, which can add latency but doesn't bring throughput down (16m10s).

Technical Reasons for Waiting in Long-Running Processes

• There are technical reasons why processes may need to wait, including asynchronous communication, where a message may not be received immediately, and failure scenarios where a message is not received at all (17m1s).
• In distributed systems, peer services may not always be available, requiring processes to wait for them to become available before proceeding (17m33s).
• A common example of a long-running process is checking in for a flight, where a user may receive an email notification to check in, but the process may fail, requiring a retry, which can be done in a stateful manner, where the retry is scheduled for a later time (18m23s).
• In the case of the flight check-in example, the user may need to wait for a few hours before retrying, and can use a calendar entry to remind them to try again, illustrating a stateful retry in a long-running process (19m24s).
• The situation can be envisioned as having a web interface, a check-in microservice, and a background process that handles the check-in, which may need to wait for certain conditions to be met before proceeding (19m51s).
• A personal experience of a failed check-in process due to a barcode generation issue is used to illustrate the importance of resiliency in distributed systems, where certain parts are always broken or network connections are always down (19m59s).
• The failure of a single component, such as barcode generation, should not bring down the entire system, and a well-designed system should be able to handle such failures locally without affecting the overall user experience (20m51s).
• A chain reaction of failures can occur when a problem is passed on to the user, making them responsible for resolving the issue, which is a bad design (21m12s).

Resilience in Long-Running Processes

• A better design would be for the check-in service to handle the issue locally, for example, by checking in the user and sending the boarding pass later, which requires long-running capabilities within the service (22m44s).
• Many teams prefer to be stateless and avoid keeping state, which can lead to rethrowing errors instead of handling them locally (23m23s).
• Customers often expect a synchronous response, such as seeing a confirmation message and receiving a boarding pass immediately, which can make it challenging to implement long-running processes (23m37s).
• A more resilient design would prioritize handling errors locally and providing a better user experience, even if it means not providing an immediate synchronous response (23m2s).

Handling Long-Running Processes in Payments

• The discussion extends the example of flight bookings to include payment collection, specifically handling credit card payments, which typically involves using an external API service like Stripe. (24m15s)
• There is a challenge with service availability when charging credit cards, as the service might not be available at the time of the transaction, necessitating alternative solutions to avoid disappointing customers. (25m10s)
• In distributed systems, remote call exceptions can arise from various issues, such as network problems or service provider failures, making it difficult to determine the exact cause of the failure. (26m0s)
• Handling exceptions in distributed systems is complex because it is unclear whether a transaction was completed, which can lead to issues like double charging if not managed properly. (26m29s)
• Solutions to these issues include using workflows or running periodic reconciliation jobs to ensure transactions are correctly processed and any discrepancies are addressed. (26m49s)
• Embracing asynchronous thinking is recommended, where APIs are designed to acknowledge requests and provide results later, using HTTP codes to communicate the status of the request. (27m16s)
• Long-running processes can extend the options of what an API can do, and making APIs asynchronous allows for better handling of long-running tasks within services, giving more freedom to implement requirements as desired (27m47s).
• Extending payment options to include customer credits on their account, similar to some companies that offer credits for returned goods, or PayPal's system of holding funds before deducting from a bank account, can provide more options for handling payments (28m11s).
• Implementing long-running processes can pose new problems around consistency, such as handling transactions across different services, like credit handling or credit card charging, and ensuring that all steps are technically transactional (29m0s).
• In distributed systems, failing a payment process can require compensating actions, such as rebooking customer credits, to maintain consistency, and this complexity can arise quickly when considering all implications (29m33s).

Service Boundaries and Long-Running Processes

• Having long-running capabilities is necessary for designing good services and service boundaries, and this technical capability should be present in the architecture (30m17s).
• A booking service can tell a payment service to retrieve payment via a message or REST call, and if the credit card is rejected, the next step would be to ask the customer to provide new details, allowing them to still book their flight (30m41s).
• Long-running processes can be used to handle scenarios where a customer needs to provide new credit card details after the initial rejection, and this can be achieved through a workflow that includes compensating actions (31m40s).
• GitHub subscriptions have a fully automated process for renewal, but if the credit card is invalid, an email is sent to update the card, introducing a long-running process that requires handling (31m59s).
• A common reaction to this requirement is to pass it to a component that already handles long-running processes, such as booking, but this can lead to domain concept leakage and added complexity (32m43s).
• Booking should not know about credit card details, as it only cares about receiving payment, and handling payment methods should be separate (33m21s).
• Domain-driven design (DDD) also emphasizes the importance of separating domain language and concepts, and in this case, the booking service should not care about credit card rejection (33m43s).
• To handle long-running requirements within payment, it's essential to make it easy for teams to implement, and potentially using workflows or orchestration can help (34m12s).
• Payment might be fast and synchronous in most cases, but handling edge cases where it's not is crucial, and designing an API that can handle both cases is necessary (34m33s).
• Using workflows or orchestration can help implement long-running processes, and having these capabilities available in different services can make it easier to distribute the process correctly among microservices (35m10s).
• Not having long-running capabilities in a service like payment can lead to monolithic design if the logic is moved to another service, such as booking, just because it has the capability (35m47s).
• Having long-running capabilities at the disposal of every service avoids the creation of monolithic "God Services" and makes it easier to distribute responsibilities correctly, as well as embracing long-running, asynchronous, and non-blocking processes (36m1s).

Organizational Strategies for Long-Running Processes

• A good architecture requires a process orchestration capability, which can be obtained as a service, either internally or externally, and can be easily implemented by a team (37m17s).
• Organizations that successfully use process orchestration often have a Center of Excellence, a dedicated team that cares about process orchestration, process automation, and related topics (37m58s).
• A Center of Excellence should focus on enablement and providing a platform, rather than building solutions, and should enable others to build things by consulting, helping, and providing technology (38m50s).
• The traditional model of central teams being involved in solution creation has been replaced by a model where central teams focus on enabling others, and this shift is driven by the need for autonomy and freedom in decision-making (39m32s).
• The creation of a Center of Excellence is not a step backward towards centralization, but rather a way to enable teams to make their own decisions while still providing guidance and support (40m10s).
• The concept of team topologies is discussed, emphasizing different types of teams to enhance development efficiency. These include stream-aligned teams focused on business logic, enabling teams with a consulting function, and platform teams providing necessary technology. (40m17s)
• Stream-aligned teams are designed to maximize productivity and reduce friction, allowing them to deliver business value effectively. (40m50s)
• Enabling teams assist by consulting across projects, while platform teams supply the technology needed, preventing teams from having to figure out everything independently. (41m11s)
• The complicated subsystem team is mentioned but not emphasized, as it deals with specialized tasks like fraud checks or AI services. (41m24s)
• Organizations can map these team structures effectively, often using a center of excellence for process orchestration and automation, with tools like Camunda or RPA tools. (41m36s)
• This approach prevents teams from spending excessive time in evaluation mode without delivering business value. (42m20s)
• Spotify's "Golden Path" concept from 2020 is highlighted, where defined solution templates are provided for building specific types of applications, making it easy and desirable for teams to use them without being forced. (42m39s)
• The "Golden Path" approach helps avoid "rumor-driven development," which is not scalable and can lead to inefficient technology use. (43m40s)
• Spotify also developed an open-source tool called Backstage.io to support this approach. (44m9s)
• Spotify's approach to development emphasizes autonomy for teams, but as the company grows, the software ecosystem becomes more complex and fragmented, leading to slower development speeds (44m28s).
• Standardization of services and tooling can help free engineers from infrastructure complexity, rather than restricting autonomy, as seen in the concept of the "standards paradox" (44m47s).
• Companies like Twilio offer pre-built services, known as the "PaaS path," that allow teams to get up and running quickly, and creating an incentive structure can encourage teams to take this path (45m10s).

Graphical Models and Long-Running Processes

• Graphical models, such as BPMN, can be used to express complex processes in a simple and powerful way, and can be used for living documentation, test cases, and operations (45m55s).
• Graphical models can also be used to discuss complex processes with different stakeholders, including non-developers, and can help elevate decisions about long-running behavior to the business level (46m50s).
• Visualizing complex processes is important for making decisions about long-running behavior, and can help redesign the customer journey to leverage new architecture (47m41s).
• Redesigning the customer journey is necessary to fully leverage new architecture, and graphical models can be a powerful tool in this process (47m59s).

Real-World Examples and Conclusion

• The airline industry has seen significant changes in customer experience over the last five years, with automation playing a key role in improving services, such as automatic check-in for flights (48m28s).
• A personal experience with a delayed and canceled flight to London demonstrated the use of automation in rebooking and providing updates through email and a mobile app, although some issues still required human intervention (48m38s).
• The use of long-running capabilities, process orchestration platforms, and workflow engines can help design better service boundaries, reduce complexity, and provide a better customer experience (50m22s).
• Embracing asynchronicity and using these technologies can also increase operational efficiency, automation, and compliance, while reducing risk and documenting processes (50m46s).
• To successfully adopt these technologies across an organization, central enablement is necessary, and resources such as books, websites, and conferences can provide more information on the topic (51m8s).