Orchestrating A Path to Success - a Conversation with Bernd Ruecker

cucon San Francisco conference

• The cucon San Francisco conference will take place from November 18 to 22nd, where engineers will share how they implement innovations in real-world scenarios, and attendees can explore tracks like architectures, engineering productivity, and generative AI in production (10s).

Introduction to Bernd Ruecker

• Bernd Ruecker is the founder and chief technologist at Camunda, a company involved with process orchestration, which now has about 500 people and clients including banks, insurance companies, and telcos (52s).
• Bernd Ruecker has contributed to various open-source workflow engines over the past 20 years and has written three books, with a fourth one in progress (1m8s).
• Bernd Ruecker started programming at the age of 13 or 14 and became interested in software architecture, which led him to study computer science and earn a Master's degree in software engineering (1m37s).

Bernd Ruecker's Early Life and Education

• Bernd Ruecker's journey into architecture began when a friend asked for help with his business, which involved selling modded graphic cards online, and he started writing an ERP system to manage the business, which taught him about software architecture and the importance of understanding the bigger picture (1m50s).

Bernd Ruecker's Journey into Software Architecture

• Bernd Ruecker's experience with the ERP system led him to think about orchestration, as part of the system's functionality involved looking at data and addressing orders (3m49s).

Process Orchestration: Definition and Importance

• Process orchestration involves getting order in certain steps, coordinating those steps, and using technology to define models that can be interpreted and run by a workflow engine or process orchestrator, with the goal of efficiently managing processes such as order fulfillment and returns handling (6m30s).
• The journey into process orchestration began with looking into visuals for workflows, which were not well-developed at the time, and exploring options such as expensive enterprise software and open-source projects, ultimately leading to contributing to an open-source project and implementing a workflow within that system (4m31s).
• Process orchestration is a type of distributed system, and people often struggle with it because it involves coordinating steps and making decisions, especially when humans are involved, and it requires a good understanding of the underlying technology and models (6m12s).
• BPMN (Business Process Model and Notation) is a graphical model used to define processes, and it is a key part of process orchestration, allowing users to define steps and sequences, and run instances of those processes through a workflow engine or process orchestrator (6m45s).
• Process orchestration is also about long-running processes, and it involves using workflow engines or process orchestrators to interpret models and run instances of those processes, making decisions and taking actions as needed (7m31s).

The Path to Process Orchestration

• The lack of good solutions for workflow management led to exploring open-source projects and eventually contributing to one, which led to freelancing and co-founding a company that specializes in process orchestration as a product (5m9s).
• The journey into process orchestration was not planned, but rather stumbled upon, and it involved exploring different options and technologies, and eventually finding a passion for visuals and workflow management (5m39s).

Challenges and Misconceptions of Process Orchestration

• A process orchestrator can wait for various events to occur, such as human input, asynchronous system responses, or customer feedback, and it can persist and provide tooling to track the current state of the process (7m36s).
• The difficulty in adopting process orchestration is largely due to perception, with many developers having a negative view of visual models, such as class diagrams and process models, which may be seen as outdated or unnecessary (8m10s).
• The use of visual models, such as UML diagrams, can be helpful in providing an overview of code and allowing for automatic documentation, but this approach has not been widely adopted (8m51s).
• Process models, such as BPMN, can be executable code and readable by both technical and non-technical stakeholders, making them a valuable tool for process orchestration (9m24s).
• The fear of process models among developers may be historical, dating back to the idea that managers could program using languages like COBOL, which was ultimately unsuccessful (9m54s).
• The paradox of programming, where software engineers must constantly reprogram and innovate, may contribute to the frustration with visual modeling tools and process orchestration (10m21s).
• The use of process orchestration tools, such as BPMN, can be seen as an abstraction, similar to programming languages like Java or C, and can provide a valuable layer of abstraction for process management (11m12s).

BPMN and its Role in Process Orchestration

• BPMN (Business Process Model and Notation) is a general-purpose language that can model various problems on a different abstraction level, and it's not just for business processes, but for any long-running problems that need to be expressed (11m26s).

Homegrown Workflow Engines vs. Specialized Products

• BPMN has certain elements that are great for expressing long-running problems, and it's recommended to use them instead of implementing homegrown bespoke workflow engines (12m3s).
• Homegrown workflow engines often start with a simple task, such as monitoring an order status column in a table, but can quickly become complex and messy (12m17s).
• The low-code movement aims to minimize the amount of code that a developer has to write, which is generally a good thing, but can sometimes make it difficult to understand what's happening when the system is not working (12m57s).
• Orchestration engines should be a black box, meaning that the core engine itself should not be touched or debugged, but rather treated as a separate entity that triggers certain activities (13m23s).

Orchestration Engines: Best Practices and Debugging

• Debugging into the core engine is not recommended, but debugging into the activities and code that are triggered by the engine is still possible (14m11s).
• Using an orchestrator does not eliminate the possibility of writing automated unit tests for the activities and code that are put into the process (14m21s).
• The relationship between an orchestrator and its user is similar to the relationship between an operating system and its user, where the user writes code on top of the operating system without stepping into it (14m32s).
• Bugs in the orchestrator should be treated similarly to bugs in the operating system, where the user reports the issue to the vendor and relies on them to fix it (14m50s).

Cloud Considerations and Security

• The cloud discussion is similar to the orchestrator discussion, where security concerns are often raised, but it's generally more secure than what most companies do on their own premises (15m1s).

Orchestration vs. Choreography in System Design

• A discussion is presented on the reasons behind using specialized products for certain tasks, comparing it to the logic of robbing banks because "that's where the money is." It is suggested that using a product designed for a specific purpose is often better than creating a custom solution, despite common objections based on simplicity or stability concerns. (15m19s)
• There is a debate between orchestration and choreography in system design, focusing on whether to control processes or allow them to evolve through event-driven methods. There was a belief that event-driven architecture is easier, more decoupled, and modern, leading some projects to avoid orchestration due to misconceptions about centralized bottlenecks. (16m12s)
• Both event-driven and orchestration approaches have their place in system architecture, particularly in distributed systems where different services need to communicate, such as CRM, ERP, and logistics systems. (17m10s)
• Orchestration involves a single component managing the end-to-end process, while choreography distributes responsibilities among components, each reacting to events. This is a core architectural decision about how to distribute responsibilities. (17m37s)

Orchestration and Choreography in Distributed Systems

• In business processes like order fulfillment or claims settlement, there is often a need for someone to be responsible for the entire process to ensure customer satisfaction. The focus is on achieving end-to-end results, understanding key performance indicators, and managing bottlenecks. (18m29s)
• In larger systems, a coexistence of both event-driven and orchestration approaches is often seen, with the ideal approach depending on how the system is laid out (19m41s).
• Event-driven architecture is preferred when the component that triggers an action is not responsible for completing it, such as in the case of opening a bank account where multiple services need to be triggered (19m50s).
• Orchestration is more suitable when one component is responsible for the entire process, such as the core account opening process, while events can be used for secondary tasks that are not critical to the main process (20m22s).
• The decision between event-driven and orchestration approaches depends on the business perspective and where the responsibility lies, with a focus on the end-to-end business process (21m24s).
• A common problem in many systems is the lack of control and visibility into the end-to-end business process, often due to uncontrolled integration between systems, making it difficult to understand and adjust the process (21m48s).

The Importance of End-to-End Process Visibility

• This lack of control and visibility can hinder the ability to drive customer experience, add new services, or infuse AI, and is a bigger problem than the choice between event-driven and orchestration approaches (22m5s).

Fragility of Software Systems and its Consequences

• The fragility of software systems can also have significant consequences, such as the example of a bug in a security vendor's system causing outages in airlines, including Delta, which took days to recover (22m41s).
• Sam Newman's example of account opening in his microservices book is mentioned as a relevant example of the tradeoff between event-driven and orchestration approaches (20m10s).
• The public often doesn't understand the fragility of software systems, and upper management may not realize how fragile these systems are, which can lead to finger-pointing and a lack of understanding between IT and business sides (22m54s).

Bridging the Gap Between Business and IT

• To address this issue, it's essential to establish a connection between business and IT, enabling the business side to understand the end-to-end process and the IT side to implement it (23m31s).
• Business processes and process orchestration play a crucial role in linking business and IT, allowing for the implementation of end-to-end business processes and business capabilities (23m27s).
• An architect acts as an interpreter between the business and technical worlds, understanding both business and technology, and can facilitate communication between the two (24m13s).

The Role of Architects in Connecting Business and Technology

• There are different levels of architects, including solution architects, enterprise architects, and business architects, each with varying responsibilities and requirements (24m30s).
• Architects need to possess a range of skills, including the ability to communicate with both technical and business stakeholders, and maintain their technical intuition despite potential loss of technical expertise due to generalization (25m0s).
• Regularly engaging with code and technical aspects is essential for architects to stay up-to-date and maintain their technical intuition (25m29s).

Organizational Changes and the Need for Holistic Process Optimization

• Organizations are undergoing significant changes due to the impact of AI and other technological advancements, leading to pressure and struggles in adapting to these changes (25m51s).
• Many organizations attempt to optimize processes locally, but this can lead to siloed teams and projects that may not make sense on a larger scale, highlighting the need for a more holistic approach to process optimization (26m10s).
• Many companies miss the bigger picture by focusing on local optimizations that do not necessarily contribute to the overall end-to-end process, and may even cement bad behaviors or patterns, making it harder to change the process in the long run (26m45s).
• Local optimizations can be a huge problem, and there is a need to elevate the discussion to understand the value stream and business process, with process orchestration being a key element in making this happen (27m32s).

The Problem of Local Optimizations and the Value of Process Orchestration

• Process orchestration can look at the process, make connections to executable software, and orchestrate different tasks together, which is powerful on an enterprise level (27m40s).
• This concept is related to the economic phenomenon of Pareto optimization, which argues that local optimizations are not enough to achieve incremental optimization, and the whole system needs to be reoptimized (28m10s).
• Process orchestration can be a way to force this issue to the front and solve the problem to some extent, and it's a concept that can be understood by CIOs, CEOs, and COOs because it's related to business processes and capabilities (28m38s).

Connecting Technology and Business Goals

• The microservices community often discusses technical aspects without considering the business process and capabilities, making it difficult to communicate with business stakeholders (29m1s).
• Business stakeholders need to be on board to make transformation happen, and technology people need to understand the business reasons behind their work (29m12s).
• Many technology people think only in terms of technology and don't understand the business reasons for doing something, which is a problem because it's the business that's paying and driving the goals (29m31s).
• It's essential for technology people to understand the business goals and work in service of those goals, rather than just focusing on technology for its own sake (29m47s).

Personal Transformation and Understanding Business Value

• Personal transformation and journey are necessary for technology people to understand the business perspective and communicate effectively with business stakeholders (29m53s).
• While technology can be exciting and fun, it's essential to communicate its value in a way that resonates with business stakeholders, rather than just focusing on technical details (30m17s).
• Understanding business value is crucial, and it's essential to learn how to explain it effectively, as it's often an indirect story about how technology enables certain capabilities in software applications, which can increase customer experience (30m31s).
• Many people go through a typical journey of understanding the importance of business value, and it's essential to keep enthusiastic young people who love technology, as they are needed, and some may stay that way for 40 years (30m58s).

Different Career Paths in Technology

• Not everyone has to become an architect or a super developer; it's great to have different paths, such as being a developer or taking on an architect role, which can provide a broader view and bigger decision-making opportunities (31m12s).
• Running a company requires finding the right co-founder, and it's essential to explain to technology people why they can't fall in love with a particular piece of technology and must do things the old-fashioned way for the business's sake (31m36s).

Choosing the Right Technology for the Business

• There's a blog post called "Choose Boring Technology" that emphasizes the importance of choosing the right technology for the business, even if it's not the most exciting one (32m0s).

Training Developers in Event-Driven, Choreography, and Orchestration

• Training developers to understand when to use events, choreograph, and orchestrate in distributed systems and asynchronous systems requires a nuanced approach, including making them aware of certain concepts and having them try things out through examples and reviews (32m15s).
• For smaller problems, such as mastering asynchronous communication and consistency problems, training can be done through examples and reviews, while bigger, more architectural questions require experienced people on board (32m42s).
• Gradual exposure to increasingly complex situations is the way people learn, and it's essential to have experienced people, such as architects, to coach and guide new developers (33m27s).

The Importance of Visual Communication and Tools

• Visual communication and visual tools are essential, and it's crucial to appreciate their value, as people often don't understand their importance (33m59s).
• Process orchestration visuals are very important, especially with long-running processes that are hard to fully comprehend, and can be effectively expressed with visuals like a flow of activities with different paths and events. (34m13s)
• Despite the importance of visuals, some people are against them, but even those who dislike visuals often use a whiteboard to draw and explain processes, indicating that visuals are necessary for understanding complex processes. (34m13s)
• The human mind has a limit to the amount of complexity it can conceive of, and processes are often non-linear with feedback, making diagrams a suitable tool for representation. (35m7s)
• Different levels of abstraction, such as subprocesses, can be used to handle complex processes, and diagrams can be one tool in the architect's tool belt. (35m22s)

The Joys and Challenges of Being an Architect

• The favorite part of being an architect is seeing the big picture, understanding how different pieces are connected, and how they relate to business strategy and technical implementation. (35m52s)
• The least favorite part of being an architect is often being an outsider, not being part of the team, and having to communicate complex ideas to others. (36m17s)
• Architecture involves creativity in finding elegant solutions to complex problems and communicating them in a way that is easy to understand. (36m51s)
• The spiritual or emotional element of being an architect comes from seeing solutions work, people understanding them, and knowing that the work is making a positive impact, even if it's just incrementally. (37m17s)
• What turns people off about architecture or being an architect is the perception that architects are naysayers or governance entities, which is not true for good architects. (37m37s)

Favorite Technologies and Architectural Preferences

• Favorite technologies include VPMN, Camunda, and Java, with a particular appreciation for type languages, dependency management, and the open-source ecosystem surrounding Java (37m57s).
• In terms of architecture, the preference is for breaking down complex problems into manageable pieces and solving them, with a dislike for lengthy and unfruitful discussions (38m28s).
• The profession of solution engineer, which involves creating proof of concepts and demonstrating their feasibility, is considered a favorite role (39m10s).
• As for the future, there is a possibility of transitioning to a more mentoring-focused role, rather than actively working as an architect, but this is not currently the case (40m11s).

Future Aspirations and Desired Feedback

• When a project is completed, the desired feedback from clients or the team is an understanding of the value created and the story behind the project's connection to business value (40m50s).