Architecture Does Not Emerge - A Conversation with Tracy Bannon

Introduction to Event-Driven Architecture and RavenDB

• Building an event-driven architecture requires a database that won't be a bottleneck, and Raven DB is a suitable option with features like compiled indexes, distributed transactions, and real-time data support (0s).
• Tracy Ben is a senior principal at Merer, a passionate software architect, and change agent who also hosts the Real Technologist podcast (31s).
• Tracy Ben did not have formal training as a software architect, as the term did not exist when she started her career, but she grew into the role with a big-picture perspective and later obtained SEI certifications (50s).

Tracy Ben's Background and Perspective on Software Architecture

• Tracy Ben believes that software architects are "woefully uneducated" as a group, which is why it's essential to have resources like podcasts to educate and reach out to the next generation (2m6s).
• As Tracy Ben became educated as an architect, she found that certain resources were useful, but they often didn't address recurring problems or only addressed them from a theoretical point of view (2m27s).
• One of the issues that still exists today is the lack of repeatability, with different teams solving problems in different ways unnecessarily, which is why Tracy Ben is a strong advocate for design patterns and practices (3m31s).
• Tracy Ben believes that the software industry has been doing a bad job of writing code for 50-60 years, and as architects, they have been trying to elevate the quality of code (3m15s).
• Tracy Ben is a fan of design patterns and practices, which she believes provide a logical way to think about architecture and help stop talking about it in vague terms (3m55s).
• Architecture is a combination of art and science, with a certain amount of repeatability and a flare of artistic creation, and involves dealing with software quality attributes, tactics, and trade-offs (4m14s).
• Design patterns are domain-specific, enterprise application, integration, continuous delivery, and integration patterns, which can be overwhelming, and it's essential to understand how to pick the right tool for the right job (4m50s).
• There are two extremes: being ignorant of the tools or knowing only one tool, and it's essential to understand the basics, such as software quality attributes, tactics, and trade-offs, to make informed decisions (5m22s).
• Core concepts, such as understanding what a software quality attribute is, what tactics are, and what trade-offs are, are essential for architects, and it's crucial to start with the basics to ensure everyone is on the same page (5m58s).
• Many developers dive right into coding without considering the bigger picture, and it's essential to take a step back and understand the problem before starting to code (6m18s).
• Not everyone is meant to be an architect, and it's okay if some people don't naturally understand the broader concepts, but mentoring and guidance can help them grow into the role (6m52s).
• Being an architect is not necessarily the next step in a developer's career progression, and architects and developers are two different roles with different responsibilities and requirements (7m15s).
• There is no direct progression from being an entry-level developer to becoming an architect, and it's essential to recognize that different roles, such as tech lead, lead engineer, and solution architect, require different skills and expertise (8m2s).

The Importance of Design Patterns and Practices

• Many people mistakenly believe that architecture emerges without design, but in reality, design patterns are essential in software development (8m4s).
• Test-driven development (TDD) is often misinterpreted as not requiring design, but in reality, designing a test requires having a class in mind, which implies prior design (8m51s).
• TDD is not about avoiding design, but rather about having a good understanding of the requirements and using tests to reflect the ultimate design (9m48s).
• The best tests are those that properly reflect the requirements, but finding out the requirements can be challenging (10m21s).

Challenges in Requirements Gathering and Communication

• One of the difficulties in software development is getting clients to clearly communicate their needs and wants (11m9s).
• There is no one-size-fits-all recipe for developing software, and attempts to create such recipes, like requirements design languages, have been unsuccessful (11m34s).
• The book "Exploring Requirements" by Weinberg and Gal highlights the importance of understanding requirements before design (11m12s).
• The story of the "guaranteed cockroach killer" illustrates the limitations of trying to create simple solutions to complex problems, and the importance of understanding the underlying requirements (11m58s).
• The problem with requirements is getting the right information to begin with, and once that is achieved, one can proceed with Test-Driven Development (TDD) or other approaches, but it requires a different and difficult skill set (12m25s).

The Role of Different Skill Sets in Software Development

• It is not feasible for a single delivery team to handle everything from vision to fielded operations, as different skill sets are required for different tasks, such as human factors engineering, facilitation, and coding (12m47s).
• There is room for people with various capabilities in software engineering and architecture, making it a wonderful field with many opportunities (13m28s).
• Facilitating conversations to draw out requirements is crucial, and it is essential to have people who can listen and not jump ahead to design (13m46s).
• Understanding the business or mission context is vital in requirements analysis, and it is necessary to start with the highest-level requirements and imperatives (14m23s).
• Hierarchical thinking is still essential in software development, and it is necessary to have a big vision and understand the business or mission needs (14m48s).
• Somebody ultimately has to pay for the development, and there needs to be a vision of what the final product should be, meeting the business or mission needs (15m8s).
• A skill set is required where somebody understands both the business language and technology, as people often think that something easy to express in English is easy to program, and vice versa (15m28s).
• Architects should be able to understand both business and technology to facilitate conversations, and this is one of the key things to look for in an architect (16m9s).
• Being an architect involves parachuting into new domain spaces and finding ways to understand the requirements and develop a solution (16m25s).
• The beauty of architecture lies in collaboration and achieving cognitive diversity at the table to make things happen, which involves having a smart person who understands the domain, a deep technologist, and others working together to guide the process forward (16m30s).

Effective Requirements Gathering Techniques

• When gathering requirements, it's essential to ask open-minded yet restricted types of questions to avoid leading people in a direction they may not want to go, which requires a specific skill that many people lack (17m6s).
• Persona-based design and Journey Maps are valuable concepts in finding requirements, as they help understand how people live their lives and apply a human lens to the design process (17m42s).
• All technologists, including engineers and developers, need to have at least some basics in design thinking and be exposed to it, even if they're not good at it or don't want to do it, as it's a part of the design process (18m18s).
• Thinking out of the box is crucial in design, as illustrated by the example of Scientific American solving their sorting problem in the 1960s by using separate post office boxes for each request instead of developing a software program (18m45s).
• Scalability is an essential consideration in design, and it's not always possible to know what the scalability requirements will be, as demonstrated by the "Wall Street Journal effect," where a small business can suddenly face a massive surge in demand after being featured in a prominent publication (19m46s).

Scalability and Forecasting in Design Decisions

• The "Wall Street Journal effect" can be seen in real-life examples, such as a small bakery that struggles to scale after being mentioned in a prominent publication or a company like Tesla cancelling an order from a small bakery due to the bakery's inability to meet the sudden surge in demand (19m54s).
• A small business received a large order for 3,000 personal pies, but they couldn't scale to meet the demand, and instead, they were touched by the idea that a small business was having a problem and turned down orders (20m24s).
• When making design choices, it's essential to think about the possibility of forecasting and the potential consequences of decisions, including the cost of changing them in the future (21m3s).
• Business forecasts are crucial in anticipating the needs of the business and understanding the trade-offs involved in making design decisions (21m37s).
• Implicit assumptions should be made explicit to avoid criticism and ensure that all stakeholders understand the reasoning behind design choices (21m53s).

Big Upfront Design vs. Emergent Design

• Big upfront design is often criticized, but it's an exercise in thinking about possibilities and making informed decisions, rather than just experimenting without a plan (22m17s).
• The concept of emergent design is often misused, as most people are not trying to discover something new, but rather operationalize existing ideas, which requires making decisions upfront (22m45s).
• Emergent architecture is suitable for experimentation and discovering new possibilities, but when operationalizing, decisions need to be made upfront to ensure scalability and efficiency (23m0s).
• Architecture involves making decisions that are too expensive to change later, and these decisions should be made upfront, rather than relying on emergent design (23m32s).

Decision Autonomy and Team Empowerment

• Decision autonomy is essential in software development, and teams should be able to make decisions without aggression or resistance from others (24m6s).
• When discussing autonomy, it's often compared to variable scope in programming, where the scope of autonomy can vary and not every team has complete autonomy, with limitations and responsibilities that need to be defined (24m30s).
• A "grownup discussion" is necessary to determine the scope of autonomy for each delivery team, including their rights, responsibilities, and limitations, which is often missing in the conversation (25m6s).
• The idea that agility provides complete autonomy is a misconception, and architects, especially Enterprise architects, may resist discussions about autonomy due to this false sense (25m34s).
• In a software factory, two teams working in close proximity chose different technologies for the front end, React and Vue, without justifying their decision, highlighting the need for a more thoughtful approach to technology selection (25m46s).

Documenting Architectural Decisions

• Writing down decisions, such as architectural decision records, is essential for transparency and understanding the context behind a decision, even if it seems insignificant (27m1s).
• Decisions are often made outside of a team's sphere of influence, and documenting these decisions can help clarify the reasoning behind them (27m19s).
• A crucial aspect missing from many people's understanding is the consideration of risk and probability when making decisions, which involves dealing with the unknown and trying to minimize the risk of failure as progress is made (27m41s).

Risk Mitigation and Trade-off Analysis

• According to Barry Boehm, software development can be viewed as trying to minimize the risk of failure, and upfront decisions should focus on the things with the highest probability of success (28m1s).
• Architecture can be described as risk mitigation, involving trade-offs and balancing different factors such as security and performance to achieve the desired outcome (28m18s).
• A developer's role is not solely focused on writing code, but also involves understanding the trade-offs and risks involved in building software within an organization (29m12s).
• To help the next generation of developers and architects understand the importance of trade-off analysis, it is essential to mentor and guide them in understanding the risks and complexities involved in software development (29m10s).
• The decision to use multiple technologies or UI designs can increase the risk of finding suitable personnel to work on the project, as it limits the pool of available candidates (30m35s).
• Architects have a responsibility to mentor and bring people along with them, helping them to understand the complexities and risks involved in software development (31m25s).

The Role of Architects in Agile Environments

• The Agile movement and the idea of emergent design have led to a perception that architects are not needed, but this is not the case, and architects play a crucial role in mitigating risks and making informed design decisions (31m53s).
• The QCon London 2024 conference will feature talks on emerging trends and best practices in software development, including AI, software architectures, and modern software security (29m41s).
• For a decade or more, there was a trend of hiring only software engineers, with some people believing that engineers are more needed than architects, which can be debated as there is a difference between the two roles (32m25s).
• Architects and developers have different skill sets, and one is not a career succession path of the other, nor is one the boss of the other (32m46s).
• As a mentor and facilitator, an architect provides pragmatic guidance but does not write code daily with the team, although they should still be able to code (32m59s).
• It is essential for architects to be able to communicate effectively and bring their expertise to the table to be trusted, as described by Gregor Hohpe's analogy of an elevator architect who can go from the boiler room to the boardroom (33m54s).
• Architects should be able to speak multiple languages, not just UML, Visio, or PowerPoint, and be aware of emerging technologies that can take intermediate language diagrams and turn them into code (34m16s).

Communication and Collaboration in Architecture

• A lack of taxonomy in an organization can be frustrating, as it leads to disagreements on the meaning of words, nuances, and muscle memory issues (35m11s).
• There is a divide in the understanding of DevOps and DevSecOps, with some people believing it only includes the automated pipeline from code commit to production, while others think it encompasses more aspects, including vision and design (35m35s).
• DevOps can be seen as a set of principles under a bigger umbrella, and elevating the conversation around it can help clarify its meaning (35m52s).
• The need to consider the broader level of digital engineering and understand the flow of value is emphasized, highlighting the importance of mapping out how processes actually work to identify waste and find solutions (35m56s).
• Architecting a system requires considering how to partition it in a way that allows for incremental improvements, making it deliverable and not a "big ball of mud" (36m38s).

Decoupling and Loose Coupling in System Design

• The concept of Dead Ops as a system of principles that need to be translated into a particular context is discussed, highlighting the importance of understanding the specific situation and goals (37m9s).
• The idea of decoupling or loosely coupling things is emphasized as a key factor in the longevity of systems, allowing for easier maintenance and updates (37m56s).
• The difference between loose and essential coupling is mentioned, with essential coupling being necessary for an application to function, but inessential coupling being something to be avoided (38m38s).
• Containers are not seen as the answer to everything, and there are times when coupling is essential, such as in certain business systems or embedded software (39m6s).
• The importance of considering trade-offs and making decisions based on the specific context and goals of a system is highlighted, with different systems requiring different approaches (39m27s).

Different Perspectives in Software Development

• The concept of archetypes is mentioned as a potential topic for further discussion, with different systems having different characteristics and requirements (39m41s).
• The world can look different from the perspective of a human factors engineer and a backend developer, with the latter often viewing humans as obstacles, while the former sees the disconnect between the API and how people work (40m1s).
• Frontend developers, backend developers, and full-stack developers have different skill sets and perspectives, with full-stack developers being seen as valuable in the past for understanding technology within their ecosystem (40m39s).
• The architect's questionnaire is a set of questions asked to architects, including their favorite part of being an architect, least favorite part, and anything creative, spiritual, or emotional about architecture (41m24s).

The Architect's Questionnaire and Personal Reflections

• The favorite part of being an architect is often the initial situational awareness and problem-solving, while the least favorite part is being called into a "dumpster fire" with aggressive people (41m30s).
• Architecture can be a creative and fulfilling field, combining form and function, and allowing architects to be both technical and artistic (42m27s).
• Having a background in building architecture and art can influence one's perspective on software architecture, with a mix of left-brain and right-brain thinking (42m36s).
• What turns some people off about architecture is the "Ivory Tower" effect, where Enterprise architects focus on reviewing others' work rather than getting involved upfront to help (43m9s).
• Effective Enterprise architects should be like urban planners, understanding the bigger picture and collaborating with various stakeholders to achieve a common goal (43m33s).
• A particular genre of architecture can be dogmatic, Machiavellian, and Draconian, which can be off-putting (43m59s).

Security in Software Development

• A different perspective on a city can be gained by looking at it from the point of view of a thief, who searches for break-in points and weak entry points (44m17s).
• To improve secure software development, technologists and developers should be taught how to think like hackers, and security professionals should understand the mindset of builders (44m39s).
• There is a need for crossover between security and development teams to create more secure software (44m50s).

Emerging Technologies and AI in Software Development

• The ChatGPT app has a feature that allows for speech-only conversations, and it includes filler words like "um" and "ah" in its responses (45m18s).
• CodeQL is a tool that can be used for more complex code reviews, making it a useful technology for developers (45m52s).
• AI is a technology that is being explored, but it's essential to avoid anthropomorphizing it (46m2s).
• The puzzle of architecture is enjoyable, as it requires looking at things from different perspectives and framing them in new ways (46m29s).

The Enjoyment and Challenges of Software Architecture

• Architecture stretches one's understanding and allows for constant learning from different folks and perspectives (46m47s).
• The tedious tasks in architecture, such as noting down details for models, can be frustrating but are necessary (47m11s).
• The process of architecture can be compared to being an athlete, where discipline and hard work are required during the week, but relaxation is allowed on weekends (47m30s).
• If given the opportunity to attempt a different profession, the alternative career path of choice would be teaching, specifically through storytelling and sharing knowledge, which is currently being done through podcasting and conversations, as it is enjoyable and helps others gain knowledge (47m47s).
• Being an architect is a significant part of one's identity, and even if not actively building software, the skills and mindset will always be present, making it unlikely to completely stop being an architect (48m36s).
• When a project is completed, the most desired feedback from clients or teams is a sincere thank you, indicating that value was brought to the project, and that the work was appreciated beyond just its monetary or throughput value (49m13s).

Conclusion and Final Thoughts

• The importance of cognitive diversity should not be overlooked, as it allows for meaningful conversations and learning opportunities, and it is essential to be aware of and appreciate different perspectives (49m55s).