Trends in Engineering Leadership: Observability, Agile Backlash, and Building Autonomous Teams

Introduction and Coric

• The upcoming Cuong conference in San Francisco will feature tracks on architectures, engineering productivity, and generative AI in production, where senior practicing software developers will share their experiences in adopting emerging trends (40s).
• Chris Cooney is the head of developer relations for Coric, a full-stack observability platform, and has spent 11 years as a software engineer, starting as a Java engineer and moving into front-end engineering, Sr, and DevOps before transitioning into engineering leadership (1m27s).
• Chris has moved into engineering leadership and has seen organizations grow and change, and now, as the head of developer relations for Coric, he gets to research and understand the trends in the market, including the adoption of observability (2m7s).
• Chris meets hundreds of people every month and collects their views and insights, which makes him excited to discuss various topics in the industry, including trends in engineering leadership (2m34s).

Agile Pushback and Observability

• One trend Chris has observed is the emotional pushback against Agile, not necessarily its core tenets, but the word itself, which has led to a certain amount of fatigue around it (3m38s).
• Another trend Chris is seeing is the move towards observability, which is a technical trend that he is closely involved with due to his work with Coric (3m40s).
• Observability in organizations is shifting from focusing on what's going wrong to broader questions about the company's performance, such as Dev hours, mean time to recovery, and reduction, blurring the lines between business, technical, and people measures (3m54s).
• Measurement in the industry is becoming more abstract, with examples including measuring rage clicks and the emotionality of user interactions, providing a microcosm of the changes in the industry (4m17s).
• Technical trends like AI and ML are also impacting people and engineering teams, creating uncertainty and excitement (4m36s).
• Building observability into software allows for the measurement of people's experiences, including user behavior and emotions (4m48s).
• The evolution of observability has moved from basic metrics like CPU and memory to more complex measures like latency, response size, and marketing metrics, and is now focusing on higher-level abstractions like user experience (5m3s).
• Web vitals are being used to measure user experience, including metrics like when a user sees the content they want, not just when the page loads (5m37s).
• The distinction between signals and insights is important, with signals being specific technical measurements and insights being amalgamations of signals that provide useful information (6m4s).
• Insights should be understandable by non-technical stakeholders, such as the example of a user having a bad experience loading a product page (6m22s).
• People experience metrics are directly answering business questions, providing a new level of abstraction and understanding of user behavior (6m55s).
• Organizations are embracing observability, which involves making broad and high-level measurements to inform decision-making across the entire organization, not just technical teams, and this trend is expected to continue (7m18s).

Challenges and Benefits of Observability

• To achieve this, organizations must overcome two main challenges: a technical barrier to making information available and a barrier to making that information accessible to users without specialist knowledge (7m46s).
• The technical barrier involves making high-performance queries available to potentially thousands of users concurrently, which is a difficult task that requires a certain level of scale (8m12s).
• Assuming the technical barrier is overcome, the next challenge is to make it easy for users to get the answers they need quickly, which may involve using AI-powered natural language query tools (8m33s).
• Once these barriers are overcome, an organization can achieve a universal language around cross-cutting insights, which can shine a spotlight on areas that may have previously been hidden (9m56s).
• This universal language can be painful for some individuals, particularly those who have been "flying under the radar" in large organizations, and can make them feel vulnerable as their performance is measured against key performance indicators (KPIs) (10m12s).
• The introduction of Service Level Objectives (SLOs) and Service Level Indicators (SLIs) can help create a universal language around operational performance, as seen in the example of a principal engineer at Sabur (9m5s).
• Effective access to and ownership of data is crucial, and this requires a cultural shift and a conversation that may be initially painful but is necessary for success (10m6s).

Impact of Metrics on Human Behavior

• Implementing metrics to measure performance can significantly change human behavior, as people tend to focus on the metrics being measured, and this can lead to unintended consequences such as competitiveness between teams and artificially lowering deployment frequency to improve metrics (10m58s).
• The introduction of metrics can make some people feel nervous and uncomfortable, while others thrive and enjoy the challenge of beating their budgets, leading to a divide in behavior and reactions (10m44s).
• Great leaders see higher visibility as an opportunity to help and get involved in conversations, while less effective leaders may use metrics as a way to force different behaviors and beat engineers, highlighting the importance of good faith and trust in leadership (11m55s).
• Surfacing metrics and being forthright about the truth can be beneficial, but it's crucial to consider the potential consequences and ensure that everyone is treated well, especially if there are leaders who may misuse the measurements (12m10s).
• The success of implementing metrics depends on the premise that everyone is doing their best, and if this premise is not established, it can lead to trouble regardless of the quality of the measurements (12m31s).

Psychological Safety and Blameless Postmortems

• The experience of rolling out metrics across an organization was largely positive, but it took time to address the growing pains and challenges that arose (12m42s).
• Creating a culture of psychological safety is essential, and this can be achieved by adopting blameless postmortem methods and moving away from lengthy and detailed incident management reviews that often lead to bickering and finger-pointing (13m39s).
• Google's Project Aristotle is an example of research on psychological safety, and educating an organization on this topic can help create a more positive and supportive work environment (13m2s).
• A successful blameless postmortem meeting was conducted in 45 minutes, compared to a previous 6-hour meeting, by starting with a 5-minute briefing on the importance of blamelessness and creating an environment where mistakes can be surfaced without fear of retribution (13m53s).
• The goal of blameless postmortems is not to avoid acknowledging mistakes, but to create an environment where mistakes can be openly discussed and learned from, and to attribute mistakes to individuals or teams without malice (14m42s).
• The language used in some postmortem documents can be too indirect, avoiding blame by not naming individuals or teams, which can make it difficult to understand the cause and effect of the mistake (14m57s).
• It's crucial to maintain the relationship between cause and effect when teaching about blameless postmortems and psychological safety, and to show a clear link between actions and their consequences (15m32s).
• Organizations often get tripped up by prioritizing psychological safety over understanding the cause and effect of mistakes, which can lead to missing the point of postmortems and not learning from mistakes (15m56s).

Agile Backlash and Misunderstandings

• The Agile backlash refers to a trend where organizations are moving away from Agile methodologies, and this is partly due to a misunderstanding of the principles of Agile and a focus on frameworks and tools rather than the underlying methodology (16m20s).
• A good understanding of Agile principles can be achieved by studying the Agile Manifesto and working with experienced coaches who understand the methodology, rather than just following frameworks and tools (16m50s).
• A fundamental misunderstanding about Agile is that it's supposed to speed up the development process, when in fact it's about incrementally delivering working software to facilitate a feedback loop and continuous improvement (17m29s).
• Agile is not about getting things done faster, but rather about delivering a working product early on, giving organizations the opportunity to go live with it if they can productionize it creatively (17m44s).
• The Agile industrial complex, which involves organizations selling Scrum certifications and other Agile-related services, has contributed to the backlash against Agile, with some certifications holding little value (18m22s).
• Not all Scrum certifications are useless, and some can provide valuable insights into Agile principles and practices, such as the idea that the Scrum guide is a guide, not a holy text (18m59s).
• There is a backlash against Agile certifications due to the varying quality of services, with some being good and others being bad (19m21s).
• Agile coaches were given a significant amount of power in organizations, and while some were great, others provided bad advice that had a significant impact due to their influence on powerful people (19m32s).
• The impact of bad advice from Agile coaches can be enormous, and it's a problem that the software industry is still trying to figure out how to address (19m59s).
• The backlash against Agile is driven by three factors: the King's taking experts at their word without questioning, the lack of understanding of software delivery, and the focus on interpersonal aspects rather than delivery. (20m24s)
• The principles of Agile are still valuable and ingrained in software engineering, but the term "Agile" has become less prevalent due to fatigue, with people instead talking about continuous delivery. (20m52s)
• The core principles of Agile are part of software engineering now and are here to stay, despite the backlash against the term. (21m18s)

Empowering Teams and the "Golden Path"

• To get teams aligned around a common goal and give them autonomy, it's essential to provide a "golden path" that allows teams to make effective decisions while moving in the same direction. (21m45s)
• A previous approach to consolidate tools and reduce costs by imposing a single tool on teams was unsuccessful, as teams didn't understand or adopt the tool. (22m27s)
• A more effective approach is to provide teams with the freedom to make meaningful decisions about their software while ensuring they're moving in the same direction, as seen in the successful migration of hundreds of developers onto a new platform. (22m51s)
• The Manchester P platform as a service, later renamed, was a successful project that allowed teams to make effective decisions while moving in the same direction, with a "golden path" in place. (23m12s)
• A platform was built using Kubernetes and Jenkins for CI/CD, with the requirement that every team in Manchester tag their resources and provide deployment information, including what was deployed and what went into it (23m27s).
• The goal was to automate the process of filling out change request forms, especially during busy periods, and make the platform easy to use and almost invisible, incentivizing teams to use it as intended (24m13s).
• The platform was designed to make the "horrible stuff" easy, including dashboard generation, alert generation, and metric generation, and was onboarded by all teams in a couple of weeks with no pushback (24m31s).
• One team was able to produce a working API and UI in their first sprint, with scalable infrastructure, using the platform's features such as HTTP service metrics and tracing (24m40s).
• The team was able to go live in weeks instead of months, and the project changed the conversation and led to a new project across the entire organization (25m24s).
• The story was shared to illustrate the importance of giving engineers autonomy and ownership of their work, and that a top-down edict approach can lead to the best people leaving and the worst people trying to work through it (25m34s).
• To give engineers autonomy, a platform should be built that is highly configurable, self-service, and automates painful tasks such as compliance and change request notifications (25m54s).
• The goal is to make the "golden path" the easy path, so that engineers can focus on their work without being bogged down by bureaucracy (26m14s).
• To encourage the right behavior, it's essential to make the easy decision the right one, which requires hard work to go against, and incentivize the right behavior by making painful parts of life easy, using both the "carrot" and the "stick" approach (26m30s).
• This approach involves incentivizing the behavior and letting individuals choose, which is a preferred method of leadership (26m47s).

Connecting with Chris Cooney

• For those who want to continue the conversation, Chris Cooney can be found on LinkedIn by searching for his name, and he is usually active on the platform for work-related topics (26m59s).
• Chris Cooney can also be found on LinkedIn by searching for variations of his name, such as "Chris Cooney Logics" or "Chris Cooney observability" (27m10s).