Mastering Observability: Unlocking Customer Insights with Gojko Adzic

CuCon San Francisco and Engineering Culture Podcast

• The upcoming CuCon San Francisco conference will take place from November 18 to 22, where engineers will share how they implement innovations in real-world scenarios, exploring tracks like architectures, engineering productivity, and generative AI in production (10s).
• Shane Hasty is hosting the InfoQ Engineering Culture podcast, where he is joined by Gojko Adzic, a developer with over 12 years of experience in building software for himself and operating his own products (45s).
• Gojko Adzic started developing software at a young age, copying and pasting code from German magazines on a Commodore 64, and has since been building software for himself, including doing pre-sales, product management, support, development, and even making coffee in the office (1m13s).
• Adzic has written nine books, mostly as a way to free up his memory, with his past life involving contract development and helping others improve their process (2m1s).

Gojko Adzic's Background and "Specification by Example"

• One of Adzic's notable books is "Specification by Example," which was driven by his anger towards a conference speaker who claimed that nobody actually uses Test-Driven Development (TDD) and Behavior-Driven Development (BDD) in the real world (2m50s).
• Adzic interviewed people with good processes and documented their names, companies, and experiences, resulting in a book that showed that many people and companies, including investment banks and startups, successfully use TDD and BDD in their projects (3m41s).

Reasons for Not Adopting TDD/BDD and Measuring Value

• Organizations may not adopt Test-Driven Development (TDD) and Behavior-Driven Development (BDD) because they have too much money and can afford to repeat the same processes over and over, with incentives focused on effort rather than actual value delivered to the market (4m25s).
• The most common metrics used to measure progress are related to effort, effectively measuring how much money is being spent rather than the value being delivered (5m13s).
• The product management community has been working on figuring out how to measure value and deliver valuable products, and prove that value is being delivered to the market (5m41s).
• Organizations may not prioritize high-quality software delivery because software is not a bottleneck for them, and they may pay lip service to the idea but not actually implement it (5m57s).
• Personal incentives, politics, and discomfort with communication can also prevent organizations from adopting TDD and BDD, as many people in the industry prefer working with computers to working with people (6m10s).
• The focus on effort can lead to people keeping themselves busy or being kept busy by organizations, making it difficult to pause and coordinate as a team to figure out what to do (6m45s).
• Investing more effort in things that should not be done in the first place is a form of waste, as noted by W. Edwards Deming (7m15s).

Measuring Value Delivery and User Behavior

• To ensure that their work is adding the most value, technologists can start looking at outcomes rather than effort, and consider models such as the value exchange system or value exchange loop presented by Melissa Perrey in her book "Escaping the Build Trap" (7m37s).
• Many organizations struggle to track the value delivered by their products, and users often get value from products in ways that are difficult to measure (8m15s).
• Ron Kohavi published a book called "Trustworthy Online Controlled Experiments," which shows that only about 30% of experiments at companies like Microsoft and Slack have positive results (8m31s).
• This means that around 70% of the work done by these companies may be wasted, and data from companies like Netflix suggests that up to 90% of their early efforts were unsuccessful (9m7s).

Improving Software by Eliminating Wasted Work

• This low success rate presents a massive opportunity for improvement, particularly in areas like integration testing and maintenance (10m3s).
• By identifying and eliminating work that does not deliver value, organizations can reduce rework, improve efficiency, and develop better products (11m10s).

Tracking User Behavior Changes and Value Delivery

• One way to measure value delivery is to track changes in user behavior, as this can indicate whether a solution is solving a problem or not (11m54s).
• If user behavior does not change or changes in a negative way, it may be a sign that the solution is not delivering value (11m31s).
• By focusing on behavior change, organizations can develop more effective solutions and improve their competitiveness (11m14s).
• Behavior changes are a model for value delivered to the market, allowing for quick observation and measurement after something is delivered, unlike metrics like profit, market share, and revenue, which have a long delayed cycle (12m47s).
• Behavior changes are leading indicators that can be measured quickly, providing insight into whether a product is likely to win in the market, rather than just indicating whether it is winning (13m25s).
• Technical people can measure behavior changes using techniques similar to those used to measure system behavior, such as observability, tracing, distributed measurements, and combining logs and error measurements (13m37s).
• These techniques can be extended to measure user behavior, helping to determine if a product is actually helping users achieve their goals faster, easier, and sooner (14m3s).

Example: Simplifying a Payment Form

• A trivial example of this is a payment form that required users to enter a tax ID, but the form's validation caused confusion and obstacles for users, particularly those from the European Union (14m21s).
• The issue was resolved by streamlining the form and removing the tax ID field, making it easier for users to complete the payment process (15m57s).
• European Union countries have different tax ID formats, with some having a prefix, and users are not expected to know these details, making it important to simplify the payment process (14m58s).
• The payment provider's validation of the tax ID caused issues for users who did not enter the prefix, leading to errors and frustration (15m23s).
• Some users even selected a different country to avoid entering their tax ID, highlighting the need for a simpler payment process (15m40s).
• A website was modified to ask users for a tax ID, which was thought to be a good idea, but it caused an even bigger problem as people were unable to enter the tax ID and were scared that their payment would not be processed (16m5s).
• By measuring behavior changes, it was discovered that the modification was causing more problems, and the idea was proven to be bad, highlighting the importance of measuring behavior changes to inform decision-making (16m33s).

Understanding "Lizard" Behavior and Product Opportunities

• The concept of a "lizard" refers to individuals who do things that no rational human would do, and they cannot be comprehended, but they still do them, as described by Scott Alexander in his blog post about Phantom Lizardman (17m11s).
• Lizards make up a percentage of users who are confused, distracted, or have other issues that affect their behavior, and they will do things that cannot be explained by rational human logic (18m21s).
• An example of a lizard behavior is people paying to convert blank PowerPoint presentations to videos, which makes no sense for a rational human, but it was discovered that they were using the text-to-speech function to extract an audio track (18m50s).
• By understanding the lizard logic, a product opportunity was discovered, as making an audio file is operationally cheaper than making a video, and a new product could be created to cater to this need (19m39s).
• A simple screen was created to allow users to upload Word documents and get an audio track, increasing product usage significantly as it made the process easier for users who were struggling, and this improvement benefited all users (19m49s).

"Lizard Optimization" and User Behavior Tracking

• The idea of "lizard optimization" involves making products better for users who are struggling in some way, which can be achieved by having analytics and observability built in to see what users are doing with the products (20m24s).
• Technical people in the industry have a massive opportunity to extend existing tools and figure out what unexpected things users are doing with their products (20m39s).
• Most analytics used by product people look for expected things like clicks, conversions, and purchases, but devops and observability tools can be used to look for unexpected things like exceptions, mismatches, and deviations (20m59s).
• These tools can be extended for user behavior tracking and to spot weird things that are happening, allowing for the extraction of signal from noise (21m40s).

Logging User Errors and Unexpected Behavior

• A simple thing that can be done is logging whenever a user sees an error screen or does something that causes an unexpected path in the application (21m54s).
• Many applications ignore this level of logging, but it can provide valuable insights into user behavior, such as when a user tries to select a file of an unknown type or a type that the software can't support (22m30s).
• For example, a text-to-speech system logs when users try to convert a JPEG file or an APK file, which can provide insights into unexpected user behavior (22m54s).
• People often try to upload incorrect file types, such as MP3 files, into a text-to-speech system, but occasionally, they attempt to upload files that make sense to support, like subtitle files, which are essentially text files with timestamps (23m26s).
• Allowing the upload of subtitle files led to complaints about the system reading timestamps, which were later removed, and then complaints about the speaking speed, revealing that users actually wanted to create alternate audio tracks for videos (23m51s).
• This feedback loop led to the development of a feature to create synchronized audio tracks when uploading subtitle files, which was already 99% functional, and this feature became highly profitable when a massive enterprise software company started using the system (24m50s).
• This example illustrates the importance of monitoring user errors and using this information as product management signals to make better products and provide more value to users (25m33s).

Five Stages of Growth and Product Evolution

• The five stages of growth, a concept from the book "Lean Analytics" by Alistair Croll and Benjamin Yoskovitz, describe how products evolve and what people should focus on at each stage, including empathy, stickiness, virality, retention, and revenue (26m14s).
• The empathy stage is about determining if a product is solving a good problem and finding a good solution, while the stickiness stage is about proving there is a market for the product (26m43s).
• Understanding these stages can help technologists and operations people provide valuable signals to improve products and make better decisions (25m47s).
• When developing a product, it's essential to determine if there's a market for it by checking if people use it frequently and if the product is sticky, meaning users stay and come back, indicating potential market demand (27m7s).
• In the early stages, iterating around different user groups can help find the right market, as seen in the example where people were initially building videos but later shifted to building more audio files when the product was repositioned (27m30s).
• The product-market fit stage is crucial, where the product is used extensively by the target market, and the next stage is growth, which involves proving that the market can be reached through various means such as advertising or word of mouth (27m58s).
• The growth stage is not just about virality but also about finding the right growth engine to reach the target customers (28m4s).
• After growth, the revenue stage involves building a sustainable business model, which may require optimizing how the product makes money (28m24s).
• The final stage is scale, which involves consolidating business operations, optimizing profit, and capturing more value from the market (28m36s).
• The value exchange loop consists of delivering value to the market and capturing value from the market, with the first two stages focusing on delivering value and the last two stages focusing on capturing value (28m49s).
• As engineers, it's essential to focus on the right stages and not optimize for performance or scalability too early, as this can be a waste of time and resources if the product is not yet profitable or market-ready (29m11s).

Consequences of Not Prioritizing Stickiness

• A past experience as a CTO of a company about 16 years ago highlights the importance of understanding these stages, as the company messed up by not focusing on the sticky part of the product and instead jumped into marketing too early (29m56s).
• The consequences of not optimizing the stickiness of the product can be severe, as seen in the example where only 1 or 2 out of every 1000 people who came to the website came back, indicating a lack of market demand (30m34s).
• A startup invested its entire marketing budget into bringing a million people to its website, but only 2,000 to 3,000 people stayed due to the product not being sticky, resulting in the company not doing well afterwards (30m40s).
• The outcome would have been different if the company had focused on making the product sticky before bringing in a large number of users (31m13s).

Using Growth Stages and Guardrail Metrics

• The five stages of growth are used to determine when to focus on certain aspects and when to stop, with one stage used to optimize and another to create guardrail metrics (31m34s).
• Guardrail metrics are used to ensure that certain aspects, such as operational costs, do not grow out of proportion while focusing on growth (31m53s).
• When scaling, it's essential to avoid damaging stickiness, and guardrail metrics can help determine if changes are having a negative impact (32m27s).

Problem Frames and Solution Frames

• A viable solution is one that solves a problem, and measuring behavior changes and value is more important than measuring the product itself (32m41s).
• The concept of problem frames, as described by Michael Jackson, involves understanding the problem and solution frames and how they interact (33m3s).
• Software developers often focus on the solution frame, but it's essential to consider the problem frame to evaluate whether the solution is sensible (33m31s).

Contacting Gojko Adzic

• Gojko Adzic can be contacted by searching for his unusual name online, but be aware that a footballer with the same name may appear in search results first (33m53s).
• Gojko Adzic can be easily found by searching for his name on Google, even in Serbia (34m15s).
• He appreciates the opportunity to talk and thanks the host for taking the time to speak with him (34m22s).