Developing Regulated Software at the Speed of Innovation: Insights from Erez Kaminski

Erez Kaminski and Krix

• Erez Kaminski is the founder and CEO of a company called Krix, which helps people develop regulated software at the speed of regular software (46s).
• He has a background in physics and started his career working on control methods and automation for fusion reactors and fusion reactions (56s).
• Kaminski worked at Wolfram Research, developing Mathematica, Wolfram Alpha, and the Wolfram Cloud, and was part of the team that developed the largest expert rule-based engine ever built (1m37s).
• He later became the head of AI for the medical device group at Amgen, a large multinational pharmaceutical company, where he worked on the design and development of safety-critical systems (1m58s).
• Kaminski left Amgen to pursue higher education at MIT, where he started Krix, aiming to combine his passion for developer tooling and the developer ecosystem with the need to develop regulated software at the same speed as regular software (2m11s).
• He believes that the math used in physics is far ahead of what is used in day-to-day products, and he wants to help apply advanced applied mathematics through software in the realm of safety-critical systems (2m52s).
• Kaminski thinks that developing safety-critical systems is a challenging journey that requires considering the potential risks and consequences of the software, which is often overlooked in non-safety-critical software development (3m32s).
• Safety-critical software development requires control, reliability, and complexity to ensure the software executes as intended, especially with featureful software that makes decisions and uses powerful algorithms (4m3s).
• The goal of safety-critical product development is to prove that the software consistently and reliably performs its intended function in a maintainable fashion (4m24s).

Safety-Critical Software and Examples

• Examples of safety-critical software include medical devices, such as the Klar implant, which replaces the human sense of sound and uses complicated algorithms to function (4m43s).
• Ensuring the safety, reliability, and serviceability of such devices over an extended period, even after the original engineers have left the company, is a significant challenge (5m2s).
• As software complexity increases, so do the potential issues and mistakes, which can be mitigated through software validation and risk management (5m33s).

Software Validation and Risk Management

• Software validation involves providing objective evidence that a system conforms to its intended use, while risk management involves understanding the hazards that emerge from the software's functions and development (5m40s).
• Risk management requires critical thinking and planning to ensure that the cost and safety profile of each component are appropriate for the intended use (7m6s).

Challenges in Safety-Critical Software Development

• The increasing complexity of safety-critical software has made it difficult to manage their life cycle and documented evidence, leading to slow development times (7m29s).
• There is a need to develop a faster way to manage the life cycle of safety-critical software, as its use is becoming more widespread in society (7m39s).
• The frequency of critical systems failing is increasing over time, resulting in more catastrophic and newsworthy events, which can be mitigated by changing the way products are built for such scenarios (7m48s).
• Developing safety-critical systems, such as pacemaker software, takes more time due to the brakes put on the process, but the current amount of time and brakes may not make sense anymore (8m21s).
• The methodologies used for developing safety-critical systems are often old school, and the tools developed to reduce software complexity, such as task management and devops tools, were not built with control for safety-critical systems in mind (8m36s).

Regulation and Societal Expectations

• Society expects quality, reliability, and safety checks in products, especially in regulated industries like medical devices, pharmaceuticals, and automotive, which is why regulators are in place to ensure safety (9m32s).
• Regulators give society a voice in ensuring that products are safe, and people generally expect safety checks, especially when it comes to products that can injure or kill, such as pacemakers (9m40s).
• While some developers may prioritize rapid release cycles, most people agree that safety-critical systems require more checks and validation to ensure they work as claimed (10m12s).
• Society has voted to regulate certain industries through their voting acts, and while not all regulation is good, many regulations are appropriate and necessary to ensure safety (10m46s).
• Being regulated means building a product that matters to society, and society has decided that there need to be rules about developing such products for specific use cases (11m11s).
• Every country has an agency that monitors the development of medical products, highlighting the importance of regulation in this field (11m28s).

Standard Writing and Committees

• Standard writing involves creating guidelines, technical reports, or standards to help others develop safe products, often requiring patience, organization, and committee work (11m41s).
• There are various committees and societies, such as ISO, AAMI, and ISP, that publish standards for different regions and industries, which companies use to show conformance (12m44s).
• Standards aim to ensure that developers do not forget essential steps, even if they lack experience, by providing a checklist of obvious things to do (13m11s).
• The standard writing process is distinct from software development, requiring more organization, committees, and patience (13m21s).
• Mathematicians and developers may find it challenging to adapt to the slow pace of standard writing, as they are accustomed to working quickly (13m30s).
• One of the standards worked on is for risk management in machine learning and medical devices, and another is for medical clouds, focusing on compliance with regulations and safety requirements (13m49s).

Modern Challenges and Validated DevOps

• The increasing automation of devices and systems, including in the medical field, raises questions about what validation looks like in these spaces (14m17s).
• Currently, validation often involves massive teams and thousands of pages of documented evidence, but there is a desire to make this process more efficient and developer-friendly (14m41s).
• Regulated software development involves producing extensive documentation, often hundreds of pages long, to provide evidence of process steps and ensure the software meets its intended use, with examples including staff training, rigorous process analysis, design verification, and testing associated with features and requirements specifications (15m21s).
• This documentation serves as objective evidence that the system can reliably and safely meet its intended use, and while it may seem burdensome, it is essential for ensuring safety and maintainability, particularly in critical applications such as pacemakers (16m32s).
• The introduction of validated DevOps, which combines development, operations, and computer automation, is expected to make the production of this evidence less burdensome and more efficient, allowing for faster development while maintaining safety and reliability (16m57s).
• Validated DevOps involves connecting different IT systems to ensure that activities done in one system are prerequisite for activities done in another, and connecting them in a robust manner to prevent errors and ensure traceability (18m26s).
• This approach is expected to become more widespread in the future, not just for safety-critical applications but also for B2B mission-critical applications, as it provides a way to prove that something works and can help prevent costly challenges (17m39s).
• Validated DevOps is integrated into CI/CD pipelines to prevent errors and ensure that modifications are assessed to prevent unintended changes, allowing for faster development while maintaining safety and reliability (19m1s).
• Implementing automated and verifiable quality assurance systems can provide teams with more freedom to work faster while ensuring compliance with regulations, as these systems can generate evidence of compliance and force developers to follow necessary controls (19m20s).

Cloud-Connected Medical Devices and Security

• Cloud-connected medical devices, such as pacemakers and infusion pumps, can provide benefits like real-time notifications and remote monitoring, but also introduce complexity and security risks (20m46s).
• Infusion pumps, in particular, have many different parts and require access to various data sources, creating complexity and potential attack vectors (21m14s).
• Connecting medical devices to different systems and using various open-source libraries and versions can lead to configuration management challenges and reliability issues (21m53s).
• Breaking down the architecture of medical devices to ensure safety-critical parts are isolated and secure is a significant challenge (22m20s).
• Ensuring the reliability and security of medical devices, especially those connected to the cloud, is crucial to prevent critical errors and ensure patient safety (22m16s).
• Regulated software development requires a significant amount of evidence to demonstrate understanding of system architecture, reliability, and security to ensure that things cannot go wrong, as required by the FDA and EU regulators (22m47s).
• Even with vulnerabilities in libraries used by manufacturers, proper system architecture and barriers can prevent bad actors from controlling devices remotely, as seen in the case of a large medical device company's infusion pump (23m14s).

Interoperability and Continuous Development

• Interoperability of modern medical devices and consumer expectations pose a challenge for manufacturers who have not considered this aspect before, requiring a different way of thinking about product development (23m55s).
• Developing safety-critical products continuously is essential, as it is hard for those who do not do so to think about safety and security, and it requires a different paradigm of thinking about software development (24m14s).
• A developer who worked on language design for languages like Rescript noted that he was taught to only increase functionality, but not to think about reducing features to ensure safety, highlighting the need for a different approach to software development (24m25s).

Future of Safe Software Development

• The need for safe and secure software development will grow in the next 10-15 years, driven by emerging technologies like deep tech, fusion reactors, autonomous vehicles, and AI for pharmaceutical development and medical devices (24m56s).
• The developer community will face a big fracture as more emphasis is placed on creating the right features in a safe way, rather than just adding more features, and some developers may struggle to adapt to this new approach (25m29s).
• The goal is to make it easier for developers to create safe and secure software, so they do not leave regulated industries like healthcare after a short time due to frustration with documentation and the desire to focus on development (25m38s).
• AI-enabled systems will continue to play a significant role in regulated software development, with both opportunities and risks that need to be considered (26m3s).
• The development of safety-critical AI systems is crucial, especially when they are deployed to millions of patients with severe illnesses, and it is essential to think about the limits of AI and how it will impact people's lives (26m11s).

AI and Automation in Regulated Industries

• In the future, most commerce, B2B interactions, and personal life will be dominated by automation, including routine automation, traditional machine learning, and generative machine learning, which will require validation to ensure they interact and work correctly (26m49s).
• Validating generative AI systems is challenging due to the numerous use cases, but companies are starting to perform proper risk management, and a huge amount of software will be developed in validated DevOps, with agents checking each other to ensure they are doing the right things (27m12s).
• The FDA has approved over a thousand medical devices with machine learning, and companies are trying to figure out how to develop these devices in the right way, considering the risk they are taking and the risk to patients (27m48s).
• Revolutionary devices, such as those developed by HeartFlow, are changing the way people work and saving lives, with their AI system being used by a quarter of a million patients every year to detect potential heart problems and provide results equivalent to an interventional lead insertion (28m21s).
• HeartFlow's AI system allows patients to get a CT scan and receive results the same day or the next day, reducing the delay and risk associated with traditional methods, and it is an example of how advanced AI systems can save lives every day (29m4s).

Need for Augmented Abilities and Automation in Healthcare

• Most software in the consumer and cloud web domain is not suitable for industries that need it the most, such as healthcare, and it's impossible to train physicians and subject matter experts fast enough for the growing population (29m38s).
• There is a need to augment the abilities of these professionals, reduce their cognitive load, and make their work easier to train and more productive (29m58s).
• Medical device development and surgeries can be physically challenging, and there is a need to make these procedures easier to learn and perform, potentially through automation and AI (30m22s).
• Automation can make certain medical procedures more accessible, such as cataract removal, which can now be done in more clinics, even in rural areas (31m10s).

The Future of Automation and Safety

• Society is heading towards automation, and it's essential to figure out how to make this automation safe and reliable, especially for the safety of children (31m45s).
• The development of autonomous vehicles has not yet earned a good reputation, but it's expected to improve over time (31m30s).
• The goal is to make automation dominant in life while ensuring it's safe and reliable (31m42s).

Continuing the Conversation

• People can continue the conversation on Twitter and LinkedIn (32m2s).