Your data isn’t ready for AI with Prukalpa Sankar from Atlan
18 Sep 2024 (1 day ago)
Introduction
- Atlan is a platform designed for data teams to collaborate and manage data within an organization.
- The platform aims to connect various data assets, including dashboards, reports, and tables, to establish a unified source of truth.
- Atlan enables users to easily search, understand, trust, and govern their organization's data.
Founding story and early challenges
- The speaker's previous company, Social Cops, built India's national data platform and worked with the United Nations on incorporating data science into the global SDG agenda.
- The speaker recounts a stressful experience where they received a call from the Prime Minister's office questioning a data discrepancy in a dashboard, highlighting the importance of data accuracy and trust with stakeholders.
- The speaker emphasizes the diversity of data teams, which can lead to communication and collaboration challenges due to differing skill sets, languages, and tools, ultimately contributing to a low success rate for data projects.
Overcoming high-stakes data issues
- A company experienced a data issue when a change in their ERP system resulted in them receiving cumulative data instead of daily data, causing a significant spike in their dashboards.
- The company was able to resolve the data issue, but it took a significant amount of time and effort, including sleepless nights and constant fire drills.
- The company realized that their existing system was not scalable and they needed to develop better tools to improve their efficiency and effectiveness.
The evolution of data and AI
- In 2016, cloud technology was not widely adopted, and the modern data and cloud stack was in its early stages of development.
- Many companies, including large tech companies, attempted to solve data management problems with internal tools or open-source solutions, but these efforts often failed.
- Atlan, a data cataloging and collaboration platform, was developed with a focus on integration with existing workflows, such as Slack, Teams, and BI tools, to address the limitations of previous solutions.
Building a product to solve key problems
- The company developed empathy for its users by working closely with them and understanding their pain points.
- The company's founding team had personal experiences with the problem they were trying to solve, which gave them a deep understanding of their users' needs.
- While building a product to solve a problem is important, it is also crucial to create a product that can succeed in the market, as these two goals are not always aligned.
Adapting to technological change
- Technological advancements have been accelerating, with innovation cycles shrinking from decades to just a few years.
- This rapid pace of change necessitates that companies prioritize adaptability and rapid learning to keep up with the evolving technological landscape.
- To thrive in this environment, companies must embrace diversity in their workforce, tools, and approaches, while remaining agile and responsive to emerging trends and customer needs.
The growing role of AI in business
- There is debate about whether AI is in a bubble, with some believing it is due to the hype and investment, while others believe it is a significant technological evolution.
- AI is expected to bring significant changes in the next five to seven years, similar to the impact of the internet and smartphones.
- The hype surrounding AI has led to a willingness to invest heavily in anything related to the field.
Preparing data for AI integration
- A recent Gartner survey of Chief Data Officers revealed that 54% are prioritizing spending on data privacy and gender, while 53% are focused on data governance.
- There is a growing recognition that operationalizing AI use cases within companies is hindered by data readiness challenges.
- A significant challenge in training AI models is ensuring that only the appropriate data is used, especially in organizations with complex data estates and varying data access requirements.
Addressing bias and ethics in AI
- Companies are developing AI readiness scores for data, considering factors like completeness, accuracy, compliance, and bias metrics such as gender distribution.
- These scores can then guide data usage, with higher scores required for business decisions and lower scores potentially acceptable for internal data science projects.
- Effective collaboration between humans and AI is crucial, with a focus on designing workflows where AI acts as a "coworker" to humans, ensuring ethical and responsible outcomes.
Fundraising and fun marketing
- Atlan has experienced significant growth, increasing 7 times in size over the past two years and 30 times in the past three years.
- The company prioritizes building long-term relationships with investors who share their vision of creating an enduring company.
- The "Taylor Swift torture data Department" video was a fun and creative way for the company to announce a fundraising round, showcasing their team's personality and global reach.
Maintaining a strong co-founder relationship
- Two companies have been started with the same co-founder, and the relationship has lasted 13 years.
- The co-founders have different leadership styles and ways of solving problems, but they share the same values.
- The co-founders believe that having aligned values is essential for building a company together and ensuring its long-term success.
Final thoughts and reflections
- The speaker found the story of how the guest launched a new company to solve the problems she encountered at her old company inspiring.
- The speaker found it interesting that the guest's company was essentially incubated within her own startup.
- The speaker found it humorous that the guest and her co-founder met on a bus and went on to launch two companies together.