How AI empowers SaaS leaders to build a new data pipeline | TechCrunch Disrupt 2024

The Importance of Data in AI

• Many companies struggle with harnessing data, and this problem is exacerbated for smaller companies with sensitive data spread across multiple locations, making managing data crucial (39s).
• The rise of AI has presented unique challenges in managing big data, with the need for scalable solutions to handle large amounts of data (1m7s).
• Data Stacks, a company founded in 2019, builds technology that fuels data-driven applications, including Netflix, Spotify, and iPhone, using a project called Apache Cassandra, a highly scalable NoSQL database (1m41s).
• There is no AI without data, and specifically, unstructured data at scale, which is where Data Stacks' technology excels (2m14s).
• Despite the hype around open AI and LLMs, the importance of context from data is exponentially increasing in the enterprise, and making this context available for AI apps is a key challenge (3m45s).
• To reduce hallucinations and increase relevancy in AI applications, it's necessary to combine content from LLMs with context from stored data, using techniques like Retrieval-Augmented Generation (RAG) (3m35s).
• The need for context and data in AI applications is driving the demand for scalable data solutions, making this a critical area of focus for companies and investors (3m56s).

Challenges and Solutions in Big Data Management

• Big enterprises face challenges in managing and harnessing their data, despite the promise of "big data" 10-20 years ago, and are now trying to figure out how to utilize the data they have stored for training their own large language models (LLMs) or other purposes (4m29s).
• One of the challenges is determining which data to use, and there are also issues with surfacing or giving context to existing LMs, hosted LLMs, or their own LMs (5m13s).
• Real-time data is seen as a key factor in unlocking the power of generative AI, but currently, it may not be as helpful as historic data for enterprises trying to solve problems in areas like customer success, HR, and IT ticketing (6m12s).
• The rise of AI has brought attention to data locality issues, particularly with global companies having customers in different regions with conflicting regulations, leading to tensions and challenges in data management (6m38s).
• To address data locality issues, companies often localize their data in the most restrictive jurisdiction, such as Europe, or set up multiple central data stores, and sometimes mask personally identifiable information (PII) to create a copy of their data (7m15s).

Leveraging Language Models for Insights

• Language models are seen as an opportunity to gain insights from text data, which was previously considered an opaque blob, and companies like FiveTran are leveraging this technology to help their customers get all their data in one place and build retrieval-augmented generation (RAG) models on top of it (8m2s).
• The fundamental innovation in AI is the ability to make meaningful use of text, which will unlock various possibilities in the future (8m19s).

Data Locality and Personalization Challenges

• LVMH, a conglomerate with multiple brands, is a customer that operates in many jurisdictions, presenting challenges due to different regulatory landscapes (8m45s).
• LVMH uses masking as a solution, but the specific details of how they solve data locality problems are not known, and they move a lot of logistics data (9m21s).
• Personalization of experiences is a key goal for next-generation commerce, but it poses challenges when dealing with data from different regions, such as China, where customer information cannot leave the country (9m38s).
• Companies operating in China often have completely parallel systems and stacks due to regulatory restrictions, and companies like the one being discussed do not operate in China (10m31s).

OpenAI's Use Case and Real-Time AI

• OpenAI is a customer that uses the service for comprehensive product analytics, moving underlying databases of their systems, and scaling their operations (10m43s).
• OpenAI's use case is challenging due to their size and desire to scale operations to infinity, but they have a relatively small number of systems of record with an enormous amount of data (11m11s).
• The concept of real-time AI is important, and the evolution of AI from predictive AI to neural networks has been significant, with the pool of people working on predictive AI tools being relatively small (11m34s).
• The promise of Geni is that it's near real-time, allowing for instant responses and making things happen in real-time or near real-time, which puts a different level of focus on using real-time systems (11m58s).
• Real-time relevancy is crucial, and without it, Geni will not take off, as it requires real-time data and context to provide relevant search results (12m58s).
• The big challenges in bringing proper real-time processing to real-time data include making sure the app is at least as good as a human response would have been, with an accuracy of around 70% (13m40s).
• To achieve this, systems like Rag are needed, which go back and forth between the LLM and contextual data, using the latest ways of creating decreasing hallucinations and increasing relevancy (14m2s).
• Relevancy is a new muscle for developers, who have never had to deal with it before, but it's essential for providing the best and most relevant results to users, which can lead to a 25-50% increase in sales (14m34s).
• The goal is to get to a point where users can ask anything, and the system will provide close to accurate responses, similar to stock market data, but it's not yet clear how far away we are from achieving this (14m49s).

The Evolution and Potential of Generative AI

• Generative AI is currently seen as just a fancy chatbot, but it has the potential to be much more, and it's not the first time we've been on this journey, as it took time for the web and mobile to develop and improve (15m7s).
• The current era of AI, referred to as "The Angry Bird session," is characterized by the presence of chatbots and language models like GPT and Gemini, but these technologies are not yet transformative. (15m36s)
• This year, many enterprises are putting AI-powered applications into production, but these are mostly small, internal projects, and companies are still working out the kinks in terms of team formation and implementation. (15m59s)
• Next year is expected to be the "year of transformation" for AI, where companies will start building applications that can change their trajectory. (16m12s)

Real-Time Data Pipelines: Myth vs. Reality

• Real-time data pipelines are often unnecessary and are a "phantom" in the industry, with most use cases not requiring up-to-the-second data. (16m31s)
• The term "real-time" is often misused, and actual real-time systems with low latency are rare, with most companies not needing data that is updated every 10 seconds. (16m46s)
• In cases where low latency is required, it's often better to build the workflow within the system of record, rather than trying to move data between systems. (17m22s)
• True low-latency use cases are rare, and most companies don't have a clear use case for real-time data pipelines, with the exception of near real-time use cases like weather events for customer support. (17m48s)
• Even in near real-time use cases, the workflows are often built within the system of record, eliminating the need for real-time data pipelines. (18m25s)
• Poor system design can sometimes lead to the need for real-time data pipelines, but this is not a desirable situation, and companies should strive to avoid it. (18m40s)

Investing in Data Pipelines for AI

• Many companies are investing in building data pipelines for their AI applications to get near real-time information and make data available, but it's unclear if there's a real return on investment at the moment (19m8s).
• New companies are investing in new data pipelines to harness the power of AI, with most companies founded today wanting to be on the latest technology, even if it's not clear which direction they're going (19m51s).
• Companies are adopting new data pipelines in the hopes of being more flexible, as the current stage is still early and the future of data pipelines is uncertain (20m17s).
• Founders have anxiety about building companies on current data pipelines, worrying that they may have to scrap them in five years due to rapid changes in technology (20m41s).
• To mitigate against this, companies are testing out new tools and sharing technical information, with many adopting open-source technologies as a form of future-proofing (21m4s).
• Startups are using technologies that provide a database and a path layer, such as Langlow, to build their data pipelines (21m34s).
• Two key strategies founders are using to future-proof themselves include building on open-source technologies and focusing on getting to product-market fit quickly, rather than worrying about scale (21m46s).
• Many startups initially build their own data pipelines, but having product-market fit (PMF) provides the necessary resources to make it happen (22m27s).

Balancing Data Quantity and Quality

• The main challenge companies face is striking a balance between the quantity and quality of data, as there is no shortage of data available (22m42s).
• To unlock the real value in their data, companies should work backwards from what they are trying to accomplish, identifying the specific problem they want to solve and the required workflow and data (23m2s).
• Starting small with internal applications and specific goals is recommended, rather than trying to implement general-purpose AI across the company without a clear plan (23m45s).
• The mantra is to only solve the problems you have today and not plan ahead, as the costs of innovation are mostly in things that didn't work out (24m0s).

Building Successful AI Projects: People, Process, and Technology

• The framework for success involves technology, people, and process, but it's recommended to focus on people and building successful projects first, rather than process (24m55s).
• The most important factor is the people, specifically the SWAT teams that build the first few projects, as they are writing the manual for how to do Gen apps (25m26s).
• Companies should focus on getting something done, whether it's an internal or external app, and not worry too much about planning for scale ahead of time (25m49s).
• When building AI applications, it's essential to focus on getting something done and making it impactful, rather than trying to create a massive, world-changing application from the start (25m51s).

Common Mistakes and Best Practices in Building Data Pipelines

• The number one mistake startups make when constructing their data pipelines is "boiling the ocean," or trying to tackle too much at once, and instead, they should start with something small (26m19s).
• Start with a use case and focus on solving a specific problem, rather than trying to tackle a big vision all at once (26m43s).
• It's crucial to start small and then expand, as the biggest waste of time is working on things that aren't successful (27m0s).

The Value of Relational Databases in AI Applications

• When building something that involves working with customer data, it's essential not to neglect permissions and to consider storing data in a relational database to handle complicated permissions problems (27m18s).
• Relational databases are valuable in the context of AI applications because they can handle permissions problems, such as users and roles, effectively (27m31s).
• Traditional technology stacks have a lot of value in the context of AI applications, as AI applications are all about permissions, just like all enterprise applications (27m40s).