AI, ML, and Data Engineering InfoQ Trends Report—September 2023

QCon San Francisco and InfoQ Podcast

• The InfoQ International Software Development Conference, QCon, will be held in San Francisco from October 2nd through 6th, featuring real-world technical talks from innovative senior software development practitioners on applying emerging patterns and practices to address current challenges (10s).
• The podcast is part of the 2023 Trends report, discussing what's happening in the AI, ML, and data engineering space, with a panel of subject matter experts and practitioners (42s).
• Shi Penchala, lead editor for the AI, ML, and data engineering community at InfoQ, is facilitating the conversation (51s).
• The panelists include Sharin Thomas, a Staff engineer at Chime, who has been building data platforms, data infrastructure, and has a keen interest in streaming, data discoverability, governance, operations, and the role data plays in new advancements in AI (1m16s).
• Roland Merton, a date and data scientist at Bumble, is working with computer vision and has a background in computer science (2m15s).
• Daniel, an engineer with experience in software development, is an AWS Community Builder on machine learning and is currently developing artificial intelligence and machine learning products for different industries (2m30s).
• Anthony Alford, director of development at Genesis, has worked on customer experience-related AI projects and has a background in robotics and intelligent robot software control (2m53s).
• The podcast aims to discuss what's happening in the AI, ML, and data engineering space, what's not happening, and what's coming up that listeners should be aware of (3m26s).
• The Trends report has two major components: the podcast and another part that will be discussed later (3m45s).
• The InfoQ Trends Report is divided into two parts: a podcast discussion and a written article available on the InfoQ website, which includes a trends graph highlighting the different phases of technology adoption and providing more details on individual technologies (3m58s).
• The trends graph is a valuable resource that offers insights into the adoption phases of various technologies and provides updates on new and updated technologies since the previous year's report (4m5s).

Generative AI Overview

• Generative AI, also known as Gen AI, has been gaining significant attention since the announcement of GPT-3 a couple of years ago, and especially with the release of GPT-4 and ChatGPT earlier this year (4m48s).
• ChatGPT was used by over 17 million people in the first two months of its release, making it one of the fastest-adopted technologies (4m58s).
• Major players in the space, including OpenAI, Google, and Meta AI, have announced their plans for Gen AI, with OpenAI releasing ChatGPT, Google announcing Bard, and Meta AI releasing Llama 1 and Llama 2 (5m19s).
• The discussion aims to highlight the value and benefits of Gen AI technologies, beyond the hype surrounding them (5m24s).
• Generative AI is defined as AI based on autocompleting a certain prompt, allowing for zero-shot learning and the ability to turn data into actionable written language (6m32s).
• The concept of generative AI is not new, with generative adversarial network (GAN) models for image generation having been around for almost 10 years, but language models have gained significant attention since 2019 with the release of GPT-2 (7m1s).
• The field of generative AI has gained significant traction since 2020, with the release of GPT-3 and other image generators (7m31s).
• Stable diffusion and generative AI for images and text are becoming increasingly popular, with people using them together to create stories and illustrations, potentially cutting humans out of the loop (7m38s).
• There have been significant improvements in text generation, with GPT 3.5 and GPT 4 showing remarkable advancements, attributed to larger networks, more data, and higher-quality data (8m9s).
• The usability experience of GPT has made it accessible to a wider audience, including non-technical individuals, with many people adopting it for various purposes (8m42s).
• Mid Journey is considered the clear winner in image generation, despite its limited usability, with users paying $35 a month for a subscription and working with it through a Discord bot (9m2s).
• Once the hurdle of usability is overcome, generative AI is expected to revolutionize the industry, making it accessible to the general public (9m34s).
• ChatGPT has become synonymous with generative AI, with many companies and individuals using it for various applications, including internal projects and client solutions (10m15s).
• A new cottage industry of companies is emerging, built on top of generative AI, offering services such as homework assignments and code generation (10m47s).
• Large companies are creating their own vision and roadmaps for generative AI, indicating a growing interest and investment in the technology (11m20s).
• Generative AI is expected to see increased adoption, with many companies applying it in various ways, and its integration into products is relatively easy (11m26s).
• The initial value proposition of GPT-3 was that it didn't require fine-tuning, but now people are fine-tuning large language models, especially smaller and open-source ones like LLaMA, for specific domains (12m12s).
• Fine-tuning these models is becoming easier, but it still requires significant hardware and a substantial dataset, although models are starting to shrink in size (12m23s).
• Companies like Xedia, Morgan Stanley, Stripe, Microsoft, and Slack are already using generative AI, and adoption is increasing (13m11s).
• Prompt engineering is becoming a discipline in its own right, and it's unclear how the responsibility for prompt engineering will be integrated into traditional software development processes (14m18s).
• The concept of "Chain of Thought" prompting has been developed, where researchers found that telling language models to explain their thoughts step-by-step leads to better results (13m42s).
• The quality of results from language models and image generation can vary greatly depending on the prompt used, highlighting the importance of prompt engineering (14m1s).
• It's unlikely that dedicated prompt engineers will be needed in every project, as people will learn to use these tools effectively on their own (14m52s).

Smaller Language Models and Fine-tuning

• The use of language models like LLaMA from Meta and B from Google, as well as Amazon's bet on generative AI, is expected to continue growing, with new products like Cloud and Amazon B Rock emerging in the market (15m32s).
• Smaller models, such as those being developed by open-source communities, are becoming increasingly popular, with people fine-tuning and distilling models like LLaMA to create new models like VUNA and Alpaca (15m59s).
• The proliferation of these smaller models is expected to continue, allowing companies to alleviate concerns about using closed APIs and sending data to unknown sources (16m53s).
• Research into increasing sequence length, which is the amount of history that can be put into a chat, is ongoing, with some models supporting up to 32,000 tokens and potentially up to a million tokens in the future (17m1s).
• The ability to support longer sequence lengths is expected to enable new applications, such as in-context learning, where a model can be given a large amount of text, such as a book or a company's knowledge base, and summarize or answer questions about it (17m40s).
• The trend of using language models for summarization is expected to continue, with applications in industries such as law, where firms are using models to summarize legal documents, and science, where researchers are using models to summarize and extract information from large numbers of papers (18m20s).

Speech Synthesis and Recognition

• There's a trend of summarizing papers and documents, and another trend is speech synthesis, which involves analyzing speech data and using solutions for it (18m45s).
• Google and Meta are working on speech synthesis, releasing several models this year, including multilingual models, with Google focusing on speech-to-speech translation (19m0s).
• Open AI released Whisper, an open-source speech recognition model, at the end of last year, which is quite good for speech sentences (19m11s).
• Meta released Voice Box, a model that can take a speech audio and replace bits of it, similar to in-painting in images, but the company is cautious about releasing it due to potential abuse (19m54s).
• The development of speech synthesis models raises ethical and responsible AI consequences, which will be discussed later (20m32s).
• Google's universal speech model has a 1,000 language initiative, which will be huge for products involved with speech, such as Google products or Alexa (20m44s).
• The implementation of these models on their own hardware and products will lead to a new way of products with improved speech recognition and synthesis capabilities (21m4s).

LLM Ops and the Operationalization of LLMs

• With the adoption of large language models, teams will need to own and maintain the operations side, leading to the emergence of a new term, LLM Ops (21m47s).
• MLOps brings rigor to the process of building and launching machine learning models, and it will be essential for LLM-based applications (21m57s).
• Operational requirements for large language models (LLMs) are similar to those for traditional machine learning pipelines, but there are nuances that make operationalizing LLMs more challenging, such as collecting human feedback for reinforcement learning and prompt engineering (22m4s).
• Performance metrics for LLMs are different and constantly evolving, making it difficult to predict how they will develop in the future (22m30s).
• The LLM development life cycle consists of data ingestion, data prep, prompt engineering, and potentially complex tasks like chaining LLM calls and external calls to a knowledge base (22m46s).
• The entire life cycle requires rigor, and LLM Ops might become its own distinct field, with ML Ops being a subset of it (23m5s).
• As more applications use LLMs, the importance of LLM Ops will increase, requiring teams to continually work on data, prompt engineering, and server architecture to ensure the AI works correctly (23m15s).

Vector Databases

• Vector database technology, including embedded stores, is gaining attention, with use cases such as using sentence embedding to create observability solutions for generative applications (24m18s).
• Vector search databases are needed because large language models have limited history, and companies may want to store and query large amounts of data, such as summaries of documents, as feature vectors (24m39s).
• This technology will become more important as companies use it to store and query large amounts of data, such as legal documents or Wikipedia articles (25m3s).
• Vector databases are becoming increasingly important, allowing for efficient similarity searches in large datasets by storing feature vectors instead of raw data, and can be used with technologies such as Pinecone or Mes (25m36s).
• There has been a significant increase in funding for vector database technologies, but a slower adoption rate among developers, which is expected to change in the coming year (26m6s).
• Chroma is another player in the vector database space, offering an open-source solution (26m49s).
• The choice of vector database technology depends on the specific use case and the characteristics of the data being searched (27m1s).
• Feature stores have become a crucial part of machine learning solutions, and vector databases are likely to follow a similar trend (27m19s).
• Similarity search applications are on the rise, with use cases such as searching for weather phenomena in satellite images (27m31s).

Robotics and Drones

• Robotics and drone technologies are also advancing, with a trend of decreasing investments, but promising developments such as the establishment of an AI Institute by Boston Dynamics (28m37s).
• Cheaper and more accessible remote control technologies are becoming available, which could further accelerate the development of robotics and drone technologies (28m58s).
• Robots are becoming more viable and affordable, with the cheapest legged balancing robots available for around $1,500, making it possible for individuals to buy a robot as a platform and integrate their own hardware on top of it (29m8s).
• The Robot Operating System (ROS) is still seen as the leading software, with more and more adoption to ROS 2, and companies like Fiam are building middleware to easily add and configure plugins (29m35s).
• Google is publishing research on using language models to control robots, using them as a user interface to perform tasks, and integrating sensor data into this process (30m1s).
• This development is interesting, especially considering how hard it was to achieve in the past, and it has the potential to revolutionize the way robots are controlled and interact with their environment (30m30s).
• The use of large language models and computer vision models can enable robots to recognize objects and locations, and understand natural language commands, making it easier to program and interact with them (31m7s).
• The manufacturing industry can benefit from the integration of robotics and virtual manufacturing platforms, allowing for the simulation and testing of processes that are too costly or unsafe to perform in the physical world (31m36s).
• As robot technology advances, it is expected that robots will become more approachable as products, rather than just research projects, with companies like Tesla working on robots like the Optimus robot (32m13s).
• The future of robotics is exciting, with the potential for robots to go places they have not gone before, and the possibility of seeing the first robot on the market in the near future (32m26s).

Ethical Considerations and AI Regulation

• The increasing power of AI, ML, and data engineering technologies brings a responsibility to be ethical and fair, highlighting the importance of considering the ethical dimension of these technologies (32m28s).
• AI ethics is crucial in addressing issues such as bias, discrimination, and the potential risks to personal data and privacy, as well as ensuring that AI systems make ethical decisions and provide accurate information (33m5s).
• The adoption of AI technologies also raises concerns about unemployment and economic impact, including the potential displacement of jobs, and the need to consider sustainability and environmental implications (33m34s).
• Governments around the world are developing their own solutions to regulate AI, but it is unclear whether these individual approaches will be effective, given the global impact of AI on humanity (33m53s).
• A unified approach to regulating AI, potentially led by the United Nations, may be necessary to address the global implications of AI on humanity (34m27s).
• There is a growing recognition of the importance of safety in generative AI, but this must be balanced against the need to avoid overly restrictive regulations that could stifle innovation (35m5s).
• The trade-off between safety and functionality in AI systems is a topic of ongoing debate, with some users expressing frustration at the limitations imposed on AI models to ensure safety (35m10s).
• The development of AI regulations is an ongoing process, with governments and experts continuing to explore the best approaches to balancing innovation with responsibility (34m41s).
• There is a growing trend of people discussing the potential risks and downsides of AI and language models, such as ChatGPT, which can sometimes produce false or hurtful information, and companies are trying to implement safeguards but are unsure how to guarantee their effectiveness (35m51s).
• Research has shown that it is possible to automatically generate attacks that can "jailbreak" language models, including ChatGPT, by creating prompts that can manipulate the model's output, highlighting the need for white hat hacking to identify and address these vulnerabilities (36m35s).
• The importance of considering all possible edge cases and both the positive and negative consequences of using language models like ChatGPT is emphasized, as well as the need to remind users that the output is generated by a large language model and may not always be accurate or reliable (37m6s).
• Discrimination, even for one demography or user, is not acceptable, and it is crucial to consider the potential risks and downsides of using language models (38m18s).

Explainable AI and Data Governance

• Explainable AI is a way to explain how a model came to a result or conclusion, and it is becoming increasingly important as governments make regulations and discuss new laws related to AI ethics (38m27s).
• Explainable AI can help address concerns about AI models becoming "dumber" and provide insights into why a model is making certain decisions, and it may play a significant role in the future of AI development (38m56s).
• The increasing importance of AI explainability, data discovery, lineage, labeling, and operation is driven by the need for transparency and accountability in AI decision-making, as well as the emergence of regulations such as GDPR and CCPA (39m22s).
• The use of AI models can lead to biased outcomes if trained on biased data, as seen in the example of Amazon's hiring model that disproportionately selected men due to being trained on a dataset of mostly men's resumes (39m44s).
• In the new world of AI, good model development practices, data discovery, lineage, labeling, and operation will become crucial to ensure AI systems are fair, transparent, and reliable (40m0s).

Data Engineering Trends

• The data engineering space is undergoing significant developments and innovations, with emerging trends and patterns that leaders should be aware of in their data management projects (40m22s).
• There is a growing emphasis on speed and low latency in data engineering, with unified batch and streaming platforms becoming more popular, and architectures like Carart gaining adoption (40m49s).
• Data mesh has become a buzzword as organizations become more complex, and it is no longer sufficient for a central data team to manage everyone's use cases (41m9s).
• Data contracts are gaining importance as a way to measure data quality, ensure data observability, and guarantee that systems and data products behave as expected (41m26s).
• Data observability is becoming a main pattern in the data side, and it is no longer sufficient to have observability only at the systems and infrastructure level, but also at different abstraction layers (41m42s).
• Companies like Monte Carlo, Anomalo, and experts like Chad Sanderson are emerging in the data observability and data contracts space (42m53s).
• The need for data observability is increasing with AI, and it's not just about systems, but also about the type of data and its distribution, with past AI failures like the Zillow debacle and Amazon recruitment model serving as examples (43m7s).
• Data disciplines and AI are two sides of the same coin, requiring end-to-end discipline to manage data and machine learning models (43m47s).
• Sharin predicts that data mesh adoption will increase in the coming years, as companies feel that data teams are becoming a bottleneck, and explainability will also see more adoption (44m32s).

Predictions and Future Outlook

• Anthony predicts that artificial general intelligence (AGI) will not be achieved in the next year, and possibly not even in his lifetime (44m50s).
• Daniel believes that AI is here to stay, and with new products and technologies emerging, such as ChatPT, AI will become more accessible to normal people, not just researchers (45m27s).
• Daniel also mentions that Elon Musk will be working on AI in his companies, making AI more approachable for everyone (45m36s).
• The field of autonomous agents is an area of excitement, where agents can independently come up with ideas, such as creating a product to sell, without needing to be fed prompts or connected to other APIs (46m15s).
• Autonomous agents may be able to perform tasks such as marketing products, emailing factories, and making reservations at restaurants, making life easier for individuals, especially in situations like planning a romantic evening while traveling to a new city (46m29s).
• These agents could potentially be used for tasks like buying Valentine's gifts, making them useful for people like Roland, who is in the dating app development business (47m8s).
• It is predicted that Large Language Models (LLMs) will become more accessible to the community, with open-source frameworks like LangChain making solutions more available, and LLMs will not be limited to closed-source solutions (47m33s).
• The development of plugins for chatbots like ChatGPT is expected to continue, with potential plugins including those that can remember things for users, similar to the "Remembrall" from Harry Potter, and plugins that can interact with users through voice commands (47m56s).
• Potential plugins for ChatGPT include a restaurant plugin, a voice-based plugin that can answer questions and send emails, and a plugin that can make decisions for users (48m40s).
• Other desired plugins include ones that can tell users what they don't know, and ones that can run users' lives for them, making decisions on their behalf (49m21s).
• The future of AI and its impact on humanity will depend on how the technology develops and is utilized, with the potential for both positive and negative outcomes (49m57s).
• AI is a new way to explore previously unexplored things, making technology more approachable to everyone (50m14s).
• The use of AI at the end of the day will be determined by humanity, and its impact will be shaped by human decisions (50m27s).

Conclusion

• InfoQ will continue to provide updates on trends and new developments in the field of AI, ML, and data engineering (50m31s).
• The conversation will be continued in the future, with a follow-up discussion to assess the accuracy of predictions made and the progress that has been achieved (51m3s).
• The participants, including Daniel, Anthony, Roland, and Sharon, concluded the discussion by thanking the listeners and looking forward to the next conversation (51m8s).