AI, ML, and Data Engineering Trends in 2024

Current State of AI and ML Technologies

• The podcast discusses the current state of AI and ML technologies, particularly focusing on generative AI and large language models (LLMs).
• The discussion features a panel of experts, including Nami Oberst, founder of the open-source library LMW Atware, and Mandy Goo, lead machine learning and data engineering at W Simple.
• The podcast highlights the rapid advancements in generative AI and LLMs, driven by the release of models like ChatGPT, Google Gemini, GPT-4, and Llama 3.
• The discussion emphasizes the increasing capabilities of LLMs, including their ability to work with audio, vision, and text in real-time, as well as their growing size and complexity.
• The podcast acknowledges the significant impact of open-source solutions like Llama on the field.
• The discussion focuses on the current state of LLMs and their recent developments, with Anthony Alford, director of software development at Genesis Cloud Services, providing insights.

Key Players in the LLM Landscape

• Large language models (LLMs) are becoming increasingly prevalent, with OpenAI being a leading player in the field.
• Google, Anthropic, and Meta are also significant contributors to the LLM landscape.
• Meta is advocating for open-source AI models, with its own models being open-weight.

Trends in LLM Development

• The trend towards larger models with more parameters, larger datasets, and increased compute budgets is continuing.
• Fine-tuning, particularly "instruct tuning," is becoming increasingly common for LLMs to follow instructions.
• Context length, the amount of data that can be input into an LLM, is increasing.
• Google's Gemini model introduced a 1+ million context window length, which has sparked a trend towards longer context windows.
• Longer context windows can simplify complex tasks like information retrieval, but they may not always be the most effective solution.
• Well-crafted retrieval augmented generation (RAG) workflows may still be more suitable for targeted information search.
• The effectiveness of longer context windows depends on the use case, and they may not be beneficial for large-scale document analysis.
• LLMs are not ideal for precise information retrieval and may not be a suitable replacement for search engines.

Retrieval Augmented Generation (RAG)

• Retrieval Augmented Generation (RAG) allows users to avoid the context length problem by retrieving information from external sources.
• RAG can be used with smaller, open-source models that can be run locally or in a private cloud, enabling companies to solve specific problems without relying on large, closed models like GPT-4.
• Open-source models are particularly useful for targeted tasks, such as automating workflows or generating reports, and can be fine-tuned for specific needs.

Multimodal LLMs

• Multimodal LLMs, like GPT-4, allow users to interact with models using various formats, including audio, video, and text.
• While multimodal LLMs are still under development, they have the potential to significantly advance the capabilities of LLMs.
• The speaker experimented with GPT-4's image generation capabilities and found that it struggled to generate a logo with the correct spelling of "LLM."
• The speaker believes that smaller companies, like Midjourney, are making significant progress in the field of image generation, demonstrating the potential for innovation outside of large corporations.
• Multimodal models, particularly those with OCR capabilities, have proven valuable for internal use cases, such as analyzing screenshots and stack traces.

The Growing Number of AI Models

• The number of available AI models is rapidly increasing, with over 800,000 models currently available on Hugging Face.

RAG for Enterprise Use Cases

• Retrieval Augmented Generation (RAG) is a promising approach for companies to leverage LLMs by integrating their own data and knowledge bases.
• RAG systems can be used to enhance employee productivity by providing access to company information and documentation through a question-and-answer interface.
• One company has successfully implemented a RAG system by integrating their self-hosted LLM with various knowledge sources, including documentation, code repositories, and public artifacts.
• This RAG system has demonstrated improved relevancy and accuracy compared to simply feeding all the context into a large language model.
• The company used open-source LLMs and models from Hugging Face to build their internal LLM platform.

Secure LLM Deployment in Enterprise Settings

• The discussion centers around the use of large language models (LLMs) in enterprise settings, specifically addressing the challenge of handling sensitive data.
• The conversation highlights the need for secure solutions that allow employees to work with sensitive information while interacting with LLMs.
• Self-hosted LLMs are presented as a solution to this challenge, enabling secure data handling within the enterprise.
• The use of Llama.cpp, an open-source framework, is mentioned as a way to integrate quantized models for efficient and secure inference.

Miniaturization and Quantized Models

• The discussion emphasizes the potential for miniaturization of LLMs, enabling deployment on devices like laptops, expanding accessibility and usability.
• The importance of quantized models for edge computing is highlighted, emphasizing the trade-off between precision and speed, with quantized models offering faster processing at the cost of some accuracy.
• The discussion acknowledges the work of Georgie Gerganov in optimizing Llama.cpp for various platforms, including Mac Metal and Nvidia Cuda.
• The conversation emphasizes the need for cross-platform compatibility, enabling data scientists to work on Macs and deploy their models seamlessly on other AI PCs.

Small Language Models (SLMs)

• The discussion transitions to the topic of small language models (SLMs), highlighting their potential for specific use cases and the development of RAG (Retrieval-Augmented Generation) frameworks tailored for SLMs.
• The conversation mentions Microsoft's research on 53 models, suggesting a growing interest in SLMs within the industry.
• The discussion emphasizes the early adoption of SLMs by the company, highlighting their experience in working with these models for over a year.

Advantages of Smaller LLMs

• Smaller language models (LLMs) are becoming increasingly powerful and cost-effective, making them suitable for highly regulated industries.
• The speaker's company has observed significant performance improvements in smaller models, particularly with Microsoft's F3 model, which outperformed larger models in accuracy.
• The speaker believes that smaller LLMs will become increasingly accessible and eventually be deployed on edge devices.

Importance of Internal Benchmarking

• The speaker emphasizes the importance of developing internal benchmarking tests to evaluate the performance of LLMs in specific business contexts.
• The speaker highlights the need for more robust evaluation methods to assess the true value of LLMs beyond published benchmarks.
• The speaker's company has developed its own benchmarking tests focused on fact-based questions and logic-based tasks relevant to enterprise use cases.
• The speaker's company's internal testing has shown promising results for the F3 model.
• The speaker expresses skepticism about the reliability of some published benchmarks and emphasizes the importance of developing tailored evaluation methods.

Data Quality and Model Performance

• The quality of data used to train machine learning models significantly impacts the performance of the models.
• There is a growing interest in understanding the impact of different data sets on the performance of machine learning models.

Smaller LLMs and Hardware Advancements

• Smaller language models are becoming increasingly popular due to their ability to run on less powerful hardware.
• The availability of new GPUs from companies like Nvidia has made it possible to run these smaller language models on a wider range of devices.

Evaluating LLM Performance

• Evaluating the performance of language models should be done based on specific tasks rather than relying solely on general benchmarks.
• The business value of a language model should be considered when evaluating its performance.

AI Agents and Automation

• There has been significant development in the area of AI agents, but there is still room for improvement.
• AI-powered coding assistants, such as GitHub Copilot, are becoming increasingly popular.
• There is a growing trend towards the development of AI agents that can perform tasks autonomously, such as the Devon AI software engineer.
• The success rate of Devon, a free AI software engineer, is currently around 20%.
• Agent GPT, an AI tool, can generate outlines for topics like trends in AI podcasts.
• The speaker believes there is significant potential for AI agents to automate tasks, such as email management and communication.
• The speaker defines AI agents as tools that can combine multiple tasks and exhibit autonomy.
• The speaker acknowledges concerns about AI safety and the potential risks of granting AI agents autonomy.
• The speaker notes that AI agents are being developed for various applications, including dating apps, small business automation, and daily workflow management.
• The speaker predicts that AI agents will become more integrated into daily workspaces and platforms like Gmail and Slack.
• The speaker believes that the integration of AI agents into existing platforms will significantly reduce user effort and drive adoption.
• The speaker suggests that AI agents could eventually provide guidance on communication strategies, such as when to send an email or make a phone call.

AI Integration and Potential Concerns

• The speaker observes that many companies have declared themselves AI companies and are developing their own chatbots.
• The speaker mentions that some individuals are using AI tools like ChatGPT to generate arguments and content, potentially reducing the need for original thought.
• The discussion focuses on the security implications of integrating AI, particularly generative AI, into business operations.
• Concerns are raised about data privacy and the potential for sensitive information to be shared with third-party providers like OpenAI without explicit user consent.
• The importance of data lineage mapping is emphasized, as it becomes more challenging to track data flow with the increasing use of AI integrations.
• The need for secure workflows that minimize the risk of data breaches is highlighted, with the suggestion that companies should make secure practices the easiest option for employees.
• The design of generative AI workflows is crucial for data security, as it can influence the handling of personally identifiable information (PII) and the potential for unintended data sharing.
• Auditability and explainability are essential components of secure AI workflows, allowing for the tracing of interactions and understanding how AI decisions are made.
• Prompt injection and data poisoning are potential attack vectors that need to be considered when deploying AI systems.
• The analogy of cable versus streaming is used to illustrate the increased attack surface created by using multiple AI integrations from different vendors.
• A positive trend is observed in the growing awareness of security concerns among vendors, with some offering solutions like hosting AI models within a company's own cloud environment to enhance data security.

Production Support of LLMs

• The production support of LLMs is a crucial aspect of AI development.
• Wall Simple has a three-pronged approach to LLMs: boosting employee productivity, optimizing client operations, and foundational LM Ops.
• Their enablement philosophy focuses on security, accessibility, and optionality, preventing the common issue of using LLMs as a "hammer looking for nails."
• They have built an LM Gateway with features like auto-trail and pi redaction, but have also enabled self-hosted models for greater flexibility.
• They have also developed a reusable API for retrieval and built scaffolding around their Vector database.
• This platformization approach has encouraged end-users to identify and implement workflows that benefit from LLMs.

Predictions for the Future of AI

• Mandy predicts that the hype surrounding LLMs will subside in the next 12 months, leading to more realistic expectations and tangible results.
• Daniel predicts that artificial intelligence and blockchain will integrate, particularly in data management and database integration.
• Roland predicts that embodied AI, or robots with AI capabilities, will become more prevalent, assisting humans in various tasks.
• Companies are expected to fine-tune their own AI models using their data and potentially sell these models.
• Smaller companies with unique data sets, such as gardeners with years of garden photos and advice, are expected to find ways to extract value from their data.
• There is a possibility of an "AI winter," where the hype surrounding AI may taper off.
• Some experts believe that the internet may be polluted with content generated by AI, leading to a decline in the quality of AI models.
• In the next 12 months, a shift towards more realistic and tangible AI use cases is anticipated, including automated workflows, agent work processes, and AI integration into edge devices like laptops and phones.
• The future of AI is expected to involve more unified, end-to-end solutions powered by small language models, RAG (Retrieval-Augmented Generation), and AI-powered hardware.
• The panelists expressed hope that the potential AI winter will be short-lived.