Namee Oberst on Small Language Models and How They are Enabling AI-Powered PCs

Introduction to AI and Small Language Models (LLMs)

• Microsoft Reactor provides events, training, and community resources to help developers, entrepreneurs, and startups build on AI technology, which can be learned more about by visiting aka.ms/infoq/reactor (9s)
• Nami Oberst is the founder of AI Blocks, the company behind the open-source LLM framework called LLMware, used for Gen AI-based applications in financial services and legal industries (34s)
• LLMware aims to make generative AI easy to use, deploy, and develop for the enterprise and regulated industries, with a focus on development and deployment securely, safely, and cost-effectively (1m40s)
• Nami Oberst's background as a corporate attorney working in big law motivated her to start an AI company to automate repetitive tasks, and her experience working with highly regulated industries led her to focus on small language models (2m22s)
• Small language models can perform focused and targeted tasks, such as contract analysis, information retrieval, and providing concrete facts, and have been found to be more reliable and predictable than open AI calls in some ways (3m29s)
• AI Blocks started working on small language models 20 months ago and launched an open-source project about a year ago, with the goal of enabling mobile devices and edge computing servers to leverage Gen solutions that were previously limited to large language models (2m1s)
• The emergence of small language models is enabling AI-powered PCs and allowing for the deployment of Gen AI solutions in industries that were previously limited by the need for large language models (1m7s)
• Open AI had significant issues with outages and accuracy, prompting the development of alternative models that can be run locally and produce comparable results for specific use cases (3m58s).
• The first models released were called Dragon, which are seven billion parameter models trained to not hallucinate and provide accurate answers with quality scores, aiming to offer users the accuracy of a corporate lawyer (4m37s).
• The use of small language models has evolved to include smaller RAG fine-tunes and function calling models with one to three billion parameters, which can automate workflows and processes (5m11s).

The Rise of Small Language Models (SLMs)

• Large language models (LLMs) have been a major innovation in AI and ML, but they require significant computing resources and raise privacy concerns, limiting their adoption (6m7s).
• Small language models (SLMs) offer an alternative to LLMs, providing a more feasible solution for companies with limited computing resources and data privacy concerns (7m26s).
• The development of SLMs has the potential to make a significant impact on the adoption of AI solutions, offering a more accessible and efficient alternative to LLMs (5m57s).
• SLMs can be used in various business and technical use cases, empowering end-users, software engineers, and devops engineers to be more productive (6m39s).
• The use of SLMs can also address the challenges associated with LLMs, such as the need for significant computing resources and data privacy concerns (7m5s).
• Small language models (SLMs) offer many of the same benefits as large language models (LLMs) but are smaller in size, trained using smaller data sets, and don't require a lot of computing resources (7m31s).
• SLMs are valuable for use cases where there are constraints on resources or a need to localize model execution, and they are opening up new opportunities to run SLMs on smartphones and other mobile devices used for Edge Computing (7m58s).
• SLMs keep data within the device, making them great candidates for use cases where privacy, latency, or other concerns exist in sending data to the cloud (8m17s).
• The definition of a small language model has changed over time, and it is now possible to run models with up to 14 billion parameters on an Intel-based AI PC (9m23s).
• The latest AI PCs, such as those from Intel, are enabling the development of SLMs that can run on commodity laptops, with prices starting from around $1,100 (10m27s).
• The small language models themselves are getting better by the week, making innovation possible, and the hardware on edge devices is also improving (10m6s).
• The innovation in training SLMs is also advancing, with new technologies such as Apple Intelligence being released at the end of the month (11m8s).

Advantages and Applications of SLMs

• SLMs are not a one-size-fits-all solution and are not suitable for every use case, but they are valuable for specific applications where resources are limited or model execution needs to be localized (7m42s).
• A 3 billion parameter model is being developed to run on-device, made possible by pruning a 6.4 billion parameter model, which involves removing unnecessary parts and using distillation techniques to create a smaller version (11m11s).
• The use of small language models is becoming increasingly innovative, making it difficult to say that large language models are necessarily better, as the choice of model depends on the specific use case and task (11m47s).
• Using the right size model for the right problem is crucial, and leveraging small models with proprietary data can result in comparable accuracy to larger models (12m21s).
• A study by Nvidia found that using 40 times fewer training tokens, small language models can achieve comparable results to larger models, and even outperform them in some cases (12m40s).
• The combination of distillation, pruning, and fine-tuning with proprietary data can result in a model that is 16% better than a model of the same size trained from scratch (13m2s).
• The ability to run these models on a laptop, without the need for a special GPU farm, is democratizing AI and making it more accessible to a wider range of users (13m14s).
• The increased accessibility of AI models is expected to revolutionize the field and open up new use cases, making it possible for people to use AI for day-to-day tasks and micro-tasks (13m43s).
• The development of small language models is also addressing concerns around data privacy and leakage, as users will be able to query their documents and chat with the model without worrying about data security (14m48s).
• Small language models can be used to bring AI to workers' fingertips, increasing productivity by automating various tasks and making information more accessible, especially for those with laptops (15m7s).
• The use of small language models can democratize AI use cases, making them more accessible to a wider range of people and organizations (15m41s).
• Basic use cases for small language models include finding information in documents, such as searching for specific details in an 80-page contract, which can be done locally on a laptop without the need for internet access (15m54s).
• Small language models can also be used for tasks like summarization, SQL queries, and transcription of voice recordings, automating microtasks that are part of day-to-day work life (16m40s).
• The use of small language models on local devices can help address data privacy concerns, as sensitive information does not need to be uploaded to the cloud (17m2s).
• Small language models can be used to automate workflows, such as creating reports for financial analysts, which can include tasks like looking up company information, stock prices, and historical data (17m42s).
• Agent workflows can be created to automate tasks, such as making API calls to services like Yahoo Finance or Wikipedia, to gather information and generate reports (17m59s).
• The promise of AI is to serve as a co-pilot for everyday working life, making it accessible to users on their devices, rather than being a behemoth use case only large companies can access (18m20s).
• Small language models and the Retrieval-Augmented Generator (RAG) are a good fit for each other, as they can be trained with a company's private information and used to ask domain-specific questions, making them suitable for commodity hardware (19m0s).

Technical Aspects and Performance of SLMs

• RAG is not necessarily better with large models, and studies have shown that large language models are not designed for complex RAG, making the key to successful RAG deployment the workflow and accuracy of the chain from document ingestion to inferencing (19m35s).
• Combining RAG with a specialized embedding model that understands the domain can lead to fast inference speeds, especially with new AI PCs that have integrated GPUs, resulting in performance differences compared to older Intel-chipped devices (20m14s).
• The difference in running PyTorch versus OpenV GPU can be significant, with a 5-second to 15-second difference for a 21-question inference test for a 1.1 billion parameter model, allowing for subsecond response times with the right inferencing technique and hardware (20m43s).
• AI is becoming more accessible, coming to users at their fingertips, and bringing value to those who can utilize it, rather than sending data to the cloud (21m1s).
• The AIML and Data Engineering Trends report, recorded in August, provides more information on the topic and will be linked for reference (21m20s).
• Small language models are being used in applications like auditing and compliance, enabling proactive compliance and regulations by design rather than by accident (21m38s).
• Features related to compliance and auditability, such as AI explainability and guardrails, are crucial for AI-powered PCs (22m3s).
• Small language models shine in AI explainability, allowing for visibility into every single step of the decision-making process (22m16s).
• Chaining workflows with small language models can create decision trees based on model inferences and answers, enabling course correction and fault identification (22m32s).
• Unlike large language models, small language models provide visibility into the decision-making process, allowing for the identification of mistakes and the rationale behind the workflow design (23m24s).
• AI explainability is critical for Enterprise applications, enabling the exposure of options considered by the model and the chosen outcome at every step of the process (24m0s).
• Observability and explainability factors are crucial for systematic and observable decision-making, especially when deploying new processes (24m36s).
• The use of small language models captures every interaction, inference, and decision, providing all necessary data for auditability and compliance purposes (24m46s).

Comparing SLMs with LLMs and Their Combined Use

• Large language models are good at preserving the context of conversations, making them suitable for customer-facing chatbots that can go on for days, due to their large context window that can keep the conversation going and preserve context over long sessions (26m40s).
• Small language models can be used for processes that run in the background, such as hourly, daily, weekly, or monthly tasks, and can be chained together on CPUs to run on inexpensive hardware (27m36s).
• Small language models can be powerful on the edge in devices, doing real-time analytics for finding defects or other use cases in manufacturing processes, and then sending results to the cloud for offline analytics to generate more insights (28m30s).
• The combination of large and small language models can be the best choice for certain use cases, where large models can capture conversations and preserve context, and small models can drive insights and analytics off of that conversation in batch processes (26m1s).
• Large language models are not necessarily better than small models, as small models can be just as good in terms of performance or accuracy, despite being smaller in size (25m51s).
• The use of small language models can provide visibility into every single step in the process, which is beneficial for auditors who want to know all the under-the-hood details (25m30s).
• Small language models can be used for specific use cases, such as Edge on-device analytics, real-time defect detection, and other use cases in manufacturing processes (28m11s).
• Local modeling can generate insights that can be sent to the cloud to train large Val models, creating a feedback loop between localized small language models and cloud-based large language models, making each other better (28m35s).

Cost-Effectiveness and Adoption of SLMs

• Using a large language model for everything can be overkill and a tremendous waste of resources, especially for startups that are financially constrained (29m5s).
• Small language models can be a first step in the learning and adoption process for companies, allowing them to learn the process, solutions, and invest in a bigger solution later (29m38s).
• Enterprises are extremely cost-sensitive and look for cost efficiency, security, safety, and performance, making small language models a viable option (29m52s).
• During the exploration phase, companies can try low-cost, high-performance, and easy-to-run models and then grow into large models (30m30s).

Tools and Resources for Using SLMs

• To adopt or try small language models, the required infrastructure or tools depend on the laptop, with different recommendations for Mac and Intel-based machines (30m48s).
• For Mac users, the GGF quantize version and a solution like AMA are recommended, while for Intel-based machines, the open Vino library is the preferred choice (31m17s).
• The open Vino library is supported by LLmware, but users need to download it themselves to work with the library (31m45s).
• The Microsoft version of the NX Onyx model is a cross-platform approach that offers a middle-of-the-road performance, with negligible differences in performance compared to the ggf model on a Dell machine (32m4s).
• Onyx is a good option for non-Mac and non-Intel based machines, while ggf is the fastest way to run inferencing on a Mac, and openVINO is the best option for Intel-based machines (32m34s).
• A four-year-old Dell machine can achieve the same performance as an M3 in terms of inference speed using the right software, highlighting the importance of matching software to hardware (32m48s).

The Future of AI and SLMs

• The development of small language models is enabling AI-powered PCs, allowing for the democratization of AI and making it as ubiquitous as regular software (34m17s).
• Model HQ is a product that allows users to run openVINO models without needing to know C++, making it easy to use and accessible to a wider audience (34m36s).
• The future of AI development and PC hardware is expected to lead to better and more ubiquitous AI, with the potential for AI to be as common as regular software in the next three years (35m5s).
• Small language models can operate at the edge, without the need for cloud connectivity, and can be used in a variety of devices, including laptops, smartphones, IoT devices, and sensors in manufacturing plants or autonomous vehicles (33m49s).
• The emergence of small models is accelerating the power of AI PCs and devices, with limitless use cases and potential applications (34m7s).
• AI will become ubiquitous in the future, making it a standard feature in applications, similar to software, with some applications having to explicitly state that they do not include AI (35m29s).
• The power of small language models is increasing due to advancements in distillation, pruning, and combination techniques, allowing them to run on smaller hardware footprints (35m46s).
• The definition of a small language model is changing, enabling the possibility of running large models, such as a 20 billion parameter model, on a laptop (36m6s).
• Smaller models, like three billion parameter models, are becoming increasingly powerful, making them a viable option for many applications (36m26s).
• Small language models and AI-powered PCs are complementary technologies that will drive innovation in each other at a rapid pace (36m39s).
• The price point of AI-powered PCs is relatively inexpensive, considering their powerful capabilities, with options available for around $1,000 to $2,000 (37m35s).
• The Lunar Lake version of AI-powered PCs is becoming available for consumers, offering powerful GPU capabilities (37m12s).

Best Practices and Recommendations for SLMs

• Best practices for using small language models include starting with Microsoft models, such as the Five series, and being aware of the rapidly evolving nature of these models (38m20s).
• It is recommended to try out small language models and experiment with different options, such as using LL mware for inference on Mac devices (38m35s).
• Small language models can be easily installed and used, and they are capable of performing various tasks, with the caveat that they may not be as good as larger models for tasks like video or image generation (38m39s).
• These models can be stacked together to create workflows, allowing for workflow automation, and can be used for tasks such as sentiment analysis, named entity recognition, and information extraction (39m24s).
• There are a dozen or so models available that have specific functions and can be chained together in a workflow, and there are also many examples and YouTube videos available to help users get started (40m1s).
• When deploying small language models to production, they are more secure in some ways because they are less susceptible to suggestions and hacks, and are less likely to respond to prompt injection attacks (40m52s).
• Small language models are also great for observability, as they can be easily tested and debugged, and can be swapped out or fine-tuned if they are not performing as desired (41m35s).
• The use of small language models can also make it easier to identify and fix issues, as it is clear where the model is failing or succeeding, and the data set can be examined to identify any problems (41m43s).

SLM Ops and Deployment Considerations

• The deployment of small language models, or "SLM Ops", is an important consideration, and users should be aware of the potential benefits and challenges of using these models in production (40m34s).
• When creating an AI workflow, it's often better to start with small language models, chain them together, and then increase the model size if necessary, rather than starting with a large model like Open AI (42m16s).
• Starting with a small model, such as a 1 billion parameter model, and then substituting larger models, like 3 billion or 7 billion, can be an effective approach (42m36s).
• A 10 billion parameter model can likely solve most workflows, except for hard exceptions like video creation and image generation (42m51s).
• It's recommended to start small, explore, and iterate when working with AI models (43m18s).

Online Resources and Communities for AI

• For online resources, Mark Tech Post is a good source for leading-edge AI research, and YouTube channels like AI Anytime and World of AI are great for tutorials and exposure to the latest AI developments (43m35s).
• Hugging Face's LinkedIn site is also a good resource for promoting new models, and InfoQ's website has good information on AI, including articles on Apple Foundation models (44m18s).
• AI is becoming ubiquitous and is being integrated into various communities, including architecture, devops, cloud, security, and machine learning (44m52s).
• The question remains when AI will become a regular thing, like having a personal website, and no longer be a topic of excitement (45m14s).
• The integration of AI in everyday technology is expected to become the norm, similar to how it is now understood that everything will have the internet in it (45m27s).
• Looking back, it will be clear when AI became an integral part of daily life, but it may not be immediately apparent when it happens (45m42s).
• Nami encourages the InfoQ community to keep experimenting and playing around with AI-powered technologies (46m0s).
• Lmware, an open-source site, offers an end-to-end solution for small language models and is free for anyone to try out (46m7s).
• Small language models have the potential to commoditize and localize language model solutions, allowing for a bigger impact on the software development community (46m24s).
• The AI/ML and data engineering community page on the InfoQ website is a resource for learning more about AI/ML topics, including recent podcasts and the AI/ML Trends report for 2024 (46m40s).
• The AI/ML Trends report for 2024 covers topics such as small language models, AI-powered PCs, coding assistance, and other trends in the field (46m56s).