Small Language Models: The Future of AI on a Budget?

How Lightweight AI Can Transform Your Site’s Chatbot Without Breaking the Bank

Over the weekend, I read an article on Small Language Models (SLMs).  Unlike their Large Language Model (LLM) counterparts, like ChatGPT, SLMs are types of generative artificial intelligence designed to be lightweight, efficient, and require fewer resources.

Because of their small size, they’re fast and don’t need as much computing power, allowing them to run on personal devices or small servers.  They’re perfect for applications that need quick responses without the complexity of large-scale data.

You might think of them as “mini-AI” models – built to handle language tasks efficiently without overloading the system they’re running on.

This idea got me thinking back to my work on a Markov Chain algorithm for the Kognetiks Chatbot for WordPress plugin earlier this year.  My goal was to use a site’s own data – pages and posts – to create a small language model specifically tuned to that site’s content.

In simple terms, a Markov Chain predicts what might happen next based on what just happened, relying on the last few words to suggest the next one.  It’s like following a breadcrumb trail, where each word informs the next.  When applied to the content of a website, the algorithm scans existing phrases and sentences to predict responses based on the patterns it finds.

The Kognetiks Chatbot already includes a feature called the Knowledge Navigator, which uses a Term-Frequency Inverse Document Frequency (TF-IDF) algorithm to help users find relevant content on your site.

However, for truly generative responses, the chatbot still relies on a Large Language Model (LLM) backend – either OpenAI’s or NVIDIA’s – to understand and respond to more complex inquiries.

Initial tests with the Markov Chain algorithm are promising, showing that even a very small language model, trained only on the Kognetiks.com website, could generate meaningful responses.  Here’s an example from our first test:

The source data consists of 34 pages and 92 posts, totaling 28,185 words.  The Markov Chain was set to use an n-gram length of 3, meaning it considers sequences of three words to predict the next word in the sequence.  The chatbot’s response is rough and needs further work on sentence structure, but the results are promising.

When the data set for training is as small as this, the Markov Chain algorithm’s output can be clunky.  For higher-quality responses, a minimum of 50,000 to 100,000 words may be needed to produce coherent and relevant sentences.

A website with well-structured, on-topic content – perhaps around 50 to 100 long-form posts averaging 2,500 words each – would allow the algorithm to generate more contextually accurate results.

More varied sentence structures and phrases help the algorithm capture different ways of expressing similar ideas.

Choosing larger n-grams (sequences of 3 or 4 words) will add context but also increase the data needs.

Response coherence will depend significantly on both the quality and quantity of data and how closely it aligns with the intended topic.

Focused data helps reduce off-topic and nonsensical responses, while a wider range of data can result in unexpected or even surprising answers.

So Why Even Offer This Option in the Kognetiks Chatbot for WordPress?

There are several compelling reasons to add a self-contained small language model alternative to the Kognetiks Chatbot.

1. Cost Savings: Using OpenAI’s APIs comes with a cost.  While models have become more affordable and capable, there’s still an expense involved.
2. Content Specificity: Models from OpenAI, NVIDIA, and Anthropic are trained on massive amounts of data scraped from across the web.  Although your site’s data may be included, it’s blended with countless other sources.  Think of using a large language model like trying to taste a single ingredient in a vast stew – each element is present, but they’re all mixed together, making it hard to distinguish one from the other.
3. Data Privacy and Security: With a small language model built directly into the Kognetiks Chatbot, none of your site’s data needs to be sent to third-party servers.  This keeps your content within your ecosystem, giving you control over data access and ensuring sensitive information remains secure.
4. Tailored to Your Content: An in-house model can be fine-tuned to the specific language and nuances of your content.  Unlike broader models trained on general web data, this tailored model delivers responses that reflect your brand’s voice, ensuring users receive accurate and relevant answers.
5. Faster Response Times: A self-contained model offers quicker response times since it operates locally without the need for external API calls.  This is especially valuable for high-traffic sites where quick, seamless interactions improve the user’s experience.
6. Flexible and Independent: An alternative model lets you make updates and improvements independently.  As the plugin evolves, having a self-contained model allows us to optimize performance without being affected by third-party changes or API updates.

Watch for the Next Release

As we continue developing the Kognetiks Chatbot for WordPress plugin, we’re excited to bring the Markov Chain option to life in an upcoming release.  This new feature will empower you to use your site’s own content to enhance chatbot responses without relying on external APIs.  Keep an eye out for the next update to unlock the potential of a small language model tailored to your website, offering a faster, cost-effective, and more privacy-conscious AI experience for your users.

#ConversationalAI #LLM #SLM #WordPress #Kognetiks