Finally, I have released the next version of the Kognetiks Chatbot for WordPress – Versions 1.9.6 and 1.9.7 (a quick fix to version 1.9.6). This release was not without challenges. It started with a complete overhaul of the Knowledge Navigator process and how it integrates with prompts submitted to OpenAI. Let me explain.
The Knowledge Navigator
The knowledge acquisition process is based on an algorithm called “Term Frequency-Inverse Document Frequency” or TF-IDF for short. TF-IDF is a widely used algorithm for text analysis determining the importance or relevance of words (or terms) in documents within a larger collection of documents, in this case the posts, pages and products published on your website.
The algorithm uses two main concepts:
- “Term Frequency” refers to how often a word appears in a specific document.
- “Inverse Document Frequency” measures how common or rare a word is across all documents in the collection.
The idea here is to highlight words that are significant in individual documents but not commonly used across all documents, thereby providing a way to understand and differentiate the content of each document.
An important feature of the TF-IDF algorithm is removal of “stop words”. These are the common words that usually carry little individual significance for understanding a document’s meaning or distinguishing it from another document. Examples of stop words in English include “the”, “is”, “at”, “which”, and “on”. These words appear frequently in almost all documents and do not contribute significantly to the uniqueness of the document. Due to their high frequency but low distinctiveness, stop words often have a low “inverse document frequency” score, indicating they are not very useful for differentiating texts. Removing stop words allows the TF-IDF algorithm to score only the most meaningful words that indicate what makes a page, post or product unique or relevant on your website.
The process is a multi-phase process that starts with removing the stop words from each published page, post, or product on your website.
Phase 1 – Acquire the words from each published page, post, or product, limit by max words
Phase 2 – Compute the TF-IDF for the acquired words
Phase 3 – Score words, limit by turning percentage, while building links and top words
Phase 4 – Generate a list of top words for the whole site, this is downloadable
The Knowledge Navigator Options
You have control over the frequency of knowledge acquisition. Depending on how often you update your site, you can rescore your site hourly, twice a day, daily, or weekly.
There are eight settings that control the Knowledge Navigator process. They are:
- Select Run Schedule: This setting determines the frequence upon which your site is index. The values include Hourly, Twice Daily, Daily, and Weekly. Two other setting – Now and Cancel – allow you to run the Knowledge Navigator as soon as possible while Cancel stop future runs of the Knowledge Navigator.
- Maximum Top Words: This setting determines the maximum number of unique words that are retained after indexing the whole of your website. The setting starts at 100 words (likely best for very small or new websites) up to 2,000 words (likely best for larger websites with diverse content and topics).
- Include Published Posts: Yes to include, No to exclude.
- Include Published Pages: Yes to include, No to exclude.
- Include Published Products: Yes to include, No to exclude.
- Include Approved Comments: Yes to include, No to exclude.
- Enhanced Response Limit: This setting determines how many links to internal content are included in the chatbot’s response. The range is from 1 to 10. This setting works in conjunction with the Suppress Learnings Messages setting on the Messages If the Suppress Learnings Messages is set to None, then no internal links will be returned as part of the chatbot’s response. If the setting is Random, there is a short list of various responses like “I am currently in the process of expanding my knowledge. For additional details, please check: here, here.”, where “here” and “here” are links to internal content on your website.
- Tuning Percentage: This setting determines the number of Top Words that will be indexed for individual pages. If the percentage is set at 25 (or 25%) then the top one quarter of words in any given document based on the high scoring words in the Maximum Top Words list will be indexed. The setting defaults to 25 (or 25%) which should be good for small to medium sized websites.
The Knowledge Navigator is a powerful way to incorporate the content on your website bringing it into the conversation with the chatbot. The output continues to evolve and will include options in a future release to select from several different styles of formatting. Currently set to “here, here and here”, future releases of the chatbot could include itemized lists with page titles, summaries, and links to your sites content.
Knowledge Navigator Analysis
Never underestimate the power of knowledge. Use the output from the Knowledge Navigator’s ranking of the most powerful words on your site to dig deep into understanding beath and dept of the content on your site. Download the CSV file from the Analysis tab. For those of you who are more adventurous, you can also find more details on one of two tables in your WordPress database:
- $wpdb and “chatbot_chatgpt_knowledge_base” – This table contains the list of URLs, Titles, Words and Scores.
- $wpdb and “chatbot_chatgpt_knowledge_base_tfidf” – This table contains the list of score words based on the TF-IDF algorithm. This is the content you can easily download from the Settings tab as a CSV.
A word of caution, managing your WordPress database directly through tools like phpMyAdmin is a complex task and should only be attempted by those with sufficient technical expertise. Making changes without a proper understanding can lead to serious issues with your website. If you’re not confident in your technical skills, it is strongly recommended that you seek assistance from a professional.
Migration from OpenAI-Beta: assistants=v1 to assistants=v2
A short while ago OpenAI announced a change to the way tools and files work in the Assistants API. The migration process from OpenAI-Best: assistants=v1 to assistants=v2 necessitated a change to the backend of the chatbot.
With V2 there are improvements and new features that should make using Assistant even more powerful, from ingesting up to 10,000 files per Assistant to controlling the maximum number of tokens an Assistant run uses.
Thanks to several alert users for bringing this to my attention, I’ve made the necessary changes necessary to migrate from V1 headers to V2. The latest version of the plugin will default to the new V2 header, but there is an option to switch it back to V1 on the GPT Assistants tab. You find an option at the bottom of the page called “Beta Assistant Version”.
By the end of 2024, the V1 beta will be deprecated. For those of you looking for more information on the Migration, you can find it here: https://platform.openai.com/docs/assistants/whats-new. In subsequent releases of the chatbot, I’ll be incorporating these advanced capabilities.
Daily Chatbot Interaction Limits
And finally, there is an option to limit daily chatbot interactions. On the API/Model tab, there is not an option called Chatbot Daily Message Limit with a range from 1 to 999 (which is the default).
Setting a daily limit to say 10, 50, 100 or more interactions could hinder visitor abuse of the chatbot. It should be noted that a prompt and response count as one interaction, not two. If a chatter reaches the limit, they’ll receive this message: “Oops! You have reached the message limit. Please try again later.” It’s a daily reset, so if they were to return tomorrow that would start fresh.
This is a good start to managing API costs. Of course, there’s also the Maximum Tokens Setting which also limits the scope of the response from OpenAI. Used together, these two settings should mitigate much of the risk of offering an AI-drive chatbot to the general public.
Thank You to Contributors
The evolution of the Kognetiks Chatbot for WordPress plugin is driven by the rapidly growing installed base of users, which I now estimate is close to 2,000 active installations. As always, I want to thank each of you who have taken the time share your valuable feedback. Keep it coming as it helps me continually improve and provide a better experience for all our users.
#WordPress #Chatbot #TFIDF #AI #PluginUpdate
About the Author
Stephen Howell is a multifaceted expert with a wealth of experience in technology, business management, and development. He is the innovative mind behind the cutting-edge AI powered Kognetiks Chatbot for WordPress plugin. Utilizing the robust capabilities of OpenAI's API, this conversational chatbot can dramatically enhance your website's user engagement. Visit Kognetiks Chatbot for WordPress to explore how to elevate your visitors' experience, and stay connected with his latest advancements and offerings in the WordPress community.
