LLMs can make your business soar or sink, so tread carefully.

What should a Business Leader know before investing in AI?

ChatGPT is everywhere. Chances are, you’re already using it for everyday tasks – its language skills are unmatched. Using it to fix spelling in your email is really easy, but let’s imagine transforming it into a frontline warrior – a chatbot that not only chats but also solves customer problems. We will use this business application of a chatbot as an example in this article.

Applying chatbots to more complex business processes is very tempting but can be full of pitfalls. There are many aspects to consider when building such an application. From the privacy of your data to the cost of implementation. And from the correctness of the responses provided by chat to the maintenance in the future. In this article, we will take all the challenges one by one, describe them and give a guideline on how to approach them pragmatically. This leads to building a successful project, providing a return on investment.

What is a Large Language Model? What LLM is not?

Remember the last time you chatted with ChatGPT? The conversation was smooth, and it even knew who the US president was. But pose a niche question about, say, ‘how to change the invoice number in my system,’ and it might stumble. Even more, it might confidently tell you something off-base. 

I bring that up because it’s essential to understand that ChatGPT or any other general LLM is not a general source of knowledge. Picture this: You wouldn’t ask a history professor about advanced rocket science, right? Similarly, out of the box, an LLM may not ace a deep dive into your specific business.

But it becomes a potent tool when you plug it with information about your domain and instruct it on what to do with that knowledge. A tool that might appear intelligent.

Getting correct answers

Imagine AI as a car and data as its fuel. Without the right fuel, it won’t take you where you want. Suppose you’ve set up your AI for customer support, and someone asks, ‘How do I get the invoice?’. Instead of the AI drawing a blank or guessing, it should know exactly where to fetch the answer – like a librarian who knows which shelf a book is on.

Data is the fuel for AI. For the chat to answer questions specific to your organisation, you need to provide it with the relevant information. Let’s say you want it to serve customer support purposes. Let’s assume the user asks the system, ‘How do I get the invoice?’ 

The most standard approach to this problem is building an application that searches for relevant information and provides it to the chat. In our case, the application seeks information about the invoicing process. All searching is done in vector databases, which are great tools for finding the appropriate information in text documents. No wonder they’re becoming more popular since the rise of LLMs. Once it finds the clue, it hands it over to our chat. 

Let’s stop here and amplify the message – general purpose LLM does not know much about your business. Here’s the gist: While AI is great with words, it only knows the specifics of your business if you tell it. Feed it the correct details to transform it from a general chat buddy to a helpful assistant. Think of it as training a new employee.

In the simple case described above, most of the ‘magic’ is done by vector databases. But in some cases, just providing the information ‘on the fly’ is not good enough. For the LLM to perform better, you must fine-tune the vector embeddings or the LLM model itself. This is more advanced, so we will describe it in a separate article.

Privacy concerns

Imagine sending a personal letter and having someone else read it. That happens when you use ChatGPT or similar tools – the data goes to their home base. No worries if you’re chatting about the weather. But what if it’s confidential customer details? That’s where things get tricky.

Picture a customer sharing personal info on your support chat. Now, where does that data go? Outside your walls? And are you breaking any rules by letting it?

Confirming the legal aspects is the minimum you should do. But if you are not allowed to send the data to third parties, you need to consider privately hosted LLM so your data never goes outside your data centre. The selection of open-source LLMs available to use as privately hosted is broad. Broad selection means you have many options, but at the same time, you need expertise to make wise selections. Especially since the market is hot and keeping up with all the changes is challenging. The right decision requires careful consideration of aspects like the model’s performance on your hardware and the ability to scale up and down if needed.

A cherry on top? Going private means you aren’t tied down by vendor lock-in. You can move your infrastructure to your own data centre or cloud. It’s a critical point in the context of mitigating the risk of a vendor making drastic changes in the pricing or in the offered service itself.

Cost and defining the scope of the project

When building a house, you wouldn’t start with the roof or fancy decor, right? You’d plan the foundation and budget from day one. It’s the same with an AI. If your supplier changes the rules, you’ve got expenses like engineering time, hardware, cloud, and possible surprise fees.

When building an AI application based on Language models, it’s essential to control the project’s scope and expected costs from day one. 

It’s easy to dream big. After all, you’ve got all the tools at your fingertips. But to deliver the business value, you need to analyse the problem you are solving and set realistic milestones for the AI project. And it’s essential to start with issues which are common and easy to solve. E.g. When building a customer support application, it makes sense to begin by addressing issues which are repeated over and over again. You’d know them if you chat with your support team or skim through past tickets. If you approach the problem correctly, it’s likely that after two or three iterations with your product, you realise that you have solved 80% of the issues. Suddenly, you’ve got a solid ROI and can decide if you want to add those fancy trims or if your house is good enough.

Maintenance

Keep in mind that the more complex the application you are building, not only the implementation cost but also the cost of using it in the future increases – the maintenance and the price you pay for hardware or API calls. The maintenance might include aspects such as deployment complexity, testing, reacting to changes introduced by third parties or feeding your application with newer data.

That’s another reason to think early about the scope and be flexible during the implementation. So, before diving in, sketch out your costs. Ask yourself: How much is each chatbot chat worth to you? How much would you pay to help one customer if it’s a support bot? What conversion rate justifies the effort and cost of maintaining a chatbot if it’s for shopping? Knowing your numbers helps you steer clear of unwanted surprises.

Summary

As outlined above, it’s critical to understand the challenges when selecting the scope of the LLM project. But with a clear understanding of potential roadblocks from the beginning, you’re paving the way to the ROI you expect. The spectrum of challenges in implementing LLM for solving business problems in your organisation is vast, and you should:

• Ensuring correct behaviour makes it essential to provide your LLM with correct, well-structured data.

• Addressing privacy concerns means understanding the legal prerequisites. Remember that you have options of hosting LLM in your Data Center, which simplifies the legal aspects but introduces extra technical challenges.

• Budgeting for a project means considering all costs: development, API charges, and maintenance. It’s important to verify the assumptions after every milestone of the project and make sure the scope and the cost are under control.

• Thinking ahead is essential, so estimate maintenance efforts. Ensure these efforts align with the expected benefits of its integration into your organisation.