Integrating AI into business software: how to start

What does integrating AI into business software actually mean?

AI in your business software is rarely a separate product you bolt on. It is a feature inside software you already run: a quote that drafts itself, a search that understands plain-language questions, a ticket that gets summarised and routed on its own. The AI sits under the hood; the user just sees a button or a text field.

In practice, LLM integration in software comes down to three recognisable paths. You can call an LLM API from your existing code, you can build RAG so the model answers from your data, or you can put an AI chatbot on your own knowledge base. They are not mutually exclusive; a serious solution often combines them.

Path 1: connecting an LLM API to your existing software

The simplest starting point is calling an LLM API from your application. You send text to a model such as those from Anthropic or OpenAI, get an answer back and use it in your flow. Think of summarising long emails, drafting a reply, or classifying incoming requests.

This path is quick to set up and easy to test. You decide which text you send and what happens with the answer. The model only knows what you put in the prompt, so for general language tasks this works well. As soon as you get questions that can only be answered with your specific data, you hit the limits and need RAG.

• Good for: summarising, rewriting, classifying, drafting.
• Fast to build and easy to scope.
• Limitation: the model only knows what is in the prompt.

Path 2: RAG on your own data (in plain terms)

RAG stands for retrieval-augmented generation. In plain terms: before the model answers, your software first looks up the relevant pieces in your documents and passes them along as context. So the model does not just make something up; it answers based on what it just found in your manuals, contracts or database.

This is how you connect ChatGPT or another LLM to your own data without retraining the model. You turn your documents into searchable chunks, store them, and for each question retrieve the few most relevant fragments. Most of the work is not in the model but in that retrieval layer: cleaning your data, chunking it well and feeding it back reliably. Building a RAG application is therefore mainly a data and integration job.

If you want to combine this with actions in your systems, it moves towards having an AI agent built. And if you are unsure whether to adapt existing software or build something new, first read about having custom software made.

Path 3: an AI chatbot on your own knowledge base

An AI chatbot on your own knowledge base is essentially RAG with a conversation interface around it. The user asks a question, the bot searches your documentation, FAQs and product pages, and gives an answer with sources. This works well for customer service, an internal helpdesk and onboarding new staff.

The trap is that people think they are building a chatbot, while the real work is in the knowledge base and the escalation rules. Decide up front which questions the bot handles on its own and when a human takes over. A bot that honestly says it does not know is more useful than one that confidently invents an answer.

If you want to use this more broadly to take work off people's hands, also look at improving business processes with AI.

Where does your data go? Being honest about GDPR

The moment you send text to an external model, that data leaves your own system. With the large US providers it usually travels to servers outside the EU. If you process personal data along the way, GDPR applies in full: you need a data processing agreement, you must know where the data sits and how long it is kept.

This is solvable, but it requires choices. Some providers offer EU data locations and a commitment that your input is not used for training. For sensitive data you can choose a model that runs inside the EU, or filter personal data out of the text beforehand. The Dutch Data Protection Authority and the explainer at gdpr.eu are good starting points. Arrange this at the start, not once it is already running.

• Know where your data goes and whether it is used for training.
• Put a data processing agreement in place for personal data.
• Consider EU hosting or filtering personal data beforehand.

When you should not add AI to your business software

Not every problem is an AI problem. If a simple rule, a filter or a decent search solves the question, a language model only adds cost and uncertainty. AI is strong at language, nuance and variation, and weak at tasks that must be exact and repeatable, such as arithmetic or fixed business rules.

Skip it too if your data is a mess. RAG on unfindable, outdated or contradictory documents gives poor answers, no matter how good the model is. And if nobody checks the output in high-stakes cases, such as legal or financial decisions, a smooth but wrong summary is more dangerous than no summary. In that case start small, with an MVP built to test whether it really works.