Knowledge base attached = slower responses?

[patrickb_de]

Hi, i wanted to quickly ask something about AI agents.

Lately I’ve noticed that they take noticeably longer to respond as soon as a knowledge base is attached. If I put that same information directly into the prompt as ##Knowledge base##, responses in many of my agents are noticeably faster.

This seems to happen regardless of which model I use.

For now, I’ve found a temporary workaround:

• Frequently asked / core information goes directly into the prompt as ##Knowledge base##
• Deeper / less commonly needed knowledge stays in the knowledge base
• I have the AI bot tell users that some answers may take a bit longer, so they know what’s happening if a response takes ~5 seconds

Does anyone have ideas on how to speed this up? Is there any way to cache parts of the knowledge base and make them directly accessible from the prompt?

 

[KyriacosS_3CX]

Hello,

The advisable strategy for fast Knowledge base retrieval is to make sure the data provided in knowledge sources is properly split and well structured.

If you sell cushions and pillows, do two sources, whose description mentions the product type, this is the index, so that the AI can zero in on the data and spent time processing the smallest set possible. Splitting between files works OKish for smaller sets, as long as the content of the file is easy to determine by the proximity metrics. Two CSVs, one for each, with the first column being the product type, or a clear line at the top, explicitly defining the content, making sure only the relevant one returns a hot enough match.

 

[patrickb_de]

Thanks, that makes sense.

For some use cases, breaking information into lots of small files is manageable, but for larger or frequently changing datasets, it becomes a bit cumbersome.

So my follow-up question is: are there any plans to support larger or not so structured files while still keeping response times fast in the future? Also, are there any plans to make knowledge base files editable directly in the browser, so they don’t need to be deleted and re-uploaded every time they need updating?

 

[KyriacosS_3CX]

Hi,

You can always try creating a (or using the PBX created) knowledge base on OpenAI (and future options), to judge the response speed and tune.
Also for file editing - the PBX would need to have build in editors for all kinds of data, processing files that it does not and should not be storing locally, so not really - again this element might be easier to streamline by manipulating knowledge bases directly with OpenAI or using a custom toolchain build to use the same API key.

 

[nikolascx]

Hi Patrick,

Your approach makes sense, but to give better advice we would need to understand how your Knowledge Base is currently structured.

Can you share a few details?

• How many files are attached to the Knowledge Base?
• What file types are you using?
• Roughly how large are the files?
• Are they split by topic/product/category, or is most information in one large file?
• If possible, can you share a small extract or anonymized sample of the content/structure?
• Do you have one knowldge base and everything in it?

This would help us see whether the delay is caused by file size, structure, unclear source descriptions, or the way the information is grouped.

As a general rule, a good Knowledge Base should be split into clear topics, with each file/source having a specific purpose. Also try to avoid very large mixed files where unrelated topics are grouped together, because that makes retrieval slower and less accurate. For example one trick is to create multiple knowledge base by segregating information in each (no overlap of info) and put a clear description so the AI will select the knowledge base based on the description.

Once we see a sample of how it is structured, we can give more specific recommendations.

 

[patrickb_de]

Hi @nikolascx, I’ll open a ticket and send you more details and example files later today. I’m just waiting for my customer to respond with the original files. I’ll ask @AvraamM_3CX to forward them to you.

 

[nikolascx]

We are actually going to release some documentation on how to make knowledge bases.

However, one additional tip - LLM's like Markdown as their favourite Language.

Microsoft release this python project - markitdown. https://github.com/microsoft/markitdown

MarkItDown currently supports the conversion from:

• PDF
• PowerPoint
• Word
• Excel
• Images (EXIF metadata and OCR)
• Audio (EXIF metadata and speech transcription)
• HTML
• Text-based formats (CSV, JSON, XML)
• ZIP files (iterates over contents)
• Youtube URLs
• EPubs
• ... and more!

Essentially what this does is, you give it all your data, and tell it to markitdown. then you can upload this marked down content to your vector storage. It's great and retrieves 3x faster!!