Receptionist response time

[steve-sydney]

I've set up a receptionist,
General feedback has been the receptionist thinks too long before answering.

Have I setup too many KB files? (7)
Do I need more CPU/RAM? (AWS t3a.small)
Higher token tier? (Tier 1)
Any advice?

 

[Kevin Attard Compagno]

One possibility is that being in Tier 1 makes you reach your token threshold very quickly - there are TPM (token-per-minute) and RPM (request-per-minute) rate limits to be taken into consideration.

We will be VERY shortly releasing a blog post that discusses this. One thing you can try is to pay $50 or $100. Typically, that shifts you into Tier 2 or Tier 3 directly.

 

[steve-sydney]

Thank you Kevin, Yes $100 spend will push us up to T3. I'll do that $20 was a bit tight.

Meanwhile i have eliminated instance sizing by upgrading t3a.xlarge. - but no change ;(

Are you confident its token throttling? the agent does respond - the delay is consistent like its being polite and making sure i have finished talking. It may be Sydney living we tend to speak before the other person has finished their sentence which makes it feel slow to respond.
Is it possible to tweak the turn based detection? or i should fiddle with politeness and instruct agent to interrupt when it thinks it has the answer

 

[Kevin Attard Compagno]

if the delay is a question of a few seconds, then it is VERY UNLIKELY to be token throttling.

Keep in mind that when the caller finishes talking, the agent needs to:

• insert a short delay to ensure the caller has finished
• send the wav to openai
• wav needs to be transcribed
• reasoning needs to be applied
• response needs to be built and sent back to the PBX

...and each hop introduces some delay. Some amount of delay is simply inevitable.

In your world, how "long" is "too long" for the AI Agent to "think"?

 

[steve-sydney]

4 seconds to start the reply feels too long to me but Im always in a rush. Is that typical or longer than usual?

Tier 3 didnt speed up the response time.

 

[Kevin Attard Compagno]

4 seconds, nothing unreasonable.

 

[steve-sydney]

So 4 seconds is the norm?

 

[Kevin Attard Compagno]

Yes

 

[steve-sydney]

But I've tested someone else's 3cx virtual receptionist and the response time was much better. About 1.5 second to respond.

 

[Kevin Attard Compagno]

As I mentioned earlier, there are several points where delays can be introduced. Your response time will be a function of those. The number of KB files could also contribute. Did you try an agent WITHOUT KB to compare times?

 

[steve-sydney]

How about some feedback from the community?
How many KB articles are people generally using? How big are they in bytes?
How long are their response times.

 

[nikolascx]

I think 4 seconds is actually quite reasonable in context. If you compare it to the “old way”, users would spend 60+ seconds spelling things out (“A for Alpha, B for Bravo…”) and then wait another couple of minutes while the system pulled CRM records – so we shouldn’t forget how big a step forward this already is compared to just a few months ago, when much larger delays were the norm.

That said, if you’re aiming to optimize further, I’d look closely at the knowledge base content and structure. If you’re feeding the system a mix of Excel sheets, PDFs, and raw text files, that can slow down embedding and retrieval. Converting everything into clean, structured markdown (or otherwise normalized text) can help improve retrieval efficiency and potentially reduce response times.

LLM's like Text.

A couple of practical tests: try disabling VS to measure baseline latency, then re-enable it to see the impact, and also experiment with simplifying or restructuring your KB (especially PDFs, which tend to perform worse).

Also try VS with one text file, and with 20 texts and compare.

OpenAI has one of the best Embedding models in the world right now. We are being supported by a very good infrastructure and it happens to have a global reach..

 

[steve-sydney]

Hi Nikolas,
I apologize if 3cx organization has taken this enquiry as a criticism, its not. I really want to sell this but i have to be as good if not better than my competitors. I have tested a demo 3cx receptionist from another company with 1 second response times and I'm just trying to figure out how i can achieve similar and where Im going wrong.

All my KB's are txt files.
Is there a particular formatting which the ai reads faster? perhaps there's a template for formatting?
is it better to have multiple smaller kb's or a single kb which holds all the info?
Im currently using 7 txt files and here is an example of the formatting and size of each file.

your feedback on how to make this more efficient is helpful.

 

[KyriacosS_3CX]

Hi,

If the knowledge files represent different content categories, then splitting them can benefit speed - the description you give the base is the index whereby the AI searches to find relevant knowledge - by splitting and describing what's in each one you help it home-in to the right data faster, and also minimises "cross-talk".