Can you clarify what specific security concerns you have?
The AI agents themselves add quite little additional concern. The relevant scenarios are along the lines of - If contact "John Smith" calls you, the name needs to make it to OpenAI so it can be voiced for the greeting.