How can i get the RAW DATA by connecting to a 3cx database

Status
Not open for further replies.

clebopo

Free User
Joined
Sep 23, 2021
Messages
4
Reaction score
0
I require the raw call data from 3cx. I have used the reporting portal but it returns less records if pulled for more than a week. I need to find a way i can connect to the 3 CX database to retrieve the raw files.
 
We generally recommend against connecting to 3CX's Database in any way, but if it is the Call Data you require I recommend using the 3CX CDR instead: https://www.3cx.com/docs/cdr-call-data-records/

This of course would mean that you will only be able to retrieve call data that has been generated after CDR has been enabled.
 
  • Like
Reactions: Evolute IT
Expanding on @ChrisC_3CX post, the CDR data is an optional feature of 3CX. You have to turn the feature on and it will not go back and provide historical information. It starts generating data AFTER it is enabled. 3CX creates a text file that summarizes each phone call. The data is saved when the call is disconnected. While it includes very useful information, it does not include things like queue statistics. You enable the feature in the 3CX management console (Settings --> CDR).

I encourage you to turn on the feature and take a look at what information is provided. Be sure to add ALL the columns available. I recommend configuring CDR to create a separate file for each phone call. If this has all the information you need, and you don't need the data in real-time, this will be a good solution. There are some minor obstacles to overcome. You will need access to the files which can be complicated if your 3CX server is hosted in the cloud, but there are ways to access the files. The other minor challenge is automating the parsing of the data and getting the data into a database.

@ChrisC_3CX's characterization that 3CX generally recommends against connecting to the 3CX database is perhaps understated. In my experience, 3CX STRONGLY OBJECTS to you accessing the PostgreSQL database directly, and for good reason.

Let's take an example I have seen personally. The most common administrative tool used to connect to PostgreSQL is PGAdmin. This tool looks harmless at first glance, but in truth you have to be VERY careful with this tool. PGAdmin will let you connect to the database and view/edit the data. What is not initially obvious is that simply clicking on a table name locks the table and all the contents. Can you imagine the amount of problems this will create if 3CX is trying to write to this table and it cannot? At worst it could crash 3CX and at best it could prohibit the data from being written to the database. Either scenario can have some pretty severe consequences all because someone simply clicked on a table name.

For this reason, I encourage you to look closely at the CDR files and see if this data meets your needs. You have already concluded that exporting report data is insufficient for you, but I would encourage others to look at this option too.

The fact is, however, that the CDR data is just a summary of the phone call. It doesn't include all the legs of the call, nor does it contain any useful information about queue statistics. If you need that data, then there is really only one option -- get the data directly from the database.

Let me dispel one misunderstanding about the PostgreSQL data. It is not necessarily real-time data. I'm not certain of the specifics of how 3CX works in this regard, but by observation it is clear to me that 3CX will at times cache writes to the database. I have seen writes to the database delayed by several minutes on occasion. So, if you are looking for real-time data, accessing PostgreSQL is NOT the answer.

The following is a shameless plug for our 3CX Exporter tool. About 10+ years ago it became very obvious that access to the underlying 3CX data was going to be necessary if we were going to build custom reports. We have spent the last 10 years refining our tool to make sure we do not have any adverse impacts on 3CX. Exporter uses a carefully crafted read-only connection to 3CX. This refinement included throttling how much data we pull at any given time to ensure we don't impact 3CX performance. Further, we replicate the data over a secure TLS (HTTPS) connection which includes the need for JWT token based authorization. In addition, Exporter also parses the optional CDR data in real-time and replicates this data to SQL/MySQL thus eliminating the complexities of remote access and automation.

But let's address the elephant in the room. Does 3CX endorse the use of Exporter? 3CX does not have any certification process for 3rd-party tools, so they don't endorse any non-3CX tools. For our part, we have met many, many times with 3CX and discussed how Exporter works and even offered to discontinue the product if that would make them happy. In every case, the answer has always been "you do not need to do that". While I cannot speak for 3CX, I can can make an educated guess at their reasons for turning a blind-eye to Exporter. The alternative is to encourage everyone to figure out all the pitfalls we have addressed over the last 10 years themselves... and all the related issues that would cause for 3CX. In contrast, Exporter is a known product that has been in use for over 10 years all around the world without any issues.

3CX has announced a new REST API and Gateway service (scheduled to be released tentatively later this year). While Exporter is a valuable tool for our company, if the new 3CX tools provide direct access to the 3CX data, I'm all for it. It will make all of our lives better. I encourage 3CX to make the data available in a fully supported way.
 
The guide you linked to involves connecting to 3CX's database something that, as mentioned previously, we strongly recommend against and that will also render the 3CX Instance unsupported. I'd refer to @VoIPTools extremely detailed response above.
 
  • Like
Reactions: Evolute IT
I found this online(below link0. Is this not helpful. I also found that if i retrieve data for a longer period ( more than 5 days) the records are limited to 20000+- rows.

https://888voip.com/getting-3cx-raw-call-detail-data/
As stated don't use pgAdmin but psql instead, this will allow you to execute non-locking queries. Try using LIMIT .. OFFSET to get the first 20k then the next 20k, if you know a programming language you can create a tool to paginate to CSV if that's what you want. Alternatively you can use row_to_json to export things in the DB as json.
We all need a REST API desperately and I hope that there is some truth to the rumor stated by VoIPTools that it's coming.
 
  • Like
Reactions: Evolute IT
We all need a REST API desperately and I hope that there is some truth to the rumor stated by VoIPTools that it's coming.
It is, and the first part is WebMeeting.

No ETAs tho.
 
We have a REST API :) if you need it now.
 
Super detailed explanation above in this thread.
 
  • Like
Reactions: Evolute IT
Super detailed explanation above in this thread.

The following is a shameless plug for our 3CX Exporter tool. About 10+ years ago it became very obvious that access to the underlying 3CX data was going to be necessary if we were going to build custom reports. We have spent the last 10 years refining our tool to make sure we do not have any adverse impacts on 3CX. Exporter uses a carefully crafted read-only connection to 3CX. This refinement included throttling how much data we pull at any given time to ensure we don't impact 3CX performance. Further, we replicate the data over a secure TLS (HTTPS) connection which includes the need for JWT token based authorization. In addition, Exporter also parses the optional CDR data in real-time and replicates this data to SQL/MySQL thus eliminating the complexities of remote access and automation.
Got it, so you _do_ access the database directly. I was thinking of creating a similar application, how do you deal with making changes and updating them live -- are you restarting 3cxConfigManager?
 
If I understand your question, you are asking how we make changes to 3CX data? The answer to that question is "never, ever". That would be instant flagging of your PBX as "unsupported". That's why we use a read-only connection.
 
If I understand your question, you are asking how we make changes to 3CX data? The answer to that question is "never, ever". That would be instant flagging of your PBX as "unsupported". That's why we use a read-only connection.
I see, so your API only supports GET requests... Hopefully 3CX's REST API will let us use this product how the clients actually want, and allow us to create some mad automation tools as well
 
Let me be more precise. Our REST API uses a read-only GET when replicating 3CX data (because there is no 3CX API for replicating history). But we also leverage the 3CX Call Control API to create and update objects. But lets be clear... that's far, far different than updating the database directly. We use THEIR API to create/update extensions for example. We simply expose their API through a REST interface. In addition, we subscribe to real-time events (again through the 3CX Call Control API) and expose those events through SignalR (websockets under the covers).

At least thus far, 3CX has not removed or replaced the long-standing 3CX Call Control API in version 18. I assume 3CX will continue to use this API for their own internal purposes. But it will be interesting to see what capabilities they expose through REST and the announced Gateway service. I'm hoping they supplant the majority of our APIs with a supported API, but I suspect they won't provide a replacement for ALL our APIs. For example, we have APIs for automating the publishing of CFD applications, and copying, deleting, and moving files, and many more things important to application developers but likely outside the scope of this new API.

For my part, I would prefer to focus on building applications rather than building APIs, so I hope 3CX provides a robust and inclusive API and Gateway service. For now, we do not have any information about what will be possible.
 
Status
Not open for further replies.

Forum statistics

Threads
111,977
Messages
590,094
Members
164,906
Latest member
Nari