MrJibus
October 17, 2022, 9:22pm
1
Hi there,
I am using the matomo API to get insights data and often it takes 20 seconds to 2 minutes to load the data. After the data is showed, the same request is very fast (cache probably).
Is the Insights API processing CPU/RAM consumming ? I am using the getBulkRequest, since i have other data to retrieve.
The cron is set every hours.
I don’t have a lot of trafic, but when i do the request and monitoring the activity on my console (htop), there seems to be no problem.
Shoud I increase the ram memory limit ? (currently set to 128 ?)
Thank you
Which request do you use?
MrJibus
October 18, 2022, 9:32am
3
Hello Philippe and than you for the response.
Please find below the full request :
http://matomo?token_auth=token_auth&module=API&method=API.getBulkRequest&language=fr&format=json&
urls[0]=method=VisitsSummary.get&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[1]=method=VisitsSummary.getActions&idSite=1&date=2022-09-18,2022-10-17&period=day&segment=dimension2==DIMENSION2&
urls[2]=method=Actions.getPageUrls&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[3]=method=Referrers.getReferrerType&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[4]=method=Referrers.getAll&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[5]=method=UserCountry.getCountry&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[6]=method=UserCountry.getCity&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[7]=method=DevicesDetection.getOsFamilies&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[8]=method=DevicesDetection.getBrowsers&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[9]=method=DevicesDetection.getType&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[10]=method=Insights.getInsightsOverview&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[11]=method=VisitsSummary.get&idSite=1&date=2022-08-19,2022-09-17&period=range&segment=dimension2==DIMENSION2&
urls[12]=method=VisitTime.getVisitInformationPerServerTime&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2&
urls[13]=method=Provider.getProvider&idSite=1&date=2022-09-18,2022-10-17&period=range&segment=dimension2==DIMENSION2
Matomo is running on a single VM with 2 CPU & 2GB RAM and a separate database (2 VCPU / 1Go)
I just saw that all websites are making a total of 450K visits / month.
Monitoring activity CPU/RAM on the database and VM seems normal.
But maybe it’s just not enough anymore ?*
Which request are too long?
MrJibus
October 19, 2022, 1:30pm
5
Sorry for the delay. Following your comment I have made a few changes.
Normally, I would request the data from today, but it seems that the data may no be archived. I have a changed the period from today to yersterday.
Cron is setup every hour.
Then, I made three tests on different websites account, this is what I got in term of response (in ms). First website is a very small one, the two others are making 40K visits.
Request :
`&module=API&method=API.getBulkRequest&language=fr&format=json&urls[0]=method=VisitsSummary.get&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[1]=method=VisitsSummary.getActions&idSite=40&date=2022-09-19,2022-10-18&period=day&segment=dimension2==DIMENSION2&
urls[2]=method=Actions.getPageUrls&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[3]=method=Referrers.getReferrerType&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[4]=method=Referrers.getAll&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[5]=method=UserCountry.getCountry&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[6]=method=UserCountry.getCity&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[7]=method=DevicesDetection.getOsFamilies&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[8]=method=DevicesDetection.getBrowsers&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[9]=method=DevicesDetection.getType&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[10]=method=Insights.getInsightsOverview&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[11]=method=VisitsSummary.get&idSite=40&date=2022-08-20,2022-09-18&period=range&segment=dimension2==DIMENSION2&
urls[12]=method=VisitTime.getVisitInformationPerServerTime&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2&
urls[13]=method=Provider.getProvider&idSite=40&date=2022-09-19,2022-10-18&period=range&segment=dimension2==DIMENSION2`
Notice that I am request VisitsSummary.get for the previous period (request 11)
VisitsSummary.get : 900 / 13330 / 11150
VisitsSummary.get & Insights.getInsights can take quite some time to get.
How I can get better results ? More CPU ? More RAM ? Different request ?
Hi
I did not see first that there was only one single request…segment=dimension2==DIMENSION2 is a segment already known (and calculated) by Matomo?
I saw that in the last version of Matomo some optimizations have been made on requests:
But not sure your problem is coverd…
What is your server configuration in term of memory/CPU/disk/nbr of servers, etc.?
MrJibus
November 2, 2022, 5:15pm
7
Hi Philippe,
Sorry for the delay (again). I have updated Matomo to the latest version (4.12).
Server configuration is : 2 CPU / 2GB - PHP 7.3 - Max memory php 128Mo.
How can I know if dimension2 is already calculated ? Is is during cron ?
Not dimension2 , but the segment using dimension2
MrJibus
November 7, 2022, 3:27pm
9
Yes it is, the form is making an autocomplete, suggesting cache data.
I don’t see what could be the problem…@innocraft can give some clues?
MrJibus
November 9, 2022, 9:40am
11
Hello Philippe,
Thank you for your time.
Let me run more test on my side and I come back to you.
For example, I have increase the php limit and it seems that i have some improvements.
1 Like
MrJibus
January 23, 2023, 9:49pm
12
Bump on this.
Turns out, the segment was NOT already calculated. I have a to create a custom segment in order to get the data with dimension2 calculated.
1 Like
Then this condition was not meet?
MrJibus
January 24, 2023, 12:34pm
14
Yes ! If you’re using the API, without creating the segment, the segment is calculated live.
