We have a large site that we are trying to pull using the API’s. Pulling realtime data using one of our Dimensions pulls just fine, but when we try to pull any other reports using the Dimensions that we have setup it just times out.
For example, if we call VisitsSummary.get with a period of day and calling for last7 days the API returns data just fine. But if we add a Dimension to that call it times out. Can anyone help with maybe a database index to allow these to run?
Additional environment details…
Daily pageviews is approx 3.9 million a day. Visits is around 1.8 million a day.
You probably mean that you segment the result for a dimension, or did I get this wrong? In this case, the result set is not fetched from the archived tables, but is created from raw data (and therefore, it’s slow/times out). To solve this, create the often-used segments in the segment editor for that site and set it to “pre-processed”. With this, the segmented result set is also processed in advance. (See https://matomo.org/docs/segmentation/#improve-the-performance-of-your-site-while-using-segments)
That might work. We have not used segments before but will play around with them. Guessing we are ok creating thousands of them without major performance impact.
I’ll have to manage your expectations here - you can’t / shouldn’t create thousands of (pre-processed) segments, since every segment will be archived on its own. This will make your archives bigger in size and take some time for processing. In your case (very high load), you’ll have to experiment what’s possible and what’s not.
Segments that don’t have to be pre-processed don’t need to be set up in the segments editor but can be injected to the request any time.
OK thanks… basically what we have is a large application that runs around 3500 websites. We are collecting analytics all together and looking to be able to report back on a site by site basis, but still keep our roll up of everything.
Curios on if we would be better breaking up into 3500 separate sites versus trying to pull individual sites like we are today. We like having rolled up as one large system for some tag management/tracking reporting but maybe we are causing our own issues by doing this.
Hard to advise here, without knowing all requirements / architectural details. But in most cases, I’d go with different profiles / site-ids per website.