Hey,
It is just a suggestion, but you may experience the downside of not caching reports - if these loading times occur everytime you reload your dashboard, then that would be an indicator that the needed reports are re-calculated instead of fetched from a cache.
This shouldn’t happen, since yesterdays reports will not change, so you shoud get to see quick loading reports from the cache after having processed them once.
To access the report cache settings, see Recalulate data for goal graph
Long loading times on the very first request for a certain period can take that long if there is a lot of data to evaluate, that really depends on your server capacities.