Hi,
we need to ingest data from the MySql database we set up for Matomo tracking and we need to understand how to do that in an effective and efficient manner.
In particular, we are interested in log_visit and log_link_visit_action tables and how it is possible to identify updated rows for those tables.
We need to understand:
- how to compute a delta of new inserted and/or updated rows?
- do we need to perform an append or an upsert operation when loading, i.e. are they append only tables or rows can be updated?
- if we need an upsert operation, do we have to use timestamp or watermarks to only take updated data?
Could anyone help us finding the right strategy to compute this delta?