Yo all,
we have a pretty big Piwik installation at our company and following continuously integrated Piwik updates need us to compare testing (updated) server with production server.
We must check for database data integrity of all the metrics on custom sites arbitrary chosen. So i wonder to write a test suite that could be released as plug-in in the future.
I would check using apis calls between the two servers we’re getting the same results, and why not, testing functional UI integrity but I bet It’s already in by Piwik’s test suite.
I would check using apis calls between the two servers we’re getting the same results
we actually have thousands of those API calls that we do for each commit in Piwik and check the XML output to expected output. For example check one of the pull request worked on and you wlil see the XML expected test files. see https://github.com/piwik/piwik/pull/6589
So piwik releases should behave well in each upgrade and if they don’t, it would be great to improve our existing test suites and you are most welcome for this. We have lots of ideas see: https://github.com/piwik/piwik/labels/c:%20Tests%20&%20QA
feel free to let us know in those tickets your feedback
Thanks Matt for your fast reply, useful as usual. Reading it I can understand you can rely on data integrity by unit-tests.
In the past we did an upgrade of mayor release from v1.9.x to v2.1.0. Our console execution maybe hung/died after about half an hour of database upgrade, doing human functional tests we discovered we lost Frequency metrics data before 2012-02-xx (can’t remember).
In the meanwhile piwik went 2.3.0 so after this sadness we restarted again upgrading 1.x to 2.3.0 this time flawless, without feeling any data integrity fault.
Due to this upgrade is now felt a bit tricky, that’s because we ran console using su www-data -c ./console but without any output on the screen during the process. (I’m always using console as root on dev-VM)
The problem is minor because there’s a backup and we can repeat the task, but you see, this operations it costs each try at least half an hour on 2 servers (application + rdbms) resources usage. More, I felt human check against data a bit messy because data is huge and it could be automated so humans could spend time doing more edifying things.
In my opinion I might try running console as root doing the core:upgrade but in the meanwhile I would also give them a visual feedback of data integrity between the two installations.
Reading Update guide, this procedure looks like easy and straightforward but in the reality environments could differs a lot making the difference. I mean: maybe someone has two or more dbms master/slave, or maybe some else has two or more balanced application servers or maybe the whole ensemble is on the cloud, so upgrade procedures could differ.
In the meanwhile piwik went 2.3.0 so after this sadness we restarted again upgrading 1.x to 2.3.0 this time flawless, without feeling any data integrity fault.
Due to this upgrade is now felt a bit tricky,
Well actually we fixed that bug after 2.1.0 so that’s why it worked afterwards… it’s not really flaky just that we fixed a bug
Sorry for taking up an old post but, they came again with the same subject “functional testing on data structure”.
Now I’m wondering I should write a plugin with no frontend, only console UI. This plugin should connect to Reports APIs in the test environment and production, then should compare different metrics between the two environments.
I agree would be un needed because of Piwik’s TDD CI but, they just want to see something tangible.
Is there some inspiring example outta there? If it can be configured, could be an interesting plugin for the Marketplace?
