You are right that there was no database upgrades between the two releases:
So in theory downgrading should indeed just be overwriting the files with the ones from https://builds.matomo.org/
The only thing is that you might need to update the database version field:
UPDATE matomo_option set option_value = '4.3.1' where option_value = '4.4.1';
But I won’t promise anything and recommend backups before.