As I mentioned in my original post I’m not a technical person but I have red the FAQ and to be honest I do not see the issue of duplicates mentioned anywhere. I apologize for asking something that maybe obvious but does it mean that import_logs.py has a built in mechanism to prevent duplicates?
Response from matthieu in a post from 2012 which can be found here Importing apache logs as long-term strategy
suggests that “duplicates are not ignored, this is a missing feature” but was it fixed since? Further responses even from 2015 imply that this is still an issue.