Friends, I have ~1000 older access.log files I’d like to import. ~20MB per file, ~50K lines each.
Should I concat the files before importing or should I import them one by one?
Site question:
Is there a duplicate checker in case I happen to import an already imported file again?
Thanks in advance!
Seems that concating files first was a good idea.
Ended up with ~25000K lines per file which worked pretty nice. Importing took ~1h per log file.
Make sure to set nginx’s client_max_body_size to a decent size as well as phpfpm’s post_max_size, upload_max_filesize, memory_limit and max_execution_time else might see a lot of 413 Request entity too large errors.
Also stopped the archiving cron job prior to importing although I’m not sure if this was required.
Cheers!