Log file import script stops working after first time

Support & Bugs
1

My server logs imported perfectly into my database the first time. Now whenever I try to import more server logs, the script runs, but no data is added to the database. I have to delete the database, re-install piwik, and then import the server logs each day in order for it to work.

Here is an example of what happens when I try to import a 2nd log file:


2486 lines parsed, 279 lines recorded, 276 records/sec (avg), 279 records/sec (current)
5051 lines parsed, 1394 lines recorded, 692 records/sec (avg), 1115 records/sec (current)
6751 lines parsed, 2978 lines recorded, 984 records/sec (avg), 1584 records/sec (current)
7459 lines parsed, 4078 lines recorded, 1013 records/sec (avg), 1100 records/sec (current)
7459 lines parsed, 5466 lines recorded, 1087 records/sec (avg), 1388 records/sec (current)
Purging Piwik archives for dates: 2014-05-12
To re-process these reports with your new update data, execute the following command: 
 `piwik/console core:archive --url=http://example/piwik/`
Reference: http://piwik.org/docs/setup-auto-archiving/ 

Logs import summary
-------------------

    5797 requests imported successfully
    5231 requests were downloads
    1662 requests ignored:
        0 invalid log lines
        264 requests done by bots, search engines, ...
        68 HTTP errors
        43 HTTP redirects
        1287 requests to static resources (css, js, ...)
        0 requests did not match any known site
        0 requests did not match any requested hostname

Website import summary
----------------------

    5797 requests imported to 1 sites
        1 sites already existed
        0 sites were created:

    0 distinct hostnames did not match any existing site:



Performance summary
-------------------

    Total time: 5 seconds
    Requests imported per second: 1101.55 requests per second
2

Did you run the archiving as per instructions in the script output:

To re-process these reports with your new update data, execute the following command:
piwik/console core:archive --url=http://example/piwik/

3

I have tried running the archive after running the python script and still no data is added to the database. Here is what happens when I run the archive:


/public_html/MYWEBSITE.com/piwik/console core:archive --url=http://www.MYWEBSITE.com/piwik/ 
INFO CoreConsole[2014-05-14 03:43:57] [2493f] ---------------------------
INFO CoreConsole[2014-05-14 03:43:57] [2493f] INIT
INFO CoreConsole[2014-05-14 03:43:57] [2493f] Piwik is installed at: http://www.MYWEBSITE.com/piwik/index.php
INFO CoreConsole[2014-05-14 03:43:57] [2493f] Running Piwik 2.2.2 as Super User
INFO CoreConsole[2014-05-14 03:43:57] [2493f] ---------------------------
INFO CoreConsole[2014-05-14 03:43:57] [2493f] NOTES
INFO CoreConsole[2014-05-14 03:43:57] [2493f] - If you execute this script at least once per hour (or more often) in a crontab, you may disable 'Browser trigger archiving' in Piwik UI > Settings > General Settings. 
INFO CoreConsole[2014-05-14 03:43:57] [2493f]   See the doc at: http://piwik.org/docs/setup-auto-archiving/
INFO CoreConsole[2014-05-14 03:43:57] [2493f] - Reports for today will be processed at most every 10 seconds. You can change this value in Piwik UI > Settings > General Settings.
INFO CoreConsole[2014-05-14 03:43:57] [2493f] - Reports for the current week/month/year will be refreshed at most every 3600 seconds.
INFO CoreConsole[2014-05-14 03:43:57] [2493f] - Archiving was last executed without error 2 days 10 hours ago
INFO CoreConsole[2014-05-14 03:44:03] [2493f] - Will process 0 websites with new visits since 2 days 10 hours 
INFO CoreConsole[2014-05-14 03:44:03] [2493f] - Will process 1 other websites because some old data reports have been invalidated (eg. using the Log Import script) , IDs: 1
INFO CoreConsole[2014-05-14 03:44:03] [2493f] ---------------------------
INFO CoreConsole[2014-05-14 03:44:03] [2493f] START
INFO CoreConsole[2014-05-14 03:44:03] [2493f] Starting Piwik reports archiving...
INFO CoreConsole[2014-05-14 03:44:05] [2493f] Archived website id = 1, period = day, 316518 visits in last 17 days, 0 visits today, Time elapsed: 1.815s
INFO CoreConsole[2014-05-14 03:44:10] [2493f] Archived website id = 1, period = week, 316518 visits in last 4 weeks, 0 visits this week, Time elapsed: 4.423s
INFO CoreConsole[2014-05-14 03:44:15] [2493f] Archived website id = 1, period = month, 316518 visits in last 4 months, 291209 visits this month, Time elapsed: 5.363s
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Archived website id = 1, period = year, 316518 visits in last 4 years, 316518 visits this year, Time elapsed: 3.036s
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Archived website id = 1, 4 API requests, Time elapsed: 14.745s [1/1 done]
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Done archiving!
INFO CoreConsole[2014-05-14 03:44:18] [2493f] ---------------------------
INFO CoreConsole[2014-05-14 03:44:18] [2493f] SUMMARY
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Total visits for today across archived websites: 0
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Archived today's reports for 1 websites
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Archived week/month/year for 1 websites
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Skipped 0 websites: no new visit since the last script execution
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Skipped 0 websites day archiving: existing daily reports are less than 10 seconds old
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Skipped 0 websites week/month/year archiving: existing periods reports are less than 3600 seconds old
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Total API requests: 4
INFO CoreConsole[2014-05-14 03:44:18] [2493f] done: 1/1 100%, 0 vtoday, 1 wtoday, 1 wperiods, 4 req, 14870 ms, no error
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Time elapsed: 14.870s
INFO CoreConsole[2014-05-14 03:44:18] [2493f] ---------------------------
INFO CoreConsole[2014-05-14 03:44:18] [2493f] SCHEDULED TASKS
INFO CoreConsole[2014-05-14 03:44:18] [2493f] Starting Scheduled tasks... 
INFO CoreConsole[2014-05-14 03:44:21] [2493f] task,output
CoreAdminHome.purgeOutdatedArchives,Time elapsed: 0.083s
PrivacyManager.deleteReportData,Time elapsed: 0.014s
PrivacyManager.deleteLogData,Time elapsed: 0.008s
Piwik\Plugins\CorePluginsAdmin\MarketplaceApiClient.clearAllCacheEntries,Time elapsed: 0.016s
CorePluginsAdmin.sendNotificationIfUpdatesAvailable,Time elapsed: 0.015s
CoreAdminHome.optimizeArchiveTable,Time elapsed: 1.963s
CoreUpdater.sendNotificationIfUpdateAvailable,Time elapsed: 0.044s
INFO CoreConsole[2014-05-14 03:44:21] [2493f] done
INFO CoreConsole[2014-05-14 03:44:21] [2493f] ---------------------------
4

When you select 2014-05-12 date in Piwik it does not show any report? what about the Visitors>Visitor log ?

5

No, there are not any visitors for the 12th or in the visitor log either. Please see attachment.

6

I upgraded to the beta version (2.2.3-b6). That didn’t fix it either. I wonder why it works the first time and then stops working…

7

what python version u running?

8

Python 2.6.6

9

I upgraded to python2.7 and it’s still not working. This is frustrating…

10

i beleive the minimum recommended is 2.7.1 or higher

11

I have upgraded to 2.7.3 - still not working.

12

It’s working now. I re-installed piwik, upgraded to the beta version and made sure to run the latest version of python both on the first import, the archive, and the subsequent imports.