NO data is showning on piwik


#1

HI,

I am trying to import nginx logs to piwik using import_logs.py tool, I am using below command to import the data, after the log process I see only few requests were imported but dont see the major of the logs being imported. Below is the command being used for the same.

python ./import_logs.py --login=piwikadmin --password=‘admin123’ --show-progress --url=http://192.168.12.173:81 --idsite=1 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static --enable-bots /home/keyur/wap.access.log-20141111

7756 lines parsed, 3632 lines recorded, 11 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 3632 lines recorded, 11 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 3632 lines recorded, 11 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 3632 lines recorded, 11 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 3843 lines recorded, 11 records/sec (avg), 211 records/sec (current)
7756 lines parsed, 3843 lines recorded, 11 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 3843 lines recorded, 11 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 3843 lines recorded, 11 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 3843 lines recorded, 11 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 3843 lines recorded, 11 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 4158 lines recorded, 12 records/sec (avg), 315 records/sec (current)
7756 lines parsed, 4158 lines recorded, 12 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 4214 lines recorded, 12 records/sec (avg), 56 records/sec (current)
7756 lines parsed, 4214 lines recorded, 12 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 4214 lines recorded, 12 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 4214 lines recorded, 12 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 4214 lines recorded, 12 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 4214 lines recorded, 12 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 4214 lines recorded, 12 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 4440 lines recorded, 12 records/sec (avg), 226 records/sec (current)
7756 lines parsed, 4440 lines recorded, 12 records/sec (avg), 0 records/sec (current)
7756 lines parsed, 4440 lines recorded, 12 records/sec (avg), 0 records/sec (current)
Fatal error: Internal Server Error
You can restart the import of “/home/keyur/wap.access.log-20141111” from the point it failed by specifying --skip=3420 on the command line.

I believe it has to do something with my log format, PFB my log format… (It has many unusual fields)

log_format  website '$http_x_forwarded_for - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$remote_addr" $request_time $upstream_response_time $geoip_country_code $pipe';

And this is how my logs looks like :

      • [10/Nov/2014:00:01:13 +0530] “HEAD /filename.php HTTP/1.0” 200 0 “-” “-” “192.168.xx.xx” 0.003 0.002 - .

66.249.79.141 - - [10/Nov/2014:00:01:13 +0530] “GET /someurl HTTP/1.1” 200 5478 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” “66.249.79.141” 0.493 0.493 US .

66.249.79.125 - - [10/Nov/2014:00:01:13 +0530] “GET /someurl HTTP/1.1” 200 5281 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” “66.249.79.125” 0.298 0.298 US .

66.249.79.125 - - [10/Nov/2014:00:01:14 +0530] “GET /someurl HTTP/1.1” 302 5 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” “66.249.79.125” 0.006 0.006 US .

66.249.79.133 - - [10/Nov/2014:00:01:14 +0530] “GET /someurl HTTP/1.1” 302 5 “-” “Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” “66.249.79.133” 0.002 0.002 US .

59.89.242.231 - - [10/Nov/2014:00:01:14 +0530] “GET /someurl HTTP/1.1” 302 5 “http://www.google.co.in/somereferrer” “Mozilla/5.0 (Linux; U; Android 4.2.1; en-us; Andi5h quadro Build/JOP40D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30” “59.89.242.231” 0.002 0.002 IN .

1.39.62.204, 37.228.104.87 - - [10/Nov/2014:00:01:15 +0530] “GET /someurl HTTP/1.1” 200 7825 “-” “Opera/9.80 (Android; Opera Mini/7.0.32026/35.5390; U; en) Presto/2.8.119 Version/11.10” “37.228.104.87” 0.378 0.378 IN .

10.90.207.32, 180.178.24.61 - - [10/Nov/2014:00:01:16 +0530] “GET /someurl HTTP/1.1” 302 5 “http://www.google.com/referrer” “Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; Karbonn A6 Build/MocorDroid2.3.5) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1” “180.178.24.61” 0.002 0.002 - .

1.39.25.108 - - [10/Nov/2014:00:01:16 +0530] “GET /someurl HTTP/1.1” 302 5 “http://www.google.co.in/somereferrer” “Mozilla/5.0 (Linux; U; Android 4.2.2; en-in; Micromax A120 Build/JDQ39) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30” “1.39.25.108” 0.002 0.002 IN .

66.249.79.141 - - [10/Nov/2014:00:01:16 +0530] “GET /someurl HTTP/1.1” 200 6316 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)” “66.249.79.141” 0.273 0.273 US .

Please let me know which exact fields I should keep to make log import successful.

Regards,
Keyur