Скрипт сбора статистики bind9
Задача
В Bind9 имеется возможность записывать в лог-файл все поступившие запросы. Грех не использовать эту возможность для сбора статистики. Ниже приведенный скрипт обрабатывает query-лог bind-9 и генерирует вот такую текстовую страницу:
Requests by hour: Hour | TOTAL | A | MX | PTR | TXT | CNAME | NS | AAAA | SRV | ANY | SOA | A6 | AXFR | IXFR | 00 | 98k | 55k | 12k | 12k | 4561 | 168 | 2331 | 9007 | 47 | 159 | 520 | | | | 01 | 89k | 48k | 8847 | 11k | 5778 | 129 | 1913 | 10k | 31 | 140 | 787 | | | | 02 | 95k | 53k | 9261 | 10k | 5640 | 145 | 2540 | 9968 | 47 | 661 | 636 | | | | 03 | 104k | 53k | 12k | 10k | 9306 | 180 | 3225 | 10k | 51 | 678 | 367 | | 1 | | 04 | 142k | 80k | 13k | 11k | 11k | 237 | 4964 | 12k | 33 | 885 | 278 | | | | 05 | 115k | 65k | 11k | 8963 | 10k | 102 | 3929 | 8734 | 26 | 718 | 236 | | | | 06 | 119k | 61k | 11k | 9883 | 12k | 132 | 3678 | 9162 | 24 | 573 | 160 | | | | 07 | 99k | 54k | 12k | 9650 | 6360 | 98 | 3829 | 10k | 27 | 441 | 125 | | | | 08 | 103k | 59k | 11k | 9435 | 6272 | 79 | 3469 | 9979 | 33 | 328 | 136 | | | | 09 | 102k | 56k | 12k | 10k | 5019 | 99 | 2562 | 10k | 40 | 1400 | 116 | | | | 10 | 98k | 53k | 12k | 9607 | 5897 | 103 | 2573 | 9728 | 44 | 2055 | 114 | | | | 11 | 96k | 51k | 12k | 9980 | 5513 | 88 | 2573 | 9918 | 53 | 1418 | 108 | | | | 12 | 84k | 45k | 10k | 9120 | 4422 | 81 | 2065 | 9660 | 55 | 1091 | 118 | | | | 13 | 96k | 51k | 11k | 9359 | 7155 | 104 | 3488 | 9657 | 39 | 802 | 112 | | | | 14 | 95k | 52k | 10k | 9028 | 6873 | 89 | 3504 | 9892 | 35 | 395 | 104 | | | | 15 | 85k | 46k | 10k | 9324 | 4478 | 82 | 2282 | 10k | 44 | 228 | 110 | | | | 16 | 79k | 44k | 8341 | 8073 | 4951 | 92 | 2509 | 9781 | 41 | 190 | 108 | | | | 17 | 79k | 43k | 9237 | 8368 | 3640 | 119 | 1862 | 10k | 43 | 208 | 415 | | | | 18 | 91k | 53k | 9278 | 8954 | 5105 | 76 | 2161 | 10k | 55 | 190 | 429 | | 1 | | 19 | 87k | 48k | 8287 | 9280 | 5004 | 128 | 2084 | 10k | 54 | 157 | 500 | | | | 20 | 89k | 51k | 8417 | 10k | 3812 | 114 | 2109 | 10k | 41 | 159 | 455 | | | | 21 | 85k | 47k | 8442 | 10k | 3331 | 163 | 2061 | 10k | 48 | 182 | 558 | | | | 22 | 108k | 60k | 13k | 14k | 4851 | 134 | 2300 | 9905 | 52 | 210 | 683 | | | | 23 | 145k | 80k | 24k | 18k | 6353 | 140 | 2726 | 10k | 51 | 175 | 598 | | 1 | | TOP 20 Clients: Client IP | TOTAL | A | MX | PTR | TXT | CNAME | NS | AAAA | SRV | ANY | SOA | A6 | AXFR | IXFR | 192.168.1.87 | 799k | 548k | 25k | 30k | 111k | | 46k | 5 | | | | | | | 192.168.1.91 | 172k | 39k | 2 | 61k | | | | 71k | | | | | | | 192.168.1.83 | 139k | 80k | 37k | 21k | | | | | | | | | | | 192.168.77.15 | 104k | 62k | 5 | 31k | 1350 | | | 9169 | 20 | | 4 | | | | 192.168.77.70 | 70k | 4 | | 70k | | | | | | | | | | | 192.168.1.85 | 58k | 43k | 12k | 2918 | | | | | | | | | | | 192.168.1.82 | 43k | 11k | 31k | 105 | | | | | | | | | | | 192.168.1.32 | 38k | 4561 | 1694 | 2 | 14k | | | | | | | | | | 192.168.1.84 | 30k | 15k | 2 | 15k | | | | | | | | | | | 192.168.77.13 | 26k | 23k | 1 | 1455 | 14 | | | 1875 | 13 | | 3 | | | | 74.116.90.15 | 19k | 4 | 19k | | | | | | | | | | | | 172.17.0.5 | 11k | | | 11k | | | | | | | | | | | 64.202.161.41 | 9369 | 4476 | 4713 | | 166 | | 14 | | | | | | | | 199.59.148.218 | 6589 | 2229 | 4296 | | | | | 64 | | | | | | | 69.252.96.24 | 6068 | 3462 | 2 | | 2 | | 15 | 2523 | 63 | | | | | | 69.252.96.22 | 5907 | 3348 | 1 | | 1 | | 16 | 2464 | 77 | | | | | | 40.139.112.2 | 5870 | 5606 | 1 | | 9 | 9 | | 231 | | | 14 | | | | 130.207.54.131 | 5531 | 5478 | | | | | | 53 | | | | | | | 130.207.54.130 | 5403 | 5317 | | | | | | 86 | | | | | | | 67.225.185.56 | 5353 | 2323 | 712 | | | | | | | 1946 | 372 | | | | TOP 20 Domains: Domain | TOTAL | A | MX | PTR | TXT | CNAME | NS | AAAA | SRV | ANY | SOA | A6 | AXFR | IXFR | spamhaus.org | 267k | 256k | | | 11k | | | | | | | | | | propertyminder.com | 262k | 133k | 6568 | | 6835 | 294 | 3395 | 107k | 87 | 317 | 14 | | | | in-addr.arpa | 249k | 9 | | 249k | | | | | | | | | | | google.com | 68k | 64k | 64 | | 890 | | 29 | 1647 | | | | | | | spamcop.net | 44k | 22k | | | 21k | | | | | | | | | | mailspike.net | 32k | 32k | | | | | | | | | | | | | propertyminder.colo | 28k | 26k | 4 | | | | | 1808 | | | | | | | surbl.org | 26k | 26k | | | | | | | | | | | | | isvr.net | 22k | 9765 | 5 | | 5 | | 351 | 12k | | 6 | 1 | | | | juliakeady.com | 21k | 7916 | 13k | | | | 500 | 45 | | 1 | 9 | | | | gmail.com | 19k | 177 | 17k | | 865 | | | 5 | | | | | | | 7jet.biz | 18k | 18k | | | | | | | | | | | | | yahoodns.net | 17k | 17k | | | | | 3 | 8 | | | | | | | habeas.com | 16k | | | | 16k | | | | | | | | | | bondedsender.org | 16k | | | | 16k | | | | | | | | | | uribl.com | 16k | 16k | | | | | | | | | | | | | support-intelligence.net | 15k | 15k | | | | | | | | | | | | | barracudacentral.org | 15k | 15k | | | | | | | | | | | | | geoplugin.net | 13k | 6867 | | | | | | 6867 | | | | | | | manitu.net | 13k | 13k | | | | | | | | | | | | | TOP 20 Requests: TYPE | TOTAL | Request PTR | 70k | 70.10.10.10.in-addr.arpa A | 31k | ns2.propertyminder.com A | 31k | ns1.propertyminder.com A | 22k | isvr.net AAAA | 22k | mx.propertyminder.com MX | 21k | juliakeady.com A | 18k | myhost5.7jet.biz PTR | 18k | 2.1.168.192.in-addr.arpa MX | 18k | gmail.com PTR | 17k | 32.1.168.192.in-addr.arpa A | 17k | mobile-api8.propertyminder.com A | 17k | propertyminder.com A | 16k | amq1.propertyminder.com AAAA | 15k | static.propertyminder.com AAAA | 13k | www.geoplugin.net ANY | 11k | prudentialnorthland.com MX | 11k | samanthaannuzzi.com MX | 10k | yahoo.com A | 10k | natellena.com AAAA | 10k | mail.propertyminder.com
Подготовка
Необходимо настроить bind, что бы он писал query-log в файл. Примерно следующим образом (в /enc/named.conf):
logging {
channel query_log {
file "/var/log/bind/query.log";
severity dynamic;
print-time yes;
};
category queries { query_log; };
}
Скрипт сохраним в /root/bin/make-bind-stat.pl
Скрипт имеет смысл запускать из logrotate скрипта.
Пример конфига для logrotate (скрипт создает файл со статистикой в каталоге /var/www/html/bind-stat/, а так же отправляет его по почте):
/var/log/bind/*log { missingok notifempty compress rotate 7 daily sharedscripts prerotate DATE=`date +%Y-%m-%d`; /root/bin/make-bind-stat.pl > /var/www/html/bind-stat/$DATE.txt; cat /var/www/html/bind-stat/$DATE.txt | /usr/bin/uuencode $DATE.txt | /bin/mailx -s "named stats from NS1.minder.com" admins@minder.com endscript postrotate /sbin/service named reload 2> /dev/null > /dev/null || true endscript }
Скрипт make-bind-stat.pl
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 | #! /usr/bin/perl use strict; # # This script parses bind9 query log, and print stats in text format on stdout # # To use set $BIND_QUERY_LOG variable and run from logrotate or cron like this: # make-bind-stat_v02.pl > /var/www/html/bind-stat-`date +%Y-%m-%d`-.txt # # You should configure bind to log queries before use. # set logging channel like in example: # # logging { # channel query_log { # file "/var/log/bind/query.log"; # severity dynamic; # print-time yes; # }; # category queries { query_log; }; # } # # version 0.1, copyright: Valynkin Pavel, 2015 # # <==== config ====> my $BIND_QUERY_LOG="/var/log/bind/query.log"; my $TOP = 20; # how many lines in tables my @QUERY_TYPES=("A","MX","PTR","TXT","CNAME","NS","AAAA","SRV","ANY","SOA","A6","AXFR","IXFR"); # query types to output in tables # <==== end of config ====> my %totals_by_hour; my %totals_by_hour_details; my %host_total_requests; my %host_details; my %domain_total_requests; my %domain_details; my %query_total_requests; my %query_type; open (IN, "<$BIND_QUERY_LOG"); # # parse log and count conters # while ( <IN> ) { chomp; my $line=$_; # Extract data from line $line =~ /^[0-9]+-[A-z]+-[0-9]+ ([0-9][0-9])+:[0-9][0-9]:[0-9][0-9]\..+client ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)#[0-9]+: query: (.+) IN ([A-Z]+) .+/; next if ( !$4 ); # next if parsing failed my $hour = $1; my $client_ip = $2; my $query = $3; my $query_type = $4; $query =~ /(?:[^\.]+\.)*?([^\.]+\.[^\.]+)$/; my $domain = $1; # count counters $totals_by_hour{$hour}++; $totals_by_hour_details{$hour."-".$query_type}++; $host_total_requests{$client_ip}++; $host_details{$client_ip."-".$query_type}++; $domain_total_requests{$domain}++; $domain_details{$domain."-".$query_type}++; $query_total_requests{$query}++; $query_type{$query} = $query_type; } close IN; my ($sorted_keys,$sorted_values); # # output queries by hour table # print "Requests by hour:\n"; print "Hour\t| TOTAL\t| "; foreach (@QUERY_TYPES) { print $_,"\t| "; } print "\n"; foreach my $hour(sort keys %totals_by_hour){ print $hour,"\t| "; print &format_value($totals_by_hour{$hour}),"\t| "; foreach (@QUERY_TYPES) { print &format_value($totals_by_hour_details{$hour."-".$_}),"\t| "; } print "\n"; } # # output TOP Clients table # ($sorted_keys,$sorted_values) = &sort_hash_by_values(\%host_total_requests); #print head print "\nTOP $TOP Clients:\n"; print "Client IP\t| TOTAL\t| "; foreach (@QUERY_TYPES) { print $_,"\t| "; } print "\n"; # print data for (my $i=0; $i < $TOP; $i++) { print $sorted_keys->[$i],"\t| "; print &format_value($sorted_values->[$i]),"\t| "; foreach (@QUERY_TYPES) { print &format_value($host_details{$sorted_keys->[$i]."-".$_}),"\t| "; } print "\n"; } # # output TOP domains table # ($sorted_keys,$sorted_values) = &sort_hash_by_values(\%domain_total_requests); #print head print "\nTOP $TOP Domains:\n"; print "Domain\t\t\t| TOTAL\t| "; foreach (@QUERY_TYPES) { print $_,"\t| "; } print "\n"; # print data for (my $i=0; $i < $TOP; $i++) { print $sorted_keys->[$i],"\t"; print "\t" if ( length($sorted_keys->[$i]) < 16 ); print "| "; print &format_value($sorted_values->[$i]),"\t| "; foreach (@QUERY_TYPES) { print &format_value($domain_details{$sorted_keys->[$i]."-".$_}),"\t| "; } print "\n"; } # # output TOP requests table # ($sorted_keys,$sorted_values) = &sort_hash_by_values(\%query_total_requests); #print head print "\nTOP $TOP Requests:\n"; print "TYPE\t| TOTAL\t| Request\t"; print "\n"; # print data for (my $i=0; $i < $TOP; $i++) { print $query_type{$sorted_keys->[$i]},"\t| "; print &format_value($sorted_values->[$i]),"\t| "; print $sorted_keys->[$i],"\t"; print "\n"; } # # Subroutines # sub sort_hash_by_values { my ($hash) = @_; my $i=0; my @keys; my @values; foreach my $key(sort {$hash->{$b} <=> $hash->{$a}} keys %$hash) { #print $key,'=',$hash->{$key},"\n"; $keys[$i]=$key; $values[$i]=$hash->{$key}; $i++; last if ($i == $TOP); } return \@keys, \@values; } sub sort_hash_by_key { my ($hash) = @_; my $i=0; my @keys; my @values; foreach my $key(sort keys %{$hash}){ #print "$key => ",$hash->{$key},"\n"; #отсортирует в алфавитном порядке по значениям ключа $keys[$i]=$key; $values[$i]=$hash->{$key}; $i++; last if ($i == $TOP); } #print %sorted_hash; return \@keys, \@values; } sub format_value { my $value = @_[0]; $value = int($value/1000)."k" if ( $value >= 10000 && $value < 1000000 ); $value = int($value/1000000)."m" if ( $value > 100000 ); return $value; } |
Enjoy!