Traffic summary using iptables

2011-03-12T16:54:40Z

192.168.0.250: Created page with "= Overview = Using iptables and a simple perl script to analyze data traffic. = Prereq = <code> Perl ver 5.6+ Iptables </code> = Iptables setup = = Perl script to grab data = ..."

= Overview =
Using iptables and a simple perl script to analyze data traffic.

= Prereq =
<code>
Perl ver 5.6+
Iptables
</code>

= Iptables setup =

= Perl script to grab data =
I'm putting the data from iptables into a local mysql database. From there I further analyze.

It's a simple two-stage process;
1. Get the data from the newly created chains;
2. Put into db and reset counter.

Mine is based around getting data every hour, so I've made a cron entry for that

Perl script:
<pre>
#!/usr/bin/perl

use strict;
use DBI;

## Setup database connection
my $dsn = 'dbi:mysql:<dbname>:<ip or localhost>:3306'; my $user = '<username>'; my $pass = '<password>';
my $dbh = DBI->connect($dsn, $user, $pass) or die "Horrible!!\n$DBI::errstr\n";

my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$mon++;
$year += 1900;

## Only grab outgoing data
my @aData = `/sbin/iptables -L TRAFFIC_ACCT_OUT -n -v -x | awk '\$1 ~ /^[0-9]+\$/ { printf \"%s, %d \\n\", \$7, \$2 }'`;

foreach (@aData)
{
chomp;
my @aSplitter = split(/, /, $_);
my $sExtraSQL = "ON DUPLICATE KEY UPDATE traffic = ".$dbh->quote($aSplitter[1]);
my $sSQL = "INSERT INTO traffic (source, year, month, day, hour, traffic) VALUES (?, ?, ?, ?, ?, ?) $sExtraSQL\n";
my $sth = $dbh->prepare($sSQL);
$sth->execute($aSplitter[0], $year, $mon, $mday, $hour, $aSplitter[1]);
}
my $bResetIptableCounter = `/sbin/iptables -Z TRAFFIC_ACCT_OUT`;
</pre>

The code is extremely straightforward and no checks really. If I miss out of one hours traffic, it's no biggie, so haven't put much work into that part.

= Cron entry =
<pre>

= Database create options =
For those wanting it; I've made a unique key formed by year+month+day+hour+source. It's highly inefficient, but I'm dealing with a relatively low amount of data on my end (checking 3 hosts, so for a year I'll have a max of 3 hosts * 24 hours * 365 days ~= 25000 entries).
<pre>

</pre>

</pre>

Mirror getdeb

2011-03-11T18:50:46Z

192.168.0.250: Created page with "== Mirror - getdeb == <pre> #/bin/sh fatal() { echo "$1" exit 1 } warn() { echo "$1" } # RSYNC source RSYNCSOURCE=rsync://rsync.getdeb.net/getdeb/ # Define where you w..."

== Mirror - getdeb ==

<pre>
#/bin/sh

fatal() {
echo "$1"
exit 1
}

warn() {
echo "$1"
}

# RSYNC source
RSYNCSOURCE=rsync://rsync.getdeb.net/getdeb/

# Define where you want the mirror-data to be on your mirror
BASEDIR=$HOME/getdeb

if [ ! -d ${BASEDIR} ]; then
warn "${BASEDIR} does not exist yet, trying to create it..."
mkdir -p ${BASEDIR} || fatal "Creation of ${BASEDIR} failed."
fi

rsync --recursive --times --links --hard-links \
--stats \
--exclude "Packages*" --exclude "Sources*" \
--exclude "Release*" \
${RSYNCSOURCE} ${BASEDIR} || fatal "First stage of sync failed."

rsync --recursive --times --links --hard-links \
--stats --delete --delete-after \
${RSYNCSOURCE} ${BASEDIR} || fatal "Second stage of sync failed."

</pre>

Mirror gentoo

2011-03-11T18:49:18Z

192.168.0.250:

[[Category: Linux]]

== Mirror - gentoo ==

<pre>
#!/bin/bash

RSYNC="/usr/bin/rsync"
OPTS="--recursive --links --perms --times -D --delete --timeout=300"
#Uncomment the following line only if you have been granted access to masterportage.gentoo.org
#SRC="rsync://masterportage.gentoo.org/gentoo-portage"
#If you are waiting for access to our master mirror, select one of our mirrors to mirror from:
SRC="rsync://rsync.de.gentoo.org/gentoo-portage"
DST="/home/mirroruser/gentoo/rsync/"

echo "Started update at" `date` >> $0.log 2>&1
logger -t rsync "re-rsyncing the gentoo-portage tree"
${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1

echo "End: "`date` >> $0.log 2>&1
</pre>

Mirror gentoo

2011-03-11T18:48:48Z

192.168.0.250: Created page with "== Mirror - gentoo == <pre> #!/bin/bash RSYNC="/usr/bin/rsync" OPTS="--recursive --links --perms --times -D --delete --timeout=300" #Uncomment the following line only if you ha..."

== Mirror - gentoo ==

<pre>
#!/bin/bash

RSYNC="/usr/bin/rsync"
OPTS="--recursive --links --perms --times -D --delete --timeout=300"
#Uncomment the following line only if you have been granted access to masterportage.gentoo.org
#SRC="rsync://masterportage.gentoo.org/gentoo-portage"
#If you are waiting for access to our master mirror, select one of our mirrors to mirror from:
SRC="rsync://rsync.de.gentoo.org/gentoo-portage"
DST="/home/mirroruser/gentoo/rsync/"

echo "Started update at" `date` >> $0.log 2>&1
logger -t rsync "re-rsyncing the gentoo-portage tree"
${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1

echo "End: "`date` >> $0.log 2>&1
</pre>

Category:Linux

2011-03-11T18:48:11Z

192.168.0.250: Blanked the page

Category:Linux

2011-03-11T18:47:54Z

192.168.0.250:

#!/bin/bash

RSYNC="/usr/bin/rsync"
OPTS="--recursive --links --perms --times -D --delete --timeout=300"
#Uncomment the following line only if you have been granted access to masterportage.gentoo.org
#SRC="rsync://masterportage.gentoo.org/gentoo-portage"
#If you are waiting for access to our master mirror, select one of our mirrors to mirror from:
SRC="rsync://rsync.de.gentoo.org/gentoo-portage"
DST="/home/mirroruser/gentoo/rsync/"

echo "Started update at" `date` >> $0.log 2>&1
logger -t rsync "re-rsyncing the gentoo-portage tree"
${RSYNC} ${OPTS} ${SRC} ${DST} >> $0.log 2>&1

echo "End: "`date` >> $0.log 2>&1

2008-12-11T23:17:47Z

192.168.0.122: /* Install services */

[[Category: Linux]]

= DRBD & Heartbeat on Debian Etch =

== Install DRBD ==
Unfortunately, drbd is not actively available as a package under Debian Etch, so it has to be build from source.

Luckily this can be done automatically with debian.

<pre>
apt-get install module-assistant build-essential drbd0.7-module-source

## Then build the module
m-a a-i drbd0.7
</pre>

=== Setup some basic drbd replication [internal meta - partition] ===
<pre>
resource drbd {
protocol C;
incon-degr-cmd "killall heartbeat";
disk {
on-io-error panic;
}
syncer {
rate 10M; # Note: 'M' is MegaBytes, not MegaBits
}
on server1 {
device /dev/drbd0;
disk /dev/hdc3;
address 192.168.0.110:7789;
meta-disk internal;
}
on server2 {
device /dev/drbd0;
disk /dev/hda3;
address 192.168.0.140:7789;
meta-disk internal;
}
}
</pre>

Of course adjust to your need.

== Install heartbeat ==
<pre>
apt-get install heartbeat
</pre>

=== Setup heartbeat (basic) ===
Edit both host files ( /etc/ha.d/authkeys )
<pre>
auth 2
1 crc
2 sha1 ThisIsASampleKeyAnythingAlphaNumericIsGoodHere
3 md5 ThisIsASampleKeyAnythingAlphaNumericIsGoodHere
</pre>

Edit HA for master node ( /etc/ha.d/ha.cf )
<pre>
keepalive 1
deadtime 10
warntime 5
initdead 60
udpport 694
baud 19200
serial /dev/ttyS0
ucast eth0 192.168.0.140 (slave node should have master ip here; 192.168.0.110)
auto_failback off
watchdog /dev/watchdog
node node-a
node node-b
</pre>

== Update systemfiles ==
Add needed servers to hosts file
<pre>
[ ... ]
192.168.0.110 server1
192.168.0.140 server2
192.168.0.180 server
[ ... ]
</pre>

chmod /etc/ha.d/authkeys files
<pre>
chmod 0600 /etc/ha.d/authkeys
</pre>

Prepare new drbd partition for data
<pre>
mkfs.ext3 /dev/drbd0
</pre>

Setup exim
<pre>
dpkg-reconfigure exim4-config
</pre>

== Install services ==
<pre>
## Apache + php5 + mysql
apt-get install apache2 php5 php5-mysql mysql-server

## pureftpd
apt-get install pure-ftpd
</pre>

####### Heartbeat controlled ressources #########
Stop services and let heartbeat handle it:
<pre>
/etc/init.d/apache2 stop
/etc/init.d/mysql stop

## Copy to
cp -p /etc/init.d/mysql /etc/ha.d/resource.d/
cp -p /etc/init.d/apache2 /etc/ha.d/resource.d/

## Remove old init-scripts for mysql/apache
update-rc.d -f apache2 remove
update-rc.d -f mysql remove

## Config to use the new storage (/data)