Rate limiting outbound mail in Zimbra, with Policyd

I've recently found a need for outbound rate limiting in Zimbra. The idea is that this will help prevent a spam outbreak, sourcing from one of my users accounts, if it's compromised. This has happened in the past, and it's finally time to put a stop to it. There are two options available to me for limiting outbound mail. First is within Zimbra, the other is via out outbound spam/virus filter which is handled by a ProofPoint Protection Server. This is basically an e-mail firewall, similar to Barracuda, or Postini. I spoke with our support personnel at ProofPoint, and they were less than helpful. Not on the whole, they're generally very good, I would in fact recommend them to others. The system is sound, and flexible, and their support is usually great. In this case, I was simply asking for something that this system cannot offer. Because this is an outbound limiter, i need to rate limit based on sender, not sending server. Apparently the PPS just doesn't have that sort of tracking. So I moved on to Zimbra. Out of the box, Zimbra does not have this feature either. However, Zimbra's MTA is build on Postfix. A relatively clean, stock postfix. Most postfix mod's also work on Zimbra's MTA. I did a little digging, a little asking around on Zimbra's forums, and a little research on my own and came up with Policyd. Policyd is a service which runs on either your local postfix box, or on a remote box, which manages lists of policies. Postfix queries policyd for every transaction, and asks it if it has a policy which pertains to this transaction. Once it gets the result of said policy, it moves on with the session. Now, there are tons of things which can be done with policyd, I'm focusing on JUST outbound, and JUST quota based rate limiting.

4.2U Wii, homebrew hack.

A while ago, a friend of mine helped me exploit my Wii, so that I could run the Homebrew channel. There are many advantages to doing this, and for lots of reasons. Not just game piracy. Homebrew lets you install 3rd party apps on your Wii, which allow you to get some creative use out of the system. You can also connect USB storage to your Wii, and use it to store your games, so that you dont need to swap discs around when you want to play a game. When we hacked the Wii, it was running update 3.6. I'd stopped updating the Wii from Nintendo's update procedure because I assumed that they would probably tromp on my hacked wii, and restore the OS to its pristine condition, and I'd then have to go through re-hacking. Well, I knew this time would come. I was forced into an update, so I could run the New Super Mario Brothers game. I guess it has something to do with the ability for the players to interact as they do in the game. So, i bit the bullet, and ran the update. It wasn't until after i ran the update that i found a number of reports on the 'net about how to update a hacked Wii, without actually running nintendo's update. This would have kept my Homebrew intact, and still gotten me the update. But, oh well, I was committed at this point. After some digging, I found a number of how-to's on the subject of hacking a 4.2U Wii. 4.2U being the most recent update. I tried a number of them, and none of them worked. Until i found this how-to, which was SOOOOO much easier to follow than some of the others. Since i'd been through so many other failed attempts at hacking this system, and since it was hacked once before, i decided now was a good time to backup my save data, and format the Wii's system memory. Luckily, i was able to backup all of my save data to an SD card, and then save that data to my PC. I then performed the format, but wasnt able to put my save data back on the Wii until after i'd run each game at least once. This meant i either had to break out all of my discs, or get homebrew and the usb loader working before i could get my save data off of my sd card. So i started down the road to Wii-freedom. In the end, it wasnt all that difficult, and i ended up with bootmii, which is a more secure replacement for Wii's boot2 loader, and the homebrew channel. I also have all of my USB capability back, and once i get all of my homebrew software reinstalled, i'll have all of the functionality that i had before the format. On top of all that, i can play Mario now. :D

Some good ol house music for your Tuesday.

I'm into quite a few types of music. Mostly anything that has a good beat, and a deep sound. I always imagine that a lot of people who have been exposed to my taste in music are baffled by the seeming variety of genre's I listen to. There's a theme believe it or not. Its hard to describe. I like bass, and I like guitar. If its got a deep sound, I like it. This can be found in a few genre's commonly. Mostly electronic music, like House, and Trance, and a lot of Rock/Metal has the same element. So, most of what I listen to falls into these few genre's. One of the DJ's that I started following while XM's "The System" was still on the satellite waves, is Zoltar. He had two shows, The Mutant Dance Party, and Subterranean. Both are shows where he plays mixes of club music. Mostly house. One of these shows can still be heard on Sirius/XM's "Area", though its never on when I'm able to listen since Sirius/XM has so royally screwed up the pricing for XM customers that I can no longer listen online (as i refuse to pay extra for a service which used to be included in my package), but i'm getting off topic. You can still download podcasts of Zoltar at http://zoltar.podomatic.com, so this is how I currently listen to Zoltar. I downloaded his most recent show, which is a year-end review, a best-of 2009 show. One of the songs that caught my eye.. er.. ear was Melleefresh, and Deadmau5 doing "Hey baby". Nice beat, suggestive lyrics, but that's ok. So I started hunting around online for an MP3, so I can listen to just that song whenever i'd like. I found one, but in my search I also found a rather amusing YouTube video. Something to think about before watching this video is this. These are usually one-off DJ tracks. A lot of them are not really professionally recorded. Most of them do not have videos produced by the artist. So, fans will create videos which sort of fit. Most of them are videos just for the sake of being videos, some feature a still image and just play the music. This one is a mash-up of a ton of different suggestive animations, which have been floating around the 'net forever. So, if you don't mind suggestive lyrics, lots of looped almost-nakedness, and a deep progressive beat, then check out this video. :D

Star Trek Online.

Well, it looks like the game's going to make it out the gate! After quite a bit of ups and downs. Cryptic recently announced a release date (02/02/2010) and the game is available for pre-order. You also get some cool extra's for pre-ordering. I've pre-ordered, you should too! :P

Categories

Just a quick note. I've taken a little time to setup categories, and then took a bit more time moving all of my entries into their appropriate categories. Given the random nature of this blog, i thought this would be helpful for those of you who like reading my tech articles, but dont really care about my other topics. Now you can subscribe via RSS to just the categories you'd like to read. Enjoy!

SLES 10, your kernel is not safe!

So, i recently came across a startling discovery. On a SLES 10 server, when you install a kernel update, the update process kindly DELETES your old kernel. It's not clear to me yet if it does this after its next succesful reboot, or if it does it during the update. In other words, the people at SuSE/Novell are _SO_ confident that you'll never have a problem with a brand spankin new kernel, that they perform without a net. I'm a very conservative sysadmin. I don't like to do anything without a backup plan. When it comes to kernel updates, that backup plan is option 1 in my grub.conf (option 1 being the second option in my boot list, generally, my old kernel). From what I'm reading, there's also no way to tell the update process NOT to delete the old kernel. So you're sort of stuck with this behaviour. This actually bit us a few days ago, when due to some rather odd circumstances, we ended up with a SLES that was trying to boot a kernel that was 1 revision old. Because SLES thought it had cleaned up this kernel, the /lib/modules// directory for this kernel was empty. This obviously caused some confusion on the kernel's part, and it refused to boot. If the update process had left the older boot/module files alone, and left it up to a responsible sysadmin to clean up old kernels when they saw fit, this wouldn't have happened. Granted, in this case, the server had other issues, but that's a different story. So I've set out to fix this. Giving yourself some peace of mind is as simple as taking your kernel, and its modules, and locking copies of them away in a safe deposit box (or at least a backup directory) during the update process. And then putting them somewhere accessible afterwards, then re-add ing the old kernel to grub. This is all well and good, if you had one, maybe two servers to worry about, go ahead and do it manually. If you have a couple dozen, this is a considerable amount of work to do manually, and it takes up your time! So, i wrote a script to do it for you! It's a perl script, and it should run on a base install of SLES (or, so it has in my testing). You can download it here. Just download it to your SLES server, and run it, it'll do the work for you. Run it before your update, and select option 1, which backs up the kernel. Then run it again after the update, and select option 2, which restores the kernel. Enjoy! -War

Zettabyte File System (ZFS)

We've been doing a lot of storage research lately, and there's been a lot of talk about ZFS. I'm going to spare you the magazine article (if you want to read more on what it is, and where it comes from, look elsewhere) and give you some guts. ZFS is a 128-bit file system, and unfortunately isnt likely to be built into the linux kernel anytime soon. You can however, use it in userspace, using zfs-fuse, similarly to how you might use NTFS on linux (for those of us still dual booting). The machine i'm running on, runs solely Fedora Core 11, and has a handsome amount of beef behind it. It's also got 500gb of local storage, so I can play around with huge files no sweat. You can do the same things i'm doing, with smaller files, if you'd like. First of all, you'll need to install zfs-fuze, this was simple on Fedora.
$ sudo yum install zfs-fuse
Next some blank disk images to toy with.
$ mkdir zfs
$ cd zfs
$ for i in $(seq 8); do dd if=/dev/zero of=$i bs=1024 count=2097152;done
This gives me 8, 2gb blobs. Make these smaller if you'd like. I wanted enough space to throw some large files at zfs. You'll see in a bit. Now let's make our first zfs pool.
$ sudo zpool create jose ~/zfs/1 ~/zfs/2 ~/zfs/3 ~/zfs/4 
I named my pool jose. I like it when my blog entries have personality. :P zfs list will give you a list of your zfs pools.
$ sudo zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
jose    72K  7.81G    18K  /jose
Creating the pool also mounts it.
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      454G  210G  221G  49% /
/dev/sda1             190M   30M  151M  17% /boot
tmpfs                 2.0G   25M  2.0G   2% /dev/shm
jose                  7.9G   18K  7.9G   1% /jose
An interesting note. I never created a file system on this pool, i just told zfs to have at it. zfs must work at a block level with the drives. Now, let's poke jose with a stick, and see what he does.
$ sudo dd if=/dev/zero of=/jose/testfile bs=1024 count=2097512
2097512+0 records in
2097512+0 records out
2147852288 bytes (2.1 GB) copied, 118.966 s, 18.1 MB/s

$ sudo zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
jose  2.00G  5.81G  2.00G  /jose

Its worth note, that with a zpool add /dev/whatever you can add space to a pool of this sort. That's all fun, but this is essentially just a large file system. No really cool features yet. Let's see what we can really so with this thing. Let's make a raid group, instead of just a standard pool. Goodbye Jose
$ sudo zpool destroy jose
From jose's ashes, lets make a new pool.
$ sudo zpool create susan raidz ~/zfs/1 ~/zfs/2 ~/zfs/3 ~/zfs/4
$ sudo zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
susan  92.0K  5.84G  26.9K  /susan
Notice that susan is smaller than jose, using the same disks. This isn't because susan has made more trips to the gym than jose, rather it's because of the raid set. This is similar to raid 5, where one disk is taken for parity. So you lose a one disk worth of capacity. Let's remedy that, by throwing more (virtual) hardware at it. You cant expand a raid group, by adding a disk, so we'll do it by recreating the group.
$ sudo zpool destroy susan
$ sudo zpool create susan raidz ~/zfs/1 ~/zfs/2 ~/zfs/3 ~/zfs/4 ~/zfs/5
$ sudo zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
susan  98.3K  7.81G  28.8K  /susan
And there you go, about 8gb again. Now let's poke susan with a stick. First, here's her status:
$ sudo zpool status
  pool: susan
 state: ONLINE
 scrub: scrub completed after 0h0m with 0 errors on Tue Oct  6 15:22:24 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   ONLINE       0     0     0
	  raidz1                ONLINE       0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/4  ONLINE       0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors
Now we'll dd another file to susan, and we'll see if we can damage the array.
$ sudo dd if=/dev/zero of=/susan/testfile bs=1024 count=2097512
Then, in another terminal...
$ sudo zpool offline susan ~/zfs/4
$ sudo zpool status
  pool: susan
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
 scrub: scrub completed after 0h0m with 0 errors on Tue Oct  6 15:22:24 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   DEGRADED     0     0     0
	  raidz1                DEGRADED     0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/4  OFFLINE      0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors
The dd is still running.
$ sudo zpool online susan ~/zfs/4
DD's still going..... DD finally finished, and it took a little longer than the first copy, but it finished, and the file appears correct. Now, let's try something else. With raid, you generally wont just take a drive offline, and then bring it right back, so let's see what happens if you replace the drive. Another dd session, and then the drive swap commands.
$ sudo dd if=/dev/zero of=/susan/testfile2 bs=1024 count=2097512
In another terminal...
$ sudo zpool status
  pool: susan
 state: ONLINE
 scrub: resilver completed after 0h0m with 0 errors on Tue Oct  6 15:26:06 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   ONLINE       0     0     0
	  raidz1                ONLINE       0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/4  ONLINE       0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors
$ sudo zpool offline susan ~/zfs/4
$ sudo zpool replace susan ~/zfs/4 ~/zfs/6
$ sudo zpool status
  pool: susan
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h1m, 25.87% done, 0h3m to go
config:

	NAME                      STATE     READ WRITE CKSUM
	susan                     DEGRADED     0     0     0
	  raidz1                  DEGRADED     0     0     0
	    /home/lagern/zfs/1    ONLINE       0     0     0
	    /home/lagern/zfs/2    ONLINE       0     0     0
	    /home/lagern/zfs/3    ONLINE       0     0     0
	    replacing             DEGRADED     0     0     0
	      /home/lagern/zfs/4  OFFLINE      0     0     0
	      /home/lagern/zfs/6  ONLINE       0     0     0
	    /home/lagern/zfs/5    ONLINE       0     0     0

errors: No known data errors
This procedure seriously degraded the speed of the dd. It also made my music chop, once. After the dd finished, the status was happy again:
$ sudo dd if=/dev/zero of=/susan/testfile2 bs=1024 count=2097512
2097512+0 records in
2097512+0 records out
2147852288 bytes (2.1 GB) copied, 356.92 s, 6.0 MB/s

$ sudo zpool status
  pool: susan
 state: ONLINE
 scrub: resilver completed after 0h4m with 0 errors on Tue Oct  6 15:35:52 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   ONLINE       0     0     0
	  raidz1                ONLINE       0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/6  ONLINE       0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors
Note that 4 is now replaced with 6. Time for some coffee........... Now lets look at some really neat things. I mentioned that you couldn't expand a raid volume. What you can do is replace the disks, with larger ones. Its unclear how this affects your data though (at least, it is unclear to me!) so I'm going to try it. First let's make some larger "disks".
for i in $(seq 9 13); do dd if=/dev/zero of=$i bs=1024 count=4195024; done
Here we are at the beginning
$ sudo zpool status
  pool: susan
 state: ONLINE
 scrub: resilver completed after 0h4m with 0 errors on Tue Oct  6 15:35:52 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   ONLINE       0     0     0
	  raidz1                ONLINE       0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/6  ONLINE       0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors

$ sudo zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
susan  4.00G  3.82G  4.00G  /susan
The new disks i created are 4GB, So we should be able to double the capacity in this pool using these disks.
$ sudo zpool replace susan ~/zfs/1 ~/zfs/9
$ sudo zpool replace susan ~/zfs/2 ~/zfs/10
$ sudo zpool status
  pool: susan
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 12.94% done, 0h6m to go
config:

	NAME                       STATE     READ WRITE CKSUM
	susan                      ONLINE       0     0     0
	  raidz1                   ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/1   ONLINE       0     0     0
	      /home/lagern/zfs/9   ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/2   ONLINE       0     0     0
	      /home/lagern/zfs/10  ONLINE       0     0     0
	    /home/lagern/zfs/3     ONLINE       0     0     0
	    /home/lagern/zfs/6     ONLINE       0     0     0
	    /home/lagern/zfs/5     ONLINE       0     0     0

errors: No known data errors
$ sudo zpool replace susan ~/zfs/3 ~/zfs/11
$ sudo zpool replace susan ~/zfs/6 ~/zfs/12
$ sudo zpool replace susan ~/zfs/5 ~/zfs/13
$ sudo zpool status
  pool: susan
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 8.21% done, 0h5m to go
config:

	NAME                       STATE     READ WRITE CKSUM
	susan                      ONLINE       0     0     0
	  raidz1                   ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/1   ONLINE       0     0     0
	      /home/lagern/zfs/9   ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/2   ONLINE       0     0     0
	      /home/lagern/zfs/10  ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/3   ONLINE       0     0     0
	      /home/lagern/zfs/11  ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/6   ONLINE       0     0     0
	      /home/lagern/zfs/12  ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/5   ONLINE       0     0     0
	      /home/lagern/zfs/13  ONLINE       0     0     0

errors: No known data errors
This took a while, and really hit my system hard. I'd recommend doing this one drive at a time.
$ top

top - 16:12:10 up 25 days,  5:27, 25 users,  load average: 11.36, 9.27, 6.20
Tasks: 280 total,   2 running, 278 sleeping,   0 stopped,   0 zombie
Cpu0  : 10.2%us,  1.3%sy,  0.0%ni, 61.0%id, 27.5%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  1.6%us,  2.9%sy,  0.0%ni,  5.5%id, 89.6%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu2  :  0.7%us,  0.7%sy,  0.0%ni, 92.7%id,  5.9%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  3.9%us,  2.0%sy,  0.0%ni, 94.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  1.0%us,  0.3%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  1.3%us,  2.0%sy,  0.0%ni,  9.8%id, 86.9%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  5.4%us,  6.8%sy,  0.0%ni, 87.3%id,  0.0%wa,  0.0%hi,  0.6%si,  0.0%st
Cpu7  :  1.6%us,  1.3%sy,  0.0%ni, 97.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4121040k total,  4004956k used,   116084k free,    13756k buffers
Swap:  5406712k total,   322328k used,  5084384k free,  1441452k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                       
11021 lagern    20   0 1417m 1.1g  35m S 14.2 26.8   2393:07 VirtualBox                                                    
  313 lagern    20   0 1077m 555m  13m R 12.6 13.8   1089:52 firefox                                                       
22170 root      20   0  565m 221m 1428 S  6.6  5.5   5:57.71 zfs-fuse     
I think i'll go read some things on my laptop while this finishes. Done! Took about 15 minutes to complete. My test files are still present in the pool,
$ ls -lh /susan
total 4.0G
-rw-r--r-- 1 root root 2.1G 2009-10-06 15:27 testfile
-rw-r--r-- 1 root root 2.1G 2009-10-06 15:35 testfile2
My pool does not yet show the new size....
$ sudo zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
susan  4.00G  3.82G  4.00G  /susan
I remounted...
$ sudo zfs umount /susan
$ sudo zfs mount susan
No change.... According to harryd a reboot is necesasry. I'm not in the rebooting mood at the moment. I'll try this, and report back if it doesnt work. So, there you have it, zfs! Oh, another note. raidz is not the only raid option. raidz2 supports two parity drives. Like raid6. You can specify this via the zpool create command, using raidz2 where raidz was. Enjoy! -War

iostat demystified

Recently, we've been looking into our options for a new SAN at work. That I'll save for a whole other post. In our search, it became apparent that we didnt truly understand how much we were utilizing our current system. Our current product requires that we purchase a license in order to check these statistics on the SAN, so we turned to the servers for some more insight. The majority (if not ALL) of our servers are running some flavour of linux, most of which are RHEL 4.x and 5.x. RHEL (and most other distro's) offer a package called sysstat, which includes an I/O reporting tool called iostat. The output of iostat looks something like:
[war@somehost ~]$ iostat -x
Linux 2.6.30.5-43.fc11.i686.PAE (somehost) 	09/24/2009

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.87    0.01    1.52    0.22    0.00   96.38

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.08    24.70    4.79    4.36   258.97   232.48    53.74     0.21   23.45   2.27   2.07
sda1              0.00     0.00    0.00    0.00     0.00     0.00    16.26     0.00    7.58   6.91   0.00
sda2              0.08    24.70    4.79    4.36   258.97   232.48    53.74     0.21   23.45   2.27   2.07
dm-0              0.00     0.00    4.83   28.95   258.63   231.64    14.51     0.04    1.21   0.61   2.07
dm-1              0.00     0.00    0.04    0.10     0.34     0.84     8.00     0.02  120.62   1.57   0.02
This is a bit daunting. Lots of info, and no real descriptions. sda{1,2} are your partitions/mounts, dm-{0,1} are virtual devices used by LVM (if you're using LVM). The rest is somewhat cryptic. The man page for iostat clears things up slightly, but you may not have a full understanding after just reading these descriptions. (from the iostat man page) rrqm/s: The number of read requests merged per second that were queued to the device. wrqm/s: The number of write requests merged per second that were queued to the device. r/s: The number of read requests that were issued to the device per second. w/s: The number of write requests that were issued to the device per second. rsec/s: The number of sectors read from the device per second. wsec/s: The number of sectors written to the device per second. rkB/s: The number of kilobytes read from the device per second. wkB/s: The number of kilobytes written to the device per second. avgrq-sz: The average size (in sectors) of the requests that were issued to the device. avgqu-sz: The average queue length of the requests that were issued to the device. await: The average time (in milliseconds) for I/O requests issued to the device to be served. svctm: The average service time (in milliseconds) for I/O requests that were issued to the device. %util: Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. Personally, I'm working on learning this output, so I'm going to use this blog entry as my notes on what these stats mean, and how they react to disk activity. I'll review all of the stats which i've been able to figure out.

rrqm/s and wrqm/s, r/s and w/s

These are all about read and write requests that had to be queued because the drive was busy when the request came in. You can drive these up with some simple tests. Use DD to write a lot of data to a local disk, and you'll see the wrqm/s, and w/s counters raise. I started iostat, and then started dd, writing a 2GB file to my home directory. dd:
[war@somehost ~]$ dd if=/dev/zero of=foo bs=8k count=262144
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 31.0321 s, 69.2 MB/s
[war@somehost ~]$ 
Now, here's the iostat command, the -x displays extended statistics, and the 1 tells it to refresh every second.
[war@somehost ~]$ iostat -x 1
Linux 2.6.30.5-43.fc11.i686.PAE (somehost) 	09/24/2009

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.71    0.00    2.99    0.00    0.00   93.30

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.44    0.00    5.78   12.15    0.00   80.63

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00  7811.00    3.00  395.00    40.00 29024.00    73.03    48.30   76.60   1.11  44.10
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00  7811.00    3.00  395.00    40.00 29024.00    73.03    48.30   76.60   1.11  44.10
dm-0              0.00     0.00    3.00 8354.00    40.00 66832.00     8.00   949.32   51.63   0.05  44.10
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.29    0.00    4.29   35.00    0.00   56.43

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00 13542.00    0.00  372.00     0.00 108336.00   291.23   141.91  350.92   2.69 100.00
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00 13542.00    0.00  372.00     0.00 108336.00   291.23   141.91  350.92   2.69 100.00
dm-0              0.00     0.00    0.00 13897.00     0.00 111176.00     8.00  5494.24  349.96   0.07 100.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.44    0.00    4.56   32.73    0.00   61.27

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00 18120.00    0.00  468.00     0.00 147456.00   315.08   138.46  316.45   2.14 100.00
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00 18120.00    0.00  468.00     0.00 147456.00   315.08   138.46  316.45   2.14 100.00
dm-0              0.00     0.00    0.00 18592.00     0.00 148736.00     8.00  5450.12  313.56   0.05 100.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda2 is a Logical Volume that contains / on my system. /dev/dm-0 must be the virtual device for that logvol (honestly, i'm guessing here, look at iostat, you'll see what i mean, look at the w/s on dm-0!) Now, let's see if we can get the read counters to raise. First i tried scping a file from my workstation, to my laptop. That didnt really get me the dramatic raise in activity that dd did. Understandably, its a much slower process. Let's see what else i can abuse. I connected my blackberry via usb 2.0. It's got 8gb of memory. This is the closest thing to a usb mass storage device i had handy. This was slightly better, but still not extremely fast. I suppose the best way to stress this would be a local drive to local drive copy. At any rate, i did see the r/s and rrqm/s counters rise while the copy was being performed. Ah Ha! /dev/null is the answer. Copy your 2gb file (created by DD earlier) to /dev/null. You'll see r/s jump. I got about 800 out of my test.

rsec/s and wsec/s

These counters are very similar to r/s and w/s, except that they deal with sectors. Whether these are useful to you are not, depends on what sort of data collection you're looking for. In our example from earlier, you can see the wsec/s rose as w/s and wrqm/s did.
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda2              0.00  7811.00    3.00  395.00    40.00 29024.00    73.03    48.30   76.60   1.11  44.10

await

This is a rather important stat. This tells us how long requests sent to the drive are being forced to wait, in milliseconds. The higher this nubmer gets, the more of a bottleneck we can see in our storage. I'm continuing to work with this utility, I'll post more progress as it comes along. I'm hoping to truly get a feel for the rest of the stats. -War

PeerGuardian lists, imported to iptables.

At home, I have a Smoothwall which connects my network to the internet. It's a very robust replacement for these soho routers that everyone seems to use. It's not quite as plug and play, but it works very well, and I have a lot more control over it.

I also run PeerGuardian, from Phoenix Labs, on my workstations to help block certain access to my machines. Peer Guardian is a great program, and most of the time it works very well. The problem is, sometimes it has issues, and to be honest, I always thought it'd be cleaner to put the firewalling, on my.... Firewall! So i set out to find a way to add peerguardian's lists to my Smoothie.

There's a Project called moblock, which is supposed to do this. Well, i've never seen it work. Thats not to say it doesnt work, i just couldnt get it working on my Smoothie. So for a very long time, i went on using peer guardian locally. Well recently I happened to be watching peer guardian run its update, and realized that it;s pulling its lists from an http address. Makes sense that i might be able to do the same, right? So i pointed my web browser there, and sure enough, i'm presented with a list of rules! Rules that dont match iptables, but look very easy to parse! So, I did just that. I started writing my own parser, and before long, i had a very long list of iptables compatible rules. By very long, I mean long! Over 226000 lines!

I decided that the best way to make this list easy to update was to create a new chain, called PGBLOCK, and put my rules in there. I also created a chain called PGALLOW which supersedes the block list. So i can add exceptions if i'd like.

So, on my Fedora 11 Test machine in added the following to /etc/sysconfig/iptables.

After the chain definitions (the :CHAINNAME [number:number] lines) i added 4 lines.
-N PGALLOW
-N PGBLOCK
-A INPUT -j PGALLOW
-A INPUT -j PGBLOCK

This adds the chains, and adds them to the iptables INPUT chain. This tells iptables to pass all inbound packets through my chains before they even touch any other rules.

At first, i tried entering all of my rules into the PGBLOCK chain. This worked, but delayed every inbound packet to the point that my network connection was almost useless.

So I made a slight change. I made a new chain for each class a. 253 in all (i skipped 10. and 127.), and then i setup more specific rules inside of the PGBLOCK chain. PGBLOCK now contains lines similar to:

-A INPUT -s 1.0.0.0/8 -j PGBLOCK1
-A INPUT -s 2.0.0.0/8 -j PGBLOCK2
..
-A INPUT -s 254.0.0.0/8 -j PGBLOCK254
-A INPUT -s 255.0.0.0/8 -j PGBLOCK255

Now each packet gets subjected to a couple hundred (or thousand) rules instead of 226000 of them.

Wondering if you can get ahold of my script?
Here it is: http://www.undrground.org/scripts/getpg.tar.gz

Making this work is pretty easy.
There are a few variables at the top of the scipt that point to where you'd like some things to be saved. It needs a scratch directory for the lists it downloads. You need write access as the user youre running as, to the directory you're running it from, and the lists directory, of course. But just set all that up, and run the script. It'll generate a file called pg.firewall. Use that along with iptables-restore to build the firewall.

iptables-restore --noflush

Now, updating the firewall is a little more tricky, you need to flush the tables manually before re-importing. I did this with a perl script that looks something like:

#!/usr/bin/perl
foreach (1..255) {
if ($_ eq 10 || $_ eq 127) {
next;
}
system("/usr/sbin/iptables -F PGBLOCK$_");
}
system("/usr/sbin/iptables-restore --noflush

This flushes the tables, and then imports the new list.

I hope this helps someone else out along the way. Enjoy!

-War

Blackberry Storm, First month or so.

I've had my Storm for a little more than a month now. I've decided it's time for that follow-up review. First of all, I've had to have the phone replaced once already. The buttons on the phone stopped working. Of the 4 buttons below the screen (the hangup, call, menu, and back buttons), two of them stopped working on me. After a quick call to Verizon's support, they happily replaced the phone. I still like the phone, but I'm a little wary about it's durability. I've had the replacement phone for about 3 weeks now. It was about a month before the first one had its problems, we'll see how this one fares. Ironically, while i was waiting for my new phone to come in the mail, the old phone slipped out of my hand, and bumped my desk (about a 2" fall) and the buttons started working! I still didnt trust the old phone, and sent it back to verizon as they asked when i got the new phone in the mail. Overall, i like the phone. The screen seems like it could use some work though. If you try to press at any of the edges, it's difficult to get a response. I did find something of a hack online to fix this. You simply take a small piece of paper, and fold it up a few times so that its just a little thicker than a sheet of paper (in my case, 2 folds seemed to do the trick, making it 4x the thickness of just one sheet). Take that folded piece of paper, remove the back of the phone, pull out the battery, and place the piece of paper between the back of the battery, and the inside panel of the phone. Then put the battery back in, and put the cover back on. This for whatever reason, helps the issue. I do not know what long-term effects this may have on the phone, but it doesnt seem like it's putting pressure on anything vital, at least, not enough to cause a problem. Themes for the phone are sometimes finicky. One of the things i really wanted for this phone was an LCARS like theme. It's a touch phone after all. I tried a handfull of them, and found one that i really liked, and used it for a while, but i eventually switched back to the standard theme. Why? because themes don't just change the look, some of them actually change some of the interface. Every one that I tried degraded the user interface, some of them slowed the phone down, some even caused lock-ups! I decided the look just wasn't worth it. Application support is nice on this phone. It's not like an iPhone. I cant find app's that will control every aspect of my life, but that's ok, i dont need that. I have found a number of cool apps. all of which thus far have been free. Including a weather app, apps for twitter, mysapce, and facebook, and even a shoutcast streaming app called flycast. One of my concerns was calendaring and syncing with Zimbra. I'm sad to report that i have NOT found a slick way to handle this. The blackberry will not talk directly to Zimbra, well, i lie, e-mail itself is great. I was able to use BIS to setup a standard imap connection Zimbra, but the calendar and contacts are not there. There's a connector to go from a BES to Zimbra, but we don't have a BES. I played around, forever, with an open source BES like server, called Funambol, but i've had no luck with it. Thus far, the best compromise i've found is syncing with outlook. Which is a bummer, because I don't use outlook regularly. So i need to boot windows, fire up outlook, sync, and then shut outlook back down. Sort of a waste for all that outlook is designed to do, you know? I do still find myself in somewhat envy of the iPhone crowd, but theyre just not an option for me at the moment. I do not want to switch providers, and verizon does not currently offer an iPhone. I wouldn't be surprised to find myself switching to an iPhone if Verizon ever offer's one. -War

Pages