Your role as a netizen.

This somewhat links to my post on Privacy Online, if you want a little bit of back story, well, there it is. I thought I'd follow up with a little bit about what we should all be responsible for in our online lives. This is all my opinion. I don't claim to be an authority, but this is my bog dammit. Read if it you want, otherwise, go look up some porn, or start a flame war on Facebook. That's what the internet's for, right? Looking up illicit material, and getting into arguments over politics, religion, pet ownership, or whatever. Right? Well, not really. That's what the internet has turned into. Here's a brief and history lesson (which admittedly comes from my memory, and cites no references). The internet, in its infancy, was used for communication, and early on, but colleges. It was used to internetwork research labs, and libraries, and pass factual information around. So a student in California could collaborate with a student in New York, without leaving his lab. One of my first memories of the internet was researching for a science paper that I had due. It was a bout black holes. My dad had just gotten an internet connection, and we decided to put this thing to use. So we hopped on our blazing fast 14.4kbps dial-up connection, and went to Webcrawler, and started searching. We found a pamphlet online on the subject, it came direct from Greenwich Observatory, in England. This was astounding to me! We'd gone into a computer in England, and gotten information and saved it on my dad's computer, all for the cost of a monthly subscription to an ISP, and a telephone line. Wow! Today, that's not so amazing. I mean, you can video chat with some dude in Asia, while downloading your porn and reading the NY Times. It's second nature to us now, its become integral to our lives. The little story about the research paper, that was only about 15 years ago. Since then, connectivity to the internet has changed, a lot! Bandwidth that would have cost a fortune, and required a leased optical line to your premises back then is now commonplace, and sold for $40/month from your digital cable or DSL provider. At that price, everyone's online now. Even if theyre at the lower end, or even still using the dreaded dial-up service. You'd be hard pressed to find someone who has NO contact with the internet. And its not just on your computer anymore, cel phones, media players, and other hand-held devices. You can pull a device out of your pocket, or off of your belt, or in some cases out of your back pack, and there you have the wealth of information that is the internet at your disposal. I use the term wealth of information loosely. Now you'll see how my little stroll down memory lane ties into the subject of YOUR role as a participant in this global thing called the Internet. See, back in 1995, when i was working on my research paper on black holes, the internet was a different place. Yes, there was porn, and there was illicit material, and even some illegal activities, but there was also this huge amount of factual information. Yea, you heard me right, information, based on fact, as in, the kind you can reference. Not ridiculous opinions presented as fact, not flat out lies presented as fact, but actual fact. Yes, it's still out there, but you need to know how to find it. Is this right? When you go to the library, do you have to read 4 books on the same subject in order to decide if the information you're reading in one book is correct? Sure, if you want to be thorough, but really, if you pick up a book at the library, and its presented to you as fact, there's a pretty good chance that it ACTUALLY IS FACT. What does this have to do as your role as a netizen? Well, not a whole lot, except that you should try to do your best to avoid propagating lies as fact. A good example is these viral chain e-mails and microblog posts about how some such event is going to change the face of (insert service here) and the only way to prevent it is to (insert action here). I can remember these from back as far as the early ICQ days. This was before AOL bought out Mirabillis, and the ICQ service became their property, mid 90's or so. There were messages that spread like wildfire saying things like "Mirabillis is going to start charging for ICQ unless you send this message to every flipping person on the planet!". Or the famous, "Bill gates will send you money if you forward this e-mail on", which went on to say how Microsoft is tracking this e-mail, and if you send it to people, you'll get your slice of some exorbitant amount of money. Nevermind the fact that it's almost impossible for someone to accurately track an e-mail like that. Its not like an e-mail can phone home. At least, not without some sort of attached application. This method of spreading panic has moved its way on to pretty much every social network i've ever been a part of. I see it on facebook, i've seen it on myspace, and so many others before them. Before you spread something like this, think about it, i mean really think about it. Take a ride over to Snopes and try to find an article referencing the item that you're about to continue to spread. If you find reason to believe it's true, then ok, post away, but if you find evidence to the contrary, stop in there, or even better, spread the correct information in its place! Now comes the good part. Protecting yourself online... Imagine the battlefields of old, where wars were waged with swords, maces, bows and arrows. Warriors went into battle wearing armour, and carrying some weapon with which to defend themselves. To do otherwise would be suicide. Now imagine the internet as that battlefield, and your computer as the warrior. If your computer isn't wearing its armour, it's going to get killed. There are a lot of threats out there. Spyware, viruses, adware, attackers... Sounds like a lot to deal with, right? Well, not if you're prepared. First, it's good to be knowledgeable of what you're getting into. Be aware of current threats. I don't mean that you need to go and get a Computer Science degree, but keep an ear to the ground for threats. There are lot of web sites which publish these things. Subscribe to an RSS feed, or just hit up one or two of these sites a few times a week, just to see what's out there. Know your enemy so to speak. Next, consider your computer. Think about what OS you're running, and what its vulnerable to. It's pretty common knowledge that Windows is a target for many viruses and malware. There are a number of reasons for this, but I could write a whole other post about that. So if you're running windows, you need to be on the alert. Keep your computer up to date. Microsoft makes this relatively easy by building automatic updates into your OS. If you cant be bothered with actively going and installing these updates ever so often, then turn on auto-updates, and let them install for you while you're sleeping. The next thing to consider is Anti-Virus. Keep one on your computer, keep it running, and keep it up to date. I personally use AVG Free on my windows machines. Is this the best on the market? No, but it's free, and it works well for me. The last thing to consider is a firewall. Personally, i run a pretty lax firewall on my pc's, but then i protect my network with a firewall in between me and the internet. I do this via a Smoothwall Express machine acting as my internet gateway. It's connected to my cable modem, and then my network accesses the internet through it. You could also use a personal firewall, such as the windows firewall, or a 3rd party application. I could get into a long talk on network security, but I'll spare you. Yea, that was all about Windows, the same ideas apply to other OS's. Personally, I like linux. I run linux on my workstation at work (solely linux, no dual-boot), and my laptop, and home pc both dual boot Windows 7 and Linux. IMHO, you cant beat the security on linux. Again, i could go nuts listing the reasons, but we'll just say that at a core level, linux is just more secure than Windows. Much of this applies to MacOS X as well. The same basic principles apply however. Anti-Virus, and Firewalls are still a good idea. So, we've covered your pc, and the spread of nonsense. How about you, as the user? It's simple really. In the words of Wil Wheaton, "Don't be a dick". Yea, I know it's fun to poke at people from across the world because they have no way to retaliate, other than poking back at you, but really... If you wouldn't do it to someone's face, don't hide behind your computer and do it. We're all guilty of this from time to time, but make an effort, and the 'net will be a better place. Other than that, just try to educate yourself on how the magical machine you're sitting behind works! Yes, I know it's daunting, I know that your dvd player, your computer, and your cel phone all hold quite a bit of mystery. A computer is not that complex of a device when you get down to it. Well, it is, but you can get a general understanding of its parts and what they do without spending 4 years in higher education. Once you have that basic knowledge, you're really much better off than most of the average users on the internet. Go read about it, and learn things, you'd be surprised how simple some of this computer magic really is! Last but not least, protect your identity. There are a lot of people out there who want to collect as much information about you as possible, and use it against you. Whether its for advertising, or identity theft. See my entry on Privacy (linked above) for a more in-depth look at protecting your privacy online. If you get an e-mail, asking you for personal information, please take a really good hard look at it before you blindly reply. Any company who's asking you for your password in an e-mail really needs to take a good hard look at their security practices. Under no circumstances should you ever send someone your personal information in an e-mail unless you've encrypted it using a method that only the intended recipient can decrypt. I mean it, seriously, if a legitimate company asks you for your password in an e-mail, you should either stop dealing with them, or call them and ask them if they're really serious. E-mail, unless you've made an effort to make it otherwise, is clear text. If someone intercepts your message, they can see what's in it. So anyway, there it is, now go, be a productive member of the 'net.

Privacy online.

So I'm listening to online radio, and one of the commercials i keep hearing is for this service called Privacy Defender. They tell you how all of this information that you thought might be private is floating around the internet. The continue to scare the crap out of you by stating that prospective employers, significant others, and others might go find this information online, and it could affect their opinion of you. First I'd like to say that this is nothing more than a company preying on your fear and dis-information to try to make a buck. The idea that you could hire someone to go about erasing your identity from the internet is ridiculous. You may or may not be aware of how "The Internet" works. So I'll give you a basic run down. There is no thing called "The Internet" which is operated or controlled by some group. The network that we perceive as this mysical thing which you can go play games on, chat with your friends, or (imagine this) research things you'd like to know more about, is a conglomeration of many computers, all containing data, which is then shared out to all the rest of the members on the network. If you're reading this blog, on your computer, sitting in your living room, you're an equal member of this network as any other machine that's connected. You could be a web server, you could be a mail server, providing that your ISP allows you to. You control your information. If you don't want others to have it.... Dont share it! Keep in mind, that a lot of your information is considered public knowledge, and can be obtained through the court house, or even your local library. If you've already jumped in both feet, and your information is everywhere, well, go try to get it back. A lot of web site administrators will cooperate with you if you go to them in a professional manner. I don't know if they're legally obligated (or, even if they should be) to remove your data, but ask them, if theyre reasonable, they'll help you out. If they won't, don't sue them. It's different if they've obtained private data about you and published it without your knowledge. If you've posted some embarrassing pictures of you and your college room mate making out at some frat house party in college, you put it there, you should have thought better of it before you posted it. If someone else posted it because they were at the frat house party, and took a picture of you, and you let it happen, then maybe you should have thought of that and tackled them and deleted the picture from their phone. Contacting the site operator is essentially what services like Privacy Defender would do. Of course, i don't know what goes on under the sheets of Privacy Defender, but knowing what i know about running a web server, what else could they do? They can't go and remove things from my server, they have to go to Me, the operator of the server, and ask nicely. The only power they have that you might not, is a knowledge of the law and perhaps sleazy lawyers that know how to exploit it. I recently watched an interesting video, where Eben Moglen spoke in front of a group about "Freedom in the cloud". I'll embed the video in a moment, if you'd like to watch it. He made some very good points regarding privacy, a user's role in the internet, and how we've all blindly given our information away. If you use Facebook, i'd highly recommend you watch this video. It's a little dry, and some of it's a bit technical, but it brings out some very important details about what the guys at facebook (and other social networking sites for that matter) can do with your data! So, why am I posting this? Well, watching that video really made me think about our society, and how intertwined our lives are with our online lives. Slowly they're becoming one and the same. This is a cool trend, but also a scary one! When companies like facebook can, via a nicely worded EULA, claim ownership of all of the data you decide to post, where does your privacy go? When i started getting involved with computers, BBS's, web communities, and things of the like. I always used an alias, an avatar, to identify myself. This is how it was done, this is how everyone did it. If someone signed up with their full name as their username, you knew they were naive, and not privacy minded. At that time, if you chose to sign up with some online community, the only person who had access to your data (other than the data you chose to make public) was the operator. Was there still a chance of that operator doing nasty things with your data? Yes, but that was your responsibility. You tried to keep track of who you were giving your data to, and what you gave them. Today, we share some of our most private data without a through on sites like MySpace, and Facebook. Sexual preference, the town you live in, how many kid's you've got, what their names are, what your favorite hobbies are, where you work, your political affiliation. I could build a profile on you based on the information you post daily on Facebook. On top of that, i can find out what you looked like from your picture gallery, or if i were a predator, what you're 14 year old daughter looked like... Combine all of that data, and someone could literally show up at your door, or stalk you at work, or god forbid, your children at school. Am i saying we should all leave social networking en-mass? No, just be careful what you make public. Keep in mind that ANYONE on the internet can find that data. Dont set the password to your bank account online to some data that i could phish out of your facebook profile. -War

Rate limiting outbound mail in Zimbra, with Policyd

I've recently found a need for outbound rate limiting in Zimbra. The idea is that this will help prevent a spam outbreak, sourcing from one of my users accounts, if it's compromised. This has happened in the past, and it's finally time to put a stop to it. There are two options available to me for limiting outbound mail. First is within Zimbra, the other is via out outbound spam/virus filter which is handled by a ProofPoint Protection Server. This is basically an e-mail firewall, similar to Barracuda, or Postini. I spoke with our support personnel at ProofPoint, and they were less than helpful. Not on the whole, they're generally very good, I would in fact recommend them to others. The system is sound, and flexible, and their support is usually great. In this case, I was simply asking for something that this system cannot offer. Because this is an outbound limiter, i need to rate limit based on sender, not sending server. Apparently the PPS just doesn't have that sort of tracking. So I moved on to Zimbra. Out of the box, Zimbra does not have this feature either. However, Zimbra's MTA is build on Postfix. A relatively clean, stock postfix. Most postfix mod's also work on Zimbra's MTA. I did a little digging, a little asking around on Zimbra's forums, and a little research on my own and came up with Policyd. Policyd is a service which runs on either your local postfix box, or on a remote box, which manages lists of policies. Postfix queries policyd for every transaction, and asks it if it has a policy which pertains to this transaction. Once it gets the result of said policy, it moves on with the session. Now, there are tons of things which can be done with policyd, I'm focusing on JUST outbound, and JUST quota based rate limiting.

4.2U Wii, homebrew hack.

A while ago, a friend of mine helped me exploit my Wii, so that I could run the Homebrew channel. There are many advantages to doing this, and for lots of reasons. Not just game piracy. Homebrew lets you install 3rd party apps on your Wii, which allow you to get some creative use out of the system. You can also connect USB storage to your Wii, and use it to store your games, so that you dont need to swap discs around when you want to play a game. When we hacked the Wii, it was running update 3.6. I'd stopped updating the Wii from Nintendo's update procedure because I assumed that they would probably tromp on my hacked wii, and restore the OS to its pristine condition, and I'd then have to go through re-hacking. Well, I knew this time would come. I was forced into an update, so I could run the New Super Mario Brothers game. I guess it has something to do with the ability for the players to interact as they do in the game. So, i bit the bullet, and ran the update. It wasn't until after i ran the update that i found a number of reports on the 'net about how to update a hacked Wii, without actually running nintendo's update. This would have kept my Homebrew intact, and still gotten me the update. But, oh well, I was committed at this point. After some digging, I found a number of how-to's on the subject of hacking a 4.2U Wii. 4.2U being the most recent update. I tried a number of them, and none of them worked. Until i found this how-to, which was SOOOOO much easier to follow than some of the others. Since i'd been through so many other failed attempts at hacking this system, and since it was hacked once before, i decided now was a good time to backup my save data, and format the Wii's system memory. Luckily, i was able to backup all of my save data to an SD card, and then save that data to my PC. I then performed the format, but wasnt able to put my save data back on the Wii until after i'd run each game at least once. This meant i either had to break out all of my discs, or get homebrew and the usb loader working before i could get my save data off of my sd card. So i started down the road to Wii-freedom. In the end, it wasnt all that difficult, and i ended up with bootmii, which is a more secure replacement for Wii's boot2 loader, and the homebrew channel. I also have all of my USB capability back, and once i get all of my homebrew software reinstalled, i'll have all of the functionality that i had before the format. On top of all that, i can play Mario now. :D

Some good ol house music for your Tuesday.

I'm into quite a few types of music. Mostly anything that has a good beat, and a deep sound. I always imagine that a lot of people who have been exposed to my taste in music are baffled by the seeming variety of genre's I listen to. There's a theme believe it or not. Its hard to describe. I like bass, and I like guitar. If its got a deep sound, I like it. This can be found in a few genre's commonly. Mostly electronic music, like House, and Trance, and a lot of Rock/Metal has the same element. So, most of what I listen to falls into these few genre's. One of the DJ's that I started following while XM's "The System" was still on the satellite waves, is Zoltar. He had two shows, The Mutant Dance Party, and Subterranean. Both are shows where he plays mixes of club music. Mostly house. One of these shows can still be heard on Sirius/XM's "Area", though its never on when I'm able to listen since Sirius/XM has so royally screwed up the pricing for XM customers that I can no longer listen online (as i refuse to pay extra for a service which used to be included in my package), but i'm getting off topic. You can still download podcasts of Zoltar at http://zoltar.podomatic.com, so this is how I currently listen to Zoltar. I downloaded his most recent show, which is a year-end review, a best-of 2009 show. One of the songs that caught my eye.. er.. ear was Melleefresh, and Deadmau5 doing "Hey baby". Nice beat, suggestive lyrics, but that's ok. So I started hunting around online for an MP3, so I can listen to just that song whenever i'd like. I found one, but in my search I also found a rather amusing YouTube video. Something to think about before watching this video is this. These are usually one-off DJ tracks. A lot of them are not really professionally recorded. Most of them do not have videos produced by the artist. So, fans will create videos which sort of fit. Most of them are videos just for the sake of being videos, some feature a still image and just play the music. This one is a mash-up of a ton of different suggestive animations, which have been floating around the 'net forever. So, if you don't mind suggestive lyrics, lots of looped almost-nakedness, and a deep progressive beat, then check out this video. :D

Star Trek Online.

Well, it looks like the game's going to make it out the gate! After quite a bit of ups and downs. Cryptic recently announced a release date (02/02/2010) and the game is available for pre-order. You also get some cool extra's for pre-ordering. I've pre-ordered, you should too! :P

Categories

Just a quick note. I've taken a little time to setup categories, and then took a bit more time moving all of my entries into their appropriate categories. Given the random nature of this blog, i thought this would be helpful for those of you who like reading my tech articles, but dont really care about my other topics. Now you can subscribe via RSS to just the categories you'd like to read. Enjoy!

SLES 10, your kernel is not safe!

So, i recently came across a startling discovery. On a SLES 10 server, when you install a kernel update, the update process kindly DELETES your old kernel. It's not clear to me yet if it does this after its next succesful reboot, or if it does it during the update. In other words, the people at SuSE/Novell are _SO_ confident that you'll never have a problem with a brand spankin new kernel, that they perform without a net. I'm a very conservative sysadmin. I don't like to do anything without a backup plan. When it comes to kernel updates, that backup plan is option 1 in my grub.conf (option 1 being the second option in my boot list, generally, my old kernel). From what I'm reading, there's also no way to tell the update process NOT to delete the old kernel. So you're sort of stuck with this behaviour. This actually bit us a few days ago, when due to some rather odd circumstances, we ended up with a SLES that was trying to boot a kernel that was 1 revision old. Because SLES thought it had cleaned up this kernel, the /lib/modules// directory for this kernel was empty. This obviously caused some confusion on the kernel's part, and it refused to boot. If the update process had left the older boot/module files alone, and left it up to a responsible sysadmin to clean up old kernels when they saw fit, this wouldn't have happened. Granted, in this case, the server had other issues, but that's a different story. So I've set out to fix this. Giving yourself some peace of mind is as simple as taking your kernel, and its modules, and locking copies of them away in a safe deposit box (or at least a backup directory) during the update process. And then putting them somewhere accessible afterwards, then re-add ing the old kernel to grub. This is all well and good, if you had one, maybe two servers to worry about, go ahead and do it manually. If you have a couple dozen, this is a considerable amount of work to do manually, and it takes up your time! So, i wrote a script to do it for you! It's a perl script, and it should run on a base install of SLES (or, so it has in my testing). You can download it here. Just download it to your SLES server, and run it, it'll do the work for you. Run it before your update, and select option 1, which backs up the kernel. Then run it again after the update, and select option 2, which restores the kernel. Enjoy! -War

Zettabyte File System (ZFS)

We've been doing a lot of storage research lately, and there's been a lot of talk about ZFS. I'm going to spare you the magazine article (if you want to read more on what it is, and where it comes from, look elsewhere) and give you some guts. ZFS is a 128-bit file system, and unfortunately isnt likely to be built into the linux kernel anytime soon. You can however, use it in userspace, using zfs-fuse, similarly to how you might use NTFS on linux (for those of us still dual booting). The machine i'm running on, runs solely Fedora Core 11, and has a handsome amount of beef behind it. It's also got 500gb of local storage, so I can play around with huge files no sweat. You can do the same things i'm doing, with smaller files, if you'd like. First of all, you'll need to install zfs-fuze, this was simple on Fedora.
$ sudo yum install zfs-fuse
Next some blank disk images to toy with.
$ mkdir zfs
$ cd zfs
$ for i in $(seq 8); do dd if=/dev/zero of=$i bs=1024 count=2097152;done
This gives me 8, 2gb blobs. Make these smaller if you'd like. I wanted enough space to throw some large files at zfs. You'll see in a bit. Now let's make our first zfs pool.
$ sudo zpool create jose ~/zfs/1 ~/zfs/2 ~/zfs/3 ~/zfs/4 
I named my pool jose. I like it when my blog entries have personality. :P zfs list will give you a list of your zfs pools.
$ sudo zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
jose    72K  7.81G    18K  /jose
Creating the pool also mounts it.
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      454G  210G  221G  49% /
/dev/sda1             190M   30M  151M  17% /boot
tmpfs                 2.0G   25M  2.0G   2% /dev/shm
jose                  7.9G   18K  7.9G   1% /jose
An interesting note. I never created a file system on this pool, i just told zfs to have at it. zfs must work at a block level with the drives. Now, let's poke jose with a stick, and see what he does.
$ sudo dd if=/dev/zero of=/jose/testfile bs=1024 count=2097512
2097512+0 records in
2097512+0 records out
2147852288 bytes (2.1 GB) copied, 118.966 s, 18.1 MB/s

$ sudo zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
jose  2.00G  5.81G  2.00G  /jose

Its worth note, that with a zpool add /dev/whatever you can add space to a pool of this sort. That's all fun, but this is essentially just a large file system. No really cool features yet. Let's see what we can really so with this thing. Let's make a raid group, instead of just a standard pool. Goodbye Jose
$ sudo zpool destroy jose
From jose's ashes, lets make a new pool.
$ sudo zpool create susan raidz ~/zfs/1 ~/zfs/2 ~/zfs/3 ~/zfs/4
$ sudo zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
susan  92.0K  5.84G  26.9K  /susan
Notice that susan is smaller than jose, using the same disks. This isn't because susan has made more trips to the gym than jose, rather it's because of the raid set. This is similar to raid 5, where one disk is taken for parity. So you lose a one disk worth of capacity. Let's remedy that, by throwing more (virtual) hardware at it. You cant expand a raid group, by adding a disk, so we'll do it by recreating the group.
$ sudo zpool destroy susan
$ sudo zpool create susan raidz ~/zfs/1 ~/zfs/2 ~/zfs/3 ~/zfs/4 ~/zfs/5
$ sudo zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
susan  98.3K  7.81G  28.8K  /susan
And there you go, about 8gb again. Now let's poke susan with a stick. First, here's her status:
$ sudo zpool status
  pool: susan
 state: ONLINE
 scrub: scrub completed after 0h0m with 0 errors on Tue Oct  6 15:22:24 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   ONLINE       0     0     0
	  raidz1                ONLINE       0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/4  ONLINE       0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors
Now we'll dd another file to susan, and we'll see if we can damage the array.
$ sudo dd if=/dev/zero of=/susan/testfile bs=1024 count=2097512
Then, in another terminal...
$ sudo zpool offline susan ~/zfs/4
$ sudo zpool status
  pool: susan
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
	Sufficient replicas exist for the pool to continue functioning in a
	degraded state.
action: Online the device using 'zpool online' or replace the device with
	'zpool replace'.
 scrub: scrub completed after 0h0m with 0 errors on Tue Oct  6 15:22:24 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   DEGRADED     0     0     0
	  raidz1                DEGRADED     0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/4  OFFLINE      0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors
The dd is still running.
$ sudo zpool online susan ~/zfs/4
DD's still going..... DD finally finished, and it took a little longer than the first copy, but it finished, and the file appears correct. Now, let's try something else. With raid, you generally wont just take a drive offline, and then bring it right back, so let's see what happens if you replace the drive. Another dd session, and then the drive swap commands.
$ sudo dd if=/dev/zero of=/susan/testfile2 bs=1024 count=2097512
In another terminal...
$ sudo zpool status
  pool: susan
 state: ONLINE
 scrub: resilver completed after 0h0m with 0 errors on Tue Oct  6 15:26:06 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   ONLINE       0     0     0
	  raidz1                ONLINE       0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/4  ONLINE       0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors
$ sudo zpool offline susan ~/zfs/4
$ sudo zpool replace susan ~/zfs/4 ~/zfs/6
$ sudo zpool status
  pool: susan
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h1m, 25.87% done, 0h3m to go
config:

	NAME                      STATE     READ WRITE CKSUM
	susan                     DEGRADED     0     0     0
	  raidz1                  DEGRADED     0     0     0
	    /home/lagern/zfs/1    ONLINE       0     0     0
	    /home/lagern/zfs/2    ONLINE       0     0     0
	    /home/lagern/zfs/3    ONLINE       0     0     0
	    replacing             DEGRADED     0     0     0
	      /home/lagern/zfs/4  OFFLINE      0     0     0
	      /home/lagern/zfs/6  ONLINE       0     0     0
	    /home/lagern/zfs/5    ONLINE       0     0     0

errors: No known data errors
This procedure seriously degraded the speed of the dd. It also made my music chop, once. After the dd finished, the status was happy again:
$ sudo dd if=/dev/zero of=/susan/testfile2 bs=1024 count=2097512
2097512+0 records in
2097512+0 records out
2147852288 bytes (2.1 GB) copied, 356.92 s, 6.0 MB/s

$ sudo zpool status
  pool: susan
 state: ONLINE
 scrub: resilver completed after 0h4m with 0 errors on Tue Oct  6 15:35:52 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   ONLINE       0     0     0
	  raidz1                ONLINE       0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/6  ONLINE       0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors
Note that 4 is now replaced with 6. Time for some coffee........... Now lets look at some really neat things. I mentioned that you couldn't expand a raid volume. What you can do is replace the disks, with larger ones. Its unclear how this affects your data though (at least, it is unclear to me!) so I'm going to try it. First let's make some larger "disks".
for i in $(seq 9 13); do dd if=/dev/zero of=$i bs=1024 count=4195024; done
Here we are at the beginning
$ sudo zpool status
  pool: susan
 state: ONLINE
 scrub: resilver completed after 0h4m with 0 errors on Tue Oct  6 15:35:52 2009
config:

	NAME                    STATE     READ WRITE CKSUM
	susan                   ONLINE       0     0     0
	  raidz1                ONLINE       0     0     0
	    /home/lagern/zfs/1  ONLINE       0     0     0
	    /home/lagern/zfs/2  ONLINE       0     0     0
	    /home/lagern/zfs/3  ONLINE       0     0     0
	    /home/lagern/zfs/6  ONLINE       0     0     0
	    /home/lagern/zfs/5  ONLINE       0     0     0

errors: No known data errors

$ sudo zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
susan  4.00G  3.82G  4.00G  /susan
The new disks i created are 4GB, So we should be able to double the capacity in this pool using these disks.
$ sudo zpool replace susan ~/zfs/1 ~/zfs/9
$ sudo zpool replace susan ~/zfs/2 ~/zfs/10
$ sudo zpool status
  pool: susan
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 12.94% done, 0h6m to go
config:

	NAME                       STATE     READ WRITE CKSUM
	susan                      ONLINE       0     0     0
	  raidz1                   ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/1   ONLINE       0     0     0
	      /home/lagern/zfs/9   ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/2   ONLINE       0     0     0
	      /home/lagern/zfs/10  ONLINE       0     0     0
	    /home/lagern/zfs/3     ONLINE       0     0     0
	    /home/lagern/zfs/6     ONLINE       0     0     0
	    /home/lagern/zfs/5     ONLINE       0     0     0

errors: No known data errors
$ sudo zpool replace susan ~/zfs/3 ~/zfs/11
$ sudo zpool replace susan ~/zfs/6 ~/zfs/12
$ sudo zpool replace susan ~/zfs/5 ~/zfs/13
$ sudo zpool status
  pool: susan
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h0m, 8.21% done, 0h5m to go
config:

	NAME                       STATE     READ WRITE CKSUM
	susan                      ONLINE       0     0     0
	  raidz1                   ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/1   ONLINE       0     0     0
	      /home/lagern/zfs/9   ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/2   ONLINE       0     0     0
	      /home/lagern/zfs/10  ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/3   ONLINE       0     0     0
	      /home/lagern/zfs/11  ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/6   ONLINE       0     0     0
	      /home/lagern/zfs/12  ONLINE       0     0     0
	    replacing              ONLINE       0     0     0
	      /home/lagern/zfs/5   ONLINE       0     0     0
	      /home/lagern/zfs/13  ONLINE       0     0     0

errors: No known data errors
This took a while, and really hit my system hard. I'd recommend doing this one drive at a time.
$ top

top - 16:12:10 up 25 days,  5:27, 25 users,  load average: 11.36, 9.27, 6.20
Tasks: 280 total,   2 running, 278 sleeping,   0 stopped,   0 zombie
Cpu0  : 10.2%us,  1.3%sy,  0.0%ni, 61.0%id, 27.5%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  1.6%us,  2.9%sy,  0.0%ni,  5.5%id, 89.6%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu2  :  0.7%us,  0.7%sy,  0.0%ni, 92.7%id,  5.9%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  3.9%us,  2.0%sy,  0.0%ni, 94.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  1.0%us,  0.3%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  1.3%us,  2.0%sy,  0.0%ni,  9.8%id, 86.9%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  5.4%us,  6.8%sy,  0.0%ni, 87.3%id,  0.0%wa,  0.0%hi,  0.6%si,  0.0%st
Cpu7  :  1.6%us,  1.3%sy,  0.0%ni, 97.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4121040k total,  4004956k used,   116084k free,    13756k buffers
Swap:  5406712k total,   322328k used,  5084384k free,  1441452k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                       
11021 lagern    20   0 1417m 1.1g  35m S 14.2 26.8   2393:07 VirtualBox                                                    
  313 lagern    20   0 1077m 555m  13m R 12.6 13.8   1089:52 firefox                                                       
22170 root      20   0  565m 221m 1428 S  6.6  5.5   5:57.71 zfs-fuse     
I think i'll go read some things on my laptop while this finishes. Done! Took about 15 minutes to complete. My test files are still present in the pool,
$ ls -lh /susan
total 4.0G
-rw-r--r-- 1 root root 2.1G 2009-10-06 15:27 testfile
-rw-r--r-- 1 root root 2.1G 2009-10-06 15:35 testfile2
My pool does not yet show the new size....
$ sudo zfs list
NAME    USED  AVAIL  REFER  MOUNTPOINT
susan  4.00G  3.82G  4.00G  /susan
I remounted...
$ sudo zfs umount /susan
$ sudo zfs mount susan
No change.... According to harryd a reboot is necesasry. I'm not in the rebooting mood at the moment. I'll try this, and report back if it doesnt work. So, there you have it, zfs! Oh, another note. raidz is not the only raid option. raidz2 supports two parity drives. Like raid6. You can specify this via the zpool create command, using raidz2 where raidz was. Enjoy! -War

iostat demystified

Recently, we've been looking into our options for a new SAN at work. That I'll save for a whole other post. In our search, it became apparent that we didnt truly understand how much we were utilizing our current system. Our current product requires that we purchase a license in order to check these statistics on the SAN, so we turned to the servers for some more insight. The majority (if not ALL) of our servers are running some flavour of linux, most of which are RHEL 4.x and 5.x. RHEL (and most other distro's) offer a package called sysstat, which includes an I/O reporting tool called iostat. The output of iostat looks something like:
[war@somehost ~]$ iostat -x
Linux 2.6.30.5-43.fc11.i686.PAE (somehost) 	09/24/2009

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.87    0.01    1.52    0.22    0.00   96.38

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.08    24.70    4.79    4.36   258.97   232.48    53.74     0.21   23.45   2.27   2.07
sda1              0.00     0.00    0.00    0.00     0.00     0.00    16.26     0.00    7.58   6.91   0.00
sda2              0.08    24.70    4.79    4.36   258.97   232.48    53.74     0.21   23.45   2.27   2.07
dm-0              0.00     0.00    4.83   28.95   258.63   231.64    14.51     0.04    1.21   0.61   2.07
dm-1              0.00     0.00    0.04    0.10     0.34     0.84     8.00     0.02  120.62   1.57   0.02
This is a bit daunting. Lots of info, and no real descriptions. sda{1,2} are your partitions/mounts, dm-{0,1} are virtual devices used by LVM (if you're using LVM). The rest is somewhat cryptic. The man page for iostat clears things up slightly, but you may not have a full understanding after just reading these descriptions. (from the iostat man page) rrqm/s: The number of read requests merged per second that were queued to the device. wrqm/s: The number of write requests merged per second that were queued to the device. r/s: The number of read requests that were issued to the device per second. w/s: The number of write requests that were issued to the device per second. rsec/s: The number of sectors read from the device per second. wsec/s: The number of sectors written to the device per second. rkB/s: The number of kilobytes read from the device per second. wkB/s: The number of kilobytes written to the device per second. avgrq-sz: The average size (in sectors) of the requests that were issued to the device. avgqu-sz: The average queue length of the requests that were issued to the device. await: The average time (in milliseconds) for I/O requests issued to the device to be served. svctm: The average service time (in milliseconds) for I/O requests that were issued to the device. %util: Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. Personally, I'm working on learning this output, so I'm going to use this blog entry as my notes on what these stats mean, and how they react to disk activity. I'll review all of the stats which i've been able to figure out.

rrqm/s and wrqm/s, r/s and w/s

These are all about read and write requests that had to be queued because the drive was busy when the request came in. You can drive these up with some simple tests. Use DD to write a lot of data to a local disk, and you'll see the wrqm/s, and w/s counters raise. I started iostat, and then started dd, writing a 2GB file to my home directory. dd:
[war@somehost ~]$ dd if=/dev/zero of=foo bs=8k count=262144
262144+0 records in
262144+0 records out
2147483648 bytes (2.1 GB) copied, 31.0321 s, 69.2 MB/s
[war@somehost ~]$ 
Now, here's the iostat command, the -x displays extended statistics, and the 1 tells it to refresh every second.
[war@somehost ~]$ iostat -x 1
Linux 2.6.30.5-43.fc11.i686.PAE (somehost) 	09/24/2009

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.71    0.00    2.99    0.00    0.00   93.30

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.44    0.00    5.78   12.15    0.00   80.63

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00  7811.00    3.00  395.00    40.00 29024.00    73.03    48.30   76.60   1.11  44.10
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00  7811.00    3.00  395.00    40.00 29024.00    73.03    48.30   76.60   1.11  44.10
dm-0              0.00     0.00    3.00 8354.00    40.00 66832.00     8.00   949.32   51.63   0.05  44.10
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.29    0.00    4.29   35.00    0.00   56.43

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00 13542.00    0.00  372.00     0.00 108336.00   291.23   141.91  350.92   2.69 100.00
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00 13542.00    0.00  372.00     0.00 108336.00   291.23   141.91  350.92   2.69 100.00
dm-0              0.00     0.00    0.00 13897.00     0.00 111176.00     8.00  5494.24  349.96   0.07 100.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.44    0.00    4.56   32.73    0.00   61.27

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00 18120.00    0.00  468.00     0.00 147456.00   315.08   138.46  316.45   2.14 100.00
sda1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00 18120.00    0.00  468.00     0.00 147456.00   315.08   138.46  316.45   2.14 100.00
dm-0              0.00     0.00    0.00 18592.00     0.00 148736.00     8.00  5450.12  313.56   0.05 100.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
/dev/sda2 is a Logical Volume that contains / on my system. /dev/dm-0 must be the virtual device for that logvol (honestly, i'm guessing here, look at iostat, you'll see what i mean, look at the w/s on dm-0!) Now, let's see if we can get the read counters to raise. First i tried scping a file from my workstation, to my laptop. That didnt really get me the dramatic raise in activity that dd did. Understandably, its a much slower process. Let's see what else i can abuse. I connected my blackberry via usb 2.0. It's got 8gb of memory. This is the closest thing to a usb mass storage device i had handy. This was slightly better, but still not extremely fast. I suppose the best way to stress this would be a local drive to local drive copy. At any rate, i did see the r/s and rrqm/s counters rise while the copy was being performed. Ah Ha! /dev/null is the answer. Copy your 2gb file (created by DD earlier) to /dev/null. You'll see r/s jump. I got about 800 out of my test.

rsec/s and wsec/s

These counters are very similar to r/s and w/s, except that they deal with sectors. Whether these are useful to you are not, depends on what sort of data collection you're looking for. In our example from earlier, you can see the wsec/s rose as w/s and wrqm/s did.
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda2              0.00  7811.00    3.00  395.00    40.00 29024.00    73.03    48.30   76.60   1.11  44.10

await

This is a rather important stat. This tells us how long requests sent to the drive are being forced to wait, in milliseconds. The higher this nubmer gets, the more of a bottleneck we can see in our storage. I'm continuing to work with this utility, I'll post more progress as it comes along. I'm hoping to truly get a feel for the rest of the stats. -War

Pages