So, here's the situation.  A client reports that he can't log in to a few nodes on their Linux cluster.  Every time he tries to ssh to the nodes in question, he gets dumped out with "Connection closed by 10.0.126.17".  The other nodes, he can get to, no problem.  The strange thing is, I updated the OS on one of the nodes (as an aside, let me just say that do-release-upgrade on Ubuntu is a beautiful thing) and then cloned it to several of the nodes via SystemImager.  Ok, that's not strange.  What is strange is that he could ssh to a couple of the nodes that were updated via the SystemImager route, as well as the golden client, but not to others!

I looked at /etc/hosts.{allow,deny}, I looked at his ~/.ssh directory (but that's mounted via NFS, so it's the SAME on all of the nodes!), I looked at several other things that shouldn't matter, but hell, when you've looked at everything possible, you have to start looking at the impossible, right?

Except, I forgot that when I was upgrading the nodes, I put /etc/nologin in place so people wouldn't log in and cause trouble.

Bloody /etc/nologin!

Well, in my defense, I thought I had deleted that already.  I finally found it by doing a 'diff -r' on the /etc hierarchy of a node to which he could log in to versus one he couldn't.

Saw this on a forum this morning.  I'd seen it before and thought it was funny so I decided to grab it so I won't forget.

ROFLMAO, and all that.

Well, not me per se, but this website.  I just googled it and it doesn't show up at all.  That's not necessarily a bad thing right now, since I don't yet have a point to this website, and since I'm not sure where I will ultimately host, but it is cool to finally have my own domain.  I'm surprised I didn't do it sooner!

For the past several weeks I've been messing around with OpenSolaris (and, to a lesser extent, Solaris).  I'm really in love with ZFS.  It's an awesome filesystem with many cool features that I'm sure I have not begun to tap.  We're hoping to run a backup service (disk-based, of course) using Solaris, and so I've been tasked with all of the setup and testing.  My idea is to basically use ZFS snapshots and rsync to make backups wicked simple.  To do backups, just do rsync to a ZFS partition (or partitions), then take a snapshot.  Bingo, instant point-in-time backup.  (Ok, not really, but it is good enough for our needs).  I've set up a test system and things have been going swimmingly.

Next, I decided to take another system and to some sanity tests to make sure things like, oh, RAID rebuilds, work correctly.

That's when the trouble started.

I have an external 24-bay JBOD (SAS/SATA, we're using SATA drives) from DataOn attached to an LSA SAS 3801E controller, installed in a SuperMicro X8DTN+ based server.  Well, as it turns out, it appears either the controller and/or the external JBOD is exhibiting some kind of incompatibility problem.  I set up all the drives in the JBOD as a ZFS raidz2 pool, and I can write to the damn thing until the cows come home without problems.  However, if I do a simple "scrub" (even without i/o!) the array will end up degrading, and I'm left with a smoked zpool, and usually the thing panics and reboots the server! That's baaaaad.

I tried the card in a different system (just a PC bought from a local MicroCenter) that has a PCIe slot, and, while I don't observe the panic/reboot problem, the array still degrades after 10 minutes or so of a "scrub" or RAID rebuild.  This was all with OpenSolaris, but I did a brief test on the PC with Solaris 10, and the result was pretty much the same.

To compound matters, the SuperMicro box is itself a 24-drive JBOD connected via three SuperMicro JBOD controllers.  I set up a zpool there in much the same way as the external array, and decided to test a simple RAID rebuild while under load.  I yanked one of the drives and the machine just froze.  Completely hung.  Dead.  Shuffled off this mortal..., well, you get the idea.

This all makes me sad, because I was really looking forward to using ZFS.  I guess I will give Linux a shot, but the backup solution I am using on that side doesn't scale very well, for reasons I don't feel like going into now.
 

I'm still trying to decide if I want to host here or move this page to a provider that will let me install something like Drupal.  One one hand, that will be a lot more work, but of course, it's also going to offer a lot more flexibility.  It's all a matter of deciding how much time I want to invest in this endeavor.

Since no one is reading this yet, I don't have to worry about disruptions.  I just checked and I'm not even in the google index yet.

Ok, I think I'm done playing for now.  Welcome to my site.  I'm one of many David Morgans in the world.  After many years messing about, I finally decided to grab my own domain name and do something with it.  I'm an IT guy and am hoping to use this as a place for posting bits of code that I think might be useful to others, as well as for my arbitrary ramblings such as this one.  If you want to leave a comment, click "Comments" on the left side-bar (unless I've changed the layout of my site, in which case you'll have to find it yourself).  Sorry for the kludge but it's the best I can do given the limitations of Google sites.

-DM

So apparently this site is going to be a one-way street, communications-wise.  I've enabled comments in this section, but apparently you need an account on my domain to be able to add them! What a crock.  Oh well, I suppose at least I won't have to worry about forum spam that way...

Update: Oh, and it's even worse.  Apparently I can't just give someone "comment" access.  They either have full edit access or only view access.  That's lame!

Update 2: Ok, there is a kludge-y option for enabling public comments.  You basically have to add a Google docs form and spreadsheet.  Create a form with "Name" and "Comment" and make it public, then insert it and the resulting spreadsheet on a regular web page.  You'll also have to publicly share the spreadsheet, and you'll want to make the "Comment" cell wider, so that it looks nice(r) on the webpage.  Still, what a lot of hoops to jump through. 

As you can see, I am exploring the capabilities of Google sites.  Of course, I could do this in private and not open it to the outside world, but what fun would that be?