oooh snap! Figures - IBM in talks to buy Sun Microsystems

The Yahoo Story.

Totally stuck up dot-com ridealong company gets bought by suit-and-tie company able to admit that they sometimes don't know what they're doing.

This will end in tears.

Let me see now, I saw a picture last night that reminds me of this situation:
















Credit to the good crew at AR15.com - they make some of the funniest pictures.

Jumpstart, X-series, T-series, oh my!

So I have been working at this gig at a client site. They're pretty much squared away, and don't question that there are huge advantages to minimizing the installed software on systems. Stopping unneeded services isn't enough. With operating systems as big as Solaris 10, you'll blow your maintenance windows if the latest patch cluster has to patch everything. Never mind that you're enabling the intruders with a rather rich tool set should they actually manage the breach.

So this brings me to my next least favorite topic: Sun's Jumpstart.

For a tool so versatile and well thought out, it's still incredibly frustrating to use. Here is why:

1. Solaris DHCP server

This piece of software is only slightly more obtuse than the Lotus Notes 3.1 API. Really. And ISC DHCPd doesn't work on the platform, because the Solaris TCP/IP stack and runtime pretty much break it. The feature to kind-of make it work (the dreaded #define USE_SOCKETS) has long been orphaned in the ISC source trees.

Watching a sysadmin's eyes bug out when he sees the script needed to initialize a Sun DHCP server for jumpstart installation correctly (note, that's NOT how Sun documents it, that's even scarier) is always fun to watch.

2. Sun's Intel servers

Clearly, Sun's product management listened to their #1 customer: the people who get paid by the hour to manage systems for others. The user interface is slightly better than that of Dell x7xx series ERA/O systems, meaning you can run TWO remote consoles with their java client before either your hosting java process runs out of memory or you run out of CPU cycles.

The firmware is so finnicky on their remote management cards you may have to downgrade, and then upgrade it in order to make the remote console see your keystrokes - with the customary three second delay (thanks, Java!).

Clearly, this was written by developers who have no idea that not everyone has a quad core processsor and 8 GB of RAM.

3. Let's break OpenBoot too

So I figured I'd turn to the beloved blades with T-series processors for some relief. Boy, was I wrong. They must have hired a professional to screw up their new OpenBoot docs. Is it "boot net - install dhcp" or "boot net dhcp - install" or "boot net dhcp - install dhcp"? How do you clear the boot environment when someone has fiddled with that stuff?

I gave up, went back to BOOTP.

Where do they think this shit up?

The client engagement contact noted that he thought it was odd to see me hitting my head on the monitor every few minutes, and canceled the LCD upgrade for the workstation...

More Sun Silver Hardware, oh my...

So as it turns out, the last couple of weeks were spent poking at the newfangled V445 servers. Just like a V440, right? Wrong. Sure, faster CPU's, smaller package, SAS drives, cool silver look (gunning to be acquired by Apple, aren't you?), plus these handy features:

• Same RAID controller as the T series. Wipe the disks, disk label missing, jumpstarts go into swedish chef mode (bork bork bork)...
  
  
• Even lower quality control than the V440's. Do you throw them all off the back of a truck a couple of times to make sure? Re-seating everything in the case is not fun.
  
  
• The OpenBoot PROM is annoying - once it detects something as failed, and you clear the error, the system still boots with a warning message "The following devices have failed: " and list nothing. I guess you saved on that if statement.

In all fairness, all is not bad. It seems that the Sun-branded drives from Seagate are no longer quality-control rejects. At my last job, on a set of 30 V440's we were replacing two hard drives every month. Not officially, of course - it's all under non-disclosure... zzzzapp! What was that, oh, a cosmic ray made everything reboot...

NO CARRIER

More Sun T2000 "minor changes"

As the Sun hardware manual for the T2000 series states that "minor changes" would need to be made to installations of Solaris 10 on that hardware, I thought I'd enlighten everyone what these changes actually are.

1. As stated previously, this is the only hardware from Sun on which hardware drive mirroring for boot drives must be configured before installing the OS. This is bad enough. Read on.
   
   
2. When you add this to a pre-installation script to create the mirror before the disk is set up in JumpStart (tm) (r), the whole install bombs out with a "unable to write VTOC label" error. This is because while the drives are being synced for the (hopefully only) first time, some weird lock gets set.
   
   
3. When the sync operation on the drives completes, both mirrored disks are wiped. At least it warns you. The problem is, they're binary-wiped, and don't have the Sun format disklabel magic at the beginning. Not fixing this causes the next attempt to JumpStart the machine to bomb out too.
   
   
4. On the T2000's, one obscure reference in this document alludes to the need to add the line "set pcie:pcie_aer_ce_mask=0x1" to /etc/system. Mmmmkay, I understand this is an issue, but for gripes' sake why put it into that document? Nobody looks there, which brings me to point 5:
   
   
5. Your search engine and user interface for docs.sun.com is probably one of the worst web sites ever created my man. It's slow as molasses, and the searches return little of value. Most people prefix "docs.sun.com:" to their Google searches because that sucks a whole lot less.
   
   
6. And why do the issue numbers listed in the above-mentioned documents not link to the bug database? Not a bug, but a feature, huh?
   

I think as long as people depend on consultants to stand these systems up, nobody will ever really know the cost of ownership of this garbage.

Ah... it's a paycheck, and it pays by the hour...

Sun, Solaris, oh my...

So this place I work for sends me over to the client site to clear up some "issues" with "following industry best practice." What a shock I was in for. Seriously. I don't blame Sun entirely for this, but they sure don't make it easy. Here is what I discovered:

1. Sun systems administration courses apparently tell you to install everything on a production machine. Everything, including the (now in Solaris 10) gnome palm-pilot development utilities. Including the kanji terminals. Security? Fuck it, let's install the Sun Web Console and webmin. Imagine the surprise when people are shown this document (pdf).
   
   
2. With that confusion out of the way, it seems that true points of pain never seem to get addressed, and in fact seem to get exacerbated with every version. Let's talk Solaris Zones. For the marketecture terminology-allergic like me, this is effectively a collection of scripts to control resources to chroot environments, with some things (like NICs) virtualized. It's not a true virtual machine, but good enough. So, try to do a minimal install that supports that... just once. Not only do you need to install half of X windows (always useful on servers without video cards!), but basically another 300MB of software. Yah, well thought out, that is. So, nicely branded userlinux on Solaris with cute create scripts, but you need to install all this other crap. Thanks.
   
   I heard this got fixed in "Solaris Express" - the Sun beta operating system. Nice, sounds like a Red Hat release with a .0 on the end.
   
   
3. Solaris package management system still sucks. You still have all these undocumented dependency chains all over the place (see complaint #2, above). Yeah, I heard of Sun's Jet system, but adding DHCP to the environment there was out of the question.
   
   
4. Every vendor for sun software (that would be PeopleSoft and Oracle...) has some jacked up dependencies they need installed in the OS. I have no idea why they do this, other than developers being too bored, and wanting to re-invent the wheel again ("but mine is better, really, it's square!"). Sure, installing everything solves this, since there is no need to experiment. Plus, you can use the X-windows installers. Yeah...
   
   
5. Now, don't get me wrong, the T1000's and T2000's from Sun are actually really cool pieces of kit, but boys, did you think about it when you selected the raid controller? Unlike all your other RAID controllers, you can't turn on drive mirroring after the fact. It needs to be done before the OS is installed. Evidence here. Thanks, you jerks.
   
   
6. Lastly, why on God's earth would you still keep your 32 and 64 bit Java packages separate, after your "What's New in Solaris 10" page states you got rid of the stupid "if the package name ends in x, it's a 64 bit package" and then still keep the 64 bit JVM as a separate one? To add insult to injury, the SUNCuser meta-cluster (collections of collections of packages to install, to the uninitiated, purportedly for "easier installations") does not install the 64 bit JVM on 64 bit hardware. You're killing me!

Alright, that's enough ranting. No matter how tempting, and no matter how much less newer releases of Solaris suck less than the preceding ones, it's an exercise in frustration each time. When my prior employer's Sun rep was told that I was to be working in an all-Solaris shop for a while she apparently almost had her BigBucks Latte coming out of her nose, she laughed so hard. Apparently I gave their sales engineers the hardest time ...