A quick DCL hack to display which CPU is managing each fastpath capable device.

A few new programming examples have been added to the site. They are for the following services: sys$get_arith_exception, sys$gettim_prec, and sys$power_control.

A second disaster recovery test was performed over the weekend. While we still have some niggling little problems (accessing the EVA's console from the test network, for example), everything worked flawlessly. I now have the recovery time down to approximately 40 minutes (which is a better deal that something like 20 hours when we did this with tape restores).

Yesterday we tested part of our DR plan. This is the bit that restores the major ERP system to functionality. Without this system we're dead in the water.

Recently, we switched this system from a "restore from tape" plan to a "replicate data with Continuous Access" plan. We originally attempted to test this plan back in July, and while this test was declared a failure due to networking setup issues, this one went like a charm.

All systems restored in just over two hours (which I'm hoping we can get under an hour with some more automation).

Recently, we acquired a two processor BL860c i2 blade for inclusion in our development cluster. We justified this by pointing out that the development cluster (which also doubles as the QA/Test cluster) had no where near enough CPU power to run our full overnight processing stream. As our overnight run consists of (nearly) identical streams of parallel processing, we had been testing small subsets and hoping that the operative word "nearly" would not bite us. The addition of the i2 will allow a full run in a timely fashion (read "less that 8 hours").

In addition, we recently had an incident where the management processor of our single BL860c in development went nuts, resulting in a 24+ hour outage while various hardware swaps were tried to rectify the problem (the large outage resulted from someone dropping the ball when it came to backing up the MP settings). It was pointed out to management that they had a development team and a number of testers from the business all waiting for the recovery of this machine, and adding a second Itanium based machine would remove this single point of failure.

There were certainly some "interesting" configuration exercises required to get the new machine up and running the way we wanted it.

Did you know that the status value in the I/O status block for most of the "get information" calls is a longword? I have the sneaking suspicion that this changed some time in recent history (where "recent history" equals "some time after VMS 5.x") when the number of $GETxxI routine return values exceeded the size of a word.

What this means in practice is that you should be testing and returning the full longword rather than just the word for calls such as $GETSYI(W), $GETJPI(W), $GETQUI(W), etc.

As an example, here's a section of code that does the wrong thing and then the correct thing:

#define __NEW_STARLET 1

#include <stdio.h>
#include <stdlib.h>
#include <ssdef.h>
#include <stsdef.h>
#include <efndef.h>
#include <quidef.h>
#include <iledef.h>
#include <iosbdef.h>
#include <descrip.h>
#include <lib$routines.h>
#include <starlet.h>

#define errchk_sig(arg) if (!$VMS_STATUS_SUCCESS(arg)) (void)lib$signal(arg);


/******************************************************************************/
int main (void) {

static IOSB iosb;
static int r0_status;
static int qflags;
static unsigned int status;
static unsigned int flags = 0x000f;
static ILE3 quiitms[] = { 4, QUI$_QUEUE_FLAGS, &qflags, NULL,
                          0, 0, NULL, NULL };

static char msg[255+1];
static struct dsc$descriptor_s msg_d = { 0,
                                         DSC$K_DTYPE_T,
                                         DSC$K_CLASS_S,
                                         msg };

    /*
    ** This call will succeed requesting information, but because our item
    ** list is missing the queue name we want info on, the IOSB will contain
    ** a status indicating we didn't pass enough parameters.
    */
    r0_status = sys$getquiw (EFN$C_ENF,
                             QUI$_DISPLAY_QUEUE,
                             0,
                             quiitms,
                             &iosb,
                             0,
                             0);
    errchk_sig (r0_status);

    /*
    ** Display the word value.  Perhaps not what we were expecting.
    */
    (void)printf ("iosb.iosb$w_status = %d\n",
                  iosb.iosb$w_status);
    msg_d.dsc$w_length = sizeof (msg) - 1;
    status = iosb.iosb$w_status;
    r0_status = lib$sys_getmsg (&status,
                                &msg_d.dsc$w_length,
                                &msg_d,
                                &flags);
    errchk_sig (r0_status);
    (void)printf ("%-.*s\n",
                  msg_d.dsc$w_length,
                  msg_d.dsc$a_pointer);

    /*
    ** Display the correct longword value.
    */
    (void)printf ("iosb.iosb$l_getxxi_status = %d\n",
                  iosb.iosb$l_getxxi_status);
    msg_d.dsc$w_length = sizeof (msg) - 1;
    status = iosb.iosb$l_getxxi_status;
    r0_status = lib$sys_getmsg (&status,
                                &msg_d.dsc$w_length,
                                &msg_d,
                                &flags);
    errchk_sig (r0_status);

    (void)printf ("%-.*s\n",
                  msg_d.dsc$w_length,
                  msg_d.dsc$a_pointer);
}

Compiling, linking and running this code produces the following output:

$ cc demo.c
$ link demo
$ run demo
iosb.iosb$w_status = 33036
%SYSTEM-F-NOMSG, Message number 0000810C
iosb.iosb$l_getxxi_status = 295180
%JBC-F-MISREQPAR, missing required parameter
$

This time on development. A little back story: the system this occurred on is running on a blade system, and the management processor in the blade system had been acting flaky. HP decided to swap out the blade, but the engineer on site forgot to back up the NVRAM before he swapped out the old blade, and we ended up with an unbootable box. After much stuffing around, we managed to boot the box in a non-standard way. A day later, we saw this crash.

While cleaning up some boot definitions with SYS$MANAGER:BOOT_OPTIONS.COM ready to relocate the system disk from one EVA disk group to another, we experienced a crash. The problem has been escalated to OpenVMS Engineering as we are running the latest and greatest update patch (UPDATE-V1000).

Here's an article with the best method I've yet seen to clean up old versions of the linux kernel from your machine.

Seeing commands like

dpkg -l 'linux-*' | 
sed '/^ii/!d;/'"$(uname -r | 
sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | 
xargs sudo apt-get -y purge

make me glad I'm not a linux systems manager :-)

The EVAs I have here at work were configured a number of years ago (by HP I suspect, with little to no input as to what they were going to be used for). One of them has a single disk group containing multiple storage for multiple Windows, both development and production (eek).

The other contains the application data for the application that runs on OpenVMS, and again, both development/QA and production data are stored on the same disk group.

This was a disaster waiting to happen, and happen it did, with a runaway process on QA causing major performance impact to the production system.

Fortunately, I'd been advocating for a number of years to split production and QA data into separate disk groups, and now (funnily enough) I have the mandate to begin just such a move.

As most of our data disks are in multi-member host based shadow sets, I can do this online with zero downtime. Management was most surprised when I informed them of this (they'd obviously overlooked that point in my previous communications). I pointed out that yes, this was VMS, not that other product that's in use here...

I mentioned doing disaster recovery via Continuous Access in the last entry. Last Sunday, we had our first attempt at performing a test with this solution in place. Previously, we had confirmed that replication is working well by mounting a replicated disk read-only, but this was the first time that we would attempt to full boot the systems.

Due to some appalling planning on both the part of my company and the people that are leasing us the EVA/CA solution, things didn't go very well I'm afraid to say.

Why is it so difficult to set up a network isolated enough so that at the flick of a switch (or the disconnection of no more than two cables) I can boot the DR machines up with their production IP addresses? I was faced with the second alternative here (disconnecting cables) but I was told that access to the iLO was switched over the same piece of copper and if we pulled it, I'd have to take a laptop into the computer room and connect directly to the blade enclosure. D'oh!

Yes, things like consoles are called "out of band" for a reason.

Secondly, we are not replicating all our disks. For our critical data disks, we run three way host based mirroring, but we only replicate the first member to reduce the bandwidth requirements for the replication link. Because the second and third members had not been initialised previously, VMS was umm, reluctant to add them to the shadowsets at boot time.

Lessons learned and DCL corrected. Hopefully the network issues will be addressed next week.

Let's hope the next test goes off a little better.

I'm working way too hard. Things currently going on include:

and this is only in the last month! I'll try and do some entries for the things on this list as there are some quite interesting things going on.

I suppose I'd rather be busy than bored.

At 24-Mar-2011 03:00 GMT, the site will be down for a couple of hours while some physical hardware is relocated. Thanks for your patience.

I've converted this web site to valid HTML5. Thrilling stuff for geeks only.

This was quite straightforward for this site, as it was already in strict XHTML 1.1. I just had to change the DTD, replace <acronym> with <abbr>, and remove the P3P compact description, which I'll look at replacing as soon as I figure out how to do it.

Easy, right?

We use BMC's Control-M scheduling product here to control our overnight processing. To this end, I've just completed a training course designed to teach the basics (and some more advanced techniques) of using this complex piece of software.

Of course, once the training was completed I was tasked with completely redesigning the way the overnight processing hangs together so that it would be far simpler to maintain and understand. You do a training course, and suddenly you're an expert...