A while ago, I was on contract for my now current employer. They wanted me to port all the functionality of their application from OpenVMS on Alpha to OpenVMS on IA64. As part of the port, we had to switch from Advanced Server (Pathworks) to Samba on IA64.

I set up the Samba shares for them and ensured it worked in testing, but by the time my contract ended, the switch still hadn't been made, and was not made until about 4 months later.

Now I find myself looking at a machine that's completely CPU bound dealing with Samba polls from the Windows Server world out there.

Looking at the scripts that run in Windows land, we find that the majority of them are something like:

if exist \\sambaserver\somedir\*.txt copy \\sambaserver\somedir\*.txt c:/somewhere/

Now, let's think about what this does. The script asks the Samba server to connect to a share point and determine if a file is present, and copy it locally if so. Simple, right? But what's happening on the (OpenVMS based) Samba server?

Well, Samba connections are created in response to the client opening a port (139 or 445). That is, there is no persistant server like Advanced Server. Each time you connect to a Samba share, OpenVMS created a new process. Now, Samba is tries to be smart, and assumes that the connection will be around for a while. So it goes and reads INDEXF.SYS for the directory you connected to so as to cache file information such as the create date and size of the file (which Windows expects each time you do a directory lookup). Unfortunately, the way the script is connecting, if there are no files to process (or even if there are) this action is a complete waste of CPU time and I/O as the share is disconnected at the end of the script.

The appropriate way to do this is to connect the server/directory combination as a network drive and then refer to the local drive letter in the script. Assuming the server/directory is mapped to drive N:, the script then becomes:

if exist n:/*.txt copy n:/*.txt c:/somewhere/

By asking the Windows admins to change to this method, I've gone from a machine that was peaking at 100% CPU to one that's now peaking at 50% CPU. A good win for just "doing it right".

No. Systems Management Homepage, a tool that is the "single system management solution for managing OpenVMS". Well, I've heard that before. Things like the OpenVMS Management Station spring to mind.

As far as I can see, this product is just a bunch of MIB extensions which feed into Insight Manager.

Do we really need the Java, Apache, and PHP on our production machines to manage them? I thought I was doing pretty well at the command line over the last 25 years. Perhaps I'm missing something?

I'd love to hear your experiences with this product, pro and con. Feel free to comment below.

Bob Gezelter has just published an article on openvms.org concerning, among other things, the proposal that HP change the licensing terms for Host Based Volume Shadowing to allow no-cost single member shadow sets.

I wholeheartedly support this proposal. HBVS is to my mind a critical component of redundant and resiliant systems, and reducing the barrier to entry can only be a good thing for customers and HP both.

The benefit to the customer is the ease of adding or transitioning storage with the purchase of the HBVS license to enable more than one physical member. If the single physical device to be transitioned is already a member of a shadowset, no downtime need result.

This concept is so useful that even at the last large site I worked at that had a full license for HBVS, we still had every disk respresented by a DSA device, even if they were a single physical disk. The time saved by being able to add and remove shadow members to copy/transition is worth the extra layer of driver code.

Quite some time ago, I was asked how much better FAST I/O is than say, accessing the data via reading blocks with $QIOW. I didn't know at the time, and if you've spent any time in this game, you'll know that "I don't know" is an unacceptable answer. The correct response is "I'll find out!".

Here are the results of reading a sequential file block by block with $IO_PERFORMW() and $QIOW(). The test file is a 500,000 block file marked no_caching residing on a privately mounted disk served up by an EVA 8100. Two programs access the file with the QIO/ACP interface, and then perform 500,000 reads of 512 bytes on the file to completely read the file. Note that the EVA is allowed to cache the disk (for both programs) so we should eliminate a lot of disk latency from the timing after the first couple of runs. LIB$INIT_TIMER() and LIB$SHOW_TIMER() are used to record results, and the average of ten alternate runs is calculated.

The average elapsed time for FAST I/O verses QIO is 51.3 verses 52.0 seconds.

The average CPU time for FAST I/O verses QIO is 3.63 verses 5.01 seconds.

As we expect, significant time is spent reading the data from the disk, and the difference between the two averages is less than 1.4% But the difference in CPU time is marked at 27.6%

So in conclusion, if you need to do extensive non-record I/O to a file (or you're looking to implement your own file structure), and you can coerce the systems manager into giving you the VMS$BUFFER_OBJECT_USER identifier, you can save significant CPU time by using the Fast I/O services.

As you know, I recently accepted permanent employment as an OpenVMS Systems Specialist (yeah, yeah, I know I have to do Un*x/Windoze server support too). As the technical support team is rather light on the ground at the moment after some positions have become vacant, I was asked if I was contactable over the holidays between Christmas and the New Year. Well, I could hardly say "no".

Needless to say, I was called multiple times for various application issues that occurred after a blade enclosure was power cycled (eek) to clear a stuck mezzanine card. Of course, Murphy's Law states that half the production cluster was in that enclosure.

On investigation of the startup issue, it became apparent that the startup on one of the two nodes that rebooted had gone into some sort of infinite loop. On tracking down the problem, I discovered a wonderful piece of DCL that, if run by multiple nodes at the same time, would undoubtably have temporary filename collisions. The command procedure created a non-unique temp file, and then deleted all versions of it.

Of course, the temp file was being used for input to sysman by doing the following horrible hack:

$ temp_file = "sys$manager:temp.tmp"
$ define/nolog sys$input 'temp_file'
$ mc sysman
$ deassign sys$input
$ delete/nolog 'temp_file';*

Erk. What happens if the file's not there? You guessed it - sysman goes into an infinite loop requesting a command with an alphabetic first character! What obviously happened in the startup was that one node deleted the file while the other was still creating it. Bad, bad, bad.

The proper way to do this is of course to use the f$unique() lexical function to ensure you have no temp file name collisions. In addition, let's use sysman's @ command to input the temp file:

$ temp_file = "sys$manager:" + f$unique() + ".tmp"
$ mc sysman @'temp_file'
$ delete/nolog 'temp_file';*

Even if the temp file doesn't exist, you at least don't get an infinite loop. You just get a nice %RMS-E-FNF, file not found message.

An extremely funny¹ parody of part of "The C Programming Language" book, with a special guest author H.P. Lovecraft can be found at http://www.bobhobbs.com/files/kr_lovecraft.html. I'm going to reproduce it below the fold in its entirety as this, like The VAXorcist, deserves preservation eternal (plus I'm going to fix the compile error in the original code ;)

[1] Funny if you're into C programming and Lovecraft. Else avoid it like the plague.

This video gives a view of the universe with all depicted objects accurate to currently measured scientific data. Rather amazing. Via Jason Kottke.

This is weird. I was trying to come up with a way to let specific users use SMTP to send email off a box, and I thought "ah, there's a generic queue, and an execution queue. I can just modify the protection of the execution queue, and add an ACL for the users I want to allow email access."

So I set up a test:

$ write sys$output f$getsyi("arch_name") + " " + f$getsyi("version")
IA64 V8.3-1H1
$ init/que/batch/start test_exe_q/prot=(s:m,o:d,g:r,w:r)/own=system
$ init/que/batch/generic=test_exe_q/start test_gen_q/own=system
$ set acl/obj=que/acl=(ident=[*,*],access=none)
$ sh que/full test_*_q
Batch queue TEST_EXE_Q, idle, on xxxxxx::
  /BASE_PRIORITY=4 /JOB_LIMIT=1 /OWNER=[SYSTEM] /PROTECTION=(S:M,O:D,G:R,W:R)
         (IDENTIFIER=[*,*],ACCESS=NONE)

Generic batch queue TEST_GEN_Q
  /GENERIC=(TEST_EXE_Q) /OWNER=[SYSTEM] /PROTECTION=(S:M,O:D,G:R,W:S)
$

Obviously, I had privileges when I did this. Note that the execution queue has no world submit, and anyway, the ACL should block anything from using the queue (with the exception of someone holding BYPASS or SYSPRV privileges).

Next, I went and disabled my privileges with authorize. My default privilege mask consisted of TMPMBX and NETMBX.

Thinking "OK, now to test this", I submitted a batch job, expecting it to hang in the generic queue. To my utter surprise, it executed. What was going on here? I'd assumed (yes, I know what assume stands for, that's why I was doing a test) that the queue manager software would look at the system authorization file and rightslist to determine access before moving the batch queue entry from the generic queue to the execution queue, but apparently not. I can only guess that some smart programmer coded a check to see if the submitting process was still there, and grabbed the privilege mask from the submitting process as an I/O optimization.

When I logged the submitting process out, then logged back in, everything worked as I expected it to in the first place.

Weird, non-intuitive behaviour.

I've recently been been working on a retrofit of a proper source code tracking and build system. I use CMS and MMS extensively for my own projects, but I'd forgot how tricky it can be for a multi-developer environment. It's slowly getting there, however.

Unfortunately, none of the forward progress is being helped by the CMS API documentation, which is riddled by cut and paste errors. You might be reading the section on CMS$REPLACE() and come across the following:

"If multiple reservations exist for the element generation, you must specify the identification number of the exact reservation to be unreserved (canceled)."

Unreserved? Don't you mean "replaced"? I was reporting these to HP as I came across them, but after 10 or so, I gave up. The entire manual needs a review.

Speaking of incorrect documentation, as part of the API, the routines require a "Library Data Block" to maintain context. This is documented as an array of 50 longwords. There is even an SDL file supplied for the API that defines the structure of this block (which is broken as delivered, but I digress). Unfortunately, it appears that the software doesn't agree with either the documentation or the SDL. Witness the following:

#include <stdio.h>
#include <stdlib.h>
#include <stsdef.h>
#include <descrip.h>
#include <lib$routines.h>
#include "cms$routines.h"

int main (void) {

static int r0_status;
static int i;
static int overwrite;
static $DESCRIPTOR (lib_d, "CMS$LIB");
static ldb_cntrlblk ldb;
static int guard[20];

    printf ("Variable    Start    Length   End\n"
            "ldb         %08X %08X %08X\n"
            "guard       %08X %08X %08X\n",
            &ldb, sizeof (ldb), (int)&ldb + sizeof (ldb) - 1,
            &guard[0], sizeof (guard), (int)&guard[0] + sizeof (guard) - 1);

    for (i = 0; i < 20; i++) {
        guard[i] = 0xdeadbeef;
    }

    r0_status = cms$set_library (&ldb,
                                 &lib_d,
                                 0,
                                 0,
                                 0,
                                 0,
                                 0,
                                 0,
                                 0);
    if (!$VMS_STATUS_SUCCESS (r0_status)) {
        lib$signal (r0_status);
    }

    overwrite = 0;
    for (i = 0; i < 20; i++) {
        if (guard[i] != 0xdeadbeef) {
            overwrite++;
        }
    }

    if (overwrite) {
        printf ("%d longwords of guard were overwritten by "
                "the call to cms$set_library!\n",
                overwrite);
    }
}

Running this after defining a CMS library demonstrates what appears to be a memory overrun within the CMS$SET_LIBRARY routine. What else is lurking, I wonder?

$ cc/version
HP C V7.2-001 on OpenVMS IA64 V8.3-1H1
$ cc demo
$ link demo
$ cms show version
CMS Version V4.5-2
$ cms set lib my_disk:[cms]
%CMS-I-LIBIS, library is MY_DISK:[CMS]
%CMS-S-LIBSET, library set
$ run demo
Variable    Start    Length   End
ldb         00010000 000000C8 000100C7
guard       000100C8 00000050 00010117
%CMS-I-LIBIS, library is MY_DISK:[CMS]
7 longwords of guard were overwritten by the call to cms$set_library!
$ 

I'm very happy to announce that I've accepted a permanent position at a company that I've done contracting for in the past. There is a lot of work to do here in the development tool space. Additionally, I believe the performance of the application could significantly benefit from an analysis of the underlying RMS files. Now I'll have time to do that (when I not doing other systems admin-y things).

As some of you perhaps remember, I was engaged to look at another port to IA64 a while ago. I haven't been much engaged with this process in recent times, as I've been waiting for the developers to fix some rather important modules that failed to compile initially.

As of yesterday, all the source compiles on both platforms, and I have the programmers looking at uninitialized decimal data issues at present.

Once the system tests out functionally, I'm sure I'll get called back in to look at the performance issues that inevitably seem to arise in Alpha to IA64 ports.

I mentioned this feature when it was unofficially announced but I haven't mentioned it since. As it's pretty cool, it deserves a little writeup.

Did you know that is you have the appropriate privileges, you can issue a CTRL-T against another process? Even another process on another node of the cluster? All you need to do is define a DCL symbol called DCL$CTRLT_PID to be the PID of the target process. Then hitting CTRL-T returns information on that process rather than your current process.

The second part of this is that you can modify the output that CRTL-T produces. To do this, you define a DCL symbol called DCL$CTRLT. Here's an example of doing this in a DCL command procedure (lifted from the release notes):

$       inner = 0
$       outer = 0
$loop:
$loop1:
$       if inner .gt. 20000
$       then
$           goto end_loop1
$       endif
$       inner = inner + 1
$       dcl$ctrlt = F$FAO ("Inner count is !SL !/ Outer count is !SL", -
                           inner, outer)
$       goto loop1
$end_loop1:
$       inner = 0
$       outer = outer + 1
$       goto loop

Even more interestingly, you can use this technique from within a program and display variable information:

#include <stdio.h>
#include <stdlib.h>
#include <descrip.h>
#include <stsdef.h>
#include <lib$routines.h>

int main (void) {

static int r0_status;
static int count = 0;
static $DESCRIPTOR (symbol_d, "DCL$CTRLT");
static char value[256];
static struct dsc$descriptor_s value_d = { 0,
                                           DSC$K_DTYPE_T,
                                           DSC$K_CLASS_S,
                                           value };

    while (TRUE) {

        value_d.dsc$w_length = sprintf (value, "Count is %d", ++count);
        r0_status = lib$set_symbol (&symbol_d, &value_d);
        if (!$VMS_STATUS_SUCCESS (r0_status)) {
            (void)lib$signal (r0_status);
        }
    }
}

Ian Cartwright wrote a piece calling for banning the debugger in the late stages of a development project. He says that "This usually leads to a big upswing in the amount of logging and it's logging we know helps to fix issues". Previously in the same article he implies that a single exception and its stack trace are not enough information to locate the problem.

Now, obviously Ian is Microsoft focused, but if I can locate the failing line of code (even down to the line of assembly code that produced the exception) on OpenVMS by reading a traceback dump, surely he can do it on his operating system of choice?

Granted, without some logging, you may not have contextual information about what the end user was doing with the software at the time, but I've found you rarely need that once you have the buggy line of code staring you in the face.

Conversely, one engineer I know writes an exception handler that will deal with all unexpected errors produced by each program he writes. The exception handler dumps appropriate context from the data structures to a file. Overkill? He thinks not, as this is first part of the code he writes, and he's convinced having this in place saves him significant development time.

Having said all that, I'm going to have to disagree with Ian. Taking away the most versatile tool the developer has to examine what's occuring in the code, and then retrofitting logging speaks of poor design up front.

P.S. Before you say "Up front design?!? What are you, a dinosaur?" here's an Agile story card for you:

"The end user will never see an unhandled error, instead they will see a message that an error has occurred and that the development team has automatically been notified of this fact. The program will then save enough contextual information to a text file to establish what the end user was doing when the error occured, and exactly where in the program the error occured. This text file will then be used to automatically establish a bug report (or increase the priority of an existant bug report) in the bug tracking system."

Recently, I've been asked to retrofit a build dependency system, a source versioning system, and redo the code promotion scripts for a rather large code repository. The obvious choice here is the CMS and MMS tools from the DECset product. I thought I'd start with the dependency stuff, as I can then use it to test the checkout and checkin, rebuilds, and promotion stuff.

The system is a little large (over 1900 standalone executable images) and I sure didn't want to hand code that many MMS files. So my thought was to use the in-built dependency generator in MMS to do a first cut, and then automagically munge the files into their final form.

This unfortunately proved to be unworkable due to a number of factors, not the least being that the code repository uses the same file name extension (that is, the bit after the dot) to represent at least five different files types, each type needing a different action to update the target. So I hand coded a few MMS files, then wrote a program to look for all CALL and COPY statements in the COBOL program tree.

Once the dependencies where detected (and the inevitable special cases dealt with), the code outputs a nicely formatted MMS file based on my hand coded templates, which I'm happy to say built the most complex of the repository's executables on about the second go.

Now to run it for the other 1900+...

As you all probably are aware, Ubuntu 9.10 was released yesterday. I took the time to upgrade my server today, and boy, was it easy. And with a codename of "Karmic Koala", how could I resist running it?

If you are connected to the net, all you have to do to upgrade is:

~$ sudo do-release-upgrade

and follow a couple of simple prompts. The only nonstandard things I had to do was shut down proftpd by hand to allow the upgrade of libc, and redo a minor modification to my php.ini file.

It was all done in less that 20 minutes. Too cool.