Re: Implementing LIBTOOL on OpenVMS

[John E. Malmberg]

Ralf Wildenhues wrote:
> * John E. Malmberg wrote on Fri, Nov 18, 2005 at 09:26:14PM CET:
>
>
> > Because of differences between *NIX and OpenVMS in this area, it looks 
> > like it may not be possible to make generating shared images on OpenVMS 
> > totally transparent to a build procedure with out some cooperation with 
> > the maintainers of the package.
>
>
> I'm not sure yet.  For a large subset of packages, it may be possible.

Item 3 below from my first post is the sticky point for shared images.

> > 3. For dot-n releases, in order to provide upward compatibility so that 
> > new shared libraries can be used by older programs, the list of public 
> > symbols must always be in the same order, with new symbols being added 
> > to the end.

> Ouch.  This is a new requirement.  So in general the author of the
> library has to provide the list of symbols, it can't be generated.
> Right?

Mostly right.  A generated approach can be taken as long as the 
generating program has the previous generation of the file as a base and 
the generated output is distributed with the source, and understands the 
rules for generating the output.

It does cause problems if the project spins off a new branch that needs 
backwards compatibility, as the list of symbols must be maintained by 
the head branch.

For full compatibility, Microsoft Windows DLL files may have the same 
requirement.  Microsoft Windows DLL routines can be invoked either by 
ordinal or by name.  Change the order of the symbols to be exported, and 
it changes the ordinal value, thus breaking any routine that calls by 
ordinal.

And on a Microsoft based system, it appears that call by ordinal would 
be faster than call by symbol name.

I do not know if the Microsoft Linker sets up images to load their DLLs 
by ordinal or by name.

Effectively the OpenVMS linker sets up calling routines by ordinal, not 
by name.

> This probably means that we should have a new version_type for OpenVMS.

> > Special handing is needed if routine foo(x,y,z) changes to be foo(p,d,q) 
> > in a newer dot-n version.  The old version of foo would get a new name, 
> > but retain it's position in the symbol table, and it needs to still 
> > return something, if only an ENOSYS error code if it was really bad to 
> > continue to use it.  The new version of foo() would go to the end of the 
> > table.
> >
> > So for OpenVMS, some history needs to be maintained for the full 
> > advantages of shared images to be used.
>
>
> Hmm.  This feature does not map on features commonly found in other
> shared library systems (altough of course there are systems with
> versioned symbols).  So, probably the first support for OpenVMS in
> Libtool should disallow and need not support symbols with changed
> interface, since for portable programs, you cannot use them either.
> We should then strive to add support for this together with adding
> support for versioned symbols on other systems.

It is pretty hard to "disallow" this feature because with out a history, 
it is impossible for the build procedure to automatically know if it is 
building a shared image that is incompatible with the previous version.

The current hack around this it to set the flags so that a relink is 
always required for a new version.  This removes one of the benefits of 
using a shared image.  Especially if it is something that only one copy 
is expected to be run at a time.

Supplying a old/new version of a public routine is something that a 
programmer needs to do.  I do not think that most change control systems 
will flag if someone makes an incompatible change to a program's parameters.


Also, public symbols on OpenVMS are traditionally case insensitive and 
stored in uppercase and limited to 31 characters.  They can also be case 
sensitive.  I have the ability to export symbols in both exact case and 
uppercase.

The 31 character limit is a special case that I am trying to get some 
tools to make easier.  What happens is that if a public symbol is more 
than 31 characters, the C compiler shortens to be much smaller and adds 
a CRC type value to the end to keep it unique.  That is the symbol name 
that I have to put in the symbol table file.

Of course the current C standard states that only the first 31 
characters of a public symbol are significant, but that the restriction 
is planned to be removed in a future version of the standard.

> > Other issues:
>
> > Normally the eventual location of the shared library will reside is not 
> > used at build time.  A construct known as a logical name is used to find 
> > it if the shared image is not in the /SYS$SHARE directory.  For shared 
> > libraries not supplied by the operating system, normally a logical name 
> > is used.
> >
> > A logical name works like a symbolic link, except that to the file 
> > system it looks like it is always present in the in either the current 
> > working directory or in the root directory when it is explicitly 
> > referenced.  A logical name can supersede access to a file of the same name.
> >
> > When an application is built against a test shared library instead of a 
> > production one, it makes a local logical name for this.
> >
> > There is a naming convention for the logical names and program names 
> > where prefixes can be reserved to prevent conflicts so that multiple 
> > libraries with similar names can coexist.  These prefixes are not 
> > directory paths.


> OK.  I think this might need a bit of code, but is doable.
> Note Libtool (in default mode) wants the user to be able to use the
> uninstalled libraries, and has means to relink libraries and/or programs
> upon either execution or installation.

With OpenVMS, no relink is needed, just change what the logical name is 
pointing to.  It is like if you could have a symbolic link mean 
something different to a process tree than it does to the rest of the 
system.

This is close to the issue that brought me to this conference, the 
generated libtool script of one of the packages I am trying to build is 
looking for the destination library path to be set up, and it does not 
have any meaning on OpenVMS, so the caller of libtool did not set it up.

The build script on OpenVMS sets a logical name that points to the 
uninstalled libraries, and the install script sets it to point to the 
installed libraries.  The linked image does not change.

A linker option file contains the shared libraries that will be 
referenced by the image being built.

$ type svn_client.opt;3
LIBSVN_SHARE/SHARE
APACHE$APR_SHR/SHARE
APACHE$APU_SHR/SHARE
[End-of-file]

During the build, LIBSVN_SHARE points to image that was just built.  At 
run-time, a system wide definition is installed to point to the 
production version of the library.

Essentially sometime before the LIBSVN_SHARE is used to build something, 
the build script must do something like "DCL DEFINE/JOB LIBSVN_SHARE 
LCL_ROOT:[LIB]LIBSVN_SHARE.EXE".  This definition must be cleaned up 
after the build is complete because it will persist for the life of the 
job tree containing the build script.  And I also need the name to be in 
VMS syntax, which is hard to do from a bash shell script at this time.

In a traditional OpenVMS build environment, all the building takes place 
in a single process, so a temporary logical name is used that 
automatically self destructs when the script ends.


Also in this case the prefix "APACHE$" is reserved to to files that are 
part of the Apache web server.  The "$" in the name indicates the name 
has been registered.

By changing APACHE$APR_SHR definition, I can change either the test or 
production version of svn_client to use a compatible version of that 
library.

> Well, Libtool has postinstall_cmds and finish_cmds and all that, and you
> can stick pretty much arbitrary code in there.

Of course that would require a programmer familiar with VMS to put in
such support for each project.

> > Incremental linking does not exist on OpenVMS.  I can fake it, but all 
> > that does is use more disk space.
>  
> Well, it's needed mostly to cope with too large command line lengths.
> Does OpenVMS have a high limit there?

Current versions support 4096 characters.  Older only 1024.

What has been done in the past for large number of files is either to 
populate an object library, or to put the filenames in a linker option 
file.  Simply a list of files, usually one per line for readability.  I 
would have to check if commas are required.

> Another question for you: I remember a VMS version for Autoconf being
> announced on its mailing list some time ago.  Do you use this in order
> to get working configure scripts on OpenVMS?  Is GNU Autoconf portable 
> to OpenVMS these days?  I also wonder how much of the shell scripting
> of Libtool will just break on VMS..

I tried to run a very recent version of Autoconf.  It failed because it 
seems to require enhancements to Perl on OpenVMS.  Currently Perl on 
OpenVMS assumes a DCL shell environment.  I am working on that, but it 
will take some time.

Currently for the project I need to run Autoconf on, I have an NFS 
client on a LINUX system that runs it for me after refreshing the 
changes in that directory from the previous day.

> A couple of notes, before you start hacking: we generally like patches
> against the CVS HEAD version of Libtool.  Unintrusive patches may be
> backported to branch-1-5.  Especially for libltdl, working with CVS HEAD
> should be easier.

A CVS client has been ported to OpenVMS.  I will see if I can find it.
Otherwise is there ftp access to the unpacked head files?

> For large contributions, the FSF requires either a copyright assignment
> or disclaimer; the assignment paperwork takes a while, so it may be
> helpful to start early with this (and to check whether you may have any
> issues with this).  Other than that, patch discussions are best handled
> on the libtool-patches mailing list.

For large contributions to open source projects, I have to go through a 
central corporate office.

> Feel free to ask questions about anything you don't understand, show
> whatever changes you're working with -- we may be able to give useful
> hints.

This is a fragment of the option file that I manually generated for zlib 
1.2.3.  An image linked against it will continue to work with newer 
versions as long as the second number on the GSMATCH is equal to or 
greater than the value present, and the first number is the same.

I encoded the minor version numbers in that section, but it might be 
more appropriate to use the commit numbers from a central change control 
library.  Using the major version number as the first number may not be 
a good idea, and I may change that again before I submit that change to 
the ZLIB maintainers.

The symbols are exported both in exact case and in upper case so that 
the users of the library do not need to change their build procedures to 
require exact case, but the library can be called by modules that need 
to be built using exact case.  The uppercase version of the symbol is 
the alias assigned by the linker.

[start of fragment]

GSMATCH=LEQUAL,1,200300
CASE_SENSITIVE=YES
 symbol_vector=(zlibVersion=PROCEDURE)
 symbol_vector=(ZLIBVERSION/zlibVersion=PROCEDURE)

[end-of-fragment]

I am not all that skilled with bash shell scripting.  I have been on 
vacation for the past week, so I have to get back to the specific build 
issues that I was seeing to be more specific.  I also am having to 
create/fix the VMS versions of some UNIX utilities.

Regards,

-John
address@hidden
Personal Opinion Only