Re: Implementing LIBTOOL on OpenVMS

[Ralf Wildenhues]

Hi John,

Sorry for the delay.  I had to read a bit about OpenVMS first.

* John E. Malmberg wrote on Tue, Nov 29, 2005 at 12:25:56AM CET:
> Ralf Wildenhues wrote:
> >* John E. Malmberg wrote on Fri, Nov 18, 2005 at 09:26:14PM CET:
> 
> >>3. For dot-n releases, in order to provide upward compatibility so that 
> >>new shared libraries can be used by older programs, the list of public 
> >>symbols must always be in the same order, with new symbols being added 
> >>to the end.
> 
> >Ouch.  This is a new requirement.  So in general the author of the
> >library has to provide the list of symbols, it can't be generated.
> >Right?
> 
> Mostly right.  A generated approach can be taken as long as the 
> generating program has the previous generation of the file as a base and 
> the generated output is distributed with the source, and understands the 
> rules for generating the output.
> 
> It does cause problems if the project spins off a new branch that needs 
> backwards compatibility, as the list of symbols must be maintained by 
> the head branch.
> 
> For full compatibility, Microsoft Windows DLL files may have the same 
> requirement.  Microsoft Windows DLL routines can be invoked either by 
> ordinal or by name.  Change the order of the symbols to be exported, and 
> it changes the ordinal value, thus breaking any routine that calls by 
> ordinal.

Ouch.  I've never heard of the 'call by ordinal' convention.  I'm pretty
sure we don't support it on w32 systems.

> And on a Microsoft based system, it appears that call by ordinal would 
> be faster than call by symbol name.

Well, that appears as an obvious advantage, but at a high price indeed.
:-/

> I do not know if the Microsoft Linker sets up images to load their DLLs 
> by ordinal or by name.

That would be quite important to know, in fact, for libtool operation on
w32 systems.

> Effectively the OpenVMS linker sets up calling routines by ordinal, not 
> by name.

OK.

[ changing public interface ]

> >>So for OpenVMS, some history needs to be maintained for the full 
> >>advantages of shared images to be used.

> It is pretty hard to "disallow" this feature because with out a history, 
> it is impossible for the build procedure to automatically know if it is 
> building a shared image that is incompatible with the previous version.

Erm.  In general, this is impossible to decide on other platforms as
well.  That's why we require the library author to provide shared
library version information.  Changing public interface such as changing
a function from
  int foo(int);
to
  int foo(double);

(which I think is what you denoted in your previous post as a change in
a newer dot-n version) is an _incompatible_ change.  For the libtool
versioning model, that means you have to increase CURRENT, set REVISION
and AGE to 0, and logically it means that programs linked against one
version of the library won't work with the other version.

I'm still not quite sure how libtool should deal with this information
on OpenVMS.  But generally, whether the changes are compatible or not
*should* be specified by the library author with the version triple, as
explained in
  info Libtool Versioning
and following chapters.  It's not exactly trivial, but it's quite
important that it be followed closely by both library authors and
libtool implementations on specific systems.  Otherwise software
packaging and use will be very frustrating.

> The current hack around this it to set the flags so that a relink is 
> always required for a new version.  This removes one of the benefits of 
> using a shared image.  Especially if it is something that only one copy 
> is expected to be run at a time.

I don't quite understand this paragraph.

Does OpenVMS only allow to load one shared library of a certain name at
a time?  In which context: per-process, per-user, or globally
system-wide?

> Supplying a old/new version of a public routine is something that a 
> programmer needs to do.  I do not think that most change control systems 
> will flag if someone makes an incompatible change to a program's parameters.

No.  The programmer is expected to know, and flag it as such!  See above.

And really, I don't think that in the first iteration you should strive
to add this effective symbol versioning support in libtool.  Doing that
right is harder than you might think, and we can still do it after the
basic support is functional.  Rather, teach your library authors to
think first about good interfaces, than have them change afterwards.

> Also, public symbols on OpenVMS are traditionally case insensitive and 
> stored in uppercase and limited to 31 characters.  They can also be case 
> sensitive.  I have the ability to export symbols in both exact case and 
> uppercase.
> 
> The 31 character limit is a special case that I am trying to get some 
> tools to make easier.  What happens is that if a public symbol is more 
> than 31 characters, the C compiler shortens to be much smaller and adds 
> a CRC type value to the end to keep it unique.  That is the symbol name 
> that I have to put in the symbol table file.
> 
> Of course the current C standard states that only the first 31 
> characters of a public symbol are significant, but that the restriction 
> is planned to be removed in a future version of the standard.

Support for the symbol length limitations may be hacked into libtool
rather easily, I think, if changes should be necessary at all.

> With OpenVMS, no relink is needed, just change what the logical name is 
> pointing to.  It is like if you could have a symbolic link mean 
> something different to a process tree than it does to the rest of the 
> system.

Hmm.  I don't quite understand logical names yet, but I am reading and
learning about OpenVMS.

> This is close to the issue that brought me to this conference, the 
> generated libtool script of one of the packages I am trying to build is 
> looking for the destination library path to be set up, and it does not 
> have any meaning on OpenVMS, so the caller of libtool did not set it up.
> 
> The build script on OpenVMS sets a logical name that points to the 
> uninstalled libraries, and the install script sets it to point to the 
> installed libraries.  The linked image does not change.

OK.  Would it be reasonable if libtool took over those steps during
  libtool --mode=link ...
and
  libtool --mode=install
respectively?

> A linker option file contains the shared libraries that will be 
> referenced by the image being built.
> 
> $ type svn_client.opt;3
> LIBSVN_SHARE/SHARE
> APACHE$APR_SHR/SHARE
> APACHE$APU_SHR/SHARE
> [End-of-file]
> 
> During the build, LIBSVN_SHARE points to image that was just built.  At 
> run-time, a system wide definition is installed to point to the 
> production version of the library.
> 
> Essentially sometime before the LIBSVN_SHARE is used to build something, 
> the build script must do something like "DCL DEFINE/JOB LIBSVN_SHARE 
> LCL_ROOT:[LIB]LIBSVN_SHARE.EXE".  This definition must be cleaned up 
> after the build is complete because it will persist for the life of the 
> job tree containing the build script.  And I also need the name to be in 
> VMS syntax, which is hard to do from a bash shell script at this time.

It may not be trivial, but it should be possible to get this working.
Work on CVS HEAD Libtool rather than branch-1-5 if possible.

> In a traditional OpenVMS build environment, all the building takes place 
> in a single process, so a temporary logical name is used that 
> automatically self destructs when the script ends.

OK.

> Also in this case the prefix "APACHE$" is reserved to to files that are 
> part of the Apache web server.  The "$" in the name indicates the name 
> has been registered.
> 
> By changing APACHE$APR_SHR definition, I can change either the test or 
> production version of svn_client to use a compatible version of that 
> library.
> 
> >Well, Libtool has postinstall_cmds and finish_cmds and all that, and you
> >can stick pretty much arbitrary code in there.
> 
> Of course that would require a programmer familiar with VMS to put in
> such support for each project.

Not really.  postinstall_cmds and finish_cmds may be filled out in
libtool.m4, depending on $host.  You could add code for VMS there, and
when other projects update their libtool, it will propagate there.  :)
Those commands will be run from within the libtool script, inside the
same shell process, so you may be able to use its variables, and even
change them (if that proves necessary).

> >>Incremental linking does not exist on OpenVMS.  I can fake it, but all 
> >>that does is use more disk space.
> > 
> >Well, it's needed mostly to cope with too large command line lengths.
> >Does OpenVMS have a high limit there?
> 
> Current versions support 4096 characters.  Older only 1024.

Ouch.  I know libraries that contain 1000 objects.
Does the linker have something similar as GNU ld's --whole-archive flag?

> What has been done in the past for large number of files is either to 
> populate an object library, or to put the filenames in a linker option 
> file.  Simply a list of files, usually one per line for readability.  I 
> would have to check if commas are required.

Oh, that would be easy to support.  We already do this for some linkers,
e.g., the GNU or the AIX linker (we may only need to clean up the
support a bit in CVS HEAD.)

> >Another question for you: I remember a VMS version for Autoconf being
> >announced on its mailing list some time ago.  Do you use this in order
> >to get working configure scripts on OpenVMS?  Is GNU Autoconf portable 
> >to OpenVMS these days?  I also wonder how much of the shell scripting
> >of Libtool will just break on VMS..
> 
> I tried to run a very recent version of Autoconf.  It failed because it 
> seems to require enhancements to Perl on OpenVMS.  Currently Perl on 
> OpenVMS assumes a DCL shell environment.  I am working on that, but it 
> will take some time.

I was talking about
http://lists.gnu.org/archive/html/autoconf/2005-03/msg00067.html
which is not GNU Autoconf, but a special version for VMS.  You might
have more luck with it.

> >A couple of notes, before you start hacking: we generally like patches
> >against the CVS HEAD version of Libtool.  Unintrusive patches may be
> >backported to branch-1-5.  Especially for libltdl, working with CVS HEAD
> >should be easier.
> 
> A CVS client has been ported to OpenVMS.  I will see if I can find it.
> Otherwise is there ftp access to the unpacked head files?

The guys at opendarwin are so nice to provide us with daily snapshots:
http://libtool.opendarwin.org/libtool.tar.gz
(search for snapshot on http://www.gnu.org/software/libtool/ ).

> >Feel free to ask questions about anything you don't understand, show
> >whatever changes you're working with -- we may be able to give useful
> >hints.
> 
> This is a fragment of the option file that I manually generated for zlib 
> 1.2.3.  An image linked against it will continue to work with newer 
> versions as long as the second number on the GSMATCH is equal to or 
> greater than the value present, and the first number is the same.
> 
> I encoded the minor version numbers in that section, but it might be 
> more appropriate to use the commit numbers from a central change control 
> library.  Using the major version number as the first number may not be 
> a good idea, and I may change that again before I submit that change to 
> the ZLIB maintainers.

Hmm.  What does the GSMATCH number convey exactly?  What are its
semantics for VMS?

> The symbols are exported both in exact case and in upper case so that 
> the users of the library do not need to change their build procedures to 
> require exact case, but the library can be called by modules that need 
> to be built using exact case.  The uppercase version of the symbol is 
> the alias assigned by the linker.

OK.

> [start of fragment]
> 
> GSMATCH=LEQUAL,1,200300
> CASE_SENSITIVE=YES
>  symbol_vector=(zlibVersion=PROCEDURE)
>  symbol_vector=(ZLIBVERSION/zlibVersion=PROCEDURE)
> 
> [end-of-fragment]

> I am not all that skilled with bash shell scripting.  I have been on 
> vacation for the past week, so I have to get back to the specific build 
> issues that I was seeing to be more specific.  I also am having to 
> create/fix the VMS versions of some UNIX utilities.

OK.

A couple of hints, before you start posting patches: Likely, some
libtool idioms such as converting relative to absolute paths, may
not work under OpenVMS.  If such things turn up, rather than fixing
ten similar occasions, create one function to do the conversion,
in the spirit of func_source or func_dirname (of CVS HEAD).  In the
long run, this is bound to be less resource-intensive.

Cheers,
Ralf