Re: Upstreaming the glibc Hurd port

[Joseph Myers]

On Fri, 19 Jan 2018, Thomas Schwinge wrote:

> Hi Joseph!
> 
> On Fri, 19 Jan 2018 00:34:42 +0000, Joseph Myers <joseph@codesourcery.com> 
> wrote:
> > On Fri, 19 Jan 2018, Samuel Thibault wrote:
> > 
> > > Joseph Myers, on jeu. 18 janv. 2018 23:15:59 +0000, wrote:
> > > > Thanks for the changes pushed to sthibaul/hurd-builds so far (I realise 
> > > > there will be more to get it into a buildable state, e.g. the actual 
> > > > libpthread implementation).
> > > 
> > > What I have pushed is basically only missing the libpthread
> > > implementation, so you already have an idea of the minimal set of
> > > modifications to get something building (and IIRC essentially passing
> > > the testsuite).
> > 
> > I'd still like to have the libpthread implementation there (with a view to 
> > seeing if I can get build-many-glibcs.py working for Hurd with this branch 
> 
> Many thanks for your offer!  As far as I'm aware indeed nobody from the
> Hurd team has spent time on that yet.

This patch adds build-many-glibcs.py support for GNU Hurd.  It is
intended for master, where the builds of the i686-gnu configuration
would fail until sufficient support is merged to master, so completing
build-many-glibcs.py coverage of all glibc ABIs and making results
accurately reflect the broken state of builds for Hurd.  Using
sthibaul/hurd-builds branch, it reaches the glibc build, which then
falls over with errors starting:

In file included from mutex-init.c:19:0:
../mach/lock-intern.h: In function '__spin_lock_init':
../mach/lock-intern.h:41:13: error: 'LLL_INITIALIZER' undeclared (first use in 
this function)
   *__lock = LLL_INITIALIZER;
             ^~~~~~~~~~~~~~~
../mach/lock-intern.h:41:13: note: each undeclared identifier is reported only 
once for each function it appears in
../mach/lock-intern.h: In function '__spin_lock':
../mach/lock-intern.h:50:3: error: implicit declaration of function 'lll_lock' 
[-Werror=implicit-function-declaration]
   lll_lock (__lock, 0);
   ^~~~~~~~
In file included from mutex-init.c:20:0:
../mach/lowlevellock.h: At top level:
../mach/lowlevellock.h:21:10: fatal error: mach/gnumach.h: No such file or 
directory
 #include <mach/gnumach.h>
          ^~~~~~~~~~~~~~~~

Could Hurd people review how this handles building for Hurd, and
indicate whether the above errors indicate problems with these
changes, or simply incompleteness of the build support on the
sthibaul/hurd-builds branch at present?

2018-01-19  Joseph Myers  <joseph@codesourcery.com>

        * scripts/build-many-glibcs.py (Context.add_all_configs): Add
        i686-gnu configurations.
        (Context.run_builds): Include mig, gnumach and hurd in components
        considered.
        (Context.checkout): Add mig, gnumach and hurd to components.
        (Context.checkout_tar): Add URL mappings for mig, gnumach and
        hurd.
        (Context.bot_cycle): Check for changes to mig, gnumach and hurd.
        (Config.build): Install gnumach headers, build mig and install
        hurd headers for 'gnu' OS.
        (Config.install_gnumach_headers): New function.
        (Config.install_hurd_headers): Likewise.
        (Glibc.build_glibc): Do not use /usr for 'gnu' OS.  Specifiy MIG
        when building for 'gnu' OS.

diff --git a/scripts/build-many-glibcs.py b/scripts/build-many-glibcs.py
index 8849574..6095584 100755
--- a/scripts/build-many-glibcs.py
+++ b/scripts/build-many-glibcs.py
@@ -193,6 +193,8 @@ class Context(object):
                                  '--with-fpu=vfpv3'])
         self.add_config(arch='hppa',
                         os_name='linux-gnu')
+        self.add_config(arch='i686',
+                        os_name='gnu')
         self.add_config(arch='ia64',
                         os_name='linux-gnu',
                         first_gcc_cfg=['--with-system-libunwind'])
@@ -461,13 +463,15 @@ class Context(object):
             old_versions = {}
             self.build_host_libraries()
         elif action == 'compilers':
-            build_components = ('binutils', 'gcc', 'glibc', 'linux')
+            build_components = ('binutils', 'gcc', 'glibc', 'linux', 'mig',
+                                'gnumach', 'hurd')
             old_components = ('gmp', 'mpfr', 'mpc')
             old_versions = self.build_state['host-libraries']['build-versions']
             self.build_compilers(configs)
         else:
             build_components = ('glibc',)
-            old_components = ('gmp', 'mpfr', 'mpc', 'binutils', 'gcc', 'linux')
+            old_components = ('gmp', 'mpfr', 'mpc', 'binutils', 'gcc', 'linux',
+                              'mig', 'gnumach', 'hurd')
             old_versions = self.build_state['compilers']['build-versions']
             self.build_glibcs(configs)
         self.write_files()
@@ -694,7 +698,10 @@ class Context(object):
                             'gmp': '6.1.2',
                             'linux': '4.14',
                             'mpc': '1.1.0',
-                            'mpfr': '4.0.0'}
+                            'mpfr': '4.0.0',
+                            'mig': '1.8',
+                            'gnumach': '1.8',
+                            'hurd': '0.9'}
         use_versions = {}
         explicit_versions = {}
         for v in versions:
@@ -829,7 +836,10 @@ class Context(object):
                    'gmp': 'https://ftp.gnu.org/gnu/gmp/gmp-%(version)s.tar.xz',
                    'linux': 
'https://www.kernel.org/pub/linux/kernel/v4.x/linux-%(version)s.tar.xz',
                    'mpc': 'https://ftp.gnu.org/gnu/mpc/mpc-%(version)s.tar.gz',
-                   'mpfr': 
'https://ftp.gnu.org/gnu/mpfr/mpfr-%(version)s.tar.xz'}
+                   'mpfr': 
'https://ftp.gnu.org/gnu/mpfr/mpfr-%(version)s.tar.xz',
+                   'mig': 
'https://ftp.gnu.org/gnu/mig/mig-%(version)s.tar.bz2',
+                   'gnumach': 
'https://ftp.gnu.org/gnu/gnumach/gnumach-%(version)s.tar.bz2',
+                   'hurd': 
'https://ftp.gnu.org/gnu/hurd/hurd-%(version)s.tar.bz2'}
         if component not in url_map:
             print('error: component %s coming from tarball' % component)
             exit(1)
@@ -949,7 +959,8 @@ class Context(object):
                 self.clear_last_build_state(a)
             self.exec_self()
         check_components = {'host-libraries': ('gmp', 'mpfr', 'mpc'),
-                            'compilers': ('binutils', 'gcc', 'glibc', 'linux'),
+                            'compilers': ('binutils', 'gcc', 'glibc', 'linux',
+                                          'mig', 'gnumach', 'hurd'),
                             'glibcs': ('glibc',)}
         must_build = {}
         for a in actions:
@@ -1148,6 +1159,10 @@ class Config(object):
         if self.os.startswith('linux'):
             self.install_linux_headers(cmdlist)
         self.build_gcc(cmdlist, True)
+        if self.os == 'gnu':
+            self.install_gnumach_headers(cmdlist)
+            self.build_cross_tool(cmdlist, 'mig', 'mig')
+            self.install_hurd_headers(cmdlist)
         for g in self.compiler_glibcs:
             cmdlist.push_subdesc('glibc')
             cmdlist.push_subdesc(g.name)
@@ -1230,6 +1245,41 @@ class Config(object):
         cmdlist.cleanup_dir()
         cmdlist.pop_subdesc()
 
+    def install_gnumach_headers(self, cmdlist):
+        """Install GNU Mach headers."""
+        srcdir = self.ctx.component_srcdir('gnumach')
+        builddir = self.component_builddir('gnumach')
+        cmdlist.push_subdesc('gnumach')
+        cmdlist.create_use_dir(builddir)
+        cmdlist.add_command('configure',
+                            [os.path.join(srcdir, 'configure'),
+                             '--build=%s' % self.ctx.build_triplet,
+                             '--host=%s' % self.triplet,
+                             '--prefix=',
+                             'CC=%s-gcc -nostdlib' % self.triplet])
+        cmdlist.add_command('install', ['make', 'DESTDIR=%s' % self.sysroot,
+                                        'install-data'])
+        cmdlist.cleanup_dir()
+        cmdlist.pop_subdesc()
+
+    def install_hurd_headers(self, cmdlist):
+        """Install Hurd headers."""
+        srcdir = self.ctx.component_srcdir('hurd')
+        builddir = self.component_builddir('hurd')
+        cmdlist.push_subdesc('hurd')
+        cmdlist.create_use_dir(builddir)
+        cmdlist.add_command('configure',
+                            [os.path.join(srcdir, 'configure'),
+                             '--build=%s' % self.ctx.build_triplet,
+                             '--host=%s' % self.triplet,
+                             '--prefix=',
+                             '--disable-profile', '--without-parted',
+                             'CC=%s-gcc -nostdlib' % self.triplet])
+        cmdlist.add_command('install', ['make', 'prefix=%s' % self.sysroot,
+                                        'no_deps=t', 'install-headers'])
+        cmdlist.cleanup_dir()
+        cmdlist.pop_subdesc()
+
     def build_gcc(self, cmdlist, bootstrap):
         """Build GCC."""
         # libsanitizer commonly breaks because of glibc header
@@ -1346,8 +1396,10 @@ class Glibc(object):
         # writing into the working directory.  To avoid possible
         # concurrency issues, copy the source directory.
         cmdlist.create_copy_dir(srcdir, srcdir_copy)
+        use_usr = self.os != 'gnu'
+        prefix = '/usr' if use_usr else ''
         cfg_cmd = [os.path.join(srcdir_copy, 'configure'),
-                   '--prefix=/usr',
+                   '--prefix=%s' % prefix,
                    '--enable-profile',
                    '--build=%s' % self.ctx.build_triplet,
                    '--host=%s' % self.triplet,
@@ -1362,6 +1414,8 @@ class Glibc(object):
                    'RANLIB=%s' % self.tool_name('ranlib'),
                    'READELF=%s' % self.tool_name('readelf'),
                    'STRIP=%s' % self.tool_name('strip')]
+        if self.os == 'gnu':
+            cfg_cmd += ['MIG=%s' % self.tool_name('mig')]
         cfg_cmd += self.cfg
         cmdlist.add_command('configure', cfg_cmd)
         cmdlist.add_command('build', ['make'])
@@ -1369,10 +1423,11 @@ class Glibc(object):
                                         'install_root=%s' % installdir])
         # GCC uses paths such as lib/../lib64, so make sure lib
         # directories always exist.
-        cmdlist.add_command('mkdir-lib', ['mkdir', '-p',
-                                          os.path.join(installdir, 'lib'),
-                                          os.path.join(installdir,
-                                                       'usr', 'lib')])
+        mkdir_cmd = ['mkdir', '-p',
+                     os.path.join(installdir, 'lib')]
+        if use_usr:
+            mkdir_cmd += [os.path.join(installdir, 'usr', 'lib')]
+        cmdlist.add_command('mkdir-lib', mkdir_cmd)
         if not for_compiler:
             if self.ctx.strip:
                 cmdlist.add_command('strip',

-- 
Joseph S. Myers
joseph@codesourcery.com