Re: [Ltib] ./ltib -m release # externally distributable error

[aaron]

Sorry for being long-winded.  I would like to take your recommendation for
building a binary iso release using the 'trelease' mode, the 'localdir'
special tag, and a MANIFEST file.  Earlier you wrote that in the case of
the 'trelease' mode, "...you still (may) need a network connection, but
errors are not fatal."

1) Even though I do have a network connection, I believe I'm running into
a bug for the packages that I've introduced into the build, of which there
are two.  I created the .spec files that I copied from the template.spec
and edited, and the packages build fine with './ltib', but they don't work
with './ltib -m trelease'.  The tail of the error I get is below. 
Included is the printout of the line you suggested to insert into
Ltibutils.pm.  I've also included the sysfsutils package printouts to
illustrate that the build proceeds for packages that were not introduced
by me.

checking: sysfsutils
======================
License: GPL/LGPL
test_remote_file: url=sysfsutils-2.1.0.tar.gz, file=sysfsutils-2.1.0.tar.gz
OK GPP_STAGE: sysfsutils-2.1.0.tar.gz

checking: SeaMAC
==================
License: GPL
test_remote_file: url=SeaMAC_0.98.1.tar.gz, file=SeaMAC_0.98.1.tar.gz
URL must be absolute at /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm
line 206.
traceback:
 LWP::UserAgent::prepare_request:206
  (eval):249
   LWP::UserAgent::simple_request:248
    LWP::UserAgent::request:263
     Ltibutils::test_remote:345
      Ltibutils::test_remote_file:406
       main::release_copy:51
        main::release_copy_pkg:95
         main::release_copy_content:160
          main::release_main:346
           main::f_trelease:1228
            main:561

2) Ideally, I could perform the 'trelease' mode build at this point in
time without needing a network connection.  Maybe the LTIB could print
warnings instead of failing in this case.  It would seem natural that once
you acquire all of the files you need to do the build, including tar
files, rpm files, patches, md5 checksums, and the files of the ltib
itself, as well as the MANIFEST file, you should not require the network
to produce a binary iso "test" release.

I hope this was more concise.

> Hi Aaron,
>
> There's too much in this email for me to read and comprehend right now.
>
> There's good reasons for requiring network access during a release
> cycle.  If you want to bypass that then there's the MANIFEST, but at
> that point you're on your own.
>
> If you want more answers then please send in question one small topic at
>   a time.
>
> Regards, Stuart
>
> address@hidden wrote:
>> I added that line to the script and it generates printouts as follows
>> this
>> email.  Is it possible to have an error in the .spec file such that it
>> completes './ltib' but not './ltib -m trelease'?  Is it possible to
>> create
>> an iso with packages added to a local LTIB tree but not uploaded to the
>> GPP?
>>
>> I'm doing a project for a company right now which is related to some
>> other
>> projects that use tools like the ELDK, so the iso format is a familiar
>> starting point for cross-platform development for this particular
>> project.
>>  In terms of rebuilding a software project from scratch, a procedure can
>> be written for installing a common Linux distribution onto a machine,
>> installing the software of the binary iso installer, and rebuilding the
>> project from source, in this case with './ltib'.  I could also tar up
>> the
>> ltib code in CVS, and create a script that copies the required tarfiles
>> and patches to the appropriate area, but I think a self contained iso
>> file
>> is much better, especially if that's a part of the functionality of the
>> LTIB itself, which I assumed it was from the beginning of my development
>> based on reading the FAQ.
>>
>> In attempting to create such a binary release, and with the mentality of
>> the avionics industry in mind, I found it a little awkward that I needed
>> to access the network for any aspect of the creation of this iso,
>> especially since I had already successfully completed many iterations of
>> building the project executing './litb'.  Of course I can read/edit the
>> perl scripts and see the printouts on the command line to know what's
>> happening, but I just think it would be nice to have a way to make a
>> non-distributable, private binary iso installer while having the network
>> interface off.
>>
>> One problem I had to work around was a firewall at work, so doing 'cvs
>> ...' was harder for me than normal.  That's why I was asking if there
>> could be a version, maybe similar to the trelease mode in combination
>> with
>> the special tag localdir, in which no network connectivity was needed in
>> order to create the iso.  I guess I'm viewing the iso file more as a
>> convenient packager than as a medium for software distribution.
>>
>> One limitation for a build with no network might be CVS itself.  We are
>> using Subversion here, and I was thinking of migrating to Git, but I was
>> trying to see if there was potentially a way to copy all of the files of
>> the LTIB under version control to the stage directory where the iso is
>> built.  With Subversion, you can do the following:
>>
>> $ svn co http://software.server/svn/project
>> $ cd project
>>
>> # Here maybe I edit some project files so they're different than the
>> # versions that are on the server.  Also, I disconnect the network.
>>
>> $ svn export . stage-with-changes
>> $ svn export -r BASE . stage-without-changes
>>
>> Now the directory "stage-with-changes" contains the version control
>> files
>> as well as the edits that I made, and "stage-without-changes" contains
>> the
>> version control files exactly as they are on the server, without the
>> changes.  I can do this without network connectivity.  It might be that
>> CVS can't do something like that, hence the need for the cvsmanifest
>> script (which also requires network connectivity at some point in time).
>>
>> I admit that I haven't briefed myself on the legal structure of the
>> LTIB,
>> but I'm assuming that you can use it to develop private applications, as
>> well as a focal point for public, networked development.  There might be
>> other reasons for wanting to build an iso without interfacing with the
>> CVS
>> repository as well, such as needing to make small changes that other
>> developers wouldn't find useful or might find detrimental but which are
>> locally necessary.
>>
>> Any input would be appreciated.
>>
>>
>> Aaron
>>
>> ----
>>
>> $ ./ltib -m trelease
>>
>> You are about to create an iso image from this working project.
>> This will include all sources and built rpms for the target platform,
>> as well as the LTIB source code.
>>
>> *** TEST RELEASE, NOT TO BE DISTRIBUTED EXTERNALLY ***
>>
>> Before doing this, you should have done the following:
>>
>>  1. Make sure you have committed any changes
>>  2. Make sure your source code is up to date
>>  3. Configured ltib for the target
>>  4. Run ltib to build all the selected packages
>>
>> Do you want to continue: Y|n ?
>> y
>> Please enter the tag name for this release
>> Giving the name 'HEAD' will use the head of the trunk, or current
>> branch.
>> In test mode, only existing tags can be used (e.g use HEAD)
>> localdir
>>
>> Checking for unsaved config changes:
>>
>> *** TEST RELEASE, NOT TO DISTRIBUTED EXTERNALLY ***
>>
>> Checking licensing and external availabity of sources
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> The following constraints/checks are enforced:
>>
>> 1/ All source/patches in the default configuration must be
>> distributable.
>>
>> 2/ Any sources/patches not in the default configuration that are not
>> distributable are not be copied to the ISO image.
>>
>> 3/ Any content copied to the ISO image has it's md5 checksum verified by
>> downloading a fresh copy of the md5sum file and using it to verify the
>> content it checksums.
>>
>> 4/ Files already present on local disk in the LPP will be used.  Any
>> missing files will be downloaded from either the GPP staging area
>> or the CPP (Clickthru) staging area.
>>
>>                   ------------------------
>>
>>
>> Copying content from default configuration (mandatory)
>> ---------------------------------------------------------
>>
>> checking: toolchain
>> ====================
>> test_remote_file: url=tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm,
>> file=tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm
>> Testing network connectivity
>> OK GPP_STAGE:
>>
>> OK GPP_STAGE: tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm
>> test_remote_file: url=tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.src.rpm,
>> file=tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.src.rpm
>> OK GPP_STAGE: tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.src.rpm
>>
>> checking: fake-provides
>> =========================
>> License: GPL
>>
>> checking: kernel-2.6.13.4-mpc8349itx
>> ======================================
>> License: GPL
>> test_remote_file: url=linux-2.6.13.tar.gz, file=linux-2.6.13.tar.gz
>> OK GPP_STAGE: linux-2.6.13.tar.gz
>> test_remote_file: url=patch-2.6.13.4.gz, file=patch-2.6.13.4.gz
>> OK GPP_STAGE: patch-2.6.13.4.gz
>> test_remote_file: url=kernel-2.6.13-mpc8349itx-all-20060713.patch,
>> file=kernel-2.6.13-mpc8349itx-all-20060713.patch
>> OK GPP_STAGE: kernel-2.6.13-mpc8349itx-all-20060713.patch
>> test_remote_file: url=kernel-2.6.13-83xx-PCI-coherency.patch,
>> file=kernel-2.6.13-83xx-PCI-coherency.patch
>> OK GPP_STAGE: kernel-2.6.13-83xx-PCI-coherency.patch
>> test_remote_file: url=mpc8349itx-linux26-20060320-gianfar-fixes.patch,
>> file=mpc8349itx-linux26-20060320-gianfar-fixes.patch
>> OK GPP_STAGE: mpc8349itx-linux26-20060320-gianfar-fixes.patch
>> test_remote_file: url=mpc8349itx-linux26-20060620-gianfar-opt.patch,
>> file=mpc8349itx-linux26-20060620-gianfar-opt.patch
>> OK GPP_STAGE: mpc8349itx-linux26-20060620-gianfar-opt.patch
>> test_remote_file: url=kernel-2.6.13-mpc8349itx-rtc-usb-20060914.patch,
>> file=kernel-2.6.13-mpc8349itx-rtc-usb-20060914.patch
>> OK GPP_STAGE: kernel-2.6.13-mpc8349itx-rtc-usb-20060914.patch
>> test_remote_file: url=kernel-2.6.13-gianfar-skb-recycle.patch,
>> file=kernel-2.6.13-gianfar-skb-recycle.patch
>> OK GPP_STAGE: kernel-2.6.13-gianfar-skb-recycle.patch
>> test_remote_file: url=kernel-2.6.13-mpc8349itx-qg8mcu-2007011a.patch,
>> file=kernel-2.6.13-mpc8349itx-qg8mcu-2007011a.patch
>> OK GPP_STAGE: kernel-2.6.13-mpc8349itx-qg8mcu-2007011a.patch
>> test_remote_file: url=kernel-2.6.13-gcc4.patch,
>> file=kernel-2.6.13-gcc4.patch
>> OK GPP_STAGE: kernel-2.6.13-gcc4.patch
>>
>> ...
>> ...
>> ...
>>
>> checking: SeaMAC
>> ==================
>> License: GPL
>> test_remote_file: url=SeaMAC_0.98.1.tar.gz, file=SeaMAC_0.98.1.tar.gz
>> URL must be absolute at
>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm
>> line 206.
>> traceback:
>>  LWP::UserAgent::prepare_request:206
>>   (eval):249
>>    LWP::UserAgent::simple_request:248
>>     LWP::UserAgent::request:263
>>      Ltibutils::test_remote:345
>>       Ltibutils::test_remote_file:406
>>        main::release_copy:51
>>         main::release_copy_pkg:95
>>          main::release_copy_content:160
>>           main::release_main:346
>>            main::f_trelease:1228
>>             main:561
>>
>> Exiting on error or interrupt
>> Please see | tee RELEASE_COPY_CONTENT for details
>>
>>
>>> Hi Aaaron,
>>>
>>> Either there's problem in the SeaMAC.spec, or some bug in Ltibutils.pm.
>>>
>>> It would probably be easiest to put an 'warn' in just after the start
>>> of
>>> test_remote file to see what URL is getting called out.  So it would
>>> look like this (around 400)
>>>
>>> sub test_remote_file
>>> {
>>>      my ($url, $cf) = @_;
>>>      warn("no url passed"), return unless $url;
>>>      my ($file) = $url =~ m-/?([^/]+)$-;
>>> warn "test_remote_file: url=$url, file=$file\n";
>>>      my $ua  = LWP::UserAgent->new;
>>>
>>> Regards, Stuart
>>>
>>> address@hidden wrote:
>>>> I'll write a few more details tomorrow, but I was able to create the
>>>> binary release iso that I wanted to.  I used the 'trelease' mode with
>>>> the
>>>> special tag 'localdir' and a MANIFEST file.  However, I still needed
>>>> to
>>>> hack the file bin/Ltibutils.pm to build the iso, because I was getting
>>>> errors such as the following.  The SeaMAC package is one that I
>>>> introduced
>>>> to the build by creating a dist/lfs-5.1/SeaMAC/SeaMAC.spec file and
>>>> incorporating it into the configuration, etc.  Only after hacking the
>>>> file
>>>> (and I'm not a perl programmer) was I able to arrive at the iso.
>>>> Thanks
>>>> for the clue about the uClibc toolchain being distributable.  After
>>>> updating my ltib CVS files I got passed the toolchain check.
>>>>
>>>> ---
>>>>
>>>> $ ./ltib -m trelease
>>>> Updating lpp from local packages
>>>>
>>>> You are about to create an iso image from this working project.
>>>> This will include all sources and built rpms for the target platform,
>>>> as well as the LTIB source code.
>>>>
>>>> *** TEST RELEASE, NOT TO BE DISTRIBUTED EXTERNALLY ***
>>>>
>>>> Before doing this, you should have done the following:
>>>>
>>>>  1. Make sure you have committed any changes
>>>>  2. Make sure your source code is up to date
>>>>  3. Configured ltib for the target
>>>>  4. Run ltib to build all the selected packages
>>>>
>>>> Do you want to continue: Y|n ?
>>>> y
>>>> Please enter the tag name for this release
>>>> Giving the name 'HEAD' will use the head of the trunk, or current
>>>> branch.
>>>> In test mode, only existing tags can be used (e.g use HEAD)
>>>> localdir
>>>>
>>>> Checking for unsaved config changes:
>>>>
>>>> *** TEST RELEASE, NOT TO DISTRIBUTED EXTERNALLY ***
>>>>
>>>> Checking licensing and external availabity of sources
>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>
>>>> The following constraints/checks are enforced:
>>>>
>>>> 1/ All source/patches in the default configuration must be
>>>> distributable.
>>>>
>>>> 2/ Any sources/patches not in the default configuration that are not
>>>> distributable are not be copied to the ISO image.
>>>>
>>>> 3/ Any content copied to the ISO image has it's md5 checksum verified
>>>> by
>>>> downloading a fresh copy of the md5sum file and using it to verify the
>>>> content it checksums.
>>>>
>>>> 4/ Files already present on local disk in the LPP will be used.  Any
>>>> missing files will be downloaded from either the GPP staging area
>>>> or the CPP (Clickthru) staging area.
>>>>
>>>>                   ------------------------
>>>>
>>>>
>>>> Copying content from default configuration (mandatory)
>>>> ---------------------------------------------------------
>>>>
>>>> checking: toolchain
>>>> ====================
>>>> Testing network connectivity
>>>> OK GPP_STAGE:
>>>>
>>>> OK GPP_STAGE: tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm
>>>> OK GPP_STAGE: tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.src.rpm
>>>>
>>>> ...
>>>> ...
>>>> ...
>>>>
>>>> checking: SeaMAC
>>>> ==================
>>>> License: GPL
>>>> URL must be absolute at
>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm
>>>> line 206.
>>>> traceback:
>>>>  LWP::UserAgent::prepare_request:206
>>>>   (eval):249
>>>>    LWP::UserAgent::simple_request:248
>>>>     LWP::UserAgent::request:263
>>>>      Ltibutils::test_remote:345
>>>>       Ltibutils::test_remote_file:404
>>>>        main::release_copy:51
>>>>         main::release_copy_pkg:95
>>>>          main::release_copy_content:160
>>>>           main::release_main:346
>>>>            main::f_trelease:1228
>>>>             main:561
>>>>
>>>> Exiting on error or interrupt
>>>> Please see | tee RELEASE_COPY_CONTENT for details
>>>>
>>>>
>>>>
>>>>> Hi Aaron,
>>>>>
>>>>> I re-read this again and there are a number of points/issues to
>>>>> comment
>>>>> on:
>>>>>
>>>>> 1/ You can release whatever the toolchain.  The problem you seem to
>>>>> have
>>>>> had in the first instance is that there was no network connection
>>>>> available and so the checks to see that the file to be copied is on
>>>>> the
>>>>> GPP failed.
>>>>>
>>>>> 2/ If your BSP is different from that in CVS and/or it contains
>>>>> content
>>>>> that is not on the GPP, then you can't make an 'official' release.
>>>>> There's a long story behind this to do with signing off approved
>>>>> content.  However, you have some options:
>>>>>
>>>>> a) use ./ltib -m trelease
>>>>> b) use the tag 'localdir' and a MANIFEST file
>>>>>
>>>>> In the case of a), you still (may) need a network connection, but
>>>>> errors
>>>>> are not fatal.  The ISO image will be marked as 'non-distributable'
>>>>>
>>>>> In the case of b) 'localdir' is a special tag that says read the
>>>>> file called MANIFEST and use that to copy as the ltib source (not the
>>>>> packages).  You can generate a static manifest file from CVS using:
>>>>> bin/cvsmanifest > MANIFEST.  However, I still think a network is
>>>>> needed
>>>>> to copy some of the package content if its missing from your LPP.
>>>>> This
>>>>> is not really an option I want to support.
>>>>>
>>>>> 3/
>>>>>  >> Undefined subroutine &main::tag_repository called at
>>>>>  >> /var/backup/20091012-ltib/ltib/bin/Ltibrelease.pm line 353.
>>>>>  >> traceback:
>>>>>
>>>>> This is a bug, at some point the export of tag_repository got removed
>>>>> from Ltibutils.pm.  I've put it back.  Note that if you use the
>>>>> tagname
>>>>> 'HEAD' no CVS tagging is attempted.
>>>>>
>>>>>
>>>>> Recommendations:  if it's possible/useful get your packages into LTIB
>>>>> and then you can simply run:  ./ltib -m release and use the tag HEAD,
>>>>> this is the best way.  Failing that I'd go with ./ltib -m trelease
>>>>>
>>>>> Question: if you can say, what do you have in your LTIB tree that
>>>>> can't
>>>>> be put into the Savannah CVS?  Also why is making an ISO useful to
>>>>> you?
>>>>>   The problem with these is they can't be updated easily.
>>>>>
>>>>> Regards, Stuart
>>>>>
>>>>>
>>>>> Stuart Hughes wrote:
>>>>>> Hi Aaron,
>>>>>>
>>>>>> I'll look into this and get back to you,  all the content on the GPP
>>>>>> is
>>>>>> distributable so the problems you raise are bugs.
>>>>>>
>>>>>> Regards, Stuart
>>>>>>
>>>>>> address@hidden wrote:
>>>>>>> At some point in my usage of the LTIB tool I switched from using
>>>>>>> the
>>>>>>> glibc
>>>>>>> C library to uclibc to save space or because of compiler errors, I
>>>>>>> forget
>>>>>>> which.
>>>>>>>
>>>>>>> Now I'm trying to bundle up my ltib tree into a binary release
>>>>>>> using:
>>>>>>>
>>>>>>> ./ltib -m release
>>>>>>>
>>>>>>> I'm getting the following error:
>>>>>>>
>>>>>>> checking: toolchain
>>>>>>> ====================
>>>>>>> Testing network connectivity
>>>>>>> No network download connection available
>>>>>>>
>>>>>>> tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm is not externally
>>>>>>> distributable
>>>>>>> Error: tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm  required by
>>>>>>> defconfig
>>>>>>> traceback:
>>>>>>>  main::release_copy_content:167
>>>>>>>   main::release_main:353
>>>>>>>    main::f_release:1198
>>>>>>>     main:552
>>>>>>>
>>>>>>> If possible, I would like to create a binary iso release of my tree
>>>>>>> that
>>>>>>> includes the packages in the local package pool that are required
>>>>>>> to
>>>>>>> reproduce my images from source.  This iso file is not for
>>>>>>> distribution,
>>>>>>> but for internal records.  Do I need to switch back to using the
>>>>>>> glibc
>>>>>>> based cross-toolchain?  Also, it seems like this mode of ltib
>>>>>>> requires
>>>>>>> contact with the remote CVS repository, but I would like to be able
>>>>>>> to
>>>>>>> create this iso without network connectivity.
>>>>>>>
>>>>>>> In an attempt to produce any type of binary iso release, I started
>>>>>>> with
>>>>>>> a
>>>>>>> fresh copy of the ltib today and did the following:
>>>>>>>
>>>>>>> $ ./ltib --preconfig config/platform/mpc8349itx/defconfig
>>>>>>>
>>>>>>> (here I had to get around two things that stopped the build, one
>>>>>>> being
>>>>>>> the
>>>>>>> cuImage error, and the other de-selecting the samba package which
>>>>>>> had
>>>>>>> an
>>>>>>> error in the configuration (attempted execution of cross-compiled
>>>>>>> test
>>>>>>> code))
>>>>>>>
>>>>>>> $ cp config/platform/mpc8349itx/.config
>>>>>>> config/platform/mpc8349itx/defconfig
>>>>>>> $ cp
>>>>>>> config/platform/mpc8349itx/nas_linux-2.6.13.4-mpc8349itx.config
>>>>>>> config/platform/mpc8349itx/nas_linux-2.6.13.4-mpc8349itx.config.dev
>>>>>>> $ ./ltib -m release
>>>>>>>
>>>>>>> This last command errors with:
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>> checking: sparse
>>>>>>> ==================
>>>>>>> License: Open Software License
>>>>>>> OK GPP_STAGE: sparse-0.4.tar.gz
>>>>>>> OK GPP_STAGE: sparse-0.4-array-fix.patch
>>>>>>>
>>>>>>> checking: git
>>>>>>> ===============
>>>>>>> License: GPL
>>>>>>> OK GPP_STAGE: git-1.5.6.5.tar.gz
>>>>>>> OK GPP_STAGE: git-1.5.6.5-no-perl-install.patch
>>>>>>>
>>>>>>> checking: tunctl
>>>>>>> ==================
>>>>>>> License: GPL
>>>>>>> OK GPP_STAGE: tunctl-1.5.tar.gz
>>>>>>>
>>>>>>> checking: mux_server
>>>>>>> ======================
>>>>>>> License: LGPL
>>>>>>> OK GPP_STAGE: mux_server.c
>>>>>>> OK GPP_STAGE: mux_sever-1.0-fs-decl.patch
>>>>>>> Undefined subroutine &main::tag_repository called at
>>>>>>> /var/backup/20091012-ltib/ltib/bin/Ltibrelease.pm line 353.
>>>>>>> traceback:
>>>>>>>  main::release_main:353
>>>>>>>   main::f_release:1235
>>>>>>>    main:560
>>>>>>>
>>>>>>> I guess it would be nice if there were some method of producing the
>>>>>>> binary
>>>>>>> iso file without having to contact the remote CVS repository.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> LTIB home page: http://ltib.org
>>>>>>>
>>>>>>> Ltib mailing list
>>>>>>> address@hidden
>>>>>>> http://lists.nongnu.org/mailman/listinfo/ltib
>>>>>>>
>>>>>> _______________________________________________
>>>>>> LTIB home page: http://ltib.org
>>>>>>
>>>>>> Ltib mailing list
>>>>>> address@hidden
>>>>>> http://lists.nongnu.org/mailman/listinfo/ltib
>>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> LTIB home page: http://ltib.org
>>>>
>>>> Ltib mailing list
>>>> address@hidden
>>>> http://lists.nongnu.org/mailman/listinfo/ltib
>>>>
>>
>>
>>
>