Regards, Stuart
Stuart Hughes wrote:
> Hi Aaron,
>
> Okay, I'll need to fix the .md5 bug, I think I know what that is.
>
> Regarding network access during making an ISO, this is essential. There
> could only be a problem if someone has tampered with your copy, but this
> is exactly why all the network checking is done. The same goes for the
> content. With a local release anything not on the remote network is
> untrusted essentially. Note that there's provision in the release
> mechanism for git (for the future), this would give even greater
> security as you can check the local git tree against the remote (in case
> the remote had been tampered).
>
> I do agree however for customers (end users) the ISO should not require
> a network connection. In fact that's really the only reason for making
> ISO releases.
>
> I'll look into the localdir_nobuild, but to be honest I don't actually
> use this any more. It might be an idea to remove it.
>
> So to summarise:
>
> You should be able to make a test release, but there must be network
> access so that existing (network) content can be checked. Content that
> is not on the network should be warned about, but not prevent the
> creation of a test release.
>
> For the output ISO, it should be usable without a network (for now).
>
> I'll try to fix the remaining .md5 error soon.
>
> Regards, Stuart
>
>
> address@hidden wrote:
>> In the same way that you are concerned about the possibility that someone
>> might distribute a bugus iso release, people working with mission
>> critical
>> software would be worried about needing to connect to the Internet at the
>> time when the LTIB is creating the test release iso. I can see where a
>> technical expert would be less concerned, but this would be a red flag to
>> someone who oversees a software build process.
>>
>> Another thing I thought I would point out is that you can actually
>> kind of
>> get around this "checking" mechanism which ensures that packages are real
>> when constructing an iso test release. I will illustrate this at the
>> same
>> time as I describe the remaining part of the bug with the trelease
>> option.
>> Now I'm getting passed the http:// ftp:// thing that you fixed, but I'm
>> getting stuck on the MD5 checksum step. I do have the files:
>>
>> /opt/ltib/pkgs/u-boot-1.2.0-dl1000.tar.bz2.md5
>> /opt/ltib/pkgs/SeaMAC_0.98.1.tar.gz.md5
>>
>> However, the LTIB deletes my local copies during its checking in the
>> trelease mode, I presume because it couldn't find them on the GPP
>> server. So this remains a problem. The first error I get is:
>>
>> checking: u-boot-1.2.0-dl1000
>> ===============================
>> License: GPL
>> FAILED gpp_stage: u-boot-1.2.0-dl1000.tar.bz2
>> u-boot-1.2.0-dl1000.tar.bz2 not yet approved for distribution
>> WARN: skipping md5sum check for
>> /opt/ltib/pkgs/u-boot-1.2.0-dl1000.tar.bz2, md5 file was not found
>> cp: cannot stat `/opt/ltib/pkgs/u-boot-1.2.0-dl1000.tar.bz2.md5': No such
>> file or directory
>> Died at
>> /home/ria/non-distributable-ltib-mpc8349itx-20091015/bin/Ltibrelease.pm
>> line 55.
>> traceback:
>> main::release_copy:55
>> main::release_copy:61
>> main::release_copy_pkg:94
>> main::release_copy_content:159
>> main::release_main:345
>> main::f_trelease:1228
>> main:561
>>
>> Exiting on error or interrupt
>> Please see | tee RELEASE_COPY_CONTENT for details
>>
>>
>>
>> Now, let's say I reconfigure with -m config, and I deselect my custom
>> U-Boot package. Now, I get to this point in trelease mode:
>>
>>
>>
>> checking: SeaMAC
>> ==================
>> License: GPL
>> FAILED gpp_stage: SeaMAC_0.98.1.tar.gz
>> SeaMAC_0.98.1.tar.gz not yet approved for distribution
>> WARN: skipping md5sum check for /opt/ltib/pkgs/SeaMAC_0.98.1.tar.gz, md5
>> file was not found
>> cp: cannot stat `/opt/ltib/pkgs/SeaMAC_0.98.1.tar.gz.md5': No such
>> file or
>> directory
>> Died at
>> /home/ria/non-distributable-ltib-mpc8349itx-20091015/bin/Ltibrelease.pm
>> line 55.
>> traceback:
>> main::release_copy:55
>> main::release_copy:61
>> main::release_copy_pkg:94
>> main::release_copy_content:159
>> main::release_main:345
>> main::f_trelease:1228
>> main:561
>>
>> Exiting on error or interrupt
>> Please see | tee RELEASE_COPY_CONTENT for details
>>
>>
>>
>> Let's say I to a -m config once again, and deselect this custom
>> package. Now, if I do ./ltib -m trelease, followed by
>> "localdir_nobuild", I can
>> effectively bypass the checking mechanism of the LTIB with the GPP. The
>> first thing that happens after it checks all the packages that I have
>> selected in my file
>>
>> config/platform/mpc8349itx/.config
>>
>> with the data in the GPP is that it drops me to the main configuration
>> menu
>>
>> GNU/Linux Target Image Builder : Platform Selection
>>
>> Now, I can select my platform, and I can also select u-boot-1.2.0-dl1000,
>> as well as SeaMAC for my build. This bypasses the network checks, and I
>> can build my test release iso. I haven't tried it, but I assume I could
>> also replace the standard packages with bogus copies in this way, like
>> the
>> kernel for instance, and easily create a iso release with my version of
>> whatever package I like.
>>
>> Anyway, I still think it might actually be a benefit the LTIB to allow a
>> loose mechanism for building a test release iso that does not require
>> network checking. Like I said, this is from a source code control
>> standpoint, and might sway a project manager one way or another in
>> deciding whether or not to use this tool for development. I think
>> marking
>> the test release very clearly as non-distributable is sufficient in this
>> case, and that anyone who really wants to package up something bogus can
>> do it anyway, even using the existing functionality of the LTIB with the
>> localdir_nobuild special tag. I view the test release iso more as a
>> convenient installer than as something to be distributed on the Internet.
>>
>> So, there's just one more MD5 sum problem, but I can almost do the build
>> with trelease/localdir/MANIFEST. Thanks again for all your help.
>>
>> Aaron
>>
>>> Hi Aaron,
>>>
>>> What you were running into here was that later LWP::UserAgent modules
>>> die unless they have the correct 'http:// ftp://' prefix. It's odd as
>>> even Fedora 10 has this, but not Debian Lenny. Anyhow the bug was in
>>> LTIB.
>>>
>>> I've committed a change that addresses both issued discussed:
>>> 1/ fix the bug in LTIB
>>> 2/ enhance the release so that in trelease mode files don't have to
>>> exist on the GPP (or package pools to be more accurate)
>>>
>>> This means that provided you have all the content locally you can make
>>> your ISO image.
>>>
>>> Regarding your points below I agree except that a network connection to
>>> the GPP *must* be available while making a release. If it isn't it
>>> would be possible to put in bogus copies of files that do exist on the
>>> real GPP and send them out in an ISO to unsuspecting people. Also it's
>>> important to record what content on your ISO is 'approved' (on the GPP
>>> or your own private CPP) versus that which only exists locally.
>>>
>>> Please test and let me know it it does what you expect (or not).
>>>
>>> Regards, Stuart
>>>
>>> address@hidden wrote:
>>>> Thanks for all your help Stuart. Sorry it took so long to debug. I
>>>> guess
>>>> it got a little confusing because it looks like LWP::Debug has changed
>>>> somehow. When I type man LWP::Debug, the top of the file is:
>>>>
>>>> ----
>>>> LWP::Debug(3) User Contributed Perl Documentation LWP::Debug(3)
>>>>
>>>> NAME
>>>> LWP::Debug - deprecated
>>>> ----
>>>>
>>>> Besides the bug, my original thought was that I should be able to
>>>> produce
>>>> the binary test relase iso without network connectivity at the end
>>>> of my
>>>> development. I was hoping that you could make the test for network
>>>> connectivity non-fatal in some cases, such as the
>>>> trelease/{localdir,localdir_nobuild}/MANIFEST combo. Since it is a
>>>> test
>>>> release, and since packages can be introduced that are private, I don't
>>>> feel it necessary for the LTIB to check the GPP for the distribution
>>>> approval of the packages and fail the build if there's no connection.
>>>> By
>>>> nature and name the iso is non-distributable. My issue here doesn't
>>>> necessarily stem from a privacy standpoint, although this could be
>>>> important to developers using the LTIB, but from a source code control
>>>> standpoint. At a very basic level, if I've been using the LTIB for
>>>> some
>>>> time, going through multiple build cycles, target testing, and settled
>>>> on
>>>> something I want to encapsulate in non-distributable iso form, the last
>>>> thing that I want at the end of all that testing is for the LTIB to
>>>> require access to the Internet in order to construct the iso file (even
>>>> if
>>>> I can understand what is happening). It would be much more reassuring
>>>> from a source code control standpoint to be able to unplug the network,
>>>> build the iso, take this iso to another machine that is also removed
>>>> from
>>>> the network, and rebuild my target image. Anyway, if you could include
>>>> this functionality by issuing a warning rather than an error, I think
>>>> that
>>>> would make more sense, only in the case of the non-distributable iso.
>>>> If
>>>> I unplug my network cable, the printouts I get for trelease/localdir
>>>> are
>>>> as follows. Please let me know what you think.
>>>>
>>>> Thanks again Stuart, and have a good weekend yourself.
>>>>
>>>> checking: toolchain
>>>> ====================
>>>> Testing network connectivity
>>>> No network download connection available
>>>>
>>>> tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm not yet approved for
>>>> distribution
>>>> URL must be absolute at
>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm
>>>> line 206, <FN> line 2154.
>>>> traceback:
>>>> LWP::UserAgent::prepare_request:206
>>>> (eval):249
>>>> LWP::UserAgent::simple_request:248
>>>> LWP::UserAgent::request:263
>>>> Ltibutils::test_remote:342
>>>> Ltibutils::test_remote_file:400
>>>> main::release_copy:51
>>>> main::release_copy:61
>>>> main::release_copy_content:155
>>>> main::release_main:346
>>>> main::f_trelease:1228
>>>> main:561
>>>>
>>>> Exiting on error or interrupt
>>>> Please see | tee RELEASE_COPY_CONTENT for details
>>>>
>>>>
>>>>> Hi Aaron,
>>>>>
>>>>> It's a bug, I'll fix and let you know when to re-test. Might not be
>>>>> until Monday now.
>>>>>
>>>>> Have a good w/e.
>>>>>
>>>>> Regards, Stuart
>>>>>
>>>>> Stuart Hughes wrote:
>>>>>> Hi Aaron,
>>>>>>
>>>>>> Thanks for that, I can see the difference, the URL is missing the
>>>>>> http://... in the one that fails:
>>>>>>
>>>>>>
>>>>>> prepare_request: url is:
>>>>>> 'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349emdstu.tgz' at
>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
>>>>>>
>>>>>> versus:
>>>>>>
>>>>>>
>>>>>> 'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349emdstu.tgz' at
>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
>>>>>> prepare_request: url is: 'u-boot-1.1.3-mpc8349emdstu.tgz' at
>>>>>>
>>>>>>
>>>>>> I'll dig into this and get back to you.
>>>>>>
>>>>>> Regards, Stuart
>>>>>>
>>>>>>
>>>>>>
>>>>>> address@hidden wrote:
>>>>>>> I don't think we have a proxy server in our network topology, but I
>>>>>>> can
>>>>>>> double-check if need be. I changed my configuration a little to
>>>>>>> debug
>>>>>>> this problem. I did a CVS checkout of the latest copy of the LTIB.
>>>>>>> From
>>>>>>> there, I copy
>>>>>>>
>>>>>>> dist/lfs-5.1/u-boot/u-boot-1.1.3-mpc8349itx.spec
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> dist/lfs-5.1/u-boot/u-boot-1.1.3-mpc8349itxyz.spec
>>>>>>>
>>>>>>> and
>>>>>>>
>>>>>>> /opt/ltib/pkgs/u-boot-1.1.3-mpc8349emds.tgz
>>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>> /opt/ltib/pkgs/u-boot-1.1.3-mpc8349emdstu.tgz
>>>>>>>
>>>>>>> and I generate the md5sum file for the copied tar file. I create a
>>>>>>> MANIFEST file and add an entry for the new "xyz" spec, and I edit my
>>>>>>> config/platform/mpc8349itx/main.lkc to build the new U-Boot project,
>>>>>>> which
>>>>>>> can be found in my local file system, but not on in the GPP. When I
>>>>>>> type
>>>>>>> ./ltib --configure, I select the uClibc cross-tools rather than
>>>>>>> Glibc.
>>>>>>> I
>>>>>>> kill the build after it successfully cross-compiles this new package
>>>>>>> and
>>>>>>> moves on to the kernel build.
>>>>>>>
>>>>>>> My perl RPM package is:
>>>>>>>
>>>>>>> # rpm -q perl
>>>>>>> perl-5.10.0-82.fc11.i586
>>>>>>>
>>>>>>> Given this new, more or less pristine configuration, I insert the
>>>>>>> debug
>>>>>>> statement you suggested into the file UserAgent.pm file.
>>>>>>>
>>>>>>> Now, typing ./ltib --dltest, I get:
>>>>>>>
>>>>>>> $ ./ltib --dltest
>>>>>>> Updating lpp from local packages
>>>>>>>
>>>>>>> Processing platform: Host support packages
>>>>>>> ...
>>>>>>> ...
>>>>>>> Processing: u-boot-1.1.3-mpc8349itxyz
>>>>>>> =======================================
>>>>>>> prepare_request: url is:
>>>>>>> 'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349emdstu.tgz' at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
>>>>>>> FAILED : u-boot-1.1.3-mpc8349emdstu.tgz
>>>>>>> prepare_request: url is:
>>>>>>> 'http://bitshrine.org/gpp/u-boot-1.1.3_mpc8349e_pciagent.patch' at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
>>>>>>> OK GPP: u-boot-1.1.3_mpc8349e_pciagent.patch
>>>>>>> prepare_request: url is:
>>>>>>> 'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349itx-all-20060628.patch'
>>>>>>>
>>>>>>> at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
>>>>>>> OK GPP: u-boot-1.1.3-mpc8349itx-all-20060628.patch
>>>>>>> prepare_request: url is:
>>>>>>> 'http://bitshrine.org/gpp/u-boot-1.1.3-large-image-boot.patch' at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
>>>>>>> OK GPP: u-boot-1.1.3-large-image-boot.patch
>>>>>>> ...
>>>>>>> ...
>>>>>>> Started: Fri Oct 23 11:08:36 2009
>>>>>>> Ended: Fri Oct 23 11:09:49 2009
>>>>>>> Elapsed: 73 seconds
>>>>>>>
>>>>>>> These packages would not have complete downloads available:
>>>>>>> u-boot-1.1.3-mpc8349itxyz:
>>>>>>> u-boot-1.1.3-mpc8349emdstu.tgz
>>>>>>>
>>>>>>> Build Failed
>>>>>>>
>>>>>>>
>>>>>>> Typing ./ltib -m trelease, followed by y, and localdir, I get:
>>>>>>>
>>>>>>> checking: toolchain
>>>>>>> ====================
>>>>>>> Testing network connectivity
>>>>>>> prepare_request: url is: 'http://bitshrine.org/gpp/' at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211, <FN>
>>>>>>> line
>>>>>>> 2154.
>>>>>>> OK GPP_STAGE:
>>>>>>>
>>>>>>> prepare_request: url is:
>>>>>>> 'http://bitshrine.org/gpp/tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm'
>>>>>>> at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211, <FN>
>>>>>>> line
>>>>>>> 2154.S
>>>>>>> OK GPP_STAGE: tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm
>>>>>>> prepare_request: url is:
>>>>>>> 'http://bitshrine.org/gpp/tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.src.rpm'
>>>>>>> at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
>>>>>>> OK GPP_STAGE: tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.src.rpm
>>>>>>>
>>>>>>> checking: fake-provides
>>>>>>> =========================
>>>>>>> License: GPL
>>>>>>>
>>>>>>> checking: u-boot-1.1.3-mpc8349itxyz
>>>>>>> =====================================
>>>>>>> License: GPL
>>>>>>> prepare_request: url is:
>>>>>>> 'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349emdstu.tgz' at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
>>>>>>> prepare_request: url is: 'u-boot-1.1.3-mpc8349emdstu.tgz' at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
>>>>>>> URL must be absolute at
>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm
>>>>>>> line 212.
>>>>>>> traceback:
>>>>>>> LWP::UserAgent::prepare_request:212
>>>>>>> (eval):255
>>>>>>> LWP::UserAgent::simple_request:254
>>>>>>> LWP::UserAgent::request:269
>>>>>>> Ltibutils::test_remote:342
>>>>>>> Ltibutils::test_remote_file:400
>>>>>>> main::release_copy:51
>>>>>>> main::release_copy_pkg:95
>>>>>>> main::release_copy_content:160
>>>>>>> main::release_main:346
>>>>>>> main::f_trelease:1228
>>>>>>> main:561
>>>>>>>
>>>>>>> Exiting on error or interrupt
>>>>>>> Please see | tee RELEASE_COPY_CONTENT for details
>>>>>>>
>>>>>>> The line "use LWP::Debug qw(+);" is uncommented, but does not
>>>>>>> generate
>>>>>>> debug statements.
>>>>>>>
>>>>>>>> Hi Aaron,
>>>>>>>>
>>>>>>>> This does not help. The problem is that the expected failure is
>>>>>>>> easy
>>>>>>>> to
>>>>>>>> fix, but *first* the other problem (1/) needs to be resolved. The
>>>>>>>> reason is that your other machine actually dies at
>>>>>>>> LWP::UserAgent::prepare_request:206, so we need to find out why.
>>>>>>>> Even if I put the code in to make the missing remote file just a
>>>>>>>> warning, you'd still die in the same place as the failure is in
>>>>>>>> UserAgent.pm (not part of LTIB).
>>>>>>>>
>>>>>>>> So please go back to the other machine and try to figure out
>>>>>>>> what is
>>>>>>>> wrong with its copy of
>>>>>>>> /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm
>>>>>>>> put some prints in there and try to see why it doesn't like the
>>>>>>>> URL.
>>>>>>>> So
>>>>>>>>
>>>>>>>> If you do not then we'll never know why your FC11 is failing (BTW I
>>>>>>>> know
>>>>>>>> have FC11 and its fine on there for me using a simple ./ltib
>>>>>>>> --dltest).
>>>>>>>> Can you try running on an unmodified LTIB:
>>>>>>>> ./ltib --dltest
>>>>>>>> this runs the same checking logic as the release and is quicker to
>>>>>>>> try.
>>>>>>>> In /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm you could add
>>>>>>>> a
>>>>>>>> print here (around line 206):
>>>>>>>>
>>>>>>>> sub prepare_request
>>>>>>>> {
>>>>>>>> my($self, $request) = @_;
>>>>>>>> die "Method missing" unless $request->method;
>>>>>>>> my $url = ""
>>>>>>>> die "URL missing" unless $url;
>>>>>>>> warn "prepare_request: url is: '$url'";
>>>>>>>> die "URL must be absolute" unless $url->scheme;
>>>>>>>>
>>>>>>>> One thought also, are you running via a proxy? if so maybe your LWP
>>>>>>>> doesn't like that. I have not proxy to try with (but in the past
>>>>>>>> when
>>>>>>>> I
>>>>>>>> did it worked fine, but not tried on FC11).
>>>>>>>>
>>>>>>>> Regards, Stuart
>>>>>>>
>>>>
>>>>
>>
>>
>>
>
>
> _______________________________________________
> LTIB home page: http://ltib.org
>
> Ltib mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/ltib
>
_______________________________________________
LTIB home page: http://ltib.org
Ltib mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/ltib