In the same way that you are concerned about the possibility that someone
might distribute a bugus iso release, people working with mission critical
software would be worried about needing to connect to the Internet at the
time when the LTIB is creating the test release iso. I can see where a
technical expert would be less concerned, but this would be a red flag to
someone who oversees a software build process.
Another thing I thought I would point out is that you can actually kind of
get around this "checking" mechanism which ensures that packages are real
when constructing an iso test release. I will illustrate this at the same
time as I describe the remaining part of the bug with the trelease option.
Now I'm getting passed the http:// ftp:// thing that you fixed, but I'm
getting stuck on the MD5 checksum step. I do have the files:
/opt/ltib/pkgs/u-boot-1.2.0-dl1000.tar.bz2.md5
/opt/ltib/pkgs/SeaMAC_0.98.1.tar.gz.md5
However, the LTIB deletes my local copies during its checking in the
trelease mode, I presume because it couldn't find them on the GPP server.
So this remains a problem. The first error I get is:
checking: u-boot-1.2.0-dl1000
===============================
License: GPL
FAILED gpp_stage: u-boot-1.2.0-dl1000.tar.bz2
u-boot-1.2.0-dl1000.tar.bz2 not yet approved for distribution
WARN: skipping md5sum check for
/opt/ltib/pkgs/u-boot-1.2.0-dl1000.tar.bz2, md5 file was not found
cp: cannot stat `/opt/ltib/pkgs/u-boot-1.2.0-dl1000.tar.bz2.md5': No such
file or directory
Died at
/home/ria/non-distributable-ltib-mpc8349itx-20091015/bin/Ltibrelease.pm
line 55.
traceback:
main::release_copy:55
main::release_copy:61
main::release_copy_pkg:94
main::release_copy_content:159
main::release_main:345
main::f_trelease:1228
main:561
Exiting on error or interrupt
Please see | tee RELEASE_COPY_CONTENT for details
Now, let's say I reconfigure with -m config, and I deselect my custom
U-Boot package. Now, I get to this point in trelease mode:
checking: SeaMAC
==================
License: GPL
FAILED gpp_stage: SeaMAC_0.98.1.tar.gz
SeaMAC_0.98.1.tar.gz not yet approved for distribution
WARN: skipping md5sum check for /opt/ltib/pkgs/SeaMAC_0.98.1.tar.gz, md5
file was not found
cp: cannot stat `/opt/ltib/pkgs/SeaMAC_0.98.1.tar.gz.md5': No such file or
directory
Died at
/home/ria/non-distributable-ltib-mpc8349itx-20091015/bin/Ltibrelease.pm
line 55.
traceback:
main::release_copy:55
main::release_copy:61
main::release_copy_pkg:94
main::release_copy_content:159
main::release_main:345
main::f_trelease:1228
main:561
Exiting on error or interrupt
Please see | tee RELEASE_COPY_CONTENT for details
Let's say I to a -m config once again, and deselect this custom package.
Now, if I do ./ltib -m trelease, followed by "localdir_nobuild", I can
effectively bypass the checking mechanism of the LTIB with the GPP. The
first thing that happens after it checks all the packages that I have
selected in my file
config/platform/mpc8349itx/.config
with the data in the GPP is that it drops me to the main configuration menu
GNU/Linux Target Image Builder : Platform Selection
Now, I can select my platform, and I can also select u-boot-1.2.0-dl1000,
as well as SeaMAC for my build. This bypasses the network checks, and I
can build my test release iso. I haven't tried it, but I assume I could
also replace the standard packages with bogus copies in this way, like the
kernel for instance, and easily create a iso release with my version of
whatever package I like.
Anyway, I still think it might actually be a benefit the LTIB to allow a
loose mechanism for building a test release iso that does not require
network checking. Like I said, this is from a source code control
standpoint, and might sway a project manager one way or another in
deciding whether or not to use this tool for development. I think marking
the test release very clearly as non-distributable is sufficient in this
case, and that anyone who really wants to package up something bogus can
do it anyway, even using the existing functionality of the LTIB with the
localdir_nobuild special tag. I view the test release iso more as a
convenient installer than as something to be distributed on the Internet.
So, there's just one more MD5 sum problem, but I can almost do the build
with trelease/localdir/MANIFEST. Thanks again for all your help.
Aaron
Hi Aaron,
What you were running into here was that later LWP::UserAgent modules
die unless they have the correct 'http:// ftp://' prefix. It's odd as
even Fedora 10 has this, but not Debian Lenny. Anyhow the bug was in
LTIB.
I've committed a change that addresses both issued discussed:
1/ fix the bug in LTIB
2/ enhance the release so that in trelease mode files don't have to
exist on the GPP (or package pools to be more accurate)
This means that provided you have all the content locally you can make
your ISO image.
Regarding your points below I agree except that a network connection to
the GPP *must* be available while making a release. If it isn't it
would be possible to put in bogus copies of files that do exist on the
real GPP and send them out in an ISO to unsuspecting people. Also it's
important to record what content on your ISO is 'approved' (on the GPP
or your own private CPP) versus that which only exists locally.
Please test and let me know it it does what you expect (or not).
Regards, Stuart
address@hidden wrote:
Thanks for all your help Stuart. Sorry it took so long to debug. I
guess
it got a little confusing because it looks like LWP::Debug has changed
somehow. When I type man LWP::Debug, the top of the file is:
----
LWP::Debug(3) User Contributed Perl Documentation LWP::Debug(3)
NAME
LWP::Debug - deprecated
----
Besides the bug, my original thought was that I should be able to
produce
the binary test relase iso without network connectivity at the end of my
development. I was hoping that you could make the test for network
connectivity non-fatal in some cases, such as the
trelease/{localdir,localdir_nobuild}/MANIFEST combo. Since it is a test
release, and since packages can be introduced that are private, I don't
feel it necessary for the LTIB to check the GPP for the distribution
approval of the packages and fail the build if there's no connection.
By
nature and name the iso is non-distributable. My issue here doesn't
necessarily stem from a privacy standpoint, although this could be
important to developers using the LTIB, but from a source code control
standpoint. At a very basic level, if I've been using the LTIB for some
time, going through multiple build cycles, target testing, and settled
on
something I want to encapsulate in non-distributable iso form, the last
thing that I want at the end of all that testing is for the LTIB to
require access to the Internet in order to construct the iso file (even
if
I can understand what is happening). It would be much more reassuring
from a source code control standpoint to be able to unplug the network,
build the iso, take this iso to another machine that is also removed
from
the network, and rebuild my target image. Anyway, if you could include
this functionality by issuing a warning rather than an error, I think
that
would make more sense, only in the case of the non-distributable iso.
If
I unplug my network cable, the printouts I get for trelease/localdir are
as follows. Please let me know what you think.
Thanks again Stuart, and have a good weekend yourself.
checking: toolchain
====================
Testing network connectivity
No network download connection available
tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm not yet approved for
distribution
URL must be absolute at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm
line 206, <FN> line 2154.
traceback:
LWP::UserAgent::prepare_request:206
(eval):249
LWP::UserAgent::simple_request:248
LWP::UserAgent::request:263
Ltibutils::test_remote:342
Ltibutils::test_remote_file:400
main::release_copy:51
main::release_copy:61
main::release_copy_content:155
main::release_main:346
main::f_trelease:1228
main:561
Exiting on error or interrupt
Please see | tee RELEASE_COPY_CONTENT for details
Hi Aaron,
It's a bug, I'll fix and let you know when to re-test. Might not be
until Monday now.
Have a good w/e.
Regards, Stuart
Stuart Hughes wrote:
Hi Aaron,
Thanks for that, I can see the difference, the URL is missing the
http://... in the one that fails:
prepare_request: url is:
'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349emdstu.tgz' at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
versus:
'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349emdstu.tgz' at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
prepare_request: url is: 'u-boot-1.1.3-mpc8349emdstu.tgz' at
I'll dig into this and get back to you.
Regards, Stuart
address@hidden wrote:
I don't think we have a proxy server in our network topology, but I
can
double-check if need be. I changed my configuration a little to
debug
this problem. I did a CVS checkout of the latest copy of the LTIB.
From
there, I copy
dist/lfs-5.1/u-boot/u-boot-1.1.3-mpc8349itx.spec
to
dist/lfs-5.1/u-boot/u-boot-1.1.3-mpc8349itxyz.spec
and
/opt/ltib/pkgs/u-boot-1.1.3-mpc8349emds.tgz
to
/opt/ltib/pkgs/u-boot-1.1.3-mpc8349emdstu.tgz
and I generate the md5sum file for the copied tar file. I create a
MANIFEST file and add an entry for the new "xyz" spec, and I edit my
config/platform/mpc8349itx/main.lkc to build the new U-Boot project,
which
can be found in my local file system, but not on in the GPP. When I
type
./ltib --configure, I select the uClibc cross-tools rather than
Glibc.
I
kill the build after it successfully cross-compiles this new package
and
moves on to the kernel build.
My perl RPM package is:
# rpm -q perl
perl-5.10.0-82.fc11.i586
Given this new, more or less pristine configuration, I insert the
debug
statement you suggested into the file UserAgent.pm file.
Now, typing ./ltib --dltest, I get:
$ ./ltib --dltest
Updating lpp from local packages
Processing platform: Host support packages
...
...
Processing: u-boot-1.1.3-mpc8349itxyz
=======================================
prepare_request: url is:
'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349emdstu.tgz' at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
FAILED : u-boot-1.1.3-mpc8349emdstu.tgz
prepare_request: url is:
'http://bitshrine.org/gpp/u-boot-1.1.3_mpc8349e_pciagent.patch' at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
OK GPP: u-boot-1.1.3_mpc8349e_pciagent.patch
prepare_request: url is:
'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349itx-all-20060628.patch'
at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
OK GPP: u-boot-1.1.3-mpc8349itx-all-20060628.patch
prepare_request: url is:
'http://bitshrine.org/gpp/u-boot-1.1.3-large-image-boot.patch' at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
OK GPP: u-boot-1.1.3-large-image-boot.patch
...
...
Started: Fri Oct 23 11:08:36 2009
Ended: Fri Oct 23 11:09:49 2009
Elapsed: 73 seconds
These packages would not have complete downloads available:
u-boot-1.1.3-mpc8349itxyz:
u-boot-1.1.3-mpc8349emdstu.tgz
Build Failed
Typing ./ltib -m trelease, followed by y, and localdir, I get:
checking: toolchain
====================
Testing network connectivity
prepare_request: url is: 'http://bitshrine.org/gpp/' at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211, <FN>
line
2154.
OK GPP_STAGE:
prepare_request: url is:
'http://bitshrine.org/gpp/tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm'
at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211, <FN>
line
2154.S
OK GPP_STAGE: tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.i386.rpm
prepare_request: url is:
'http://bitshrine.org/gpp/tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.src.rpm'
at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
OK GPP_STAGE: tc-fsl-x86lnx-ppc-uclibc-4.2.4-1.src.rpm
checking: fake-provides
=========================
License: GPL
checking: u-boot-1.1.3-mpc8349itxyz
=====================================
License: GPL
prepare_request: url is:
'http://bitshrine.org/gpp/u-boot-1.1.3-mpc8349emdstu.tgz' at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
prepare_request: url is: 'u-boot-1.1.3-mpc8349emdstu.tgz' at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm line 211.
URL must be absolute at
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm
line 212.
traceback:
LWP::UserAgent::prepare_request:212
(eval):255
LWP::UserAgent::simple_request:254
LWP::UserAgent::request:269
Ltibutils::test_remote:342
Ltibutils::test_remote_file:400
main::release_copy:51
main::release_copy_pkg:95
main::release_copy_content:160
main::release_main:346
main::f_trelease:1228
main:561
Exiting on error or interrupt
Please see | tee RELEASE_COPY_CONTENT for details
The line "use LWP::Debug qw(+);" is uncommented, but does not
generate
debug statements.
Hi Aaron,
This does not help. The problem is that the expected failure is
easy
to
fix, but *first* the other problem (1/) needs to be resolved. The
reason is that your other machine actually dies at
LWP::UserAgent::prepare_request:206, so we need to find out why.
Even if I put the code in to make the missing remote file just a
warning, you'd still die in the same place as the failure is in
UserAgent.pm (not part of LTIB).
So please go back to the other machine and try to figure out what is
wrong with its copy of
/usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm
put some prints in there and try to see why it doesn't like the URL.
So
If you do not then we'll never know why your FC11 is failing (BTW I
know
have FC11 and its fine on there for me using a simple ./ltib
--dltest).
Can you try running on an unmodified LTIB:
./ltib --dltest
this runs the same checking logic as the release and is quicker to
try.
In /usr/lib/perl5/vendor_perl/5.10.0/LWP/UserAgent.pm you could add
a
print here (around line 206):
sub prepare_request
{
my($self, $request) = @_;
die "Method missing" unless $request->method;
my $url = $request->url;
die "URL missing" unless $url;
warn "prepare_request: url is: '$url'";
die "URL must be absolute" unless $url->scheme;
One thought also, are you running via a proxy? if so maybe your LWP
doesn't like that. I have not proxy to try with (but in the past
when
I
did it worked fine, but not tried on FC11).
Regards, Stuart