[Wp-mirror-list] Attempting to mirror on laptop

[wp mirror]

Dear Benjamin,

Thank you very much for spotting the issue with the Debian Almquist
Shell (DASH).

0) Preamble

I have since learned that Debian and Ubuntu are moving to DASH as the
default shell for scripts because it is smaller and faster.  The GNU
Bourne Again Shell (BASH) remains the default for login.  The Debian
policy manual has long mandated that "shell scripts specifying
'/bin/sh' as interpreter must only use POSIX features".

The article <https://wiki.ubuntu.com/DashAsBinSh> is helpful.

Scripts need to be tested with CHECKBASHISMS.  This is to increase
portability.  This is not just an issue for ichunks, but for all
places where I fork a shell.

1) ichunks

Here is what an ichunk looks like before the fix:

shell> checkbashisms enwiki-20120802-pages-articles-p000000000-c000001000.sh
<snip>
possible bashism in
enwiki-20120802-pages-articles-p000000000-c000001000.sh line 3085
(test with unary -a (should be -e)):
                if [ -a $IMAGE./water_reflectivity.jpg ]; then
<snip>

I have fixed this (replaced -a with -e) in the code for WP-MIRROR 0.4,
and the ichunks pass.

2) images

Downloaded images are stored under a [0-9a-f] directory tree.  The
exact directory is computed from a hash of the file name, which I
computed by forking a shell command.

Please confirm that the following shell commands work:

The old shell command was:

shell> echo -n Arc_en_ciel.png | openssl md5
00135a44372c142bd509367a9f166733

The first two hex characters, `00' in this case, are used.  So
Arc_en_ciel.png should be stored under
/var/lib/mediawiki/images/0/00/.

Well it turns out that the options to ECHO, -n in this case, may no
longer be supported in DASH.

The new shell command is:

shell> env printf %s Arc_en_ciel.png | openssl md5
00135a44372c142bd509367a9f166733

Please also show the output of:

shell> dash -c "env printf %s Arc_en_ciel.png | openssl md5"
00135a44372c142bd509367a9f166733

3) pipelines

I recall from your emails of 2012-07-13, that redirecting pipes was a
problem. It turns out that the redirection "|&" is a bashism for "2>&1
|".

Please show me your output of the following commands:

shell> dash -c "/usr/sbin/apache2ctl -S |& grep mediawiki"
dash: Syntax error: "&" unexpected

shell> dash -c "/usr/sbin/apache2ctl -S 2>&1 | grep mediawiki"
         port 80 namevhost mediawiki.site
(/etc/apache2/sites-enabled/mediawiki.site.conf:1)

4) block devices

I have yet to figure out the following error that you reported 2012-07-13:

-----sys-pathname-slaves-----done-----
physical-block-device-list        : (/sys/dev/block/8:5/)
/dev/8:5: No such file or directory
*** - read: input stream #<input buffered pipe-input-stream character 6421>
      has reached its end

    z) commented out lines 3443-3456 of wpmirror

5) ichunks again

Thanks for pointing out the following line:

IMAGEPATH=http://upload.wikimedia.org/wikipedia/nil/

The appearance of 'nil' is a bug.  The line should read

IMAGEPATH=http://upload.wikimedia.org/wikipedia/simple/

I have corrected this in the code for WP-MIRROR 0.4.

Sincerely Yours,
Kent