[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Wp-mirror-list] Attempting to mirror on laptop
From: |
Dr. Kent L. Miller |
Subject: |
[Wp-mirror-list] Attempting to mirror on laptop |
Date: |
Wed, 15 Aug 2012 19:02:55 -0400 (EDT) |
User-agent: |
Alpine 2.00 (DEB 1167 2008-08-23) |
---------- Forwarded message ----------
Date: Wed, 15 Aug 2012 17:50:59 -0400
From: Benjamin Goldsmith <address@hidden>
To: wp mirror <address@hidden>
Subject: RE: Attempting to mirror on laptop
Thank you for your continued assistance with wp-mirror, Kent.
Here is the information you requested:
1)images
I will not post the results of:
root-shell> ls -l /var/lib/mediawiki/images/
The list included tens of thousands of image files:
address@hidden:~# ls -l /var/lib/mediawiki/images/ | wc -l
40356
Here is a list of the directories ONLY. You'll note that the [a-f][0-9]
subdirs are missing:
address@hidden:~# ls -l /var/lib/mediawiki/images/ | egrep '^d'
drwxr-xr-x 18 root root 4096 Jul 11 00:03 bad-images
drwxr-xr-x 18 root root 4096 Aug 11 18:11 math
drwxrwxrwx 2 root root 4096 Jul 12 19:08 temp
drwxrwxrwx 2 root root 4096 Jul 12 19:08 thumb
drwxrwxrwx 2 root root 4096 Aug 12 13:50 tmp
drwxr-xr-x 2 root root 36864 Aug 12 13:50 wp-mirror
address@hidden:~# du --human --total /var/lib/mediawiki/images/[0-9a-f] | tail
-n 1
du: cannot access `/var/lib/mediawiki/images/[0-9a-f]': No such file or
directory
0 total
address@hidden:~# du --human --total /var/lib/mediawiki/images/thumb | tail -n
1
4.0K total
address@hidden:~# du --human --total /var/lib/mediawiki/images/math | tail -n
1
18M total
2) work files
address@hidden:~# du --human --total /var/lib/mediawiki/images/wp-mirror/ |
tail -n 1
1.1G total
address@hidden:~# ls
/var/lib/mediawiki/images/wp-mirror/simplewiki-latest-md5sums.txt* | wc -l
2
address@hidden:~# ls
/var/lib/mediawiki/images/wp-mirror/simplewiki-20120805-pages-articles* | wc
-l
315
address@hidden:~# ls /var/lib/mediawiki/images/wp-mirror/[0-9a-f]
ls: cannot access /var/lib/mediawiki/images/wp-mirror/[0-9a-f]: No such file
or directory
#!/bin/sh
IMAGE=
OUTPUT=
IMAGEPATH=http://upload.wikimedia.org/wikipedia/nil/
COMMONSPATH=http://upload.wikimedia.org/wikipedia/commons/
/bin/mkdir -p $OUTPUT./thumb
/bin/chmod 777 $OUTPUT./thumb
/bin/mkdir -p $OUTPUT./temp
/bin/chmod 777 $OUTPUT./temp
/bin/mkdir -p $OUTPUT./tmp
/bin/chmod 777 $OUTPUT./tmp
if [ -a $IMAGE./0/0f/Dibuix_de_Leo.png ]; then
echo 0/0f/Dibuix_de_Leo.png already exists >> exists.log
else
curl --retry 0 -f -O $COMMONSPATH./0/0f/Dibuix_de_Leo.png
if [ -a $IMAGE./Dibuix_de_Leo.png ]; then
/bin/mkdir -p $OUTPUT./0/0f/
/bin/mv ./Dibuix_de_Leo.png $OUTPUT./0/0f/
echo ./0/0f/Dibuix_de_Leo.png downloaded >> download.log
else
curl --retry 0 -f -O $IMAGEPATH./0/0f/Dibuix_de_Leo.png
if [ -a $IMAGE./Dibuix_de_Leo.png ]; then
/bin/mkdir -p $OUTPUT./0/0f/
/bin/mv ./Dibuix_de_Leo.png $OUTPUT./0/0f/
echo ./0/0f/Dibuix_de_Leo.png downloaded >>
download.log
else
echo ./0/0f/Dibuix_de_Leo.png failed >> failed.log
fi
fi
fi
3) MediaWiki
To that end, please open a web browser to:
<http://simple.mediawiki.site/index.php/Main_Page>
Worked but with issues. I saw messages such as:
a) Error: image is invalid or non-existent
b) {{:Main Page/Article Division by zero}}
c) File:Gnome-applications-science.svg
Please let me know if you see images on this page:
<http://simple.mediawiki.site/index.php/Dinosaur>
The page loaded but without images.
Please let me know if you see nicely formatted math symbols on this page:
<http://simple.mediawiki.site/index.php/Pythagorean_theorem>
The math symbols appear and look great.
Thank you for your continued assistance. I really appreciate it.
Best,
-Ben
-----Original Message-----
From: wp mirror [mailto:address@hidden
Sent: Wed 8/15/2012 5:21 AM
To: Benjamin Goldsmith
Cc: address@hidden
Subject: Attempting to mirror on laptop
Dear Benjamin,
0) Preamble
Thank you for providing output from `wp-mirror --text'. When
WP-MIRROR runs in monitor mode (--text, --screen, or --gui) it
collects state information by forking shell processes that run MySQL
as well as certain BASH commands. This suggests that MySQL problems
have been solved, and that file system permissions are in order.
Thank you for the chown error messages. It appears that WP-MIRROR is
not locating the image files that were downloaded when the i-chunks
were processed. You should have over 50,000 images occupying about
40G. So where are the image files?
1) images
Downloaded image files should be stored in a directory tree under
/var/lib/mediawiki/images/[0-9a-f]/. Resized image files should be
stored under /var/lib/mediawiki/images/thumb/. Mathematical
expressions should have been converted to PNG image files and stored
under /var/lib/mediawiki/images/math/.
To that end, please let me see your output for the following commands:
root-shell> ls -l /var/lib/mediawiki/images/
total 6235
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 0
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 1
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 2
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 3
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 4
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 5
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 6
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 7
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 8
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 9
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 a
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 b
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 bad-images
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 c
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 d
-rw-r--r-- 1 root root 1507005 Aug 12 03:09 download.log
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 e
-rw-r--r-- 1 root root 278194 Aug 12 03:06 exists.log
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 f
-rw-r--r-- 1 root root 4527367 Aug 12 03:10 failed.log
drwxr-xr-x 18 www-data www-data 432 Nov 27 2011 math
drwxrwxrwx 2 www-data www-data 48 Nov 27 2011 temp
drwxrwxrwx 18 www-data www-data 432 Nov 29 2011 thumb
drwxrwxrwx 2 www-data www-data 48 Aug 12 09:54 tmp
drwxr-xr-x 2 www-data www-data 62936 Aug 14 23:37 wp-mirror
root-shell> du --human --total /var/lib/mediawiki/images/[0-9a-f] | tail -n
1
45G total
root-shell> du --human --total /var/lib/mediawiki/images/thumb | tail -n 1
1.1G total
root-shell> du --human --total /var/lib/mediawiki/images/math | tail -n 1
16M total
2) work files
WP-MIRROR stores its work files (dump files, x-chunks, i-chunks, etc.)
under the directory /var/lib/mediawiki/images/wp-mirror/. I would
like to see if files are there, and if the i-chunks are properly
formed.
To that end, please let me see your output for the following commands.
root-shell> du --human --total /var/lib/mediawiki/images/wp-mirror/ | tail
-n 1
1.5G total
root-shell> ls
/var/lib/mediawiki/images/wp-mirror/simplewiki-latest-md5sums.txt*
| wc -l
2
root-shell> ls
/var/lib/mediawiki/images/wp-mirror/simplewiki-20120805-pages-articles*
| wc -l
315
root-shell> ls /var/lib/mediawiki/images/wp-mirror/[0-9a-f]
ls: cannot access /var/lib/mediawiki/images/wp-mirror/[0-9a-f]: No
such file or directory
root-shell> cat
/var/lib/mediawiki/images/wp-mirror/simplewiki-20120805-pages-articles-p000
000000-c000001000.sh
| head -n 33
#!/bin/sh
IMAGE=
OUTPUT=
IMAGEPATH=http://upload.wikimedia.org/wikipedia/nil/
COMMONSPATH=http://upload.wikimedia.org/wikipedia/commons/
/bin/mkdir -p $OUTPUT./thumb
/bin/chmod 777 $OUTPUT./thumb
/bin/mkdir -p $OUTPUT./temp
/bin/chmod 777 $OUTPUT./temp
/bin/mkdir -p $OUTPUT./tmp
/bin/chmod 777 $OUTPUT./tmp
if [ -a $IMAGE./c/c7/PB050006.JPG ]; then
echo c/c7/PB050006.JPG already exists >> exists.log
else
curl --retry 0 -f -O $COMMONSPATH./c/c7/PB050006.JPG
if [ -a $IMAGE./PB050006.JPG ]; then
/bin/mkdir -p $OUTPUT./c/c7/
/bin/mv ./PB050006.JPG $OUTPUT./c/c7/
echo ./c/c7/PB050006.JPG downloaded >> download.log
else
curl --retry 0 -f -O $IMAGEPATH./c/c7/PB050006.JPG
if [ -a $IMAGE./PB050006.JPG ]; then
/bin/mkdir -p $OUTPUT./c/c7/
/bin/mv ./PB050006.JPG $OUTPUT./c/c7/
echo ./c/c7/PB050006.JPG downloaded >> download.log
else
echo ./c/c7/PB050006.JPG failed >> failed.log
fi
fi
fi
3) MediaWiki
Please let me know if you can browse <simple.mediawiki.site>.
To that end, please open a web browser to:
<http://simple.mediawiki.site/index.php/Main_Page>
Please let me know if you see images on this page:
<http://simple.mediawiki.site/index.php/Dinosaur>
Please let me know if you see nicely formatted math symbols on this page:
<http://simple.mediawiki.site/index.php/Pythagorean_theorem>
Sincerely Yours,
Kent
- [Wp-mirror-list] Attempting to mirror on laptop, wp mirror, 2012/08/05
- [Wp-mirror-list] Attempting to mirror on laptop, wp mirror, 2012/08/07
- [Wp-mirror-list] Attempting to mirror on laptop, Dr. Kent L. Miller, 2012/08/07
- [Wp-mirror-list] Attempting to mirror on laptop, wp mirror, 2012/08/08
- [Wp-mirror-list] Attempting to mirror on laptop, wp mirror, 2012/08/09
- [Wp-mirror-list] Attempting to mirror on laptop, Dr. Kent L. Miller, 2012/08/13
- [Wp-mirror-list] Attempting to mirror on laptop, wp mirror, 2012/08/15
- [Wp-mirror-list] Attempting to mirror on laptop,
Dr. Kent L. Miller <=
- [Wp-mirror-list] Attempting to mirror on laptop, wp mirror, 2012/08/17
- [Wp-mirror-list] Attempting to mirror on laptop, Dr. Kent L. Miller, 2012/08/19
- [Wp-mirror-list] Attempting to mirror on laptop, wp mirror, 2012/08/19
- [Wp-mirror-list] Attempting to mirror on laptop, wp mirror, 2012/08/20