[Wp-mirror-list] en.wikipedia image sizes and templates

[wp mirror]

Dear Jason,

Thank you very much for your comments regarding the installed wikipedia.

1) Math

This surprised me a bit, because the Math extension works fine on my
platform, and I thought the three config options were default values
anyway.

Even so, I have added the three config options to
`/etc/mediawiki/LocalSettings_wpmirror.php' as you recommended, and
posted the new version of wp-mirror-0.6.

2) Templates

The gradual erosion of page rendering has been a growing problem all
year.  Compare

<http://simple.wikipedia.org/wiki/John_Adams>
<http://simple.wikipedia.site/index.php/John_Adams>

2.1) Interlanguage links.  These all vanished early in 2013.  These
links vanished not only from the navbar displayed on the left side of
each page, but also from the underlying database.  This is due to the
`wikidata' project, which centralized all interlanguage links into one
wiki, and then purged said links from the dump files of all other
wikis.  Displaying wikidata content now requires a set of extensions
that only works with mediawiki 1.21+.  Debian still uses mediawiki
1.19, which is the long term support (LTS) version from WMF.

2.2) Infoboxes.  These have vanished.  The information is certainly
there in the underlying database.  This can be seen by executing the
command:

mysql> select * from
simplewiki.page,simplewiki.revision,simplewiki.text where
rev_text_id=old_id and rev_page=page_id and page_title like
'John_Adams'\G

Rendering an infobox, however, now requires the Scribunto extension
which only works with mediawiki 1.20+.

2.3) Transclusions.  Many, if not most, templates now result in red
links.  A red link is supposed to mean that mediawiki can not find the
page or template in the underlying database.  Yet the templates are
certainly there in the underlying database. For example
`Template:USPresidents' appears as a red link, yet it can be seen by
executing the command:

mysql> select * from
simplewiki.page,simplewiki.revision,simplewiki.text where
rev_text_id=old_id and rev_page=page_id and page_title like
'Template:USPresidents'\G

3) Thumbs

Presently, the full-size image dumps do not include any thumbs.  This
can be seen by executing the command:

(shell) rsync rsync://ftpmirror.your.org/wikimedia-images/wikipedia/simple/

The thumb dumps for XOWA store thumbs in a four-level directory tree

.../thumb/a/b/c/d/<image file name>/<thumb file name>

rather than the two-level default for mediawiki

../thumb/a/b/<image file name>/<thumb file name>

Provision of thumb image dumps is something that I do want to follow
up with the folks at WMF.

4) Mediawiki version lifecycle

I think I am confronted by an `impedance mismatch' between the
development policies of WMF and of Debian.  According to
<http://www.mediawiki.org/wiki/Version_lifecycle>:

`MediaWiki operates on a "continuous integration" development model,
where software changes are pushed live to Wikimedia web sites such as
Wikipedia on a regular basis.'

However, LTS versions are released every two years, the next of which,
mediawiki 1.23 (LTS), is slated for May 2014.  Thereafter, several
months could elapse before this version finds its way into a Debian
distribution.

5) Debian version lifecycle

Debian prefers LTS versions such as mediawiki 1.19 and mysql 5.5.

6) Defining the problem

When wp-mirror is used to build a mirror, page rendering erodes over
time.  This is because WMF follows a `continuous integration' policy
while wp-mirror instead relies on Debian to satisfy dependencies.
Poor page rendering discourages use of wp-mirror as a utility for
building mirrors, and calls into question the suitability of
distributing such a utility via Debian.

7) Planning for wp-mirror 0.7

I think I need to address the `impedance mismatch' in development cycles.

One option would be to remove mediawiki 1.19 and related extensions as
dependencies in the wp-mirror DEB package; and instead include the
latest version of mediawiki and relevant extensions, as part of the
wp-mirror DEB package.  This would have some consequences.

Size. The wp-mirror 0.7 DEB package would become much larger.  Current
file sizes are:

wp-mirror_0.6-1_all.deb - 3.7M (most of which is the Reference Manual)
mediawiki-1.22.0.tar.gz - 20M

Updates.  There would be no updates from Debian.

Release cycle. Instead of reacting to upgrades at Debian, I would have
to react to upgrades at WMF.  There are a couple of ways of doing
that.

7.1) Manual updates.  One possibility is that wp-mirror could be
updated whenever a release at WMF caused a significant erosion of page
rendering.

7.2) Automatic updates.  Alternatively, wp-mirror could be designed to
look for the latest mediawiki tarball, and perform an automatic update
whenever a more recent version was detected.  I suspect, however, that
this later approach would be brittle.  For example, for mediawiki
1.18, all the math functionality was ejected from the mediawiki core
and relegated to an extension.  Conversely, there are cases when the
functionality of an extension has been merged into the core.
Detecting such cases would be chore to program.

Comments are welcome as always.  I shall start work on wp-mirror 0.7
soon, and shall keep you posted.

Best wishes for the holidays.

Sincerely Yours,
Kent