Re: [Wp-mirror-list] [PATCH RFC] hdparm: tolerate being in a Virtual Machine

[wp mirror]

Dear Jason,

1) Check-pointing

For several reasons, wp-mirror uses check-pointing:  1) Robustness.
Loading large dump files, such as those found in the `enwiki,' takes
days and almost always fails.  Therefore, wp-mirror breaks dump files
into smaller pieces (called `xchunk's, `schunk's, and `ichunk's).  The
failure of a few chunks to load is not tragic because the vast
majority load successfully. 2) Resume.  Mirror building can take days
or even weeks.  Check-pointing allows the user to shutdown and resume
later.  This is necessary for laptop application.  Wp-mirror resumes
from the last known check-point.  3) Concurrency.  Wp-mirror stores
its state information in an InnoDB table `wpmirror.file'.  Since each
database transaction is Isolated (the `I' in `ACID') from every other,
multiple instances of wp-mirror can run without stepping on one
another.

2) State information

Mirror building involves a great number of tasks.  As mentioned above,
wp-mirror tracks the state of each task in an ACID compliant database
table.  You can see this by executing:

(shell)$ mysql --host=localhost --user=root --password
Enter password: **********
...
mysql> select name,type,state from wpmirror.file where state<>'done' limit 10;
+------------------------------------------------+--------+-------+
| name                                           | type   | state |
+------------------------------------------------+--------+-------+
| enwiki-20131202-local-media                    | ichunk | start |
| enwiki-20131202-page-p009350000-c000010000.sql | schunk | valid |
| enwiki-20131202-page-p009360000-c000010000.sql | schunk | start |
| enwiki-20131202-page-p009370000-c000010000.sql | schunk | start |
| enwiki-20131202-page-p009380000-c000010000.sql | schunk | start |
| enwiki-20131202-page-p009390000-c000010000.sql | schunk | start |
| enwiki-20131202-page-p009400000-c000010000.sql | schunk | start |
| enwiki-20131202-page-p009410000-c000010000.sql | schunk | start |
| enwiki-20131202-page-p009420000-c000010000.sql | schunk | start |
| enwiki-20131202-page-p009430000-c000010000.sql | schunk | start |
+------------------------------------------------+--------+-------+
10 rows in set (0.01 sec)

3) Scheduling

The order in which tasks are executed is determined by a Finite State
Machine (see WP-MIRROR 0.6 Reference Manual, 5.1.5 The Finite State
Machine) which is most easily seen by looking at:

Figures 5.1 FSM State Transition Diagram
Table 5.2 FSM Tables *state-transition* and *type-state-function*
Table 5.3 FSM Priority Table *type-state-priority*

4) Upgrade

You mentioned that you are in the midst of building a mirror of `en'.
This is not a problem:  1) Kill wp-mirror and any processes that it
has spawned; 2) update wp-mirror; and 3) restart wp-mirror.

If you ran wp-mirror from the command line, just use Control-C.
Probably wp-mirror was waiting for a spawned process (e.g. wget,
bunzip2, gunzip, mysql, ...) to complete, so you should kill that one
too.

If wp-mirror was started as a `cron' job, then you will have to first
find the PID of the process by executing:

(root-shell)# ps -wef | grep wp-mirror
root     10365 30637  0 12:23 pts/8    00:00:25
/usr/lib/clisp-2.49/base/lisp.run -B /usr/lib/clisp-2.49 -M
/usr/lib/clisp-2.49/base/lispinit.mem -N /usr/share/locale -ansi -q -q
/usr/bin/wp-mirror --mirror
(root-shell)# kill -15 10365

Again, you may have to kill one or two spawned processes.  For example:

(root-shell)# ps -wef | grep mysql
root     25099 10365  0 16:56 pts/8    00:00:00 /bin/sh -c
/usr/bin/mysql --host=localhost --user=wikiadmin
--password=xxxxxxxxxxxx --database=enwiki  <
'enwiki-20131202-page-p009150000-c000010000.sql'
root     25100 25099  0 16:56 pts/8    00:00:00 /usr/bin/mysql
--host=localhost --user=wikiadmin --password=x xxxxxxxxxxx
--database=enwiki
root     25105  8560  0 16:57 pts/27   00:00:00 grep mysql
(root-shell)# kill -15 25099 25100

Next upgrade wp-mirror and finally restart.

Sincerely Yours,
Kent