Re: [Savannah-hackers-public] Changes to the CVS repo don't propagate to the website

[Bob Proulx]

I think thinkgs are working as they were before again.

Bob Proulx wrote:
> > Is this related to migration to the new VM?
> 
> Almost certainly it must be related.  But when I test uploading to a
> test project I do see the web site updated with the changes.

Maybe yes and maybe no.  The stuck process timestamps from about the
time of the move.  But the problem was entirely on www.gnu.org and it
is a problem that has happened before.  So maybe no.

> The way this works is a little bit of a "Rube Goldberg" device.  CVS
> updates trigger the loginfo hook processing which is simply a curl hit
> of a URL on the web server machine.  A python script on the web server
> then goes and updates the web site copy of the project.

Actually the python script only touches a file in a directory
indicating that a project was updated.  Then a cron script uses flock
on /tmp/update-cvs.lock to semaphore running update-cvs.sh every two
minutes to do the update.  It's update-cvs.sh that was stuck.

> Can you point me to a specific repository that is seeing this
> behavior?

With help from bandali on IRC I started looking at this page which was
not updated.

  https://www.gnu.org/software/recent-releases-include.html

Yet the repository clearly had an update.

  
http://web.cvs.savannah.gnu.org/viewvc/www/www/software/recent-releases-include.html?r1=1.973&r2=1.974

So then I started poking around on www.gnu.org to figure out where
things were stuck.  I found this process tree.

  wwwcvs   29893 29889  0 Jun20 ?        00:00:00       /usr/bin/flock -w2 -x 
/tmp/update-cvs.lock /usr/local/bin/update-cvs.sh
  wwwcvs   29896 29893  0 Jun20 ?        00:00:03         /bin/bash 
/usr/local/bin/update-cvs.sh
  wwwcvs   26857 29896  0 Jun20 ?        00:00:00           /usr/bin/cvs -d 
:pserver:address@hidden:/webcvs/pdsplibrary checkout -P pdsplibrary

The last log line from it:

  Thu Jun 20 16:15:09 EDT 2019
  pdsplibrary: Beginning checkout (type non-gnu)
  cvs2 checkout: Updating pdsplibrary
  ...
  cvs2 checkout: Updating pdsplibrary/libpdssn/refman/html

It is Fri, 21 Jun 2019 18:21:55 -0600 now.  So that one checkout has
been running for 24+ hours?  That seems to have been the problem as to
why web pages are not updating.

And this problem has happened before.  There is another cronjob that
runs once a day and sends an email to sysadmin opening a ticket saying
the following when this happens.  I remember Lisa setting that up for
me on a previous occasion when this had happened before.

  /usr/local/bin/update-cvs.sh is stuck on gnu.org

This is all on www.gnu.org which was not part of the migration.  I
killed process 26857 above since that was stuck.  That allowed the
update-cvs.sh to move on.  Exiting in this case.  Then the */2 cron to
run it started again.  This run it walked through the queued work.  It
is still running.  But I think it will be able to do what it does now.
Looking at this page I see it is updated now.

  https://www.gnu.org/software/recent-releases-include.html

I am going to keep an eye on it until it is through running.  It
appears to be making progress through the queued checkout work.  I am
surprised by number of directories with work queued.  Literally
thousands because it was still over a thousand and reducing before I
thought to look to see how many.

Bob