[Yum] Re: yum mirroring

Robert G. Brown rgb at phy.duke.edu
Fri Aug 8 10:54:14 UTC 2003


On 8 Aug 2003, Peter Peltonen wrote:

> --<snip>--
> i386/XFree86-Mach64-3.3.6-29.i386.rpm is uptodate
> i386/XFree86-Mono-3.3.6-29.i386.rpm is uptodate
> i386/XFree86-75dpi-fonts-3.3.6-29.i386.rpm
> i386/XFree86-8514-3.3.6-29.i386.rpm
> i386/XFree86-S3V-3.3.6-29.i386.rpm is uptodate
> i386/XFree86-AGX-3.3.6-29.i386.rpm
> i386/XFree86-FBDev-3.3.6-29.i386.rpm
> i386/XFree86-Mach8-3.3.6-29.i386.rpm
> i386/XFree86-VGA16-3.3.6-29.i386.rpm is uptodate
> --</snip>--
> 
> How come i386/XFree86-AGX-3.3.6-29.i386.rpm is not uptodate? This seems
> to happen always with the same files.

I'm clueless, but here are some debugging questions in case you haven't
already looked or tried them.

Is there anything odd about the stat data of these files?  Do the
timestamps, ownerships, and perms all coincide with those on the server
(as far as you can tell) and are they the same before and after?  What
happens if you e.g.

  mv i386/XFree86-AGX-3.3.6-29.i386.rpm /tmp

(wait overnight for the cron script to run)

  ls -al i386/XFree86-AGX-3.3.6-29.i386.rpm /tmp/XFree86-AGX-3.3.6-29.i386.rpm  
  diff i386/XFree86-AGX-3.3.6-29.i386.rpm /tmp/XFree86-AGX-3.3.6-29.i386.rpm

looking for timestamp changes and/or diffs.  If the first night reveals
nothing I'd add -c to your transfer arguments for that night as well --
a checksum comparison should give you a separate/direct observation of
what is "different", maybe.

I suggest this because one possible explanation is server side stuff
beyond your control -- something that touches these files on the server
and continually alters e.g. their timestamp or makes some sort of change
that rsync interprets as the files not being identical, so it updates
them.  It is quite fussy -- I don't know for sure what it's criterion
for update is, but it seems to involve most of the stat data except e.g.
access times.

This isn't impossibly implausible.  Perhaps the server side people are
using a nightly cron script to synchronize their repository with
"something", maybe a mounted CD, maybe somebody else's site, and are
using plain old rcp or scp or cp without a -p so that the timestamp
changes every night for certain files.  

> If I run the cronjob manually the behaviour does not occur.

Note that a server site bug would probably exhibit this behavior.  Their
update (that re-timestamps these files) happens at 2 am.  Your cron
script runs at 3 am, so it updates the files as their timestamp has
changed and it doesn't use a full checksum to determine "identity".  The
next day, perplexed, you run the update by hand but of course nothing
has changed since their 2 am update and your 3 am update, so the files
aren't updated.

This too can be tested, both by the ritual above and by simply skipping
the nightly cron altogether and running the update by hand the next day.
If the hypothesis is approximately correct, the FIRST time you run the
update script after their nightly cron you'll get an unnecessary update,
but of course the second and so forth you won't.

> Maybe someone here can figure out what is wrong in my scripts and
> explain me what is happening?

Well, the above is ONE possible situation.  I'm not sure that we've
reached the Holmesian situation where all that remains, however
implausible, must be the truth but I have a fair bit of faith in rsync
doing the right thing and suspect that it must be something like the
above.  It is also extremely odd that it works "by hand" and fails "by
cron", where they only possible difference I can think of between them
might be inherited environment (but shouldn't be as you use su -).

The last suggestion is to try playing with the su - ayo part of your
script.  Either let the nightly cron run as root (including the rsync)
or set up a cron for the user ayo to run the mirror.  The reason I
suggest this is that you're defining a bunch of variables in the update
script and then su'ing to a new login environment where those variables
may not be inherited.  Also, if I'm not mistaken the su -c line is a bit
ambiguous -- shouldn't the "'s around SRPMS be escaped, as they are
inside the "" pair for the -c line?  You might check to be sure this
string is expanding the way you think it is.

   rgb

> 
> My /etc/cron.daily/mirroring script:
> --<snip>--
> #!/bin/bash
> MIRRORDIR=/scratch/yum/scripts
> rm -f $MIRRORDIR/updates.log
> echo "Beginning mirrorin at `date`. " > $MIRRORDIR/updates.log 2>&1
> $MIRRORDIR/get-updrh62.sh >> $MIRRORDIR/updates.log 2>&1
> $MIRRORDIR/get-updrh72.sh >> $MIRRORDIR/updates.log 2>&1
> $MIRRORDIR/get-updrh73.sh >> $MIRRORDIR/updates.log 2>&1
> touch /scratch/yum/updates-fetched.txt
> --</snip>--
> 
> 
> And for example the get-updrh73.sh looks like this:
> 
> --<snip>--
> #!/bin/bash
> MIRRORDIR=/scratch/mirror
> VERBOSE=1
> MIRROR=ftp.funet.fi
> REMOTE="$MIRROR::ftp/pub/mirrors/ftp.redhat.com/pub/redhat/linux/updates/7.3/en/os/"
> LOCAL="$MIRRORDIR/redhat/linux/updates/7.3/en/os/"
> if [ ! -z $VERBOSE ]; then
>         VRB="-vv --stats"
> fi
> echo "Fetching RH73 updates."
> su - ayo -c "rsync $VRB -a --partial --exclude "SRPMS" --delete \
> $REMOTE $LOCAL"
> echo " "
> --</snip>--
> 
> Regards,
> 

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu






More information about the Yum mailing list