[Yum-devel] [UG] XML file for mirrors, checksums (Metalink)

[Anthony Bryan]

Hello,

>From the feature list of urlgrabber, I thought maybe someone would be
interested in Metalink. These features in particular:

# byte ranges - fetch only a portion of the file
# reget - for a urlgrab, resume a partial download
# batched downloads using threads - download multiple files simultaneously
(feature still in progress)
# retries - automatically retry a download if it fails. The number of
retries and failure types are configurable
# mirror groups - treat a list of mirrors as a single source, automatically
switching mirrors if there is a failure

Metalink lists mirrors, P2P, checksums, and other metadata in an XML format
for programs that download. (There are five programs for making these
Metalinks). There are also eight clients for Mac, Unix, and Windows. The
clients can use the multiple URLs for higher reliability (in case a server
goes offline, the transfer automatically uses another mirror) or
segmented/multi-threaded downloads which are much faster. There can be full
file, and segment/partial file checksums listed. The segment checksums allow
segments with errors to be re-transmitted, instead of the whole file. This
makes it useful for distribution of large files like ISO disk images.
Downloads can be filtered by location, OS, or language, if that information
is provided.

>From using yum, it seems like it can already switch mirrors, but it doesn't
do segmented downloads. You already have the byte range fetching, I guess
you are avoiding segmented downloads.

Metalink is used by OpenOffice.org, openSUSE, around ten Linux and BSD
distributions, and other open source and commercial organizations that use
mirrors and/or P2P. Here's what the format looks like:

<?xml version="1.0" encoding="UTF-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/">
  
<files>  
   <file name="FC-6-i386-DVD.iso">
   <os>Linux-x86</os>
   <size>3525195776</size>
   <verification>
    <hash type="md5">89d337f5543474532027950b44ecbdd5</hash>
   </verification>
   <resources>
      
      <url type="http"
           location="gb"
           preference="10">
 
http://zeniiia.linux.org.uk/pub/distributions/fedora/linux/core/6/i386/iso/F
C-6-i386-DVD.iso
      </url>
      <url type="ftp"
           location="ua"
           preference="10">
 
ftp://fedora.org.ua/pub/linux/fedora/core/6/i386/iso/FC-6-i386-DVD.iso
      </url>
      <url type="http"
           location="tr"
           preference="10">
 
http://mirrors.cs.hacettepe.edu.tr/fedora/linux/core/6/i386/iso/FC-6-i386-DV
D.iso
      </url> 
 
   </resources>
   </file>
</files> 
</metalink>

There's is more information available at

http://en.wikipedia.org/wiki/Metalink
http://www.linux.com/article.pl?sid=06/11/01/1641247
http://en.opensuse.org/Released_Version#Metalink

(( Anthony Bryan
 )) Metalink [ http://www.metalinker.org ]