[Yum] Yum and Bittorrent

[Jarret R]

Konstantin Ryabitsev wrote:
> Well, there is another issue which your method does not address. If you 
> only torrent large files (say, over 50 Mb in size), then the vast 
> majority of packages is still downloaded from the server. What bogs down 
> large mirrors is not bandwidth -- we have it out the proverbial wazoo. 
> What bogs us down is processor and io load. If most requests are still 
> made to the servers (plus there is additional tracker load), does that 
> actually improve the situation?

This is part of what we want to test. We are assuming that the BT 
overhead is to high for smaller files because of the design decisions, 
but we can't quantify it. A .torrent file will almost always be smaller 
than the file that it defines. If we can reduce the upload burden of a 
server from 1 copy of the file for every downloader to the torrent file 
+ maybe 5 copies for every 100 peers, I think it will have a serious impact.

Like I've said. We are planning this public beta to flesh out these 
issues and see what kind of an impact we can have.

> I still think we're chasing the wrong goose trying to unite yum and 
> bittorrent, and it is better to have a bittorrent-like protocol for 
> syncing mirrors and a special download method for urlgrabber that would 
> grab byte-ranges from multiple servers. This would allow the mirror load 
> to drop significantly, thus improving the situation when half the world 
> sends us HTTP requests.

I assume that most mirrors use rsync as it is, so I am not sure how 
useful switching to BT would be, but it is something we will look into. 
Another point to remember is that the server doesn't have to be the 
tracker or the only initial seed.

A tracker requires little bandwidth and could be run by a smaller 
mirror, like my university (Lehigh University). We don't have the 
bandwitdh to run a full mirror, but we could run a tracker. Then as many 
repositories as we can get can start the inital seeding, they already 
have the file. When a new peer enters the swarm, it will contact all the 
repsoitories and request pieces of the file, which would reduce overall 
load on the server dramatically. This scenario assumes a lot, but I 
think if the system is viable, we will start to see things like this.

> But I would like to see and play with your work. There are some 
> interesting points there.

Thanks. I will post here when we are ready to open the beta to the 
public. Also, you mentioned that you run a mirror. One thing that would 
help us is to get our hands on a download log for a real mirror. This 
would allow us to make a more educated guess on which files should be 
served via BT instead of just using size as the deciding factor.

PS - There are several new features coming down the pike for BitTorrent 
that could significantly impact this system.

(1) Multi-Tracker specification - this would allow us to specify more 
than one tracker for each torrent. If the first went down, the 
downloader would just switch to the next one.

(2) Peer Sharing - Two trackers running the same torrent can share 
peers. This will allow larger, longer lasting swarms.

(3) Torrent File Sharing - Instead of a BT Instance for every file being 
shared via BT, we start one up for all the files we want to share via 
bt. Then the downloader only requests pieces from the files that it 
wants. This should reduce the load on the server and just maintain one 
swarm. We are looking into this one, but we don't have anything yet.

Thanks,
Jarret Raim