[Yum-devel] API Changes: Was: plugings modifying the URLGrabber instance

Sat Jul 2 23:52:48 UTC 2005

On Sat, Jul 02, 2005 at 03:18:45AM -0400, seth vidal wrote:
> On Sat, 2005-07-02 at 15:49 +1000, Menno Smits wrote:
> > seth vidal wrote:
> > 
> > >>Okay, just note that I'm sort of poking at this from two directions.
> > >>Making Yum speak RHN, and adding features to Yum so that RHN/Current
> > >>could be modified to work with Yum.
> > > 
> > > 
> > > So one thing I'm not interested in - I'm not interested in yum emulating
> > > or talking over xml-rpc just like up2date/rhnsd does.
> > > 
> > > I wouldn't mind learning from problems they may have encountered but
> > > making yum compatible with rhn's xml-rpc command stack is NOT terribly
> > > interesting to me.
> > > I can see how it might be valuable to be able to, eventually, but
> > > targeting that first is a bad idea, imo.
> > 
> > I agree. We should aim for incremental refactoring steps that result in 
> > a sufficiently clean abstraction of repositories and packages so that 
> > talking to stuff like RHN is possible. I can envisage other repository 
> > types as well.

I agree here as well for the most part.  Any other management issues
(example: registration, client profiles, actions, etc.) should be done
completely outside of Yum.  Yum serves only as an RPM agent.

However, there are parts of Yum that need improvement or additional
features to make Yum integrate better in an advanced management system
that I hope I'm not the only one that sees the future bringing the Linux
community.

> 
> I can think of a bunch of repository types that I'm not interested in
> supporting, tbh. not interested in:
>  red carpet's format
>  apt's format
>  urpmi's format
>  yum 2.0.x headers/ format
>  
>  to be perfectly honest I'm not interested in RHN's format, I'm only
> interested in making it more possible for other formats to be added by
> other people. But I have ZERO inclination to maintain 57 different
> format types just for fun. 
> 

Then don't.  I really, honestly, can't blame you for not wanting to
maintain everyones pipe dream for Yum.  Instead, lets do support for
multiple repository types /right/.  Have specific classes that you must
derive from and specific APIs you must implement.  After we get that
done it wont be hard to allow plugins to implement repository types.

I have interest in RHN style repos.  I'll maintain mine.

> 
> > To be clear, we are aiming for Yum to be able to support different 
> > repository types simultaenously right?
> 
> yes. Though, again, I don't see a lot of a point to support formats
> beyond the standard one created via createrepo. Considering how trivial
> it is to make repomd repositories my sympathy for those folks who feel
> they can't is pretty low.
> But if we make it possible for other formats to be supported on the way
> to cleaning up the repo/packageSack code in yum, that's fine.
> 
> 
> Maybe someone should explain it to me but I don't really see much virtue
> in supporting other repository formats. I've heard the 'compatibility'
> argument but  if it cost something to make the createrepo format I'd
> understand but it costs users nothing. The sole case I've heard so far
> is Jack's argument: 
> "I have an rhn satellite server and its my only mechanism for accessing
> updates, how can I use it". That situation makes _some_ sense to me.
> Supporting current doesn't really, though. If you're using current then
> you already have all the packages in some directory on some server you
> control anyway. Why would you bother setting up the extra complexity of
> it when you could just setup a webserver and be done with it? What does
> it buy yum to have this code in the tree? I guess that's my basic
> question.

Fedora, and the Linux community in general need a modern, advanced
management system.  The Red Hat Network is the only thing out there that
I know of.  It's proprietary.  I think we all understand the
disadvantages to that.

So what do we use?  Cron jobs and cfengine?  Seth, you're always the one
talking about moving into the 21st century.  :-)

These management systems require machines to authenticate rather than
just pull information anonymously, as Yum supports now.  Yum does not
have any support to present authentication to its server.  This is where
I was going with some earlier remarks about having Yum include an
arbitrary HTTP header per repository in ALL HTTP GET requests.  How you
get your session ID, login to the server, what commands Yum should do
are examples of things that are outside the scope of Yum and should not
be supported by Yum.  Yum is just the RPM agent.

As far as Current goes, I'd like Current to support the Repo Metadata
format because it really makes a lot of since.  However, it just doesn't
work unless Current can authenticate Yum's GET requests.  Not just a GET
request for a config file, but every GET request.  Current is an attempt
at this so called advanced management system.  I'd like to use Yum as my
RPM agent.

Now, away from my fantasies and back to reality.  I and many folks
deploy RHEL where Yum doesn't work.  That's a big need.  Up2date is
quite behind Yum in features, and simply not as good.  However, Red Hat
has been interested in supporting Yum in RHEL environments for years.
Not only have I been encouraged from Red Hat folks to work on Yum
supporting RHN repositories, but I actually have perl code on my RHN
Satellite that is the beginnings of old-style Yum support in the
Satellite.

In summary, for RHEL alone there are many reasons that Yum needs
additional functionality.  I would very much like to replace my
dependence on up2date with Yum.  The obvious approach is support for
multiple repository types.
> 
> 
> > I think different package sacks are still required but Jack's right in 
> > that we do need some way for Yum to be able to use multiple packagesack 
> > types at the same time. At the moment Yum either uses the the repomd or 
> > the sqlite package sack. Some sort of "MetaSack" class is required to 
> > hold different packagesacks and aggregrate operations on them.
> 
> The benefit of the single representation of packages is just searching.
> Now if we wanted to do this an 'easier' way it would be to have the
> metasack really just perform the same routines on the rest of the
> packageSacks. One packageSack for each repository.
> 
> so MetaSack.searchProvides() would really do:
>    for repo in repos:
>       repo.sack.searchProvides()
> 

I like the idea of the MetaSack.  Its very clear why Yum needs all the
packages to wind up in a SuperSack.  However, I see this SuperSack as
yet another implementation of PackageSack.  It should support the same
API.

>    then return those results into a ListPackageSack or into whatever is
> the expected return format.
> 
> > By "shared methods" I assume you mean "shared implementation". Shared 
> > implementation isn't as important here as shared interface. As long as 
> > PackageObjects look and act the same we can mix and match them 
> > interchangably.
> 
> by shared methods I meant more like a base abstract class with methods
> that all of the derived classes would have.
> 

In my experience a base abstract class has very little code in it.
OO style programming has shared implementation as a design feature as
you should never have to duplicate code.  However, the reason I like OO
style programming is the ability to create a hierarchy of different
objects, yet all with the same API.

> 
> > > Something like:
> > > YumPackageObject
> > >     YumRepoMDPackageObject
> > >     YumRHNPackageObject
> > [snip]
> > 
> > These classes seem about right to me. The main thing I'd add is the 
> > MetaSack as mentioned above and Repository classes to abstract the 
> > mechanics of talking to the repository and populating package sacks or 
> > whatever. Hell, repository classes could probably handle parsing their 
> > own bits sections of yum.conf. Each repository class should have 
> > particular PackageObject and PackageSack subclasses associated with them.
> 
> why would they parse their own sections in the config file? we would end
> up needing to parse it once from the main config class and then parse it
> again to pitch it off to each repository format to parse themselves.
> That seems inelegant. 
> 

I agree with Seth on this one.  Repo objects should not parse
configuration information.  However, we must keep in mind that different
types of repositories will require a few additional bits of config
information.  Using RHN as the example, it makes the most since to have
the baseurl be the URL of the XMLRPC interface:

    https://rhn.linux.ncsu.edu/XMLRPC

However, there's no way to tell that URL apart from a URL that points to
a repomd style repository.

> I'd say the repository classes just take a simple dict of information
> for their own information and we keep any configuration file parsing in
> a single class.
> 
> So we end up with a base config class that stores global/main
> configuration data. Then a configparser class which deals with the file
> format, the RepositoryStorage class which holds and accesses all of the
> repositories and finally repository classes which actually store the
> repo information. That way the repository classes don't care if the
> yum.conf file is in ConfigParser format or xml or some other format.
> 

Indeed. 

> The other side of the argument is the need for some additional
> repository type to neeed some wacky config variable in the config file.
> Given the restrictions of ConfigParser as-is we might simply be able to
> consider all other items in each repo section as key-value pairs, grab
> them as simple strings and let the repository class deal with what it
> gets in the config dict. That's not much cleaner, though. hmm.
> 
> 
> > Ideas for things to do from here (maybe this should go on the wiki?):
> > 
> > - merge the repomd stuff into the rest of yum and clean it up. The whole
> >    repomd directory can probably go unless there's still some need to
> >    keep it separate.
> 
> agreed - it was originally separate for various organizational reasons
> that have since gone away.
>

dito

> 
> > - remove the older repo packagesack stuff so that we only have the
> >    sqlite cache code left
> well I want to make sure we can still read the xml completely into
> memory, directly.
> 
> > - experiment with the MetaSack idea and see if its feasible to use
> >    multiple packagesacks at the same time
> 
> I think if we treat it like a wrapper for  a sack for each repository
> and just require that the repository sack object have certain methods we
> should be in reasonably good shape.
> 

Not a wrapper.  A subclass of PackageSack.  

I've got to say that I think the term 'meta' is becoming overused.  I'd
go for a different name.</opinion>

> 
> > - develop a Repository base class and implement repo metadata repository
> >    as a subclass. This may be just refactoring what's already in
> >    repos.py.
> 
> <nod>
> 

Need to create base abstract classes for PackageSack and PackageObject
as well.  Should be simple.

Jack Neely

> -sv
> 
> 
> _______________________________________________
> Yum-devel mailing list
> Yum-devel at linux.duke.edu
> https://lists.dulug.duke.edu/mailman/listinfo/yum-devel

-- 
Jack Neely <slack at quackmaster.net>
Realm Linux Administration and Development
PAMS Computer Operations at NC State University
GPG Fingerprint: 1917 5AC1 E828 9337 7AA4  EA6B 213B 765F 3B6A 5B89