[PATCH] Basic support for non-ascii package names. BZ 1039276

[Toshio Kuratomi]

On Dec 10, 2013 1:55 AM, "Zdenek Pavlas" <zpavlas at redhat.com> wrote:
>
> Not sure how useful is this, but individual changes are probably ok,
> and the test RPM did install, reinstall and remove just fine.

At one time I thought that we checked whether yum handled Unicode package
names and it was capable of that.  I suppose that could have broken due to
diuse, though.

Looking at your patch, it seems dangerous.  It converts the name to Unicode
so every place that uses the name should have its other inputs changed to
Unicode strings as well (e, v, and r.  The format strings that combine
them.)  You also have to track the variables that are derived from name to
and do the same thing with everything that they're combined with (as those
variables are now Unicode strings).

When you test this, be sure to test it with locales with different
encodings.  The easiest locale to test with is the C locale as it will trip
over any non-ascii data that isn't explicitly changed to bytes.  For
example:

LC_ALL=C ./yum install $rpmfile

Note: since yum worked with Unicode names before you may find that it is
easier to deal with name as a byte string and fix whatever code has
regressed recently.  In fact, some code probably must use the bye str
version of name.  Rpm likely stores the byte version of name in its
database.  So you could end up with interoperability problems when yum
encounters a name that it cannot decode in the current locale if you
translate name to a Unicode string.

-Toshio

> ---
>  output.py              | 7 +++++--
>  yum/history.py         | 1 +
>  yum/packages.py        | 3 +++
>  yum/transactioninfo.py | 3 +++
>  4 files changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/output.py b/output.py
> index 041910c..de77d69 100755
> --- a/output.py
> +++ b/output.py
> @@ -1482,6 +1482,7 @@ class YumOutput:
>
>          def _add_line(lines, data, a_wid, po, obsoletes=[]):
>              (n,a,e,v,r) = po.pkgtup
> +            n = to_unicode(n)
>              evr = po.printVer()
>              repoid = po.ui_from_repo
>              pkgsize = float(po.size)
> @@ -1708,6 +1709,7 @@ Transaction Summary
>                  out += '\n%s:\n' % action
>                  for txmbr in pkglist:
>                      (n,a,e,v,r) = txmbr.pkgtup
> +                    n = to_unicode(n)
>                      msg = "%s.%s %s:%s-%s" % (n,a,e,v,r)
>                      msgs.append(msg)
>                  for num in (8, 7, 6, 5, 4, 3, 2):
> @@ -2894,7 +2896,7 @@ class DepSolveProgressCallBack:
>
>          yb = self.ayum
>
> -        prob_pkg = "%s (%s)" % (reqPo, reqPo.ui_from_repo)
> +        prob_pkg = u"%s (%s)" % (reqPo, reqPo.ui_from_repo)
>          msg = _('Package: %s') % (prob_pkg,)
>          ui_req = formatRequire(needname, needversion, needflags)
>          msg += _('\n    Requires: %s') % (ui_req,)
> @@ -3211,13 +3213,14 @@ class YumCliRPMCallBack(RPMBaseCallback):
>      def _out_event(self, te_current, te_total, ts_current, ts_total,
>                     percent, process, pkgname, wid1):
>          if self.output and (sys.stdout.isatty() or te_current ==
te_total):
> +            pkgname = to_unicode(pkgname)
>              (fmt, wid1, wid2) = self._makefmt(percent, ts_current,
ts_total,
>
 progress=sys.stdout.isatty(),
>                                                pkgname=pkgname, wid1=wid1)
>              msg = fmt % (utf8_width_fill(process, wid1, wid1),
>                           utf8_width_fill(pkgname, wid2, wid2))
>              if msg != self.lastmsg:
> -                sys.stdout.write(to_unicode(msg))
> +                sys.stdout.write(msg)
>                  sys.stdout.flush()
>                  self.lastmsg = msg
>              if te_current == te_total:
> diff --git a/yum/history.py b/yum/history.py
> index 6f60f54..6fc1ecd 100644
> --- a/yum/history.py
> +++ b/yum/history.py
> @@ -726,6 +726,7 @@ class YumHistory:
>              except (sqlite.OperationalError, sqlite.DatabaseError):
>                  self.conf.readable = False
>                  return None
> +            self._conn.text_factory = str
>
>              #  Note that this is required due to changing the history DB
in the
>              # callback for removed txmbrs ... which happens inside the
chroot,
> diff --git a/yum/packages.py b/yum/packages.py
> index cc1f1e3..247ca6d 100644
> --- a/yum/packages.py
> +++ b/yum/packages.py
> @@ -330,6 +330,9 @@ class PackageObject(object):
>                                     self.arch)
>      envra = property(fget=lambda self: self._envra())
>
> +    def __unicode__(self):
> +        return str(self).decode('UTF-8')
> +
>      def __str__(self):
>          return self.ui_envra
>
> diff --git a/yum/transactioninfo.py b/yum/transactioninfo.py
> index d34f3a8..2cabfc9 100644
> --- a/yum/transactioninfo.py
> +++ b/yum/transactioninfo.py
> @@ -864,6 +864,9 @@ class TransactionMember:
>      def __hash__(self):
>          return object.__hash__(self)
>
> +    def __unicode__(self):
> +        return str(self).decode('UTF-8')
> +
>      def __str__(self):
>          return "%s.%s %s:%s-%s - %s" % (self.name, self.arch, self.epoch,
>                                          self.version, self.release,
self.ts_state)
> --
> 1.7.11.7
>
> _______________________________________________
> Yum-devel mailing list
> Yum-devel at lists.baseurl.org
> http://lists.baseurl.org/mailman/listinfo/yum-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.baseurl.org/pipermail/yum-devel/attachments/20131210/d444145e/attachment.html>