[Yum-devel] [PATCH 3/4] pkglist: use streaming-friendly pkg split

[James Antill]

On Fri, 2012-11-23 at 16:28 +0100, Zdeněk Pavlas wrote:
> ---
>  createrepo/__init__.py |    6 ++++--
>  createrepo/utils.py    |   15 ---------------
>  2 files changed, 4 insertions(+), 17 deletions(-)
> 
> diff --git a/createrepo/__init__.py b/createrepo/__init__.py
> index 622e442..0f1cfee 100644
> --- a/createrepo/__init__.py
> +++ b/createrepo/__init__.py
> @@ -47,7 +47,7 @@ except ImportError:
>      pass
>  
>  from utils import _gzipOpen, compressFile, compressOpen, checkAndMakeDir, GzipFile, \
> -                  checksum_and_rename, split_list_into_equal_chunks
> +                  checksum_and_rename
>  from utils import num_cpus_online
>  import deltarpms
>  
> @@ -612,7 +612,9 @@ class MetaDataGenerator:
>              self._worker_tmp_path = tempfile.mkdtemp() # setting this in the base object so we can clean it up later
>              if self.conf.workers < 1:
>                  self.conf.workers = num_cpus_online()
> -            worker_chunks = split_list_into_equal_chunks(pkgfiles, self.conf.workers)
> +            worker_chunks = [[] for i in range(self.conf.workers)]
> +            for i, pkg in enumerate(pkgfiles):
> +                worker_chunks[i % self.conf.workers].append(pkg)

 NAK. But...

>              worker_cmd_dict = {}
>              worker_jobs = {}
>              base_worker_cmdline = [self.conf.worker_cmd, 
> diff --git a/createrepo/utils.py b/createrepo/utils.py
> index cfe68e6..af9f741 100644
> --- a/createrepo/utils.py
> +++ b/createrepo/utils.py
> @@ -190,21 +190,6 @@ def encodefiletypelist(filetypelist):
>          result += ftl[x]
>      return result
>  
> -def split_list_into_equal_chunks(seq, num_chunks):

 We can't remove this API. But we can change it to:

def split_list_into_equal_chunks2(seq, num_chunks):
    out = [[] for i in range(num_chunks)]
    for i, data in enumerate(seq):
        out[i % num_chunks].append(data)
    return out

...which my tests show that it does basically the same thing but with a
different ordering:

In [3]: s = createrepo.utils.split_list_into_equal_chunks2

In [4]: s([1, 2, 3, 4, 5, 6, 7, 8, 9], 4)
Out[4]: [[1, 5, 9], [2, 6], [3, 7], [4, 8]]

In [5]: s = createrepo.utils.split_list_into_equal_chunks

In [6]: s([1, 2, 3, 4, 5, 6, 7, 8, 9], 4)
Out[6]: [[1, 2], [3, 4], [5, 6], [7, 8, 9]]

...what is the intention of changing the order?