Pieter Steyn on Mon, 10 Nov 2008 16:39:03 +0200 (SAST)


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

[GLUG-tech] Re: The case of the missing lines from xargs --max-procs .. md5sum


Hi Andrew,

xargs will take each file it found and append it to the sort...md5sum.

It first builds an argument list, and then executes the commands.

Have a look at find's exec function.

Hope I understood correctly. :-)

Cheers,
Pieter


On Mon, Nov 10, 2008 at 4:29 PM, Andrew McGill <glug@xxxxxxxxxxxx> wrote:
> Dear Glug,
>
> What would you expect this to do --:
>
>    find -type f -print0 |
>        xargs -0 -n 8 --max-procs=16 md5sum >& ~/md5sums
>
>    sort -k2 < md5sums > md5sums.sorted
>
> Compared to this?
>
>    find -type f -print0 |
>        xargs -0                     md5sum >& ~/md5sums
>
>    sort -k2 < md5sums > md5sums.sorted
>
> I was a little surprised that on my system running in parallel (the first
> version) loses around 1 line of output per thousand (~22Gb in mostly small
> files).
>
> Is there a correct way to do md5sums in parallel without having a shared
> output buffer which eats output (I presume) -- or is losing output when
> haphazardly combining output streams actually strange and unusual?
>
> &:-)
> --
> To unsubscribe: send the line "unsubscribe glug-tech" in the
> subject of a mail to "glug-tech-request@xxxxxxxxxxxx".
> Problems? Email "glug-tech-admins@xxxxxxxxxxxx". Archives are at
> http://www.linux.org.za/Lists-Archives/
> RULES: http://www.linux.org.za/glugrules.html
>
>
-- 
To unsubscribe: send the line "unsubscribe glug-tech" in the
subject of a mail to "glug-tech-request@xxxxxxxxxxxx".
Problems? Email "glug-tech-admins@xxxxxxxxxxxx". Archives are at
http://www.linux.org.za/Lists-Archives/
RULES: http://www.linux.org.za/glugrules.html