Andrew McGill on Mon, 10 Nov 2008 16:29:57 +0200 (SAST)


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

[GLUG-tech] The case of the missing lines from xargs --max-procs .. md5sum


Dear Glug,

What would you expect this to do --:

    find -type f -print0 | 
	xargs -0 -n 8 --max-procs=16 md5sum >& ~/md5sums

    sort -k2 < md5sums > md5sums.sorted

Compared to this?

    find -type f -print0 | 
	xargs -0                     md5sum >& ~/md5sums

    sort -k2 < md5sums > md5sums.sorted

I was a little surprised that on my system running in parallel (the first 
version) loses around 1 line of output per thousand (~22Gb in mostly small 
files).  

Is there a correct way to do md5sums in parallel without having a shared 
output buffer which eats output (I presume) -- or is losing output when 
haphazardly combining output streams actually strange and unusual?

&:-)
-- 
To unsubscribe: send the line "unsubscribe glug-tech" in the
subject of a mail to "glug-tech-request@xxxxxxxxxxxx".
Problems? Email "glug-tech-admins@xxxxxxxxxxxx". Archives are at
http://www.linux.org.za/Lists-Archives/
RULES: http://www.linux.org.za/glugrules.html