Re: file descriptor diet

[Paul Eggert]

[Adding bug-tar.]

On 09/13/10 03:52, Jim Meyering wrote:

> He means merely that when tar traverses a tree of depth N
> it will no longer require N open file descriptors.

Yes, that's the point.  I should clarify that the only reason
I had to put tar on a diet is that a week or so ago, I installed
a patch that made tar a file descriptor glutton.  This gluttony
problem doesn't exist in any shipping version of tar.

As part of the diet, whenever a system or library call C fails with
errno == EMFILE, and that call occurs deeply in the traversal,
tar chooses a cached file descriptor D that points to an ancestor
directory, closes D, and then retries C.  Also, tar now sets the
CLOEXEC flag on directories' file descriptors; this avoids problems
when tar does a fork+exec, because we can't assume the execed program
can deal with EMFILE that way.

This diet should work with any program that recursively traverses
a directory hierarchy.  Coreutils uses a different approach, which
caches at most 4 directory file descriptors.  I cached more
aggressively in tar partly for the fun of it, and partly because
caching more fds makes race conditions less likely.  (The last
couple of weeks, I've been fixing some race condition bugs in
tar.)