Implement asynchronous readdir

Accessing directories with many (several thousand) files over sshfs is
slow, because most SFTP server implementations only send a fixed amount
of entries per READDIR command (e.g. OpenSSH SFTP: 100 entries). This
patch implements sending several READDIR commands in parallel, in order
to speed up directory listing in these cases.

An option (sync_readdir) is also added so that users can easily switch
on the old behaviour.

The performance improvement is astonishing. Accessing a directory with
30k files in from a remote server that has a RTT of 15ms via OpenSSH
SFTP:

Synchronous readdir:
 $ ./sshfs -o sync_readdir host:/tmp /mnt/temp
 $ time "ls" -1 /mnt/temp/test | wc -l
 30000
 "ls" -1 /mnt/temp/test  0.07s user 0.01s system 1% cpu 6.928 total

Asynchronous readdir:
 $ ./sshfs host:/tmp /mnt/temp
 $ time "ls" -1 /mnt/temp/test | wc -l
 30000
 "ls" -1 /mnt/temp/test  0.07s user 0.01s system 12% cpu 0.605 total

Accessing a directory with 100k files shows even more dramatic
improvement:

Synchronous readdir:
 $ ./sshfs -o sync_readdir host:/tmp /mnt/temp
 $ time "ls" -1 /mnt/temp/test2 | wc -l
 100000
 "ls" -1 /mnt/temp/test2  0.67s user 1.22s system 0% cpu 3:31.56 total

Asynchronous readdir:
 $ ./sshfs host:/tmp /mnt/temp
 $ time "ls" -1 /mnt/temp/test2 | wc -l
 100000
 "ls" -1 /mnt/temp/test2  0.20s user 0.03s system 14% cpu 1.631 total

This can easily be reproduced by creating a directory on a server and
touching a lot of files in it:
 $ mkdir /tmp/test
 $ cd /tmp/test
 $ for i in $(seq 1 30000); do touch $i; done

Signed-off-by: Alexander Neumann <alexander@bumpern.de>

1 files changed, 4 insertions, 0 deletions

diff --git a/ChangeLog b/ChangeLog
index 63c999b..fa71225 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -3,6 +3,10 @@
 	* Add -o disable_hardlink option (debian bug #670926).  Reported
 	by Louis-David Mitterrand
 
+	* Optimize readdir by sending multiple requests in parallel.  Add
+	-o sync_readdir to restore old behavior.  Patch by Alexander
+	Neumann
+
 2014-01-07  Miklos Szeredi <miklos@szeredi.hu>
 
 	* Map SSH2_FX_FAILURE to ENOTEMPTY for rmdir.  Reported by Ross