Directories with large number of files aren't listed correctly.

[islamgalileo]

If you try to get the files in a directory that has say more than 500 hundred files you won't get the complete list. The main issue is the buffer size that is 65K.

[bchelli]

Hi @islamgalileo,
Sorry for the late response.
Could you point me the file containing the 65k limit?
Thanks,

[sfyfedotcom]

I'm having a similar issue. Trying to get a list of 9129 pdfs from a folder, but it's only returning the first 346

[bigfei]

Me too, my Download directory contains more than 3000 files, only returning the first 374 filenames.

smb2Client.readdir('Download',function(err, files){
    if(err) console.error(err);
    console.log(files.length);
});

[bchelli]

Hi guys,
Do you think this is related to the issue #6 ?
Thanks

[marsaud]

Thus may be a timeout problem fixed by #12

[marsaud]

@islamgalileo , @sfyfe31 , @bigfei does your problem remains with 0.2.7 ?

[nickperkinslondon]

I am having this problem. ( on 0.2.7 )
I have several folders with up to 999 sub-folders ( each with a 3-digit name )
It only lists up to 583 sub-folders -- if there are more than that, it shows only the first 583.

The number (for me) is always 583, which is oddly consistent, but remember my folder names are all the same length, so if the problem were some 64k limit, that would be about 110 bytes per entry -- which seems reasonable. ( or a 32k limit, with 55 bytes per entry ) Maybe it's like reading a file, where you get back the first 32k and you have to keep going back to get more data. ( it's definitely not a time-out )

Should be easy to reproduce, just create a folder with 1000 files and try to list them all.
I hope this can be fixed -- I am using it for an important project!

[wadjeroudi]

@nickperkinslondon did you find a solution ?

[mauricioszabo]

Did somebody found a solution for it? I'm thinking about using it in a project, but can't because of this limit...

[ZackCoddington]

I have found a small workaround that in limited situations can help. The issue is definitely due to the buffer size, which I can't figure out how to increase. However since I only needed the file names from the directories I am reading, I was able to dramatically decrease the amount of buffer usage by each file. This was done by going into .\structures\query_directory.js and changing ['FileInformationClass', 1, 0x25] to ['FileInformationClass', 1, 0x0C].

Then I had to change the parseFile function within .\messages\query_directory.js to

function parseFile(buffer){
var file = {}
, offset = 0
;

// index
file.Index = buffer.readUInt32LE(offset);
offset+=4;

// FilenameLength
file.FilenameLength = buffer.readUInt32LE(offset);
offset+=4;
// Filename
file.Filename = buffer.slice(offset, offset+file.FilenameLength).toString('ucs2');
offset+=file.FilenameLength;

return file;
}

With this I am still limited to roughly 1057 files

[ZackCoddington]

I have done some more investigations into this, and it looks like the way QUERY_DIRECTORY works in SMB2 with large file systems is that the next QUERY_DIRECTORY command will then start off where the last query stopped. However it looks like readdir only reads the directory once, and then sends a CLOSE command the the SMB server. Because of this close command we start again at index 0 if we try to query the directory again.

So while we can't increase the buffer size and decreasing the buffer space each file is of limited use. This could be resolved by either modifying readdir.js, or creating a new readLargeDir.js in a way that waits until all the files have been read before closing the SMB connection.

[wadjeroudi]

@ZackCoddington nice debug ! Tried a year ago a long debug, couldn't find the reason.

[bchelli]

Thanks to @ZackCoddington
This is fixed in the v0.2.9 of the package