oPromessa/flickr-uploader

Idea: Exclusion performance

Closed this issue · 2 comments

malys commented

Hi again :)
I 'm finding out performance improvements.
in my case, there are many excluded folders. I'm excluded foldes including past year's photos yet synchronized.

I think that "exclude" method can be improved replacing:

for f in filenames:

by
`
.....
os.walk(FILES_DIR, followlinks=True):

            # We want to exclude folders, It's a waste of time to check every time files included in a  exclued folder 
            if os.path.basename(os.path.normpath(dirpath)).encode('utf-8') in EXCLUDED_FOLDERS:
                dirnames[:] = []
                filenames[:] = []   
                logging.warning('Folder {!s} ignored.'
                    .format(os.path.basename(os.path.normpath(dirpath)).encode('utf-8'))) 
                
            for f in filenames:
                filePath = os.path.join(dirpath, f)
                #if self.isFileIgnored(filePath):
                #    logging.debug('File {!s} in EXCLUDED_FOLDERS:'
                #                .format(filePath.encode('utf-8')))
                #    continue

`
Inspired by: http://cyluun.github.io/blog/manipulating-python-oswalk

I hope that will be useful .

With the last version and my fix, the performance is fantastic !!.

Thanks. Performance of V2.6.7 is much better but, yes, when you have a large number or sizeable EXCLUDED_FOLDERS, I agree it can still waste some time going thru them. Will take a look at this code!

Thanks. Available on V2.7.0