Using Chunk?

[Mannshoch]

I try to use pysmb on a router.
If a file I try to copy exceed a certain size, the RAM is full. Internet says that I should transfer the file in chunks. Is there an example somewhere how to do it?

[miketeo]

@Mannshoch : Can you provide an estimate on the file size before your app runs out of RAM?

[Mannshoch]

I have planned to backup a ~16GB ZIP form a WebDAV with as cron job once a month onto a NAS. My Router has 2GB RAM (around 1.4GB is unused). It is possible to ask WebDAV (webdavclient3) for the file size.

[miketeo]

The issue might be that your app is trying to read the entire zip into memory before uploading to your NAS.
If you can read your zip file in smaller chunks from your WebDAV server, you can use pysmb's storeFileFromOffset to upload each chunk of data at the last offset.

[Mannshoch]

that's my current code:

for filename in webdav_fs.ls(webdav_pfad, detail=False):
    webdav_file_path = os.path.join(filename)
    smb_file_path = os.path.join(smb_path, filename)
    file_size = webdav_fs.info(webdav_file_path)['size']
    offset = 0
    chunk_size = 8 * 1024  # 8KB
    f = io.BytesIO()
    webdav_client.download_fileobj(webdav_file_path, f)
    f.seek(0)  
    while True:
        chunk = f.read(chunk_size)
        if not chunk:
            break
        smb_conn.storeFileFromOffset(smb_service_name, smb_file_path, f, offset=offset)
        offset += len(chunk)

[miketeo]

webdav_client.download_fileobj(webdav_file_path, f)
This line will download the entire WebDAV file into the memory.
Please check if your client can download the file in chunks from the remote WebDAV server.