EBLOB: No space left on device

[odiszapc]

I see a lot of "No space left on device" entries in node logs, but I actually have 103GB of free disk sapce and this space is utulizable (I checked I can still write data)

2018-12-24 13:28:06.456769 25c95fe8b56ab182/2902/1363 INFO: 1:b5d060ab6d16...ded5b0d70d7f: WRITE: client: 10.10.9.67:56206, trans: 50431473, cflags: 0x1 [need_ack], io-flags: 0x0 [], io-offset: 0, io-size: 17428/0, io-user-flags: 0x0, io-num: 0, ts: '2018-12-24 13:28:06.455600', time: 858 usecs, err: -28. 'backend_id': 1
2018-12-24 13:28:06.456854 25c95fe8b56ab182/2457/1363 INFO: 1:b5d060ab6d16...ded5b0d70d7f: WRITE: sending trans: 50431473 -> 10.10.9.67:56206/1: size: 0, cflags: 0x200 [reply], start-sent: 0/120
2018-12-24 13:28:06.456978 25c95fe8b56ab182/2457/1363 INFO: 1:b5d060ab6d16...ded5b0d70d7f: WRITE: sending trans: 50431473 -> 10.10.9.67:56206/1: size: 0, cflags: 0x200 [reply], finish-sent: 120/120
2018-12-24 13:28:06.514657 367697aa9f795441/2904/1363 INFO: blob: 04f9098df705: stat: bctls: 549, no-sorted-index: 12, search-on-disk: 537, bloom-no-key: 537, found-index-block:0, no-index-block: 0, bsearch-reached: 0, bsearch-found: 0, additional-reads: 0, err: -2 'backend_id': 1
2018-12-24 13:28:06.514767 367697aa9f795441/2904/1363 ERROR: blob: 04f9098df705: i-1: eblob_write_prepare_disk: position: 0, offset: 0, size: 17476, flags: 0x140 [exthdr|chunked_csum], total data size: 0, disk-size: 0, data_fd: 0, index_fd: 0, bctl: (nil): -28 'backend_id': 1
2018-12-24 13:28:06.514833 367697aa9f795441/2904/1363 ERROR: blob: 04f9098df705: i-1: eblob_writev: finished: position: 0, offset: 0, size: 17476, flags: 0x140 [exthdr|chunked_csum], total data size: 0, disk-size: 0, data_fd: 0, index_fd: 0, bctl: (nil): -28 'backend_id': 1
2018-12-24 13:28:06.514893 367697aa9f795441/2904/1363 ERROR: 04f9098df705: EBLOB: blob-write: WRITE: -28: No space left on device 'backend_id': 1

$ df -h
Filesystem               Size  Used Avail Use% Mounted on
...
/dev/sdb1                1.0T  922G  103G  91% /srv/node/partition1
...

$ df -i
Filesystem                 Inodes IUsed     IFree IUse% Mounted on
...
/dev/sdb1               429260768  1666 429259102    1% /srv/node/partition1
...

Configuraton:

        "logger": {
            "frontends": [
                {
                    "formatter": {
                        "type": "string",
                        "pattern": "%(timestamp)s %(request_id)s/%(lwp)s/%(pid)s %(severity)s: %(message)s %(...L)s"
                    },
                    "sink": {
                        "type": "files",
                        "path": "/var/log/elliptics/node.log",
                        "autoflush": true,
                        "rotation": {
                            "move": 0
                        }
                    }
                }
            ],
            "level": "info"
        },
        "options": {
                "join": true,
                "flags": 4,
                "remote": [
                        "10.10.9.67:1025:2"
                ],
                "address": [
                        "10.10.9.66:1025:2"
                ],
                "wait_timeout": 60,
                "check_timeout": 60,
                "io_thread_num": 8,
                "nonblocking_io_thread_num": 4,
                "net_thread_num": 4,
                "daemon": true,
                "auth_cookie": "cookie-demo",
                "cache": {
                        "size": 1073741824
                },
                "indexes_shard_count": 16,
                "monitor": {
                        "port": 20000,
                        "call_tree_timeout": 0
                }
        },
        "backends": [
                {
                        "backend_id": 1,
                        "type": "blob",
                        "group": 1,
                        "history": "/srv/node/partition1/history",
                        "data": "/srv/node/partition1/data/blob",
                        "sync": "-1",
                        "blob_flags": "513",
                        "blob_size": "8G",
                        "records_in_blob": "10000000",
                        "iterate_thread_num": 8,
                        "defrag_percentage": 5
                }
        ]
}

[odiszapc]

Also I see:

2018-12-24 13:34:06.441967 3082765aa60034cf/31314/31276 ERROR: OUT OF FREE SPACE: available: 104799 Mb, total: 1048063 Mb, blob size: 8192 Mb 'backend_id': 1

[odiszapc]

/**
 * eblob_check_free_space() - checks if there is enough space for another 2 blobs
 * (2 blobs are needed for sorting) -  or there is at least 10% of free space available on this FS.
 */

Ok, it's pretty clear now