Deleting PV in kubernetes is not deleting volume on gluster w/ heketi

Question

Deleting PV in kubernetes is not deleting volume on gluster w/ heketi

debianmaster opened this issue 7 years ago · 10 comments

i have a setup a storage class backed by gluster w/ heketi.
I have provisioned few volumes with this dynamic storage volume but when i delete PV its not deleting underlying volume automatically

debianmaster commented 7 years ago

❓

Answer 1 · 2017-04-23T18:52:07.000Z

@debianmaster can you provide me more details and logs ? also which version of kube is in use here ?

Answer 2 · 2017-04-24T07:15:10.000Z

OpenShift Master:
v1.5.0+031cbe4
Kubernetes Master:
v1.5.2+43a9be4

My disks were 50GB previously and i got no storage error, (which seems right) , but now i increased it to 100GB. and restarted heketi pod, i'm still getting following error.

Failed to provision volume with StorageClass "default": glusterfs: create volume err: error creating volume Error calling v.allocBricksInCluster: database is in read-only mode.

Answer 3 · 2017-04-24T11:35:58.000Z

i have restarted gluster DaemonSets (by killing) and also heketi still see that database ready only mode error

Answer 4 · 2017-04-24T14:26:26.000Z

@debianmaster Can you restart the heketi pod and paste the logs from the point where it starts coming back up?

Answer 5 · 2017-04-24T15:49:28.000Z

Heketi v4.0.0-57-g27e7ea4
[heketi] INFO 2017/04/24 11:32:18 Loaded kubernetes executor
[heketi] WARNING 2017/04/24 11:32:18 Unable to open database.  Retrying using read only mode
[heketi] INFO 2017/04/24 11:32:18 Loaded simple allocator
[heketi] INFO 2017/04/24 11:32:18 GlusterFS Application Loaded
Authorization loaded
Listening on port 8080
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 735.351µs
[asynchttp] INFO 2017/04/24 11:32:24 Started job 5ac838d25ba64ecfe7e08315674713c4
[heketi] INFO 2017/04/24 11:32:24 Creating volume 21dc2b01f95b207ccd224ab09a07954a
[heketi] INFO 2017/04/24 11:32:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:24 Completed job 5ac838d25ba64ecfe7e08315674713c4 in 2.078337ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 240.783µs
[asynchttp] INFO 2017/04/24 11:32:24 Started job 33692bcea29f867d165ccdeef20f6fbd
[heketi] INFO 2017/04/24 11:32:24 Creating volume 918a11c044201679511ead21dbfb182f
[heketi] INFO 2017/04/24 11:32:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:24 Completed job 33692bcea29f867d165ccdeef20f6fbd in 698.725µs
[negroni] Started GET /queue/5ac838d25ba64ecfe7e08315674713c4
[negroni] Completed 500 Internal Server Error in 88.756µs
[negroni] Started GET /queue/33692bcea29f867d165ccdeef20f6fbd
[negroni] Completed 500 Internal Server Error in 43.042µs
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 169.386µs
[asynchttp] INFO 2017/04/24 11:32:39 Started job cb43ed7de7bc3bcb9cd34365d7038a41
[heketi] INFO 2017/04/24 11:32:39 Creating volume 86a96c805ef47e35e7cf514ee49e73fd
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:32:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:39 Completed job cb43ed7de7bc3bcb9cd34365d7038a41 in 1.01767ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 104.023µs
[asynchttp] INFO 2017/04/24 11:32:39 Started job 84b1a9d3d5eaa0c7f5e3195ea6a272bd
[heketi] INFO 2017/04/24 11:32:39 Creating volume 9ba7cd1ae936acfa8e211faf9c7a833e
[heketi] INFO 2017/04/24 11:32:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:39 Completed job 84b1a9d3d5eaa0c7f5e3195ea6a272bd in 607.641µs
[negroni] Started GET /queue/cb43ed7de7bc3bcb9cd34365d7038a41
[negroni] Completed 500 Internal Server Error in 75.599µs
[negroni] Started GET /queue/84b1a9d3d5eaa0c7f5e3195ea6a272bd
[negroni] Completed 500 Internal Server Error in 67.341µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 181.19µs
[asynchttp] INFO 2017/04/24 11:32:54 Started job f42076002f8dde52b68463b5885b35ba
[heketi] INFO 2017/04/24 11:32:54 Creating volume 54f38956790fce0b5981f9defcfcfdbc
[heketi] INFO 2017/04/24 11:32:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:54 Completed job f42076002f8dde52b68463b5885b35ba in 4.385998ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 163.506µs
[asynchttp] INFO 2017/04/24 11:32:54 Started job c9658d6f9e9a5fb6fa954f39b794b5b5
[heketi] INFO 2017/04/24 11:32:54 Creating volume 69b3a566f6b013af7aa749d43b466f68
[heketi] INFO 2017/04/24 11:32:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:32:54 Completed job c9658d6f9e9a5fb6fa954f39b794b5b5 in 610.964µs
[negroni] Started GET /queue/f42076002f8dde52b68463b5885b35ba
[negroni] Completed 500 Internal Server Error in 83.412µs
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:32:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started GET /queue/c9658d6f9e9a5fb6fa954f39b794b5b5
[negroni] Completed 500 Internal Server Error in 64.039µs
[negroni] Started POST /volumes
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Completed 202 Accepted in 549.541µs
[asynchttp] INFO 2017/04/24 11:33:09 Started job 8b6b253f80de8afe06d741ab88d322e8
[heketi] INFO 2017/04/24 11:33:09 Creating volume c18795c0a62d0378d204fa9b0a676d23
[heketi] INFO 2017/04/24 11:33:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:09 Completed job 8b6b253f80de8afe06d741ab88d322e8 in 737.056µs
[negroni] Started GET /queue/8b6b253f80de8afe06d741ab88d322e8
[negroni] Completed 500 Internal Server Error in 92.453µs
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 184.729µs
[asynchttp] INFO 2017/04/24 11:33:09 Started job fd5bf24afa83f977f448208b1a5d8507
[heketi] INFO 2017/04/24 11:33:09 Creating volume 58ebd326b87ff38f535ddf3a92dca362
[heketi] INFO 2017/04/24 11:33:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:09 Completed job fd5bf24afa83f977f448208b1a5d8507 in 727.503µs
[negroni] Started GET /queue/fd5bf24afa83f977f448208b1a5d8507
[negroni] Completed 500 Internal Server Error in 86.055µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 176.067µs
[asynchttp] INFO 2017/04/24 11:33:24 Started job 5b73e827dc21fac22ec342633bb4165c
[heketi] INFO 2017/04/24 11:33:24 Creating volume ce1d02813672a7aa433492012bf71ebb
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:33:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:24 Completed job 5b73e827dc21fac22ec342633bb4165c in 1.003952ms
[negroni] Started GET /queue/5b73e827dc21fac22ec342633bb4165c
[negroni] Completed 500 Internal Server Error in 80.524µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 98.002µs
[asynchttp] INFO 2017/04/24 11:33:24 Started job 7639172d29537fc11390f723c82e2641
[heketi] INFO 2017/04/24 11:33:24 Creating volume 06bd6964d8a99856b240dd90fc544ea1
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:33:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:24 Completed job 7639172d29537fc11390f723c82e2641 in 717.965µs
[negroni] Started GET /queue/7639172d29537fc11390f723c82e2641
[negroni] Completed 500 Internal Server Error in 59.848µs
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 245.159µs
[asynchttp] INFO 2017/04/24 11:33:39 Started job 798c2ae3ba8b6480d43b6c11022b446f
[heketi] INFO 2017/04/24 11:33:39 Creating volume 5fb706cfdf55c5ef0bfd6c464d42455d
[heketi] INFO 2017/04/24 11:33:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:39 Completed job 798c2ae3ba8b6480d43b6c11022b446f in 844.851µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 147.371µs
[asynchttp] INFO 2017/04/24 11:33:39 Started job 44247a00d6b0ed7a93d59e1d26cf2abf
[heketi] INFO 2017/04/24 11:33:39 Creating volume c495a3009cb8761b0575d859b4a374bf
[heketi] INFO 2017/04/24 11:33:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:39 Completed job 44247a00d6b0ed7a93d59e1d26cf2abf in 632.587µs
[negroni] Started GET /queue/798c2ae3ba8b6480d43b6c11022b446f
[negroni] Completed 500 Internal Server Error in 85.886µs
[negroni] Started GET /queue/44247a00d6b0ed7a93d59e1d26cf2abf
[negroni] Completed 500 Internal Server Error in 75.35µs
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 219.09µs
[asynchttp] INFO 2017/04/24 11:33:54 Started job ab96a60c47a7592fa8c8693c78ddba38
[heketi] INFO 2017/04/24 11:33:54 Creating volume 6785440ed64a6ee617d360f377ce37a7
[heketi] INFO 2017/04/24 11:33:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:54 Completed job ab96a60c47a7592fa8c8693c78ddba38 in 884.413µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 130.356µs
[asynchttp] INFO 2017/04/24 11:33:54 Started job 2e5e94b16e36ce3588910f0eea551acc
[heketi] INFO 2017/04/24 11:33:54 Creating volume 03e1ad5f5c02d99730608e7540dcfb26
[heketi] INFO 2017/04/24 11:33:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:33:54 Completed job 2e5e94b16e36ce3588910f0eea551acc in 705.094µs
[negroni] Started GET /queue/2e5e94b16e36ce3588910f0eea551acc
[negroni] Completed 500 Internal Server Error in 78.332µs
[negroni] Started GET /queue/ab96a60c47a7592fa8c8693c78ddba38
[negroni] Completed 500 Internal Server Error in 45.787µs
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:33:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 178.808µs
[asynchttp] INFO 2017/04/24 11:34:09 Started job f2bf7860f572b0bc796e88333886a687
[heketi] INFO 2017/04/24 11:34:09 Creating volume eb2db6279f0e02236954550e827315c4
[heketi] INFO 2017/04/24 11:34:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:09 Completed job f2bf7860f572b0bc796e88333886a687 in 924.276µs
[negroni] Started GET /queue/f2bf7860f572b0bc796e88333886a687
[negroni] Completed 500 Internal Server Error in 75.65µs
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 133.054µs
[asynchttp] INFO 2017/04/24 11:34:09 Started job d48f5fb0ffd46da615f3bc28bc4754b3
[heketi] INFO 2017/04/24 11:34:09 Creating volume 74978a5cfc4aa369b46839b70d88c142
[heketi] INFO 2017/04/24 11:34:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:09 Completed job d48f5fb0ffd46da615f3bc28bc4754b3 in 914.321µs
[negroni] Started GET /queue/d48f5fb0ffd46da615f3bc28bc4754b3
[negroni] Completed 500 Internal Server Error in 65.367µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 509.384µs
[asynchttp] INFO 2017/04/24 11:34:24 Started job 9d1a4333db1f9f67b0dac8d1566c1f4d
[heketi] INFO 2017/04/24 11:34:24 Creating volume 6d4c6f5a89cc92b2d40957ae6b76ac25
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:34:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:24 Completed job 9d1a4333db1f9f67b0dac8d1566c1f4d in 1.169951ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 101.554µs
[asynchttp] INFO 2017/04/24 11:34:24 Started job 231ce63a4a9d985640d8848514cb3694
[heketi] INFO 2017/04/24 11:34:24 Creating volume 943e5fb8ff4fab389afbff6e4dcaeeae
[heketi] INFO 2017/04/24 11:34:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:24 Completed job 231ce63a4a9d985640d8848514cb3694 in 654.238µs
[negroni] Started GET /queue/9d1a4333db1f9f67b0dac8d1566c1f4d
[negroni] Completed 500 Internal Server Error in 75.882µs
[negroni] Started GET /queue/231ce63a4a9d985640d8848514cb3694
[negroni] Completed 500 Internal Server Error in 77.351µs
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 176.093µs
[asynchttp] INFO 2017/04/24 11:34:39 Started job adbd582658367c9656f485e25698a4ed
[heketi] INFO 2017/04/24 11:34:39 Creating volume 57c7e3392c205a6eb5d8e58c52f5d119
[heketi] INFO 2017/04/24 11:34:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:39 Completed job adbd582658367c9656f485e25698a4ed in 843.757µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 116.165µs
[asynchttp] INFO 2017/04/24 11:34:39 Started job 4121196d91a554fb533092afa1def5d7
[heketi] INFO 2017/04/24 11:34:39 Creating volume 477d6f0a1023ec1a2c46fbd90a17401e
[heketi] INFO 2017/04/24 11:34:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:39 Completed job 4121196d91a554fb533092afa1def5d7 in 635.208µs
[negroni] Started GET /queue/adbd582658367c9656f485e25698a4ed
[negroni] Completed 500 Internal Server Error in 94.771µs
[negroni] Started GET /queue/4121196d91a554fb533092afa1def5d7
[negroni] Completed 500 Internal Server Error in 41.669µs
[negroni] Started POST /volumes
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:34:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Completed 202 Accepted in 67.095634ms
[asynchttp] INFO 2017/04/24 11:34:54 Started job f9c1150695b709cccf49e53de579f211
[heketi] INFO 2017/04/24 11:34:54 Creating volume d7e7177c45b2bf90c9e6ab40a076eca5
[heketi] INFO 2017/04/24 11:34:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:54 Completed job f9c1150695b709cccf49e53de579f211 in 836.735µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 202.707µs
[asynchttp] INFO 2017/04/24 11:34:54 Started job f692c9a476109fee6fc9436ccd696e85
[heketi] INFO 2017/04/24 11:34:54 Creating volume 23c04ae3d36580b92fb2e9bf7da097fd
[heketi] INFO 2017/04/24 11:34:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:34:54 Completed job f692c9a476109fee6fc9436ccd696e85 in 650.293µs
[negroni] Started GET /queue/f9c1150695b709cccf49e53de579f211
[negroni] Completed 500 Internal Server Error in 88.178µs
[negroni] Started GET /queue/f692c9a476109fee6fc9436ccd696e85
[negroni] Completed 500 Internal Server Error in 40.837µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 613.505µs
[asynchttp] INFO 2017/04/24 11:35:09 Started job 21c7c7e96a5bfc62e054be7c7e934689
[heketi] INFO 2017/04/24 11:35:09 Creating volume 481456f74c49510ad7bbd0e32eb2a563
[heketi] INFO 2017/04/24 11:35:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:09 Completed job 21c7c7e96a5bfc62e054be7c7e934689 in 1.216958ms
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 173.166µs
[asynchttp] INFO 2017/04/24 11:35:09 Started job 4be651549ddbc8cc657f47638c5ee3e6
[heketi] INFO 2017/04/24 11:35:09 Creating volume e72e2e1856d4989f7c65de6e1a332721
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:35:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:09 Completed job 4be651549ddbc8cc657f47638c5ee3e6 in 767.981µs
[negroni] Started GET /queue/21c7c7e96a5bfc62e054be7c7e934689
[negroni] Completed 500 Internal Server Error in 78.358µs
[negroni] Started GET /queue/4be651549ddbc8cc657f47638c5ee3e6
[negroni] Completed 500 Internal Server Error in 42.179µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 168.881µs
[asynchttp] INFO 2017/04/24 11:35:24 Started job 1a3e19fbc3a1ba32c1e290bb1b712374
[heketi] INFO 2017/04/24 11:35:24 Creating volume c0f1803e5359e9430533986abd1afcd8
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:35:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:24 Completed job 1a3e19fbc3a1ba32c1e290bb1b712374 in 1.109376ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 759.801µs
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:35:24 Started job 01fcdd0fb72c0943f059638e95c96126
[heketi] INFO 2017/04/24 11:35:24 Creating volume 0061d2b32cf18d12e7cc882c156e5568
[heketi] INFO 2017/04/24 11:35:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:24 Completed job 01fcdd0fb72c0943f059638e95c96126 in 865.098µs
[negroni] Started GET /queue/01fcdd0fb72c0943f059638e95c96126
[negroni] Completed 500 Internal Server Error in 99.422µs
[negroni] Started GET /queue/1a3e19fbc3a1ba32c1e290bb1b712374
[negroni] Completed 500 Internal Server Error in 74.521µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 352.344µs
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:35:39 Started job 139d045eff0289ded3251988b25eb0f9
[heketi] INFO 2017/04/24 11:35:39 Creating volume 4639880c01b30f616ad9eca2aeabc66c
[heketi] INFO 2017/04/24 11:35:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:39 Completed job 139d045eff0289ded3251988b25eb0f9 in 819.578µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 774.621µs
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:35:39 Started job 5401cad00308a3bfc8f1b301ede47631
[heketi] INFO 2017/04/24 11:35:39 Creating volume 0e8fb6bc6245f52a63f1b4bd4f922979
[heketi] INFO 2017/04/24 11:35:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:39 Completed job 5401cad00308a3bfc8f1b301ede47631 in 708.601µs
[negroni] Started GET /queue/139d045eff0289ded3251988b25eb0f9
[negroni] Completed 500 Internal Server Error in 106.851µs
[negroni] Started GET /queue/5401cad00308a3bfc8f1b301ede47631
[negroni] Completed 500 Internal Server Error in 43.412µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 172.159µs
[asynchttp] INFO 2017/04/24 11:35:54 Started job 4736e63d1795d994aae1d2a5f2c4303c
[heketi] INFO 2017/04/24 11:35:54 Creating volume 89425a5eb21019ae0675c225e313434f
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:35:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:35:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:54 Completed job 4736e63d1795d994aae1d2a5f2c4303c in 1.003371ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 101.368µs
[asynchttp] INFO 2017/04/24 11:35:54 Started job 4930e85fd29def20576c3c46a77b4a6b
[heketi] INFO 2017/04/24 11:35:54 Creating volume cc3c9b565807c8f4e7c656bae2982cc4
[heketi] INFO 2017/04/24 11:35:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:35:54 Completed job 4930e85fd29def20576c3c46a77b4a6b in 716.502µs
[negroni] Started GET /queue/4930e85fd29def20576c3c46a77b4a6b
[negroni] Completed 500 Internal Server Error in 80.269µs
[negroni] Started GET /queue/4736e63d1795d994aae1d2a5f2c4303c
[negroni] Completed 500 Internal Server Error in 63.702µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 515.754µs
[asynchttp] INFO 2017/04/24 11:36:09 Started job 3d8820343f5ff2fa8b5b60e29f4a416b
[heketi] INFO 2017/04/24 11:36:09 Creating volume 99edcd2ee905ee9130a5009fb96cfe70
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:36:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:09 Completed job 3d8820343f5ff2fa8b5b60e29f4a416b in 1.380268ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 181.663µs
[asynchttp] INFO 2017/04/24 11:36:09 Started job 95219ae40cd191c3a17ea271afb82b11
[heketi] INFO 2017/04/24 11:36:09 Creating volume 668106e46d644a828e6434acddbb0f55
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:36:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:09 Completed job 95219ae40cd191c3a17ea271afb82b11 in 885.904µs
[negroni] Started GET /queue/3d8820343f5ff2fa8b5b60e29f4a416b
[negroni] Completed 500 Internal Server Error in 78.19µs
[negroni] Started GET /queue/95219ae40cd191c3a17ea271afb82b11
[negroni] Completed 500 Internal Server Error in 70.674µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 188.918µs
[asynchttp] INFO 2017/04/24 11:36:24 Started job 8161accbd0ebbe904fa04a5d5c6d92dc
[heketi] INFO 2017/04/24 11:36:24 Creating volume 2045bf9eef11d28aa32632fd7446521c
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:36:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:24 Completed job 8161accbd0ebbe904fa04a5d5c6d92dc in 1.099057ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 647.185µs
[asynchttp] INFO 2017/04/24 11:36:24 Started job 1ee2370cd54f9fcd1dbc3f00650d6c7e
[heketi] INFO 2017/04/24 11:36:24 Creating volume 4e076ec13cc5052bd210bae74d5d2311
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:36:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:24 Completed job 1ee2370cd54f9fcd1dbc3f00650d6c7e in 936.396µs
[negroni] Started GET /queue/8161accbd0ebbe904fa04a5d5c6d92dc
[negroni] Completed 500 Internal Server Error in 102.638µs
[negroni] Started GET /queue/1ee2370cd54f9fcd1dbc3f00650d6c7e
[negroni] Completed 500 Internal Server Error in 322.754µs
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 169.805µs
[asynchttp] INFO 2017/04/24 11:36:39 Started job 93d88b7e03555b2999b152efcb01f667
[heketi] INFO 2017/04/24 11:36:39 Creating volume a771bdc21e7b66dd578c4383c0fa7780
[heketi] INFO 2017/04/24 11:36:39 brick_num: 0
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:36:39 Completed job 93d88b7e03555b2999b152efcb01f667 in 781.398µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 159.636µs
[asynchttp] INFO 2017/04/24 11:36:39 Started job 4854e7d420f42e5cb38550c92b61b586
[heketi] INFO 2017/04/24 11:36:39 Creating volume ddd53f2269b530be9c6105d0c8f35c64
[heketi] INFO 2017/04/24 11:36:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:39 Completed job 4854e7d420f42e5cb38550c92b61b586 in 625.261µs
[negroni] Started GET /queue/93d88b7e03555b2999b152efcb01f667
[negroni] Completed 500 Internal Server Error in 77.727µs
[negroni] Started GET /queue/4854e7d420f42e5cb38550c92b61b586
[negroni] Completed 500 Internal Server Error in 71.383µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 529.85µs
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[asynchttp] INFO 2017/04/24 11:36:54 Started job 3cde3b0cc9a8a4f2d241345176a8ea28
[heketi] INFO 2017/04/24 11:36:54 Creating volume 5983d8c2e019b20946e7212ac29d4224
[heketi] INFO 2017/04/24 11:36:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:54 Completed job 3cde3b0cc9a8a4f2d241345176a8ea28 in 765.055µs
[negroni] Started GET /queue/3cde3b0cc9a8a4f2d241345176a8ea28
[negroni] Completed 500 Internal Server Error in 87.44µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 127.96µs
[asynchttp] INFO 2017/04/24 11:36:54 Started job 3a6080a63ec05714473a25466d14e48b
[heketi] INFO 2017/04/24 11:36:54 Creating volume b9b5ead78eaa876d23ee2d9676563770
[heketi] INFO 2017/04/24 11:36:54 brick_num: 0
[asynchttp] INFO 2017/04/24 11:36:54 Completed job 3a6080a63ec05714473a25466d14e48b in 632.469µs
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:36:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started GET /queue/3a6080a63ec05714473a25466d14e48b
[negroni] Completed 500 Internal Server Error in 71.521µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 275.707µs
[asynchttp] INFO 2017/04/24 11:37:09 Started job f1d79f1503fb8c186469cb3ccb1aecb4
[heketi] INFO 2017/04/24 11:37:09 Creating volume 401642815509420268fc1d7d698a39ee
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:37:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:09 Completed job f1d79f1503fb8c186469cb3ccb1aecb4 in 1.100964ms
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:09 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 125.011µs
[asynchttp] INFO 2017/04/24 11:37:09 Started job 9bbeac8e22895b9aac49127f9864e51a
[heketi] INFO 2017/04/24 11:37:09 Creating volume 872f9d8c85328568a9683d859a7bd6b6
[heketi] INFO 2017/04/24 11:37:09 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:09 Completed job 9bbeac8e22895b9aac49127f9864e51a in 1.03045ms
[negroni] Started GET /queue/f1d79f1503fb8c186469cb3ccb1aecb4
[negroni] Completed 500 Internal Server Error in 115.502µs
[negroni] Started GET /queue/9bbeac8e22895b9aac49127f9864e51a
[negroni] Completed 500 Internal Server Error in 1.92336ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 251.474µs
[asynchttp] INFO 2017/04/24 11:37:24 Started job a88de7ab81a2807eb1be8ec25c9d598d
[heketi] INFO 2017/04/24 11:37:24 Creating volume cfe19ba109cf871b2e3814c144a80e7b
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:37:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:24 Completed job a88de7ab81a2807eb1be8ec25c9d598d in 896.621µs
[negroni] Started GET /queue/a88de7ab81a2807eb1be8ec25c9d598d
[negroni] Completed 500 Internal Server Error in 82.529µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 130.861µs
[asynchttp] INFO 2017/04/24 11:37:24 Started job 1160009e6699c103ae2f74f6953b1f59
[heketi] INFO 2017/04/24 11:37:24 Creating volume 697cd65b8f94b0ec1bf0153920d849af
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:24 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:37:24 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:24 Completed job 1160009e6699c103ae2f74f6953b1f59 in 750.409µs
[negroni] Started GET /queue/1160009e6699c103ae2f74f6953b1f59
[negroni] Completed 500 Internal Server Error in 69.751µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 192.712µs
[asynchttp] INFO 2017/04/24 11:37:39 Started job c641cac7b2f257b668ed26836aa09efe
[heketi] INFO 2017/04/24 11:37:39 Creating volume b40185487ab5b1719816dadfd41d6dd3
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:39 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] INFO 2017/04/24 11:37:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:39 Completed job c641cac7b2f257b668ed26836aa09efe in 1.138811ms
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 224.507µs
[asynchttp] INFO 2017/04/24 11:37:39 Started job e7e6472b5376453372da25d2d8c533b6
[heketi] INFO 2017/04/24 11:37:39 Creating volume 36f33d0bcd0ba8335db19780534230ef
[heketi] INFO 2017/04/24 11:37:39 brick_num: 0
[asynchttp] INFO 2017/04/24 11:37:39 Completed job e7e6472b5376453372da25d2d8c533b6 in 662.948µs
[negroni] Started GET /queue/c641cac7b2f257b668ed26836aa09efe
[negroni] Completed 500 Internal Server Error in 117.395µs
[negroni] Started GET /queue/e7e6472b5376453372da25d2d8c533b6
[negroni] Completed 500 Internal Server Error in 51.534µs
[negroni] Started POST /volumes
[negroni] Completed 202 Accepted in 423.765µs
[asynchttp] INFO 2017/04/24 11:37:54 Started job ae9450ecfe2387fa8d4e0205b3ce648d
[heketi] INFO 2017/04/24 11:37:54 Creating volume 9e7c9b03acfe05d31e1ab3f0b687da63
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/app_volume.go:148: Failed to create volume: Error calling v.allocBricksInCluster: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:61: database is in read-only mode
[heketi] ERROR 2017/04/24 11:37:54 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:308: Error calling v.allocBricksInCluster: database is in read-only mode

i did restart couple of times

Answer 6 · 2017-04-24T20:14:07.000Z

any pointers where i can look?

Answer 7 · 2017-06-01T17:09:41.000Z

@debianmaster This comment might help you: heketi/heketi#677 (comment)

Restarting the heketi pod didn't help in our case but scaling down the heketi deployment, waiting for a few minutes, then scaling it up again fixed the underlying issue with BoltdDB file locks heketi/heketi#731 (comment)

kubectl scale --replicas=0 deploy/heketi

... wait a few minutes for the write file locks to be released, then

kubectl scale --replicas=1 deploy/heketi

Answer 8 · 2017-06-02T21:36:38.000Z

@saidimu thanks for your help, i will give a try. closing for now.

Answer 9 · 2018-06-08T09:10:30.000Z

heketi/heketi#1215 looks related