For every two volumes brick pids getting changed after brick mux enabled on gcs setup
rmadaka opened this issue · 2 comments
rmadaka commented
Observed behavior
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| BRICK ID | HOST | PATH | ONLINE | PORT | PID |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| fbc0933b-43ca-4d15-9198-41b9006d2500 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-969368e4-09c3-11e9-b866-525400745346/subvol1/brick1/brick | true | 37232 | 2893 |
| e6cc8315-0324-4501-ac27-ae25102fb41a | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-969368e4-09c3-11e9-b866-525400745346/subvol1/brick2/brick | true | 41399 | 1364 |
| ace06c6f-f372-47a6-9b30-1fafd42bddd2 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-969368e4-09c3-11e9-b866-525400745346/subvol1/brick3/brick | true | 41897 | 1269 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-9de6404b-09c3-11e9-b866-525400745346
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| BRICK ID | HOST | PATH | ONLINE | PORT | PID |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| 1458f2fe-dbea-4502-95f1-d54749806559 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-9de6404b-09c3-11e9-b866-525400745346/subvol1/brick1/brick | true | 37232 | 2893 |
| e07add02-3c3a-421d-95b7-8c03156a295b | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-9de6404b-09c3-11e9-b866-525400745346/subvol1/brick2/brick | true | 41399 | 1364 |
| 78fc8769-356a-45aa-8759-c796e3689142 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-9de6404b-09c3-11e9-b866-525400745346/subvol1/brick3/brick | true | 41897 | 1269 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-9f5f251c-09c3-11e9-b866-525400745346
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| BRICK ID | HOST | PATH | ONLINE | PORT | PID |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| ff98ef67-502d-468b-bb48-25e0fee624e6 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-9f5f251c-09c3-11e9-b866-525400745346/subvol1/brick1/brick | true | 41453 | 3450 |
| d48b0ded-a38b-40fd-a524-fadb397d12dc | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-9f5f251c-09c3-11e9-b866-525400745346/subvol1/brick2/brick | true | 39647 | 1532 |
| 43a856a1-bd70-4cf7-b122-5d1ae0d02ee7 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-9f5f251c-09c3-11e9-b866-525400745346/subvol1/brick3/brick | true | 35099 | 1432 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-a7314692-09c3-11e9-b866-525400745346
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| BRICK ID | HOST | PATH | ONLINE | PORT | PID |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| af522db0-3adc-4577-b943-828af3f56261 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-a7314692-09c3-11e9-b866-525400745346/subvol1/brick1/brick | true | 41453 | 3450 |
| 65d9f12a-ec17-4407-9db5-44a185256c6b | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-a7314692-09c3-11e9-b866-525400745346/subvol1/brick2/brick | true | 39647 | 1532 |
| b28aa665-4718-4406-9d21-4f2088968538 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-a7314692-09c3-11e9-b866-525400745346/subvol1/brick3/brick | true | 35099 | 1432 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-a8b641e3-09c3-11e9-b866-525400745346
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| BRICK ID | HOST | PATH | ONLINE | PORT | PID |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| 95256a89-bd3f-4d64-80fc-6d1006762b7a | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-a8b641e3-09c3-11e9-b866-525400745346/subvol1/brick1/brick | true | 34844 | 4063 |
| 3592f763-935f-4636-93ec-0b6632f4d340 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-a8b641e3-09c3-11e9-b866-525400745346/subvol1/brick2/brick | true | 39396 | 1710 |
| 68d547a9-dea9-4e0f-bcd1-3b9e62c8c618 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-a8b641e3-09c3-11e9-b866-525400745346/subvol1/brick3/brick | true | 36994 | 1614 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-b030e878-09c3-11e9-b866-525400745346
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| BRICK ID | HOST | PATH | ONLINE | PORT | PID |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| b1c51881-8f3b-4134-bea6-313e048ad1e8 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-b030e878-09c3-11e9-b866-525400745346/subvol1/brick1/brick | true | 34844 | 4063 |
| 70b7adaf-0b5e-46b6-8306-d231e051fd45 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-b030e878-09c3-11e9-b866-525400745346/subvol1/brick2/brick | true | 39396 | 1710 |
| 656e1fd0-eed0-4092-8ca4-6e75090e86dc | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-b030e878-09c3-11e9-b866-525400745346/subvol1/brick3/brick | true | 36994 | 1614 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Expected/desired behavior
After brick-mux enabled , for all volumes brick pid should be same.
Details on how to reproduce (minimal and precise)
-> On Fresh gcs setup , Brick-mux enabled
-> Started creating PVCs.
-> Then logged into gd2 pod and observed volume status behaviour..
-> For every two volumes brick pids are getting changed
-> ```[root@gluster-kube1-0 /]# curl -X GET http://gluster-kube1-0:24007/v1/cluster/options | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 496 100 496 0 0 29779 0 --:--:-- --:--:-- --:--:-- 31000
[
{
"default": "50000",
"key": "cluster.op-version",
"modified": false,
"value": "50000"
},
{
"default": "50000",
"key": "cluster.max-op-version",
"modified": false,
"value": "50000"
},
{
"default": "off",
"key": "cluster.brick-multiplex",
"modified": true,
"value": "on"
},
{
"default": "250",
"key": "cluster.max-bricks-per-process",
"modified": false,
"value": "250"
},
{
"default": "off",
"key": "cluster.localtime-logging",
"modified": false,
"value": "off"
},
{
"default": "off",
"key": "cluster.shared-storage",
"modified": false,
"value": "off"
}
]
### Information about the environment:
- Glusterd2 version used (e.g. v4.1.0 or master): glusterd version: v6.0-dev.93.gitd4ca2cb
- Operating system used:
- Glusterd2 compiled from sources, as a package (rpm/deb), or container: container
- Using External ETCD: (yes/no, if yes ETCD version):
- If container, which container image:
- Using kubernetes, openshift, or direct install: kubernetes
- If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside: inside
[glusterd2.log](https://github.com/gluster/glusterd2/files/2712569/glusterd2.log)
Note: Attached GD2 pod logs
### Other useful information
- glusterd2 config files from all nodes (default /etc/glusterd2/glusterd2.toml)
- glusterd2 log files from all nodes (default /var/log/glusterd2/glusterd2.log)
- ETCD configuration
- Contents of `uuid.toml` from all nodes (default /var/lib/glusterd2/uuid.toml)
- Output of `statedump` from any one of the node
### Useful commands
- To get glusterd2 version
```
glusterd2 --version
```
- To get ETCD version
```
etcd --version
```
- To get output of statedump
```
curl http://glusterd2-IP:glusterd2-Port/statedump
```
vpandey-RH commented
@rmadaka Can I get output of volume info for all these volumes ?
atinmu commented
Not reproducible with latest master. Closing it. If you happen to reproduce it again, please reopen.