Permission denied to persist data to Docker named volumes

[rusbob]

Given Docker compose file

docker-compose.yml:

version: '3.5'

services:
    grafana:
        image: grafana/grafana:5.2.1
        volumes:
            - "./grafana/:/etc/grafana/provisioning"
            - "grafana_data:/var/lib/grafana"
        environment:
            - "GF_AUTH_ANONYMOUS_ENABLED=true"
            - "GF_USERS_ALLOW_SIGN_UP=false"
        ports:
            - "3000:3000"
        networks:
            - monitoring

volumes:
    grafana_data: {}

networks:
    monitoring:
        driver: overlay

Run above docker compose:

$ docker stack deploy --prune -c docker-compose.yml metrics

Fail to create grafana playlist to persist data:

Inspect used volume:

$ docker volume ls                   
DRIVER              VOLUME NAME
local               metrics_grafana_data

$ docker inspect metrics_grafana_data
[
    {
        "CreatedAt": "2018-07-24T14:28:58+03:00",
        "Driver": "local",
        "Labels": {
            "com.docker.stack.namespace": "metrics"
        },
        "Mountpoint": "/var/lib/docker/volumes/metrics_grafana_data/_data",
        "Name": "metrics_grafana_data",
        "Options": null,
        "Scope": "local"
    }
]

List docker host rights on named volumes:

$ ls -l /var/lib/docker/volumes
drwxr-xr-x 3 root root  4096 Jul 23 17:57 metrics_grafana_data

List grafana container volume rights:

# ls -l /var/lib/grafana
total 412
-rw-r--r-- 1 grafana grafana 409600 Jul 24 11:28 grafana.db
drwxrwxrwx 7 grafana grafana   4096 Jul 23 14:57 plugins
drwx------ 3 grafana grafana   4096 Jul 24 06:57 sessions

Docker version:

Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: hlnrrxvpqhcrc2lxdz86ibiuq
 Is Manager: true
 ClusterID: yr0dl4x3fka3ba2f0q77i7bi9
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
...
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-23-generic
Operating System: Ubuntu 18.04 LTS
OSType: linux
Architecture: x86_64
...
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
...
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[xlson]

The development of the docker image for Grafana has been moved to the main Grafana repo. This issue can be tracked here: grafana/grafana#13188