Plugin failure after Upgrading to Marathon 1.4.0

[karthimohan]

According to Marathon Release notes. Looks like there is a breaking Plugin API change in this version. We currently heavily rely on this plugin for authorization.
https://github.com/mesosphere/marathon/releases/tag/v1.4.0
Currently I use the latest version of ldap plugin.
I think the main think it broke is /v2/tasks endpoint

curl -X GET "http://xxx:xxx@localhost:8080/v2/tasks"
{"tasks":[]}

But i have n number tasks running. If i disable the Plugin feature in Marathon and query /v2/tasks endpoint, i do see the running tasks.

@gondor Is there a plan / timeline to add support for Marathon 1.4+

[erickrmz]

Sharing a bit on how we faced this issue.

We were working with Marathon 1.3.9 using the marathon-ldap plugin (latest), and using Traefik-Marathon integration as well. Everything was working fine until yesterday when we decided to upgrade to Marathon 1.4.2 to get the latest goodies. We thought the Marathon upgrade went fine, but our Traefik integration broke (we could no longer see any backends).

In Traefik logs we could see:

time="2017-04-12T15:39:10-07:00" level=debug msg="Configuration received from provider marathon: {}"

After inspecting different Marathon endpoints, we noticed that we would get the expected hash from calling /v2/apps, however when we tried to access /v2/tasks we noticed the result was an empty hash. We tried different things to work around this, when we tried disabling the plugins (we were using marathon-ldap), we started to get the expected results from /v2/tasks, and Traefik started working again

With plugins disabled

root 29061 27832 1 Apr12 ? 00:09:31 java -Djava.library.path=/usr/local/lib:/usr/lib:/usr/lib64 -Djava.util.logging.SimpleFormatter.format=%2$s%5$s%6$s%n -server -Djava.awt.headless=true -Xms2048M -Xmx2048M -Xss256M -Xmn512M -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Dlogback.configurationFile=/etc/marathon/logback.xml -cp /tmp/marathon-5f0afe9cf782f22f11c81c848d23ca70d59fcbda.jar mesosphere.marathon.Main --enable_features task_killing --ssl_keystore_path /etc/marathon/hb.org.jks --mesos_authentication --mesos_authentication_principal mt --mesos_authentication_secret_file /etc/maratho/marathon_mesos_principal_secret_file --framework_name marathon --event_subscriber http_callback --mesos_role marathon --zk_max_versions 50 --max_tasks_per_offer 5 --http_max_concurrent_requests 1000 --logging_level info --master zk://10.20.0.201:2181,10.20.10.201:2181,10.20.20.201:2181/mesos --ssl_keystore_password xxxxx

$ curl localhost:8080/v2/tasks {"tasks":[{"ipAddresses":[{"ipAddress":"172.17.0.12","protocol":"IPv4"}],"stagedAt":"2017-04-13T05:02:45.387Z","state":"TASK_RUNNING",.... .... ...
With plugins enabled

root 29061 27832 1 Apr12 ? 00:09:31 java -Djava.library.path=/usr/local/lib:/usr/lib:/usr/lib64 -Djava.util.logging.SimpleFormatter.format=%2$s%5$s%6$s%n -server -Djava.awt.headless=true -Xms2048M -Xmx2048M -Xss256M -Xmn512M -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Dlogback.configurationFile=/etc/marathon/logback.xml -cp /tmp/marathon-5f0afe9cf782f22f11c81c848d23ca70d59fcbda.jar mesosphere.marathon.Main --enable_features task_killing --ssl_keystore_path /etc/marathon/hb.org.jks --mesos_authentication --mesos_authentication_principal mt --mesos_authentication_secret_file /etc/maratho/marathon_mesos_principal_secret_file --plugin_dir /etc/marathon/plugins --plugin_conf /etc/marathon/plugins/plugin_conf.json --framework_name marathon --event_subscriber http_callback --mesos_role marathon --zk_max_versions 50 --max_tasks_per_offer 5 --http_max_concurrent_requests 1000 --logging_level info --master zk://10.20.0.201:2181,10.20.10.201:2181,10.20.20.201:2181/mesos --ssl_keystore_password xxxxx

$ curl http://XXXX:XXXX@localhost:8080/v2/tasks {"tasks":[]}

Plugin configuration

# cat /etc/marathon/plugins/plugin_conf.json { "plugins": { "authorization": { "plugin": "mesosphere.marathon.plugin.auth.Authorizer", "implementation": "io.containx.marathon.plugin.auth.LDAPAuthorizor" }, "authentication": { "plugin": "mesosphere.marathon.plugin.auth.Authenticator", "implementation": "io.containx.marathon.plugin.auth.LDAPAuthenticator", "configuration": { "users": [ { "username": "XXXX", "password": "XXXX", "groups": [ "Admin" ] }, { "username": "guest", "password": "XXXX", "groups": [ "Developers" ] } ], "authorization": { "access": [ { "group": "Admin", "permissions": [ { "allowed": "*", "type" : "app" }, { "allowed": "*", "type" : "group" } ] }, { "group": "Developers", "permissions": [ { "allowed": "view", "type": "app", "path": "/" }, { "allowed": "view", "type": "group", "path": "/" } ] } ] } } } } }

[karthimohan]

@gondor sorry to bug you. will you be able to take a look at this issue? If not, we might need to consider other options.

[minyk]

Just to clarify, after Marathon 1.4.4 this issue should be addressed. See this: https://github.com/mesosphere/marathon/releases/tag/v1.4.4