Reconciles take a long time

[mook-as]

Describe the bug
Apologies for the really vague bug report.

We've been noticing that sometimes resources will take a long time to upgrade. Unfortunately we are not sure yet under what circumstances this occurs.

I've seen this with QuarksStatefulSet not being regenerated, as well as QuarkJobs not creating the Job when triggered.

To Reproduce
(As an example):

• Deploy a QuarksJob
• Trigger it by setting the strategy to now.
• Notice that the Job isn't created for a long time

Expected behaviour

• The Job (or whatever) is updated soon-ish.

Actual behaviour

• The Job (or whatever) is only triggered after a long time (possibly on the order of an hour).

Environment

• CF-Operator Version: v6.1.10-0.g7b1a5f2f
• KubeCF 2.5.4.
• minikube

Additional context

• Deleting the controller (e.g. the cf-operator / quarks-job / quarks-secret pod) and letting it restart seems to (at least some of the time) make things happen faster.
• Will attempt to fetch logs the next time this happens.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175135832

The labels on this github issue will be updated when the story is started.

[manno]

There was a bug with updating the status.