Extend Health model with error cause

Question

Extend Health model with error cause

runebarikmo opened this issue 5 years ago · 10 comments

Hi,

Thanks for a great library.

I was wondering if we could extend the Health model to capture the error cause when the health check fails?

I can use Tagged to annotate the check with static metadata, but I don't think there is a way to capture the result of the health check, is it?

I'm currently looking at publishing the results of the health checks into a monitoring system which
accepts:

error code
error message
severity (Minor,Major,Critical)

Would it make sense to extend the Health ADT with more Sick types?

For example:

sealed trait Health extends Product with Serializable

case object Healthy extends Health

sealed trait Sick extends Health
case object Sick extends Sick
final case class SickWithErrorMessage(message: String) extends Sick
final case class SickWithThrowable(cause: Throwable) extends Sick

val fromErrorMessage: String => Health = SickWithErrorMessage
val fromThrowable: Throwable => Health = SickWithThrowable

// mods

def recoverToSickWithErrorMessage[F[_], H[_]: Applicative, E](f: E => String)(implicit F: ApplicativeError[F, E]): HealthCheckEndoMod[F, H] =
  _.transform {
    _.handleError(e => HealthResult.const[H](Health.fromErrorMessage(f(e))))
  }

def recoverToSickWithThrowable[F[_], H[_]: Applicative](implicit F: ApplicativeError[F, Throwable]): HealthCheckEndoMod[F, H] =
  _.transform {
    _.handleError(e => HealthResult.const[H](Health.fromThrowable(e)))
  }


// redis ping test (capture error text during recovery)

def pingCheckWithErrorHandler[F[_], E](handleError: E => String)(implicit cmd: Ping[F], F: ApplicativeError[F, E]): HealthCheck[F, Id] =
  HealthCheck
    .liftF {
      cmd.ping.as(HealthResult.one(Health.healthy))
    }
    .through(sup.mods.recoverToSickWithErrorMessage(handleError))


// call test
//...
val errorToString: String => String = identity
val healthCheck = modules.redis.pingCheckWithErrorHandler(errorToString)

healthCheck.check shouldBe Right(HealthResult.one(Health.SickWithErrorMessage("error")))

Not sure if this is the best way of modelling it, but it would be nice to have a richer Sick model.

What do you think? I can work on a PR if you are interested.

Answer 1 · 2020-05-19T20:37:45.000Z

Thanks for the issue!

In general I agree that it's worth to have this kind of information when your checks fail, but I'm not sure sup is meant to be used that way. I originally meant it to be a library that exposes the information so that other programs can ask for it (and not pass that data somewhere for other programs to interpret it).

I also wouldn't recommend to couple your systems to the representation used by Sup, as that could change (it only has to stay consistent with other sup modules, so that e.g. remote checks can be bridged together).

You can actually do this kind of thing already by tagging the result with the appropriate information. It's not perfect (e.g. you won't have a type-level guarantee of consistency in having errors <=> Sick, or having no errors <=> Healthy), but we can definitely improve on that. Maybe by making Tagged a monad (if it's possible) and using EitherT we could get rid of that issue altogether. It could be interesting :)

So back to your original problem - if we can get the kind of information you need in a Tagged (or some other wrapper that could have a place in sup), you should be able to convert from that to a data structure you can send to the monitoring system.

Answer 2 · 2020-05-19T21:54:39.000Z

Thanks for your answer

It's probably a good idea to keep the model simple.

Like you say, I can already use tagging to store the error information, but I won't have the type system guarantees that you mention. I can work with this for now, but yes it would be nice if we could find a better solution for this :) I will think about it some more.

Answer 3 · 2020-05-20T01:04:10.000Z

Here's something that kind of seemed to work...

sealed trait TaggedT[+L, +R] extends Product with Serializable {

  def fold[A](incorrect: L => A, correct: => A): A = this match {
    case Incorrect(error, _) => incorrect(error)
    case Correct(_)          => correct
  }

  def toEither: Either[L, Unit] = fold(Left(_), Right(()))
}

object TaggedT {
  final case class Incorrect[L, R](error: L, value: R) extends TaggedT[L, R]
  final case class Correct[R](value: R) extends TaggedT[Nothing, R]

  def incorrect[L](error: L): TaggedT[L, Health] = Incorrect(error, Health.Sick)
  val correct: TaggedT[Nothing, Health] = Correct(Health.Healthy)

  implicit def catsReducibleForTaggedT[L]: Reducible[TaggedT[L, *]] =
    new Reducible[TaggedT[L, *]] {

      def foldLeft[A, B](fa: TaggedT[L, A], b: B)(f: (B, A) => B): B =
        fa match {
          case Incorrect(_, a) => f(b, a)
          case Correct(a)      => f(b, a)
        }

      def foldRight[A, B](fa: TaggedT[L, A], lb: Eval[B])(f: (A, Eval[B]) => Eval[B]): Eval[B] = fa match {
        case Incorrect(_, a) => f(a, lb)
        case Correct(a)      => f(a, lb)
      }

      def reduceLeftTo[A, B](fa: TaggedT[L, A])(f: A => B)(g: (B, A) => B): B =
        fa match {
          case Incorrect(_, a) => f(a)
          case Correct(a)      => f(a)
        }

      def reduceRightTo[A, B](fa: TaggedT[L, A])(f: A => B)(g: (A, Eval[B]) => Eval[B]): Eval[B] = fa match {
        case Incorrect(_, a) => Eval.later(f(a))
        case Correct(a)      => Eval.later(f(a))
      }

    }

}

object Demo extends App {
  final case class MyErr(msg: String)
  implicit val showMyErr: Show[MyErr] = e => show"MyErr(${e.msg})"

  def demoL: TaggedT[MyErr, Health] = TaggedT.incorrect(MyErr("woops"))
  def demoR: TaggedT[MyErr, Health] = TaggedT.correct

  implicit def show[A: Show, B]: Show[TaggedT[A, B]] = t => t.fold("Sick: " + _.show, "Healthy")
  println(demoL.reduce) //Sick
  println(demoR.reduce) //Healthy
  println(demoL.show) //Sick: MyErr(woops)
  println(demoR.show) //Healthy

  val hc1 = HealthCheck.liftF(IO(HealthResult(demoL)))
  val hc2 = HealthCheck.liftF(IO(HealthResult(demoR)))

  val reporter = HealthReporter.fromChecks(hc1, hc2)
  println(reporter.check.unsafeRunSync().value.health) //Sick
  println(reporter.check.unsafeRunSync().value.checks.map(_.toEither)) //NonEmptyList(Left(MyErr(woops)), Right(()))
}

Something roughly like this. I don't know if it's a lawful reducible, though, so there's some further work required. But it works quite nicely :)

Basically we "smuggle" a value of the right type internally, but don't expose that to the user in the methods (only in the pattern match, if they really want it...).

In the methods, we only provide a fold (and an equivalent toEither), in which you either get the error (and the error alone) or nothing (unit, or whatever you pass to the second parameter of fold). So in the public-facing API you have the consistency guarantees.

Answer 4 · 2020-05-20T01:13:55.000Z

Okay, looks like that breaks * Reducible[TaggedT[String, ?]].reducible.ordered constistency...

(yeah, there's a typo in the law)

I'll think more about this.

Edit: Apparently, it was just a flaw in the definition of equality (fold doesn't work as it ditches the value passed at construction). I don't think that'll be a problem... and using the right definition made it pass.

WIP branch at https://github.com/kubukoz/sup/tree/wip-taggedt

Answer 5 · 2020-05-20T06:39:28.000Z

Awesome :) This looks really nice. Thanks.
Good that the new type its lawful as well.

I will play around with the wip branch version and start using TaggedT in my health checks.

Answer 6 · 2020-05-21T19:00:26.000Z

After playing around with this for a while I realize that it's sufficient to use the
standard Tagged type. All I want to do is to capture the result of the health check regardless
if the check fails or succeeds. That's what the H[_] container in HealthResult lets you do which is really awesome.

Here is a rewritten version of the DB health check where I define my own type to capture the results.
This achieves exactly what I want :)

Thanks again for your help.

object demo extends IOApp {

  // application specific tag type
  final case class HealthCheckId(value: String) extends AnyVal
  final case class ApplicationHealthCheck[R](id: HealthCheckId, result: R)
  object ApplicationHealthCheck {
    def taggerForId[R](id: HealthCheckId): R => ApplicationHealthCheck[R] =
      ApplicationHealthCheck(id,_)
  }

  override def run(args: List[String]): IO[ExitCode] = {
    // DB healthcheck
    def connectionCheck[F[_]: Bracket[*[_], Throwable]](
      xa: Transactor[F]
    )(
      timeout: Option[FiniteDuration]
    ): HealthCheck[F,Tagged[ApplicationHealthCheck[String],*]] = {
      val actualTimeoutSeconds = timeout.foldMap(_.toSeconds.toInt)
      val tagger = ApplicationHealthCheck.taggerForId[String](HealthCheckId("Db"))
      val tagSuccess = tagger.map(Tagged(_,Health.healthy)).map(HealthResult(_))
      val tagError = tagger.map(Tagged(_,Health.sick)).map(HealthResult(_))

      HealthCheck.liftF {
        FC.isValid(actualTimeoutSeconds)
          .transact(xa)
          .as(tagSuccess("Ok"))
          .handleError(err => tagError(err.getMessage))
      }
    }

    // run health check
    def goodTransactor[F[_]: Async: ContextShift]: Transactor[F] =
      Transactor.fromDriverManager[F]("org.h2.Driver", "jdbc:h2:mem:")

    def badTransactor[F[_]: Async: ContextShift]: Transactor[F] =
      Transactor.fromDriverManager[F]("org.h2.Driver", "blabla")

    val hc1 = connectionCheck[IO](badTransactor)(5.seconds.some)
    val hc2 = connectionCheck[IO](goodTransactor)(5.seconds.some)

    val reporter = HealthReporter.fromChecks(hc1,hc2)

    val printResults =
      reporter.check.map(_.value.health).flatMap(health => putStrLn(s"health: ${health}")) *>
        reporter.check.map(_.value.checks).flatMap(checks => putStrLn(s"results: ${checks}"))

    printResults.as(ExitCode.Success)
    // Output:
    // health: Sick
    // results: NonEmptyList(Tagged(ApplicationHealthCheck(HealthCheckId(Db),No suitable driver found for blabla),Sick), Tagged(ApplicationHealthCheck(HealthCheckId(Db),Ok),Healthy))
  }
}

Answer 7 · 2020-05-21T19:59:09.000Z

Awesome! Glad you got it to work.

btw. You can make the code highlight as Scala if you add scala after the three `-quotes.

Answer 8 · 2020-05-21T20:11:28.000Z

Ah yes. Will remember next time. Cheers :)

Answer 9 · 2020-11-28T05:45:40.000Z

just curious, @runebarikmo do you use sup in production?

Answer 10 · 2020-11-28T10:13:57.000Z

just curious, @runebarikmo do you use sup in production?

I'm running sup in a test environment at the moment, but anticipate it will go into a production eventually. It's working well :)