getindata/flink-http-connector

flink task not fail when HTTP connection not available

shmilygkd opened this issue · 4 comments

shmilygkd commented

As the title suggests, when the http service configured in the http sink is launched after starting the flink app, the flink application will not try to restart or fail because the connection is rejected or the connection is unavailable. Can I get a reasonable explanation? thanks.

kristoffSC commented

In other words,
if we got HTTP error code or HTTP connection was disconnected (network issues etc) or the endpoint is unavailable - in that case flink job is not restarted right?

And you are asking whether this is a feature or a bug correct? :)

shmilygkd commented

Yes, the flink task does not seem to have any perception on the surface, but the downstream http can no longer receive data， i'm not sure if this is a bug.

kristoffSC commented

If I remember correctly this is by design (so not a bug :) )

The reasons for this were:

  1. end client requirerments for which we start initial implementation
  2. generally speaking, http endpoint can be unstable. So its a matter of decision - if endpoint is flaky do you want your job to be flaky.

The second point is missing two things to be perfect:

  1. option to choose - whether job should be restarted or not if connection fails or something like that.
  2. retry mechanism. Currently we dont have any retry mechanism (its on TODO list). With retry, the "flaky" nature of the endpoint could be mitigated.

What you have however is a metric of failed requestest. So you can monitor this and react up on its value.
There is a test that covers "FailledConenction" HttpSinkConnectionTest:testFailedConnection

shmilygkd commented

Thank you for your patient reply. Let me think about it again~