PrefectHQ/prefect

3.1.8 zombie crash automation doesn't work and deployed flows don't generate heartbeats

Opened this issue · 1 comments

Bug summary

I followed the instruction to enable automation for crashing zombie flows.
When I tried it on flow that was deployed using .serve(), it generated heartbeats, and when I tried deploying same flow using prefect --no-prompt deploy --prefect-file prefect.yaml -n Zombie, it didn't
image
image

But even with flow that was generating heartbeats successfully, automation from the same instruction didn't work. My procedure was:

  1. Serve the flow
  2. Run it
  3. Wait for the first heartbeat to happen
  4. Stop serving process.

Result: flow continues to run as if automation wasn't there.

Screenshots:
image
image
image

Setup description:

Flow code:

import time

from prefect import flow, task, get_run_logger

@task
def task_a():
    logger = get_run_logger()
    logger.info("Hello")
    time.sleep(5)

@flow
def flow_a():
    while True:
        task_a.submit().result()

if __name__ == "__main__":
    flow_a.serve()

prefect.yaml:

name: Zombie

work-queue-default: &default
  work_pool:
    name: work_pool_1
    work_queue_name: default
    job_variables:
      stream_output: true


deployments:
- name: Zombie
  entrypoint: zombie.py:flow_a
  <<: *default

prefect config view:

🚀 you are connected to:
http://127.0.0.1:4200
PREFECT_PROFILE='default'
PREFECT_API_KEY='********' (from profile)
PREFECT_API_URL='http://127.0.0.1:4200/api' (from profile)
PREFECT_LOCAL_STORAGE_PATH='C:\Users\ihor.ramskyi\PycharmProjects\Prefect_Tutorial\local_storage' (from profile)
PREFECT_RUNNER_HEARTBEAT_FREQUENCY='30' (from profile)

Worker was started using command prefect worker start -p work_pool_1 -t process -q default
Flow was deployed using command prefect --no-prompt deploy --prefect-file prefect.yaml -n Zombie

Version info

Version:             3.1.8
API version:         0.8.4
Python version:      3.10.11
Git commit:          53a83ebc
Built:               Tue, Dec 17, 2024 10:20 AM
OS/Arch:             win32/AMD64
Profile:             default
Server type:         server
Pydantic version:    2.7.1

Additional context

No response

Thanks for the issue @ihor-ramskyi-globallogic! My hunch is that the heartbeat setting is getting set in your execution environment. I think the easiest way to do that is to add a prefect.toml file in the root of your project that looks like this:

runner.heartbeat_frequency = 30

Once that file is created, you can redeploy your flow, and you should see heartbeats coming from flow runs for that deployment.

Let me know if you run into any more issues with this! If creating a prefect.toml works for you, we should update the docs to include that step.