openfaas/faasd

Functions are scaled to zero after a reboot

wimdecorte opened this issue · 3 comments

wimdecorte commented

Due diligence

My actions before raising this issue

Before you ask for help or support, make sure that you've consulted the manual for faasd. We can't answer questions that are already covered by the manual.
Done. Bought the book, read the deployment and troubleshooting sections, inspected the various logs.

Why do you need this?

I want to make sure my functions are available when the RPI4 boots.

Expected Behaviour

On startup, the functions should go to the ready state.

Current Behaviour

Instead they remain "not ready" until I remove them and redeploy them

Are you a GitHub Sponsor (Yes/No?)

One-time $65 contribution.

Check at: https://github.com/sponsors/openfaas

  • [ x] Yes
  • No

List All Possible Solutions and Workarounds

N/A

Which Solution Do You Recommend?

N/A

Steps to Reproduce (for bugs)

  1. N/A

Your Environment

  • OS and architecture:
    RPI4 8GB
  • Versions:
    Ubuntu 22 LTS server arm64
go version
ubuntu@u22faasd:~$ go version
Command 'go' not found, but can be installed with:

containerd -version
ubuntu@u22faasd:~$ sudo containerd -version
containerd github.com/containerd/containerd v1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16

uname -a
ubuntu@u22faasd:~$ uname -a
Linux u22faasd 5.15.0-1013-raspi #15-Ubuntu SMP PREEMPT Mon Aug 8 06:33:06 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

cat /etc/os-release
ubuntu@u22faasd:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

faasd version
ubuntu@u22faasd:~$ faasd version
  __                     _ 
 / _| __ _  __ _ ___  __| |
| |_ / _` |/ _` / __|/ _` |
|  _| (_| | (_| \__ \ (_| |
|_|  \__,_|\__,_|___/\__,_|

faasd version: 0.16.2   commit: b7be42e5ec47bc9a52eb3459b0f3084d61c55e58

Perhaps relevant, from the various logs:

ubuntu@u22faasd:~$ faas-cli describe xml-to-json
Name: xml-to-json
Status: Not Ready
Replicas: 0
Available Replicas: 0
Invocations: 0
Image: docker.io/wimdecorte/xml-to-json:latest
Function Process:
URL: http://127.0.0.1:8080/function/xml-to-json
Async URL: http://127.0.0.1:8080/async-function/xml-to-json
Environment:
NODE_VERSION: 16.17.0
YARN_VERSION: 1.22.19
write_debug: false

ubuntu@u22faasd:~$ faas-cli logs xml-to-json
Cannot connect to OpenFaaS on URL: http://127.0.0.1:8080

ubuntu@u22faasd:~$ sudo journalctl -t openfaas-fn:xml-to-json
Aug 17 21:33:22 ubuntu openfaas-fn:xml-to-json[2288]: xml-to-json app listening at http://localhost:8080
-- Boot 4c9f71a726c24640ad2a90dbd9a66108 --
Aug 18 21:15:46 u22faasd openfaas-fn:xml-to-json[1829]: xml-to-json app listening at http://localhost:8080

--> but the RPI was restarted on Aug 22

ubuntu@u22faasd:~$ sudo service faasd status
● faasd.service - faasd
Loaded: loaded (/lib/systemd/system/faasd.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-08-21 15:49:55 UTC; 22h ago
Main PID: 669 (faasd)
Tasks: 9 (limit: 9237)
Memory: 27.5M (limit: 500.0M)
CPU: 3.995s
CGroup: /system.slice/faasd.service
└─669 /usr/local/bin/faasd up

Aug 22 12:43:55 u22faasd faasd[669]: 2022/08/22 12:43:55 Resolver: "faasd-provider"="10.62.0.1"
Aug 22 12:43:55 u22faasd faasd[669]: 2022/08/22 12:43:55 Resolver: "basic-auth-plugin"="10.62.0.2"
Aug 22 12:43:55 u22faasd faasd[669]: 2022/08/22 12:43:55 Resolver: "nats"="10.62.0.3"
Aug 22 12:43:55 u22faasd faasd[669]: 2022/08/22 12:43:55 Resolver: "prometheus"="10.62.0.4"
Aug 22 12:43:55 u22faasd faasd[669]: 2022/08/22 12:43:55 Resolver: "gateway"="10.62.0.5"
Aug 22 12:43:55 u22faasd faasd[669]: 2022/08/22 12:43:55 Resolver: "queue-worker"="10.62.0.6"
Aug 22 12:43:55 u22faasd faasd[669]: 2022/08/22 12:43:55 Looking up IP for: "gateway"
Aug 22 12:43:55 u22faasd faasd[669]: 2022/08/22 12:43:55 faasd: waiting for SIGTERM or SIGINT
Aug 22 12:43:55 u22faasd faasd[669]: 2022/08/22 12:43:55 Proxy from: 0.0.0.0:8080, to: gateway:8080 (10.62.0.5)
Aug 22 12:43:56 u22faasd faasd[669]: 2022/08/22 12:43:56 Proxy from: 127.0.0.1:9090, to: prometheus:9090 (10.62.0.4)

Attached screenshot shows the openfaas UI is up, but the function is not.
2022-08-22_10-45-20

alexellis commented

Hi @wimdecorte thanks for the one-time sponsorship.

This behaviour is working as designed, upon reboot, all functions are scaled to zero and will be scaled up in time for the first invocation that they receive.

This should not cause you any issues that I'm aware of? Have you tried visiting the URL as displayed in the URL field in the UI?

Some people have added a /health/ endpoint to their functions and invoke them with a cron task of @reboot /home/pi/warm-up-functions.sh

Alex

alexellis commented

/set title: Functions are scaled to zero after a reboot

vnourdin commented

Hello @alexellis, I have an error 500 Can't reach service for: my-fun.namespace. on first function's call when they are scaled to 0, is this normal ?
I understand why we scale to zero at reboot, but a 500 to the end user is a problem for me.
What I would have expected is to scale up the function to one and then respond.
Thanks !