Unable to run inference service

Question

Unable to run inference service

VikasAbhishek opened this issue a year ago · 11 comments

I have followed the official documentation and deployed the serverless deployment, but after running the first inference service example. I am unable to get desired result that has shown in the page. Please find attached image here with

I followed this link for running the inference service
https://kserve.github.io/website/master/get_started/first_isvc/#5-perform-inference

VikasAbhishek commented a year ago

Resolved.

Answer 1 · 2024-04-01T14:09:31.000Z

Hi @VikasAbhishek, what the command below returns?

kubectl get isvc

Answer 2 · 2024-04-02T06:11:42.000Z

Classification: Confidential Hi Filippe, Please find attached image. From: Filippe Spolti ***@***.***> Sent: Monday, April 1, 2024 7:40 PM To: kserve/website ***@***.***> Cc: Vikas Ghunawat Meena ***@***.***>; Mention ***@***.***> Subject: Re: [kserve/website] Unable to run inference service (Issue #355) [CAUTION: This Email is from outside the Organization. Unless you trust the sender, Don't click links or open attachments as it may be a Phishing email, which can steal your Information and compromise your Computer.] Hi @VikasAbhishek<https://github.com/VikasAbhishek>, what the command below returns? kubectl get isvc - Reply to this email directly, view it on GitHub<#355 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHL3AUQVDG6GF26CPMYY6I3Y3FTDDAVCNFSM6AAAAABFRTVHB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZHAYTOOBWG4>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> ::DISCLAIMER::

…

________________________________ The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

________________________________

Answer 3 · 2024-04-02T13:37:04.000Z

there is nothing attached, just pasting the text here should be enough,.

Answer 4 · 2024-04-02T14:21:36.000Z

Classification: Confidential ***@***.***:/home/ubuntu# kubectl get isvc -n kserve-test NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE sklearn-iris http://sklearn-iris.kserve-test.svc.cluster.local<http://sklearn-iris.kserve-test.svc.cluster.local/> True 100 sklearn-iris-predictor-00003 5d6h From: Filippe Spolti ***@***.***> Sent: Tuesday, April 2, 2024 7:07 PM To: kserve/website ***@***.***> Cc: Vikas Ghunawat Meena ***@***.***>; Mention ***@***.***> Subject: Re: [kserve/website] Unable to run inference service (Issue #355) [CAUTION: This Email is from outside the Organization. Unless you trust the sender, Don't click links or open attachments as it may be a Phishing email, which can steal your Information and compromise your Computer.] there is nothing attached, just pasting the text here should be enough,. - Reply to this email directly, view it on GitHub<#355 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHL3AUXHNDP4WMWROT6GBW3Y3KYBLAVCNFSM6AAAAABFRTVHB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSGA3DQMJZGU>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> ::DISCLAIMER::

…

________________________________ The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

________________________________

Answer 5 · 2024-04-02T19:03:31.000Z

@VikasAbhishek Seems like you have not configured a domain or DNS. If you try to infer from outside the cluster it won't work

Answer 6 · 2024-04-04T06:38:49.000Z

Classification: Confidential Hi Sivanantham, Can you please guide me how to configure DNS and in which services should I make the changes. If you have any example file for the same, please share with me. Thank you, Vikas Ghunawat Meena From: Sivanantham ***@***.***> Sent: Wednesday, April 3, 2024 12:34 AM To: kserve/website ***@***.***> Cc: Vikas Ghunawat Meena ***@***.***>; Mention ***@***.***> Subject: Re: [kserve/website] Unable to run inference service (Issue #355) [CAUTION: This Email is from outside the Organization. Unless you trust the sender, Don't click links or open attachments as it may be a Phishing email, which can steal your Information and compromise your Computer.] @VikasAbhishek<https://github.com/VikasAbhishek> Seems like you have not configured a domain or DNS. If you try to infer from outside the cluster it won't work - Reply to this email directly, view it on GitHub<#355 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHL3AUQNHV5IZQ5LZM6WCC3Y3L6JTAVCNFSM6AAAAABFRTVHB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZSHA2DIMJYGQ>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> ::DISCLAIMER::

…

________________________________ The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

________________________________

Answer 7 · 2024-04-04T06:51:39.000Z

Here is the docs from knative for configuring domain. If you are just doing it for testing purposes, then you can just use this command below and you can use the kubectl port-forward to access the cluster without the need to configuring DNS.

kubectl patch cm config-domain --patch '{"data":{"example.com":""}}' -n knative-serving

If you want to configure DNS follow this doc

Answer 8 · 2024-04-04T07:19:57.000Z

Classification: Confidential ***@***.***:/home/ubuntu# kubectl logs load-testf4ps9-qznqs -n kserve-test Requests [total, rate, throughput] 30000, 500.02, 0.00 Duration [total, attack, wait] 59.999s, 59.998s, 663.663µs Latencies [min, mean, 50, 90, 95, 99, max] 427.534µs, 790.51µs, 748.113µs, 955.089µs, 1.052ms, 1.593ms, 26.421ms Bytes In [total, mean] 0, 0.00 Bytes Out [total, mean] 2460000, 82.00 Success [ratio] 0.00% Status Codes [code:count] 404:30000 Error Set: 404 Not Found I am getting these logs while running the performance test. From: Sivanantham ***@***.***> Sent: Thursday, April 4, 2024 12:22 PM To: kserve/website ***@***.***> Cc: Vikas Ghunawat Meena ***@***.***>; Mention ***@***.***> Subject: Re: [kserve/website] Unable to run inference service (Issue #355) [CAUTION: This Email is from outside the Organization. Unless you trust the sender, Don't click links or open attachments as it may be a Phishing email, which can steal your Information and compromise your Computer.] Here is the docs from knative for configuring domain<https://knative.dev/docs/serving/using-a-custom-domain/>. If you are just doing it for testing purposes, then you can just use this command below and you can use the kubectl port-forward to access the cluster without the need to configuring DNS. kubectl patch cm config-domain --patch '{"data":{"example.com":""}}' -n knative-serving If you want to configure DNS follow this doc<https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#configure-dns> - Reply to this email directly, view it on GitHub<#355 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHL3AUUXD2RMECSLURDQVV3Y3T2BBAVCNFSM6AAAAABFRTVHB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZWGMZDMNZSGU>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> ::DISCLAIMER::

…

________________________________ The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

________________________________

Answer 9 · 2024-04-04T07:56:46.000Z

Classification: Confidential ***@***.***:/home/ubuntu# curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Content-Type: application/json" http://${INGRESS_HOST}:${INGRESS_PORT}/v1/models/sklearn-iris:predict<http://$%7bINGRESS_HOST%7d:$%7bINGRESS_PORT%7d/v1/models/sklearn-iris:predict> -d @./iris-input.json * Trying 127.0.0.1:8080... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 8080 (#0)

POST /v1/models/sklearn-iris:predict HTTP/1.1 Host: sklearn-iris.kserve-test.example.com User-Agent: curl/7.68.0 Accept: */* Content-Type: application/json Content-Length: 76

* upload completely sent off: 76 out of 76 bytes * Mark bundle as not supporting multiuse < HTTP/1.1 404 Not Found < date: Thu, 04 Apr 2024 13:15:37 GMT < server: istio-envoy < x-envoy-upstream-service-time: 3 < content-length: 0 < * Connection #0 to host localhost left intact I am still getting the same issue. From: Sivanantham ***@***.***> Sent: Thursday, April 4, 2024 12:22 PM To: kserve/website ***@***.***> Cc: Vikas Ghunawat Meena ***@***.***>; Mention ***@***.***> Subject: Re: [kserve/website] Unable to run inference service (Issue #355) [CAUTION: This Email is from outside the Organization. Unless you trust the sender, Don't click links or open attachments as it may be a Phishing email, which can steal your Information and compromise your Computer.] Here is the docs from knative for configuring domain<https://knative.dev/docs/serving/using-a-custom-domain/>. If you are just doing it for testing purposes, then you can just use this command below and you can use the kubectl port-forward to access the cluster without the need to configuring DNS. kubectl patch cm config-domain --patch '{"data":{"example.com":""}}' -n knative-serving If you want to configure DNS follow this doc<https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#configure-dns> - Reply to this email directly, view it on GitHub<#355 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BHL3AUUXD2RMECSLURDQVV3Y3T2BBAVCNFSM6AAAAABFRTVHB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZWGMZDMNZSGU>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> ::DISCLAIMER::

…

________________________________ The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.

________________________________

Answer 10 · 2024-04-24T03:15:06.000Z

@VikasAbhishek

Ensure the KServe inference service (sklearn-iris) is correctly deployed and accessible within the Kubernetes cluster. The command kubectl get isvc -n kserve-test should list the service as READY.
Ensure the ingress setup and network policies allow external traffic to the service. The persistent 404 errors could indicate network misconfiguration or an incorrect URL.
lacking a proper DNS configuration could prevent external access. Using kubectl patch cm config-domain --patch '{"data":{"example.com":""}}' -n knative-serving can set a temporary domain for testing purposes.