cann't get event log from k8s deployment
wang2024666 opened this issue · 7 comments
hi all,
we deployed Exceptionless on self-hosted K8S, with helm chart from relese v8.15. All pods are running now and we changed the service type to NodePort. Tried to post the API port and it returned 202:
post http://10.122.90.50:31038/api/v2/events
but we can't get any log from APP page。
we checked the logs od API pod, it seems no log item releated to the api event.
any suggestion how can we troubleshoot it now?
api pods just process incoming requests. The event post job is what processes the saved incoming event payload.
api pods just process incoming requests. The event post job is what processes the saved incoming event payload.
the post job log shows:
[02:50:00 ERR] Job run EventPostsJob failed: Unable to retrieve payload 'q/4ad/4ad9c2a9d7ac492bb9c531ec0739918e.payload'.
[02:50:00 INF] Processing queue entry: id=933d7c6d33204d7c932e8c63945b45a9 type=EventPost attempt=2
[02:50:00 ERR] Unable to get file stream for q/472/4726b1ab644f468e8695399d375d368c.payload: Could not find a part of the path '/app/storage/q/472/4726b1ab644f468e8695399d375d368c.payload'.
System.IO.DirectoryNotFoundException: Could not find a part of the path '/app/storage/q/472/4726b1ab644f468e8695399d375d368c.payload'.
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirError)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode, Boolean failForSymlink, Boolean& wasSymlink, Func4 createOpenException) at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func4 createOpenException)
at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
at System.IO.File.OpenRead(String path)
at Foundatio.Storage.FolderFileStorage.GetFileStreamAsync(String path, CancellationToken cancellationToken) in /_/src/Foundatio/Storage/FolderFileStorage.cs:line 54
[02:50:00 INF] Item: 933d7c6d33204d7c932e8c63945b45a9, Retry attempts: 1, Retries Allowed: 1, Retry Delay: 0:03:00
[02:50:00 INF] Exceeded retry limit moving to deadletter: 933d7c6d33204d7c932e8c63945b45a9
[02:50:00 INF] Abandon complete: 933d7c6d33204d7c932e8c63945b45a9
[02:50:00 ERR] Job run EventPostsJob failed: Unable to retrieve payload 'q/472/4726b1ab644f468e8695399d375d368c.payload'.
You have to call the api for submission, the EventPostsJob picks up the saved submission to process it. Sounds like you need to configure the storage directory to point to the same folder.