[feature request] Integrating pod logs with SLS

[llinvokerl]

What would you like to be added:
I'd like to propose a new feature: integrating pod logs with Simple Logging Service (SLS) when configured by the user.

Here's the rationale:
In my current workflow, after a job is completed in kubedl, the associated pods are marked as 'Completed' but not deleted. This can lead to a scenario where large log files eventually fill up the disk space, which seems suboptimal.

My question is, given that system events can be stored through SLS configuration, why are pod logs not recorded? Capturing pod logs would allow for the deletion of pods post-job completion, addressing the potential disk space issue. Moreover, it appears that with SLS configured, users might lose access to any pod logs, which could be problematic. (I haven't verified this myself, but that's the impression I get from the code.)

I envision the pod log recording process as follows:

step1: When a pod status changes to 'Completed', the kubedl manager should immediately collect and write logs to SLS, record the pod details in a MySQL table, and then delete the pod.
step2: For log viewing in the console: 1. If the pod is running, fetch logs via apiserver; 2. If the pod is deleted (completed), fetch logs via SLS.
step3: For viewing the task instance list in the console: 1. If the pod is running, fetch the pod list via apiserver; 2. If the pod is deleted (completed), obtain the pod list from MySQL.

Why is this needed:
This approach should mitigate the issue of logs consuming excessive disk space on the node and solve the bug where logs are inaccessible post-SLS configuration.

I'm open to the possibility that my understanding of kubedl may not be fully comprehensive, so please correct me if I'm mistaken. Thanks for considering my feature request.

[SimonCqk]

@llinvokerl hi llinvokerl, would you contribute the proposed feature to the community? If so, I will assign it to you :)