Azure/aks-periscope

aks kollect does not work with AKS with disabled local accounts

Opened this issue · 5 comments

Superzer0 commented

Describe the bug
We have AKS (1.24.6) configured in a way that disables local account access (best security practice).
Currently, our AKS is configured with Azure AD authentication with Kubernetes RBAC.

To Reproduce
Steps to reproduce the behavior: (for example)

  1. Execute az aks kollect -g *** -n *** --storage-account "***"
  2. See the error:
    (BadRequest) Getting static credential is not allowed because this cluster is set to disable local accounts. Code: BadRequest Message: Getting static credential is not allowed because this cluster is set to disable local accounts.

Expected behavior
AKS periscope runs

Desktop:

  • OS: Windows 11
  • Powershell Core 7.2.7
  • { "azure-cli": "2.42.0", "azure-cli-core": "2.42.0", "azure-cli-telemetry": "1.0.8", "extensions": { "aks-preview": "0.5.114", "azure-devops": "0.25.0", "interactive": "0.4.5" } }
peterbom commented

Hi @Superzer0 - thanks for reporting this issue!

The problem seems to be with az CLI attempting to get (local account) admin credentials and using these to deploy resources to the cluster. We'll have to discuss further whether there's something we can do to change that in an upcoming release of the CLI. cc: @Tatsinnit

Right now, the only way I can see to run Periscope is to deploy the resources manually, as described here. I gave this a try and it worked for me.

Note that there are a few placeholders that need filling in, including AZURE_BLOB_SAS_KEY which is the SAS token for the storage account where logs will be uploaded. You can create a SAS using something like:

sub_id=...
stg_account=...
blob_container=...
sas_expiry=`date -u -d "30 minutes" '+%Y-%m-%dT%H:%MZ'`
sas=$(az storage account generate-sas \
    --account-name $stg_account \
    --subscription $sub_id \
    --permissions rwdlacup \
    --services b \
    --resource-types sco \
    --expiry $sas_expiry \
    -o tsv)

Also note that the value for AZURE_BLOB_SAS_KEY must be preceded by a ? character.

Would this approach work for you, at least in the meantime?

  • 🎉1
  • ❤️2
Superzer0 commented

Thank you @peterbom, I will try the manual approach. It would be great to have that option automated in CLI as it simplifies the troubleshooting. Thanks!
Could we leave the issue open, until we have feedback on whether this could be implemented in AZ CLI?

  • 👍2
peterbom commented

FYI @Superzer0: Azure/azure-cli-extensions#5539

I'll keep this issue open until the CLI is released with the above change.

(We're also working on a similar change for the VSCode Periscope integration: Azure/vscode-aks-tools#173)

peterbom commented

Hi @Superzer0, the latest aks-preview extension (v. 0.5.116) for az CLI contains the changes to enable running kollect on AAD-enabled clusters with local accounts disabled. You can run az upgrade to try that.

It's also enabled in the latest release of the AKS extension for VS Code, in case it's convenient for you to run Periscope from there.

Superzer0 commented

Hello, sorry for the delay. The periscope worked for us and we have the diagnostics uploaded to the storage account (aks-preview 0.5.137).
However, the kanalyze command is always returning no results (when using default storage account linked via diagnostics settings).

Merged "***" as current context in ***\AppData\Local\Temp\tmp2y8e4_3o
No resources found in aks-periscope namespace.
No resources found in aks-periscope namespace.
No resources found in aks-periscope namespace.
No resources found in aks-periscope namespace.
No resources found in aks-periscope namespace.
No resources found in aks-periscope namespace.
No resources found in aks-periscope namespace..
No resources found in aks-periscope namespace...
No resources found in aks-periscope namespace....
No resources found in aks-periscope namespace.....
Got 0 diagnostic results for 3 ready nodes.........
Could not get network config. Please run 'az aks kanalyze' command later to get the analysis results.
Could not get networking status. Please run 'az aks kanalyze' command later to get the analysis results.
  • 👍1