MWAA CLI - open more supported on airflow commands

[wilsonhooi86]

Overview

Currently MWAA has limited supported airflow commands. As listed here: https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-cli-command-reference.html

We have a few scenarios that requires some unsupported MWAA CLI command to work.
Example:

1. Airflow pool import
   Command: airflow pools import [-h] [-v] FILEPATH
   Link: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#pools
   Issue: This command is not supported in MWAA as it returns Response [500] error.
   Reason needed: This will help us to import a list of airflow pools at one go using json file. Currently we have to use airflow pools set command to insert/update and airflow pools delete command to delete one airflow pool at a time. This takes alot of time when we have 200++ airflow pools to insert/update/delete daily. This may cause performance issue to Airflow metadata with many airflow CLI command calls each time.

2. Airflow create / delete role
   Command: airflow roles create [-h] [-v] [role [role ...]] or airflow roles delete [-h] [-v] [role [role ...]]
   Link: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#roles
   Issue: This command is not supported in MWAA as it returns Response [500] error.
   Reason needed: To dynamically create new roles programmatically, so that we can auto assign specified roles to users who access specified dags.

3. Airflow assign role to user
   Command: airflow users add-role [-h] [-e EMAIL] -r ROLE [-u USERNAME] [-v]
   Link: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#users
   Issue: This command is not supported in MWAA as it returns Response [500] error.
   Reason needed: Based on point no.2 , after the roles are programmatically created, we would like to programmatically assign the respective airflow custom roles to the users.

Acceptance Criteria

Able to support all the airflow CLI commands for above scenarios.

Additional Info

Unlocking the unsupported Airflow CLI commands in MWAA will greatly help us in many scenarios such as custom role creation and airflow pool assignments. There are many more scenarios we wish to use Airflow CLI for automation feature in future as well.

[Kytha]

Hi Wilson, I'll look into this. For # 2, airflow roles create is listed as a supported command for v2.6+, but I will see if I can recreate your issue. For the other 2 I'll need to see what (if any) blockers there are to supporting these commands.

[wilsonhooi86]

Good Day @Kytha , thank you for looking into this issue. I tried the airflow roles create command recently but still get Response [500] error.

FYI, I used the sample code from this page to run other mwaa cli command: https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-cli-command-reference.html#airflow-cli-commands-tutorials

Also to add on, may I ask is there why MWAA blocks selected airflow CLI commands? It would be better to unlock all the airflow CLI commands in MWAA, so that we can dynamically programmatically update the MWAA from backend for various scenarios like airflow pools import list and custom roles feature (Needed commands to work: airflow roles add/delete, airflow users create/add-role/delete/remove-role)