/azuredevops-security-automation

Python-based solutions that managed Azure DevOps security via API

Primary LanguagePythonMIT LicenseMIT

Create project status Check project status pylint Score License
Create project status Check project status pylint Score License: MIT

Table of Contents

Challenge

Azure DevOps gives a nice opportunity to have dedicated projects for dedicated teams and/or projects
However default groups and permissions are too open for enterprise projects and too hard to understand for new Azure DevOps users
Typically it ends with the whole team using administrator permissions

Solution

Create common groups and set relevant permissions to keep Azure DevOps users happy
To keep enterprise security folks happy too setup security monitoring to ensure that nobody violates rules
And to keep operations happy, both projects creation and verification is automated using Azure DevOps API and Pipelines

HLD diagram

diagram

How-to use

  1. Fork this repo
  2. Put name of the new project name to projectName variable in ./cloud/create-project.yml & ./cloud/check-project.yml for Cloud and ./onprem/create-project.yml & ./onprem/check-project.yml for on-premise
  3. Clone or create new Variable groups - ados-group-names-$projectName and ados-acls-$projectName
  4. If you cloned Variable groups ensure that you're happy with values - Variable groups keys and values are described in detail below
  5. Put cloned Variable group names to ./cloud/create-project.yml & ./cloud/check-project.yml for Cloud and ./onprem/create-project.yml & ./onprem/check-project.yml for on-premise
  6. Create new Azure DevOps pipeline using ./cloud/create-project.yml for Cloud and ./onprem/create-project.yml for on-premise, rename pipeline (e.g. create-$projectName), run it and check logs
  7. Create new Azure DevOps pipeline using ./cloud/check-project.yml for Cloud and ./onprem/check-project.yml for on-premise, rename pipeline (e.g. check-$projectName), run it and check logs

Azure DevOps APIs

Azure DevOps UI vs API naming

In Azure DevOps UI naming is different from API naming, e.g. in Artifact feed UI role is called "Owner" and API name is "administrator": screenshot

Azure DevOps APIs and Versions

Azure DevOps has a few APIs and each API has a few versions, e.g.:
regular https://dev.azure.com/{organization}/_apis/projects?api-version=5.0 to create Project,
new https://vssps.dev.azure.com/{organization}/_apis/graph/groups?scopeDescriptor={projectScopeDescriptor}&api-version=5.0-preview.1 to create Group

Here's summary of scripts vs used APIs (actions should be understandable from script names):
https://dev.azure.com version 5.0 (can be switched to 5.1 if necessary):

  1. ./common/create_project
  2. ./common/check_permissions
  3. ./common/set_permissions
  4. ./common/check_git_policy
  5. ./common/set_git_policy
  6. ./cloud/export_project_info

For on-premise link will be server_url/collection, e.g. https://ados.demo.kagarlickij.com/DefaultCollection

https://vssps.dev.azure.com version 5.0-preview.1 vs TFSSecurity CLI:

  1. ./cloud/export_project_info
  2. ./cloud/get_group_members
  3. ./cloud/get_groups
  4. ./cloud/export_group_info
  5. ./cloud/delete_group
  6. ./cloud/create_group

This API in not available in Azure DevOps Server v2019.Update1.1 so TFSSecurity CLI must be used
TFSSecurity CLI can be executed on server where Azure DevOps is installed, so you have to install agent on that server (obviously Windows-based) machine
This is annoying because Python tasks on self-managed Windows agents are problematic, so I had to install Python manually and wrapped Python calls into PowerShell in tasks
Another important thing is account used to run Azure DevOps agent - it must have admin permissions to use TFSSecurity and by default it's just a network service. So carefully control who can use that agent
The last but not least complain about TFSSecurity is it's output - poorly formatted console output. No YAML. No CSV. No JSON. So I had to develop a few workarounds to manage it, e.g. ./onprem/export_group_info.py
Hope separate boiler in hell is waiting for TFSSecurity CLI developers 🙏

https://feeds.dev.azure.com version 5.0-preview.1 (can be switched to 5.1-preview.1 if necessary):

  1. ./common/export_feed_info
  2. ./common/get_feed_acl
  3. ./common/create_feed
  4. ./common/set_feed_acl

This API in not available in Azure DevOps Server v2019.Update1.1 so regular one must be used
It has an important limitation - feeds can not be created in project scope, only on collection level

https://vsrm.dev.azure.com version 5.0 (can be switched to 5.1 if necessary):

  1. ./common/create_tmp_release_pipeline
  2. ./common/delete_tmp_release_pipeline

For on-premise link will be server_url/collection, e.g. https://ados.demo.kagarlickij.com/DefaultCollection

Personal vs System access token

Default permissions of System access token are not enough to perform all CRUD actions
Those permissions can be extended, but it's too dangerous because other pipelines will use this token too
So solution is to create another "root" user and use it's Personal Access Token

Azure DevOps API throttling

I never had 429 from Azure DevOps API but I had number of situations when after received 200 code change wasn't present
So I added time.sleep(1) to ./common/set_permissions script and a few pipeline tasks with sleep 10 to wait until certain resources are created

Response codes

Python requests.exceptions.RequestException is happy with all 2** codes but some Azure DevOps APIs respond with multiple 2** codes for multiple results
So I had to add status_code of the response to match desired e.g. 204 in ./cloud/delete_group which is bit different from documentation

Organization

All scripts have --organization parameter to enter API address, e.g. https://dev.azure.com/kagarlickij for cloud or https://ados.demo.kagarlickij.com/DefaultCollection for on-premise Exception is ./cloud/export_project_info which has https://dev.azure.com and https://vssps.dev.azure.com hardcoded so you insert bare organization name instead of full link

Project scope descriptor vs Project ID

Both projectId and projectScopeDescriptor are exported as environment variables for future usage by ./cloud/export_project_info.py script
projectId value is necessary to obtain projectScopeDescriptor value

https://vssps.dev.azure.com API uses projectScopeDescriptor to identify project to work with

https://dev.azure.com API uses tokens - entity that consist of projectId and some separator characters

Tokens

Tokens worse some additional explanations: as per documentation Tokens are arbitrary strings representing resources in Azure DevOps
Token format differs per resource type, however hierarchy and separator characters are common between all tokens
To get Token for security namespace run GET https://dev.azure.com/{organization}/_apis/accesscontrollists/{securityNamespaceId}?api-version=5.1 (security namespaces are described down below)

For Project namespace token format is $PROJECT:vstfs:///Classification/TeamProject/{projectId}
For Git Repositories namespace token format is repoV2/{projectId}
For AnalyticsViews namespace token format is $/Shared/{projectId}
For Library namespace token format is Library/{projectId}
For Build and ReleaseManagement it's just {projectId}

Those are used in ./common/get_permissions and ./common/set_permissions conditions

Security namespaces

As per documentation security namespaces are used to store access control lists (ACLs) of tokens
You can get list of namespaces with GET 'https://dev.azure.com/{organization}/_apis/securitynamespaces?api-version=5.1'
As for now, there're 60 namespaces, for our purposes we'll use a few of them: 52d39943-cb85-4d7f-8fa8-c6baac873819 for Project
d34d3680-dfe5-4cc6-a949-7d9c68f73cba for AnalyticsViews
2e9eb7ed-3c0a-47d4-87c1-0ffdd275fd87 for Git Repositories
b7e84409-6553-448a-bbb2-af228e07cbeb for Library
33344d9c-fc72-4d6f-aba5-fa317101a7e9 for Builds
c788c23e-1b46-4162-8f5e-d7585343b5de for ReleaseManagement

Azure DevOps Security basics

Numbers for permissions

Permissions for majority of resources are present as sum of "bits" (numbers) for actions in namespace
e.g. in AnalyticsViews "Read" is 1 and "Execute" is 8 so "allow" field in ACL's "acesDictionary" must be 9
The same logic is fair for "deny" rules, and resulting ACL looks like this
Exception is Artifact feed permissions model, there are no "acesDictionary" and "bits", more simple concept of "roles" is used instead: screenshot

Create entity vs set permissions

As mentioned above permissions are set in resource's ACL and existing entities (user or group) must be used
So there's no way to create user/group and set permissions at the same time
To provide cross-account capabilities in ACLs descriptors are used instead of user/group bare names

Azure DevOps Artifact feeds

Artifact feed name is still "busy" after feed is deleted

Current solution creates dedicated feed for each project (but project-scoped only in the cloud)
It might be a situation when project has to be deleted and then created again
But feed with the same name will fail to create even if it was deleted from "Deleted feeds" (trashbox for feeds)
e.g. when I'm trying to create test1 feed I'm getting response "A feed named 'test1' already exists." but when I'm listing feeds test1 does not exist in both API and UI including "Deleted feeds"

create-project pipeline

create-project pipeline creates new project based on "security template" and consist of the following steps:

  1. Create project
    --processTemplate param set to be Scrum ('6b724908-ef14-45cf-84f8-768b5384da45') by default
    Process templates ids can be listed with Processes - List API
    --projectDescription param is optional and empty by default
    sourceControlType is hardcoded to be Git
    visibility is hardcoded to be Private

  2. Export project-related info
    As mentioned above PROJECT_ID and PROJECT_SCOPE_DESCRIPTOR vars are exported for further usage

  3. Create tmp Release pipeline
    Unlike builds, you have to have at least one Release pipeline to be able set permissions in ReleaseManagement namespace
    Built-in 'Release Administrators' group is also created automatically by Azure DevOps after first release pipeline is created
    So we have to create dummy release pipeline and delete it later

  4. Delete built-in groups
    Since we will use our custom groups default (built-in) groups can be deleted
    However 'Project Valid Users', 'Project Administrators' and '$projectName Team' groups can not be deleted
    I prefer to delete default groups because '$projectName Team' inherits some undesired permissions from other default groups
    If people are added to default 'Project Administrators' group check-project pipeline will alert since it is not desired

  5. Create custom groups
    Administrators - ADOS setup, Releases setup (until we have it as code), security changes (both ADOS and app related)
    Developers - develop apps, develop build pipelines, watching Builds and Releases
    Product Owners - manage Release activities
    Auditors - checking security configs (current vs desired)
    Each group has prefix with project name (e.g. prj01-developers) to make cross-project access easy if it's required

  6. Export group-related info
    Exports group SID as a variable for further usage
    Format of SID from time to time causes incorrect padding in Python so fix is applied

  7. Set Project permissions for each group
    Here's the first time when ./common/set_permissions.py script comes into the game
    It will be used to set almost all permissions so threat it with extra care

  8. Set Analytics permissions for each group
    Analytics views are located in the same UI section Project permissions but have different security namespace and token format

  9. Set Library permissions for each group
    Variable groups permissions are crucial piece, so pay extra attention to this permissions

  10. Set Git repos settings task creates cross-repo policy that requires at least 1 reviewer before merge into master branch

  11. Set Git permissions for each group
    This security namespace also has different token format and ./common/set_permissions.py knows about that

  12. Set Build permissions for each group
    Nothing too special here, token in this security namespace is equal to project id

  13. Set Release permissions for each group
    This task will fail if tmp Release pipeline was not created (Step #3)

  14. Delete tmp Release pipeline
    Dummy release pipeline has not be present anymore

  15. Create Artifact feed (optional)
    For cloud scenario createArtifactFeed parameter must be set true for feed to be created
    For on-premise parameters are not supported so createArtifactFeed is variable used in create-feed and check-feed-permissions for the same purpose
    https://feeds.dev.azure.com/{organization}/_apis/packaging/feeds?api-version=5.0-preview.1 API used for cloud and {server_ur}/{collection}/_apis/packaging/feeds?api-version=5.0-preview.1 API used for on-premise
    Feed has default capabilities and upstream disabled - all is hardcoded in ./common/create_feed.py

  16. Export feed info
    FEED_ID variable is exported for further usage

  17. Set feed permissions As described above, feeds use roles instead of permissions "bits" so ./common/set_feed_permissions.py script is used instead of ./common/set_permissions.py

check-project pipeline

  1. Export project-related info
    As mentioned above PROJECT_ID and PROJECT_SCOPE_DESCRIPTOR variables are exported for further usage

  2. Export group-related info
    Exports group SID as a var for further usage
    Format of SID from time to time causes incorrect padding in Python so fix is applied

  3. Check list of groups
    Checks if no additional groups were created and/or existing groups deleted/renamed

  4. Check group members
    Checks only default groups that can't be deleted: 'Project Valid Users' and 'Project Administrators'
    Nobody should be added to those groups, so desired quantity of members is 0
    However creator of the project (user, whose PAT was used in pipeline) is set to be member of 'Project Administrators', so for 'Project Administrators' desired quantity of members is 1

  5. Check Project permissions for each group
    ./common/check_permissions.py script will check if current Project scope permissions match desired
    As well as ./common/set_permissions.py it knows about different tokens for different security namespaces

  6. Check Analytics permissions for each group
    ./common/check_permissions.py script will check if current Analytics scope permissions match desired

  7. Check Library permissions for each group
    ./common/check_permissions.py script will check if current Library scope permissions match desired

  8. Check Git repos settings task performs the following checks on all repos in the project:
    Check if all branches follow naming standards: allowed names are master, feature/ and bugfix/
    Check if all branches are up to date: latest commit shouldn't be older than X days (see ados-git-params-$projectName variable group below for details), however master branch is excluded from this check
    Check if all Pull requests are up to date: pull requests shouldn't be older than X days (see ados-git-params-$projectName variable group below for details)
    All 3 checks generate warnings, not errors
    The latest one checks if master branch has at least 1 reviewer in the corresponding policy, and if not error will be raised
    Some repos can be excluded from checking by putting repo name to excluded_repos.txt file

  9. Check Git permissions for each group
    ./common/check_permissions.py script will check if current Git scope permissions match desired

  10. Check Build permissions for each group
    ./common/check_permissions.py script will check if current permissions match desired

  11. Check Release permissions for each group
    ./common/check_permissions.py script will check if current permissions match desired

  12. Export feed info (optional)
    FEED_ID var is exported for further usage
    Conditions are the same as for create-project

  13. Check Artifact feed permissions for each group (optional)
    ./common/check_feed_permissions.py script is used instead of ./common/check_permissions.py because https://feeds.dev.azure.com API is used
    Conditions are the same as for create-project

Pipelines execution

Triggers

create-project pipeline is executed only once, so no triggers needed
check-project pipeline must be executed on schedule, default is daily at 2AM
Execution time is about 1 minute, so schedule can be set to be hourly
However there's still a risk that person will be added to 'Project Administrators' group, perform some "bad" actions, removed from the group and check-project pipeline will not catch it because those actions are less than 1h long
This is an advanced scenario and to address it real-time log analytics must be set to watch Azure DevOps logs

Variables

Both create-project and check-project pipelines must used the same variables in variable groups
Moreover, it's recommended not to change variables after create-project execution otherwise "current desired state" will not match "initial desired state"
Azure DevOps variable groups don't have history of changes and versioning so it's recommended to use linking with Azure Key Vault

projectName is the only variable hardcoded in pipelines, change it's value right after forking this repo

ados-group-names-$projectName variable group contains names of custom groups to create and names of SIDs to use as environment variable names, example:

Name Value
administrators-group-sid $(administrators-group-name)-sid
administrators-group-name $(projectName)-administrators
auditors-group-sid $(auditors-group-name)-sid
auditors-group-name $(projectName)-auditors
developers-group-sid $(developers-group-name)-sid
developers-group-name $(projectName)-developers
product-owners-group-sid $(product-owners-group-name)-sid
product-owners-group-name $(projectName)-product-owners

If you want to add more groups or remove some of existing don't forget to change both variable group and pipeline's tasks

ados-acls-$projectName group contains "security bits" for each group per each security namespace, example:

Name Value
administrators-analytics-allow-permissions 6
administrators-analytics-deny-permissions 0
administrators-build-allow-permissions 65535
administrators-build-deny-permissions 0
administrators-feed-permissions administrator
administrators-git-allow-permissions 65534
administrators-git-deny-permissions 0
administrators-library-allow-permissions 19
administrators-library-deny-permissions 0
administrators-project-allow-permissions 11729915
administrators-project-deny-permissions 4259844
administrators-release-allow-permissions 65535
administrators-release-deny-permissions 0
auditors-analytics-allow-permissions 0
auditors-analytics-deny-permissions 6
auditors-build-allow-permissions 1025
auditors-build-deny-permissions 31742
auditors-feed-permissions reader
auditors-git-allow-permissions 2
auditors-git-deny-permissions 65532
auditors-library-allow-permissions 1
auditors-library-deny-permissions 0
auditors-project-allow-permissions 2097665
auditors-project-deny-permissions 13892094
auditors-release-allow-permissions 33
auditors-release-deny-permissions 4062
developers-analytics-allow-permissions 0
developers-analytics-deny-permissions 6
developers-build-allow-permissions 1921
developers-build-deny-permissions 30846
developers-feed-permissions collaborator
developers-git-allow-permissions 16758
developers-git-deny-permissions 48776
developers-library-allow-permissions 17
developers-library-deny-permissions 0
developers-project-allow-permissions 2104105
developers-project-deny-permissions 13885638
developers-release-allow-permissions 97
developers-release-deny-permissions 3998
product-owners-analytics-allow-permissions 6
product-owners-analytics-deny-permissions 0
product-owners-build-allow-permissions 1025
product-owners-build-deny-permissions 31742
product-owners-feed-permissions reader
product-owners-git-allow-permissions 16486
product-owners-git-deny-permissions 49048
product-owners-library-allow-permissions 1
product-owners-library-deny-permissions 0
product-owners-project-allow-permissions 11723265
product-owners-project-deny-permissions 4266494
product-owners-release-allow-permissions 2169
product-owners-release-deny-permissions 1926

ados-git-params-$projectName group contains parameters for git checks, example:

Name Value
maxCommitAge 15
maxPullRequestAge 5
minApproverCount 1

ados-secrets group contains Azure DevOps Personal Access Token that is used for all API actions, described in details above in "Personal vs System access token" section

Name Value
cloud_pat *****
onprem_pat *****
***** - because marked as secret
If you use cloud Azure DevOps you need only cloud_pat and if you use on-premise Azure DevOps you need only onprem_pat

github-secrets group contains GitHub Personal Access Token that is used for all API actions, described in details in "Build" section

Name Value
gitHubPat *****
***** - because marked as secret

Azure DevOps specific syntax in scripts

  1. All print functions have [INFO] or [ERROR] prefixes to make output more readable and properly colored in Azure DevOps logs, example
    However Azure DevOps Server v2019.Update1.1 supports only error and warning and doesn't support section, command, debug, etc.
  2. Environment variables are set via task.setvariable logging command

Contribution

pre-commit and pre-push hooks

To keep code clean the following measures are set:

  1. Pre-commit hook: before each commit code is formatted with Black in pre-commit-config config
  2. Pre-push hook: before each commit code is validated with Bandit in pre-commit-config config

To install those checks on your machine:

  1. Install pre-commit tool: pip install pre-commit
  2. Install pre-commit for the repo: pre-commit install && pre-commit install -t pre-push

If you need to remove pre-commit run: pre-commit install && pre-commit uninstall -t pre-push

Both checks are executed only if Python code was updated
Both checks require binaries to be installed in advance: pip install black and pip install bandit

Pull request validation:

  1. Lint Python code: Both Black and Bandit checks are up to developer, but before code is merged into master it must pass pylint check and Pull request will be marked as failed if score is lower than 8/10
    A few irrelevant for the project issues are added as exceptions to pylintrc
  2. Create project to verify if code in Pull request works - all steps match create-project pipeline, artifact feed is enabled
    Project name is based on Pull request number: prj-template-$(Build.BuildId)-prv
  3. Run positive project check to verify if it was created correctly and matches desired security rules - all steps match check-project pipeline, artifact feed is enabled
  4. Break project: Change security settings to make project "broken"
  5. Run negative project check to verify if security violations are noticed by checks - all steps match check-project pipeline, artifact feed is enabled
    This job's successful result is "warning", "success" will mean fail like in typical negative check
  6. Delete project when checks are done tmp project is deleted

Build

When code is merged into master branch build job generates pylint badge referenced on top of this file
generate_pylint_badge script uses pylint to generate score, anybadge to generate png and GitHub Gist API to push svg to GitHub Gist which is used for svg hosting
Build pipeline installs pylint and anybadge binaries on Azure DevOps agent for generate_pylint_badge script because Python packages for both pylint and anybadge don't work correctly: anybadge issue, epylint issue

Known issues

  1. pylint cmd in generate_pylint_badge causes Bandit [B602:subprocess_popen_with_shell_equals_true] issue
    It's better to replace pylint cmd with epylint but it has an issue with output to variable
  2. For on-premise pylint execution for all files doesn't work on windows