ianphil/git-insights

An SDK which helps engineers gain insights around how a team is collaborating towards a project's workitems and codebase.

Git-based SCM Insights SDK

• Git-based SCM Insights SDK
  • Overview
    • Supported Source Code Management Providers
    • ADO Insight Capabilities
    • Reportable Fields
  • Installation
    • Option 1 - Install with pip
    • Option 2 - Deploy to Azure with Terraform
    • Option 3 - Setup local Azure Function environment
      • Azure Function Prerequisites
      • Steps
  • SDK Usage
    • Example
  • Backlog Features - will be migrated to repo backlog
  • Contributing

git-insights is a work in progress / early days toolset enabling engineering leads to gain insights around how a team is collaborating towards a project's workitems and codebase. Devs can use this sdk to identify opportunities to collectively help other team members improve collaboration.

git-insights connects to your existing source code management repositories and work item activities to provide insights toward active and completed scheduled work, git commits, pull requests and code reviews. The dataset can also be used to collectively celebrate progress week over week.

Overview

Supported Source Code Management Providers

• Azure Dev OPS
• GitLab (coming soon)
• GitHub (coming soon)

ADO Insight Capabilities

• Active / completed workitems (incl duration averages)
• Workitem creation
• Pushed commits
• Git commit log stats (ie sum of line edits, additions, deletions)
• Active / completed story points
• Active / completed pull requests (incl duration averages)
• Code review comments + reviews
• Azure Function integration to support report scheduling and delivery

Reportable Fields

Sample Dataset

Field Name	Description	Type
contributor	The profile display name of the activity contributor / assignee	String
week	The week of year for the captured activity	int
repo	The git repository name	String
prs_merged	Sum of pull requests merged into the main branch	int
prs_submitted	Sum of pull requests submitted for review	int
pr_completion_days	Mean Average duration for pull request review (ie days diff between PR submission date and completion date)	float
pr_comments	Sum of posted pull request review comments	int
prs_reviewed	Sum of submitted pull requests reviewed	int
pr_commits_pushed	Sum of git commits pushed to active pull requests	int
commit_change_count_edits	Sum of changed lines within the commit log	int
commit_change_count_deletes	Sum of removed lines within the commit log	int
commit_change_count_additions	Sum of newly added lines within the commit log	int
user_stories_assigned	Sum of user stories assigned to the contributor	int
user_stories_completed	Sum of user stories completed or resolved by the contributor	int
user_story_points_assigned	Sum of story points asssigned to the contributor	int
user_story_points_completed	Sum of completed story points	int
user_story_completion_days	Mean Average duration for user story completion (ie days diff between story assignment and completion date)	float
user_stories_created	Sum of user stories created	int
user_story_initial_pr_submission_days	Mean average of elapsed time between the datetime of a user story activated date and the initial pull request submission date.	float

Installation

This SDK can be used either through the pip package or as an Azure Function.

Prerequisites

• An Azure Subscription
• An ADO PAT Token with read permissions scoped to Code, Work Items and Graph.
• An ADO organization, project, repo(s) and a backlog team
• All project dependencies such as Python, Azure Functions CLI and Terraform are configured as an easy to consume Development Container. To use this, install the Remote Development Extension and then follow these instructions for opening the workspace in the containerized environment.

Option 1 - Install with pip

pip install gitinsights

Option 2 - Deploy to Azure with Terraform

This application can be easily deployed to Azure using Terraform by following the Terraform Getting Started Guide

Option 3 - Setup local Azure Function environment

Azure Function Prerequisites

• VSCode
• Azure Blob Storage Account
• Keyvault Resource
• A secret containing the value of your PAT token.
• Service Principal with read level role assignments into the above Keyvault resource

Steps

1. Clone the repo

git clone https://github.com/erikschlegel/git-insights.git
cd git-insights
pip install -r requirements.txt && pip install -r requirements-dev.txt

2. Once the Azure extension is installed, sign into your Azure account by navigating to the Azure explorer, select Sign in to Azure under Functions, and follow the prompts in the browser.

After signing in, verify that the status bar says Azure: Signed In and your subscription(s) appears in the Azure explorer:

3. To verify that all the Azure Functions tools are installed, open the Visual Studio Code Command Palette (F1), select the Terminal: Create New Integrated Terminal command, and once the terminal opens, run the command func

4. Configuration Setup

• Rename local.settings.json.template -> local.settings.json
• Provide the Azure storage account's primary connection string for both the gitinsights_STORAGE and AzureWebJobsStorage settings.
• Provide the service principal credentials for the following settings: AZURE_CLIENT_SECRET, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_SUBSCRIPTION_ID
• Provide the keyvault name for the KeyvaultName setting, and the secret containing the PAT token for the PatSecretName setting
• Provide the ADO organization name, project name, repositories (comma delimited for multiple) and backlog team name.

5. Set a breakpoint in the code and hit F5 to debug locally.

6. The function will write the results to your blob storage account prior to completion.

JFYI: This particular Azure Function is scheduled to run every friday at 9AM. Edit functions.json if you'd like to change the NCRONTAB schedule.

SDK Usage

Example

from gitinsights.mods.managers.ado import AzureDevopsClientManager

adoProject = "Seinfeld-Trivia"
adoOrg = "Best-Shows"
repos = ["a-repo-about-nothing"]
teamId = "Team LD"
patToken = "Kramers-secret"

groupByColumns = ['contributor', 'week', 'repo']

client = AzureDevopsClientManager(adoOrg, adoProject, repos, teamId, patToken)
dataframe = client.aggregatePullRequestActivity(groupByColumns)

print(dataframe)

Backlog Features - will be migrated to repo backlog

• Azure Functions Integration
  • Dockerize Azure Function
  • Add Azure Function Continuous Delivery Pipeline
  • Integrate with PowerBI
  • Terraform integration to automate scaffolding of keyvault, azure function and blob storage resources
• Add E2E Integration Tests
• Add pre-commit hook to block pushes including creds
• Add a new provider to support Git Lab integration
• Integrate Sphinx to autogen SDK docs

Contributing

============

You can contribute to the project in multiple ways:

• Write documentation
• Implement features
• Fix bugs
• Add unit and functional tests
• Everything else you can think of