Feature request: custom git clone command.

[ivan-elude]

Describe Request:

We have a large monorepo with a variety of projects that only require a small subset of the repo to build. A regular git clone command can take up to 30 seconds due to our large repo size and high commit count. After some deep investigation into git, I found custom commands using git's --no-checkout and sparse-checkout options. Local testing shows usage of these commands drops down the git clone time to 3 seconds (10x improvement).

Currently, our usage of this orb has a 60 second runtime on circleCI. I think an option "fast-checkout" or "sparse-paths" would really help.

Example usage:

- path-filtering/filter:
  base-revision: main
  sparse-paths: # if set, assumes fast-checkout option is enabled.
    - /foo
    - /bar/baz

Examples:

The exact git commands are as follows:

# clone our repo. Note the blob filter and --no-checkout
git clone --no-checkout --filter=blob:none ssh://git@example.com:22/monorepo.git

# Pop into the folder. If you ls, there is only the .git/ foldere
cd monorepo

## Set up our sparse checkout
git sparse-checkout init --cone

# Specify the paths we care about 
git sparse-checkout set /your_path/ /some/other/path

# Pull the files.
git read-tree -mu HEAD

Supporting Documentation Links:

• https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/
• https://dev.to/patricksevat/optimize-your-git-clone-fetch-strategy-for-ci-pipeline-3pka
• https://stackoverflow.com/questions/9572407/git-sparse-checkout-with-exclusion

[Averylamp]

I'd also love this feature

[kelvintaywl]

hi @ivan-elude and @Averylamp ,

This is Kelvin from CircleCI's support team, and I'd like to thank you for sharing your feedback here!

I understand you would like options for how the git clone is run.

Right now, the path-filtering/filter job simply uses CircleCI's checkout step:

path-filtering-orb/src/jobs/filter.yml

Line 49 in 9b229fa

- checkout

As such, this will clone the full repository as its behaviour.

We have a feature request on extending the checkout step to allow for shallow cloning.

On this note, I want to share that you can also modify your use of the path-filtering Orb to customize its checkout.

Essentially, the main magic in the path-filering/filter job is actually the path-filtering/set-parameters command.
We can see the job's composition here:

path-filtering-orb/src/jobs/filter.yml

Lines 49 to 63 in 9b229fa

	- checkout
	- when:
	condition:
	not:
	equal: ["", << parameters.workspace_path >>]
	steps:
	- attach_workspace:
	at: << parameters.workspace_path >>
	- set-parameters:
	base-revision: << parameters.base-revision >>
	mapping: << parameters.mapping >>
	- continuation/continue:
	configuration_path: << parameters.config-path >>
	parameters: "/tmp/pipeline-parameters.json"
	circleci_domain: << parameters.circleci_domain >>

As such, we can customize a filter job so that you can then "bring your own checkout".

Here is an example of a change (diff) to showcase this:
(i am using a third-party orb for shallow-clone, but you can swap it out as needed).

kelvintaywl-cci/path-filtering-code-changes-only@7c31bc3...0ea4f34

Hope this helps! 🙇

[ivan-elude]

@kelvintaywl this is PERFECT! Thank you. Our filter step now takes 8 seconds, rather than 38.

Here's the code using my fast-checkout orb, as a sample for others.

version: 2.1
setup: true
orbs:
  path-filtering: circleci/path-filtering@0.1.3
  fast-checkout: issmirnov/fast-checkout@1.2.0
  continuation: circleci/continuation@0.3.1

jobs:
  filter:
    executor: path-filtering/default
    steps:
      - fast-checkout/get:
          branch: main
          sparse-paths: "foo/ bar/ .circleci/"
      - path-filtering/set-parameters:
          base-revision: main
          # mapping: <regexp applied to file list> <build property> <property value>
          mapping: |
            foo/.*                     foo-modified true
            bar/.*                     bar-modified true
            .circleci/workflows.yml    everything-modified true
      - continuation/continue:
          configuration_path: .circleci/workflows.yml
          parameters: /tmp/pipeline-parameters.json

workflows:
  setup:
    jobs:
      - filter

[kelvintaywl]

@ivan-elude thank you so much for sharing the config, and for making your fast-checkout Orb public too 💯

I will be checking out (pun intended) your public Orb indeed!
glad to have helped 😁

[sibelius]

      - git-shallow-clone/checkout:
          depth: 1

git-shallow-clone breaks the checkout

[kelvintaywl]

hi @sibelius i believe the error (exit code 128) is permissions-related and thus re: your SSH key (to git clone via git+ssh protocol).

can you try recreating your Deploy key and trigger a new build?