Azure/apiops

[FEATURE REQ] Allow only product extraction and not enforce that apis are also extracted in the same run

Opened this issue · 4 comments

erikhjensen commented

Please describe the feature.

When I run a product extractor and have no apis listed. It will not write the apis association file. I see in the Trace that the apis are being listed but i believe the CLI is getting that list and perhaps trying to validate that the APIs exist or, that it tries to reconcile them with its own list of APIs. And when it doesn't find a match, it doesn't write any associations.

We have a workflow whereby we manage products separately. This pretty much means I cannot run the product extractor or, if I do, i have to also extract the APIS, and then cherry pick the src content and move it into a branch that I can then reconcile w/ the rest of the codebase.

Perhaps if we can't just trust the api-names in the product (and I don't know why you couldn't.. as they're saved in the platform) then you could list APIs and check the names for matches.

But I think you can just get the API names from the list products for API as you do and just trust the names: https://learn.microsoft.com/en-us/rest/api/apimanagement/product-api/list-by-product?view=rest-apimanagement-2022-08-01&tabs=HTTP

github-actions commented 
  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.
guythetechie commented

Hi @erikhjensen,

If I understand correctly, you want to extract product API associations, but not the APIs themselves. In other words, you want these folders:

  • /artifacts/products/productA/apis/apiX/.json
  • /artifacts/products/productA/apis/apiY/.json
  • /artifacts/products/productb/apis/apiX/.json
  • /artifacts/products/productA/apis/apiY/.json

But not these folders

  • /artifacts/apis/apiX/...
  • /artifacts/apis/apiY/...

If that's the case, I would suggest using a .gitignore file in your Git repository and ignore anything under /artifacts/apis. The extractor will still extract those files when it runs, but they should not be included in the autogenerated PR back into your repository.

erikhjensen commented

We extract the APIs to a different folder and the products to their own root due to some environmental particularities. If I had a .gitignore, my feeling is that this would prevent the apis from being extracted during their pipeline run/configuration. The REST api link I posted does suggest that the API-names are retrievable for a product to be able to create the apis.json file. I guess I don't understand why the CLI will only produce an empty json array in that file if the same pipeline-run doesn't also extract ALL the APIs that are associated to that product. The product REST API in azure will give you the response required to produce apis.json, groups.json and productinformation.json. Perhaps if the CLI had a shallow-product capability that understands if it already extracted apis and if it doesn't, it still can produce the fleshed-out product schema.

We get around this by extracting to a target-branch that isn't the intended target but rather an intermediary branch (much like the auto-generated branch) and then I cull the API content after extracting it and the product data in the same run. It's an extra step but technically works.

guythetechie commented

I still don't fully understand the question, but we've changed how product API associations work in our latest release. Please review the release notes for more details.

  1. There is no longer an apis.json. Instead, each product API association will be mapped to a file. Search for Subfolder structure in the release notes for details on how this works.
  2. The API filters will now be applied to products too. If your configuration says to only extract apiA, then the association between productB and apiB will not be extracted.

If the statements above don't address your question, please rephrase it with a specific example. Let's say productA is associated with apiX and apiY, and productB is associated with apiX and apiY. What do you have in your configuration.extractor.yaml? And what result would you like to see?