awslabs/soci-snapshotter

indexer C component needs to be rewritten to Go

vkuzniet opened this issue · 7 comments

vkuzniet commented

Is your feature request related to a problem? Please describe.
Right now soci snapshotter has indexer component, which is implemented in pure C. It needs to be rewritten in Go, which will provide a convenient abstraction layer over zlib library. The reasoning: C code is less maintainable than respective Go implementation and that may lead to bugs eventually sneaking in.

Describe the solution you'd like
Indexer components needs to be rewritten in Go.

wmesard commented

Copying this relevant comment here:

We are still implicitly leaking details about the underlying implementation. This should be addressed in future PRs before we attempt to define a real gzip-neutral interface (that is, before we add Zstd support).

  1. NewGzipIndex and NewGzipIndexFromFile would both be taking Readers if the underlying implementation were in native Go.
  2. HasBits is an implementation detail of Gzip, and probably this implementation of the gzip-ztoc technology. So we should not be exposing this concept to the caller.
wmesard commented

Cf. #60

Kern-- commented

I've seen a couple references to this issue (e.g. #80 (comment)) that imply that this is scheduled work, but I'm not totally sold that this is something we want to do.

Specifically, my concern is whether there is an existing go library that exposes gzip/inflate APIs at the level we need. If doing this means writing and maintaining our own inflate library in go, then I'm not sure it's worth it.

rdpsin commented

There is no existing Go library that exposes the zlib API at the level we want to. The task isn't to create our own Go zlib library, but build an abstraction layer on top of the zlib API, so it's more Go-ish.

  • 👍1
Kern-- commented

In that case, can we reword the issue? The current description sounds like the end result is a project with no c dependencies.

vkuzniet commented

In that case, can we reword the issue? The current description sounds like the end result is a project with no c dependencies.

Updated the wording.

dgr237 commented

We are using the soci-snapshotter in an existing lambda which is used to copy images from our on premise container registry to ecr.

The reliance on the gzip_zinfo c code causes issues for us testing and debugging our lambda code on developer desktops (Mac and Windows). To test at the moment we have to commit the code and wait for the code to be compiled and tested on our Linux based build servers.

Rewriting the c code in golang would enable us to speed up development and testing of any changes to our lambdas.