ipfs/distributed-wikipedia-mirror

End-to-end automated Dockerfile

lidel opened this issue · 1 comments

lidel commented

The version from #77 creates a basic environment with all dependencies for dev and debugging, but is not useful for end-to-end automation like one described in #58

The complexity involved is way too big for anyone sane to dig into this, so if we want to build snapshots more often than every 3-6 months, we must simplify the entire process and produce Docker image that:

  • takes inputs
    • a ZIM file
    • file system path for persisting IPFS_PATH with the output
  • returns
    • CID of produced snapshot
    • populates IPFS_PATH with all blocks

This way IPFS_PATH is independent of the Docker image and can be persisted/managed by other means (could be an ephemeral repo, or a permanent one).

alzinging commented

I can do this, with some changes ... this month. Faster if I see interest.

How often is Wikipedia snapshotred, I'd like to "ideally" see this done on the same schedule.

This thing is fragile. It needs love.