OpenCV 3.3.0 & Git LFS - Quota Limits
ahasbini opened this issue · 4 comments
Due to the file limit of 100MB enforced by GitHub, I got an error stating that I should look into Git LFS for large binary files while pushing the OpenCV SDK v3.3.0. I quickly viewed the tutorials and have gotten all binary files (*.a and *.so) to be tracked by Git LFS on GitHub. However I failed to realize that there is a quota on the use of Git LFS and bypassing that would restrict certain things. The issue appeared today and for now I believe that certain users are facing an issue in cloning or forking this repo. GitHub's quota limits on Git LFS are quite strict and not useful and requires users to install an extra tool. Hence I'll have to look into other solutions to store these large binaries. For now I'll be trying to revert the SDK back to 3.2.0 to keep the repo working and capable of being cloned/forked by users while I look for further solutions for this. If anyone has a suggestion/solution I'll be happy to check it out.
For further information check the following:
Felt I need to give a status update on the issue. Still looking on solutions to have a more flexible Git LFS service. Found lots of implementations, including from Git LFS - Implementations, that use Amazon S3. However I feel that they are sort of outdated with API specs for Git LFS and also need to dig in more on details about the security. If anyone has some helpful feedback I'd highly appreciate.
Felt coming back to this after a while. Found a solution posted by @alanedwardes using Amazon S3. Needs some thinking and decision making to do on the usage and load requirements as there is a cost to it. Here's the link and repo in case anyone is interested.
Hi @ahasbini! I'm still using this implementation for my game binary assets for just over a year, to date with 38,908 objects and 67.91GB of data. My costs for this are the outright S3 storage costs and bandwidth - I don't pay anything for the Lambda executions and API Gateway is a few cents.
What I didn't investigate with this is security - as it's me and a couple of others using the private repo for my game, I just committed the .lfsconfig like this:
[lfs]
url = https://<username>:<password>@lfs.iamestranged.com/
There is basic support for pluggable authentication in the lib, but I didn't explore this a great deal in terms of what Git LFS itself has to offer.
I'd love to extend the library to support better authentication mechanisms/access control etc though!
So I've finally found a way to come around this albeit it's a big change. I've developed a new Gradle Plugin and uploaded it's code here: AndroidOpenCVGradlePlugin. The plugin does exactly what this repository tries to achieve but without the use of Git and GitLFS. This means it will no longer face quota issues (I sure hope I didn't miss another quota while developing the plugin 🙈) as it will be downloading the sources straight from OpenCV pre-built release sites.
There are changes needed to be done to migrate to the plugin, basically reverting the git-submodule integration and removing configs related to the OpenCV in different *.gradle files. But rest assured that once the migration is complete, the compilation will not become any different and it will still build successfully as it used to before (in-case it was successfully integrated and was building before).
I'll posting the steps and update the README.md soon.