dat-ecosystem-archive/datproject-discussions

Dat Module Wishlist

max-mapper opened this issue · 37 comments

max-mapper commented

This issue is to discuss ideas for dat modules that would be awesome to have. If you want to write a module leave a comment to notify others that you are working on it (preferably also leave a link to the repo).

I'll keep this post updated to link to all of the modules (until we have too many to fit here).

To learn more about dat modules check out the Importing data guide

If a module doesn't have a link w/ it it means nobody is working on it yet

Read about Import Methods to determine which type of module should be used for each case.

List

(not in any particular order)

In Progress/Done

Up for grabs

webmaven commented

Some other storage possibilities (in no particular order, and I am not working on any of these):

max-mapper commented

@webmaven awesome list, thanks! I definitely have @jbenet on the hook to write an IPFS backend once it's ready

webmaven commented

You're very welcome, @maxogden.

astro commented

Here's one uploader for ElasticSearch based on the module by @missinglink: https://github.com/astro/dat-elasticsearch-upload

thadguidry commented

Support the CKAN API, in particular through http://datahub.io (part of CKAN - Open Knowledge Foundation) - Latest docs here: http://docs.ckan.org/en/latest/api/index.html

calvinmetcalf commented

pouch-dat is a connector for pouchdb/couchdb and dat

knownasilya commented

rethinkdb might be a good one to add as well

wardi commented

+1 on CKAN datastore API support, or even a CKAN extension that uses Dat instead of CKAN's datastore.

RichardLitt commented

Neo4j might be another possibility.

+1 for CKAN.

coleww commented

https://github.com/coleww/s3-blobs I make no promises, but i shall try!

doowb commented

Not sure if this fits here, but I just created vinyl-dat to read/write dat records in a gulp workflow. It's basically just a way to convert dat records into vinyl files. I think it still needs some work and possibly a better way to just let dat do conversions on contents (like choosing csv, json, etc...)

doowb commented

firebase - I'll update if I start on this.

jcblw commented

I think this should work for redis as a backend https://www.npmjs.org/package/redisdown

finnp commented

I am going to have a look at CKAN as an import/export target as well as a possible storage back-end. ckanDOWN: https://github.com/finnp/ckanDOWN

pvgenuchten commented

Would https://www.npmjs.org/package/gdal be a usefull lib for dat to connect to a huge number of formats (http://gdal.org/ogr_formats.html)?

max-mapper commented

@pvgenuchten that is awesome, thanks for linking it. I'm not sure what the best place to plug gdal into dat would be at the moment but I'll think about it. It's also great that the gdal module is using node-pre-gyp to make it easy to install cross-platform

markhahnel commented

We'd be interested in throwing http://figshare.com/ in the ring here, also have some devs who could help out :)

calvinmetcalf commented

So this is a pouchdb want originally but applies to dat.

Leveldown can be tricky to get working on all platforms and while pre-built ones ones are helpful there are always going to be weirder platform combinations.

It would be great to have a pure js leveldown backend (for node) that was not restricted to memory.

Being able to install dat or pouchdb with no c deps would be super useful.

Medea seems the best of what is out there but has a requirement that keys be kept in memory which combined with the node.js hard 2GB mem limit means that there are hard limits on the datasets you'd be able to put into it.

So really the thing that would be super helpful is a pure js nodejs db with spill to disk.

missinglink commented

Check out http://geogig.org/

"GeoGig is an open source tool that draws inspiration from Git, but adapts its core concepts to handle distributed versioning of geospatial data."

kevinsimper commented

Has many of the basic one not been covered? :) As seen here https://github.com/rvagg/node-levelup/wiki/Modules#storage

Or am I am wrong? I am a little confused because how is this dat related, isn't is more of Leveldb related, that is needs more adapters?

Or is there a good example on a plugin there is written?

taterbase commented

Just published dat-mongo to handle realtime updates from MongoDB with the listen hook.

yoshuawuyts commented

@calvinmetcalf If you're looking to build a pure JS levelDB this might be a good place to start -- https://github.com/rvagg/leveljs

calvinmetcalf commented

Ooo I like, thanks
On Oct 23, 2014 2:04 PM, "Yoshua Wuyts" notifications@github.com wrote:

@calvinmetcalf https://github.com/calvinmetcalf If you're looking to
build a pure JS levelDB this might be a good place to start --
https://github.com/rvagg/leveljs

—
Reply to this email directly or view it on GitHub
#5 (comment)
.

wikier commented

any interest to RDF / Linked Data support? I can work on a dat-marmotta implementation for testing it.

yueyingcui commented
  • Couchbase
almereyda commented

@wikier RDF / Marmotta here, please! We'll need it for http://transformap.co

wikier commented

so @almereyda, should we create an issue to organize the work, define the goals, etc?

almereyda commented

@wikier Yes, of course! We'll be squatting the CouchDB Meetup in Berlin next week, where I'm also inviting @finnp right now, to precise the requirements of the TransforMap data integration pipeline.

Please feel free to get started with your perspective, but I will only then be able to respond meaningfully.

almereyda commented

In the dawn of tomorrow, I'm stumbling over dat browser.

shawnbot commented

I can't believe nobody's mentioned SQLite. So I'll do it.

balupton commented

How about JSON, RDF, and Atom feed URLs? :-D

almereyda commented

Yeah, then I could transform Discourse' API into RSS :) Or we just plug dat
on SocketHub. Or these feeds better live there.

On 7 June 2015 at 16:10, Benjamin Lupton notifications@github.com wrote:

How about JSON, RDF, and Atom feed URLs? :-D

—
Reply to this email directly or view it on GitHub
#5 (comment)
.

fabianvf commented

Guide links in the issue are broken.

onbjerg commented

links to the guides are still broken

webmaven commented

@maxogden, perhaps edit the original comment on this issue to use [ ] [x] checkboxes instead of bullets?

patcon commented

+1 on CKAN