Export Data from Google Docs to Github and or Datahub.io to make it easy to use

Question

Export Data from Google Docs to Github and or Datahub.io to make it easy to use

Opened this issue 11 years ago · 10 comments

It may be easier to see and reuse the data if we can store it locally in this repo or else export it to datahub.io -

It looks like datahub.io can store a csv file so it may be a simple export and import and I beleive that it might be fairly easy to create an interface with recline.js

Also recline.js looks like it would be pretty easy to make an interface either to datahub or to the google spreadsheet

I am not sure about github - maybe just a link to the spreadsheet would suffice

Answer 1 · 2014-02-04T16:28:56.000Z

Sounds like a good strategy to export it to datahub, and recline.js for an
interface. I'm not sure whether/how github deals with csv, almost certainly
won't be user-friendly to edit. I may have a little time tonight, but
otherwise busy until Thursday.

On Tue, Feb 4, 2014 at 4:08 PM, Tom Brooke notifications@github.com wrote:

It may be easier to see and reuse the data if we can store it locally in
this repo or else export it to datahub.io -

It looks like datahub.io can store a csv file so it may be a simple
export and import and I beleive that it might be fairly easy to create an
interface with recline.js

Also recline.js looks like it would be pretty easy to make an interface
either to datahub or to the google spreadsheet

I am not sure about github - maybe just a link to the spreadsheet would
suffice

Reply to this email directly or view it on GitHubhttps://github.com//issues/2
.

PGP ID: EF2F2BB1
PGP Fingerprint: http://reubenbinns.com/mykey.html

Answer 2 · 2014-02-04T16:46:13.000Z

Found this re: basic search interface - this has a google docs backend but
datahub might be just as easy or easier

I saw good docs but didn't see anything to fork to github - looks like
fairly easy javascript - I have javascript pro I should be able to pull in
if we need help

if we can find online links to recline we may be able to host it on a
github page

http://okfnlabs.org/recline/demos/search/?backend=gdocs&url=https://docs.google.com/spreadsheet/ccc?key=0Aon3JiuouxLUdExXSTl2Y01xZEszOTBFZjVzcGtzVVE

On Tue, Feb 4, 2014 at 11:28 AM, RDBinns notifications@github.com wrote:

Sounds like a good strategy to export it to datahub, and recline.js for an
interface. I'm not sure whether/how github deals with csv, almost certainly
won't be user-friendly to edit. I may have a little time tonight, but
otherwise busy until Thursday.

On Tue, Feb 4, 2014 at 4:08 PM, Tom Brooke notifications@github.com
wrote:

It may be easier to see and reuse the data if we can store it locally in
this repo or else export it to datahub.io -

It looks like datahub.io can store a csv file so it may be a simple
export and import and I beleive that it might be fairly easy to create an
interface with recline.js

Also recline.js looks like it would be pretty easy to make an interface
either to datahub or to the google spreadsheet

I am not sure about github - maybe just a link to the spreadsheet would
suffice

Reply to this email directly or view it on GitHub<
https://github.com/Open-Notice/on-msa/issues/2>
.

PGP ID: EF2F2BB1
PGP Fingerprint: http://reubenbinns.com/mykey.html

Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-34077621
.

Answer 3 · 2014-02-05T19:45:41.000Z

Datasets on datahub need to be attached to an 'organisation'. So we could either create a new org, or attach this to my existing organisation account on datahub which is called 'Privacy and Data Protection'. This dataset definitely fits into the scope so I'm happy to put it there if you all are?

Answer 4 · 2014-02-05T20:01:30.000Z

Yes.. Reuben, that sounds great for this prototype period.

mary

On Feb 5, 2014, at 1:45 PM, RDBinns wrote:

Datasets on datahub need to be attached to an 'organisation'. So we could either create a new org, or attach this to my existing organisation account on datahub which is called 'Privacy and Data Protection'. This dataset definitely fits into the scope so I'm happy to put it there if you all are?

—
Reply to this email directly or view it on GitHub.

Answer 5 · 2014-02-06T00:01:18.000Z

Ruben I noticed that they needed to be attached to organizations and yours would be perfect

Answer 6 · 2014-02-06T09:10:44.000Z

So it's relatively simple to get the spreadsheet to serve JSON. The document has to be made public to everyone, then file -> publish to the web. We get a key for the spreadsheet, which we put into the following URL:

https://spreadsheets.google.com/feeds/cells/[KEY GOES HERE]/od6/public/basic?hl=en_US&alt=json

I copied a pared down version of the spreadsheet and tried this out - here are the results:

https://spreadsheets.google.com/feeds/cells/0AlqMaTMcz6UzdEtvaGhIMy1VVjlwV3l4Q1ZTZko0T1E/od6/public/basic?hl=en_US&alt=json

As you can see, the downside is it generates a load of additional ugly JSON data that we don't want. But if we can iron out that we have a basic JSON API to our spreadsheet that will be continually updated whenever someone makes a change to the google spreadsheet.

Answer 7 · 2014-02-06T14:06:38.000Z

For the hardware hack, we are going to put the relevant data on a postgres
dB on an amazon server (that should be running this eve)
That data will be available as json via an https REST request to drupal (or
I guess WordPress, we just know drupal better)
On 6 Feb 2014 09:10, "RDBinns" notifications@github.com wrote:

So it's relatively simple to get the spreadsheet to serve JSON. The
document has to be made public to everyone, then file -> publish to the
web. We get a key for the spreadsheet, which we put into the following URL:

https://spreadsheets.google.com/feeds/cells/[KEY GOES
HERE]/od6/public/basic?hl=en_US&alt=json

I copied a pared down version of the spreadsheet and tried this out - here
are the results:

https://spreadsheets.google.com/feeds/cells/0AlqMaTMcz6UzdEtvaGhIMy1VVjlwV3l4Q1ZTZko0T1E/od6/public/basic?hl=en_US&alt=json

As you can see, the downside is it generates a load of additional ugly
JSON data that we don't want. But if we can iron out that we have a basic
JSON API to our spreadsheet that will be continually updated whenever
someone makes a change to the google spreadsheet.

Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-34304466
.

Answer 8 · 2014-02-06T14:13:04.000Z

Hi Chris! That's fantastic. Would it be OK for other teams taking part to make use of that data source? Is there anything we can do in the meantime to help with that aspect of your project?

Answer 9 · 2014-02-06T14:32:15.000Z

But of course! We'll publish the credentials and access URI's as soon as
the machine is rocking. It won't be perfect but I can hack new views while
we are working.
So can anyone else, when it comes that.
But I want to restrict access to https and SSH just to stop drive-by
scriptkiddies, who can be a bore.
More details when Simon si@stealthservices.co.uk fires it up, this eve I
hope.
Any other services needed on this thing, just ask. It'll be a Linux machine
with the standard stuff there, http, database, Drupal, WordPress, Java, ant
, python, sendmail blah blah
On 6 Feb 2014 14:13, "RDBinns" notifications@github.com wrote:

Hi Chris! That's fantastic. Would it be OK for other teams taking part to
make use of that data source? Is there anything we can do in the meantime
to help with that aspect of your project?

Reply to this email directly or view it on GitHubhttps://github.com//issues/2#issuecomment-34326000
.

Answer 10 · 2014-02-06T17:44:36.000Z

If it helps, I've written a simple python script to take the relevant bits of the consent legal map and put them into a JSON file (see csv2json.py in the repo).