How to make this work with some hosted database?
sergueif opened this issue · 9 comments
Hi, I really love this project and would love to write a whole bunch of good stuff on my wiki node, but because pages are stored in flat files, a single EC2 crash wiped half of my pages.
Rather than cobble together backup/restore processes, I'd love to use a durable database that's hosted and backed up / monitored by someone else (like Amazon or some db-as-a-service host)
Hence the question: Has anyone gotten fedwiki to work with a hosted database?
Preferably Amazon, but some other provider is fine too.
I'm sorry you have lost pages.
The ReadMe describes support we have for databases other than flat files.
https://github.com/fedwiki/wiki#datastore-options
This is the script I use to make daily backups of my farms.
# nightly backup of asia.wiki.org sites
# 40 4 * * * (cd wiki/asia; sh backup.sh)
when=`date +%a`
how='tar -chz .wiki'
ssh root@asia.wiki.org $how > store/$when.tgz
There are some storage plugins, but they have not been looked at for a very long time, and do not work with the wiki server in farm mode.
An alternative might be to look at using an EBS-backed instance to provide persistence.
I notice that all the existing storage plugins are blob-store like.
Is there anything about the fedwiki that would combine poorly with postgres/rds relational database? If it's really json blobs that fit fedwiki best, Posgres got the blob column type.
I'd like to take a stab within next 1-3 months of contributing either a RDS/PostgresSQL storage plugin or DynamoDB, both from Amazon, fully-managed, so they offer to take scheduled backups to s3.
Has anyone you know already made any efforts on either storage plugin?
We don't share the server's store other than by web requests for json structures. The structures are more like sentences in a grammar than rows in a table which is what leads us to blob storage.
There is a page that sketches this grammar and calls it the 'json schema' for pages.
http://ward.asia.wiki.org/json-schema.html
This and a few small details, like how we convert titles to slugs, are the essence of the federation.
I didn't fully answer your question because I've not used those amazon services. If you know your way around AWS and you are just doing personal stuff, you might try just backing up your EC2 instance to S3 on a timer. I find that I go to my own backups when I accidentally trash a page and want to just recover it. That is just one ssh/tar command with my system. I don't know how that would work with RDS/PostgresSQL automatic backups, probably full restores, nothing surgical.
We do offer end-user full site backups to a single json file. These can be dropped onto any site to bring up a page viewer from which backed-up pages can be 'forked' into the site. Here, for example, is an export of our glossary:
http://glossary.asia.wiki.org/system/export.json
If that were your site you could 'Save As' to, say, glossary-dec-2016.json to have a full backup with support for selective restore.
Thank you @WardCunningham . I am now backing up my wiki farm to an s3 bucket from a cronjob.
It was uncomfortable at first to settle on something like cron and scripts, because, I guess, coming from the industry, it is standard to separate the software such as this wiki from a database host for the purposes of scaling them out. But for this application, that is perhaps overkills.
On the other hand, I do wish for adoption of federated wiki in corporate settings, where such deployment practice might be a policy.
I'll keep this issue open, because technically farm mode is not or is poorly supported with a remote external database. But I'm ok if activity dies down, because the responses have been very helpful and I'm now running said solutions.
The linked issues in #24 suggest that at least the CouchDB adaptor is capable of farming. fedwiki/wiki-storage-couchdb#4
Is anyone using any database adapter successfully? How?