Database version or last updated time in the API?

[zx900930]

Can we get a database version or last updated date & time from the API?
So we can cache the data and update it when there is an update.

[RaphaelDDL]

It's kinda tricky.

I use MongoDB as the database because I didn't want the hassle of writing schemas (aka Mongoose and/or SQL). That means that anything I try import would import (or merge), even if all documents doesn't match structure. So ever since the beginning, I simply use mongoimport with --drop flag to drop current collections when importing updated data. That avoids having me to check documents in case we changed the json structure, as import would merge, thus making the document reflect old and new changes.

MongoDB doesn't store changed dates, and it only stores created dates and only if you use ObjectId as the _id, but in my case, I use my custom _id, which is a normalized name of the hero/artifact/item/etc.

So yeah, when there are updates, the DB is always "fresh", aka every document has same create date. In that way, it's not possible to have a way of showing when it was modified, in order to create a "latest changes" or something..

idk, I still haven't got a good idea on how implement it on current workflow

[zx900930]

Did some research

1.Using $currentDate

2.Using the System Variable NOW to get the current datetime value.

3.https://stackoverflow.com/a/61227379

Or simply don't store the date for every record. Only store the create date of every "fresh" DB, and make that date avaliable in the API as the database last udpated time.

[RaphaelDDL]

I've seen those, but doesn't work in my case:

1: I don't insert documents into DB programatically (aka calling insert()) with an UI or something of the like.. It's quite manual labor actually, I use the CLI mongoimport (from mongo-tools) where I pass the connection string and the final json to import, so the json must exist (which is the translated json I create on gamedatabase).

2: Same as above, since $NOW is a variable, means I must use it during insertion.

3: That one is the creation date, based on the standard ObjectId in the _id I mentioned on the previous reply, which I use a custom one (a string with normalized name) so can't extract creationdate from it.

Or simply don't store the date for every record. Only store the create date of every "fresh" DB, and make that date avaliable in the API as the database last udpated time.

Maybe that's the thing.. I guess I can make my translation script create a fake collection (of one item, lol) where is the date of the last translation build.

from datetime import datetime
buildTimestamp = datetime.now().timestamp()

and then use the unix buildTimestamp as the buildVersion. Then I could add that to the api as a $lookup and put it as part of the meta, like:

  "meta": {
    "requestDate": "Mon Jun 08 21:59:39 UTC 2020",
    "apiVersion": "2.1.0",
+    "buildVersion": 1591655141.886421
  }

[Permagate]

How about calculating checksum of the data? Mongodb has dbHash command that can be used for this purpose: https://docs.mongodb.com/manual/reference/command/dbHash. I'm not sure how big the mongo data is, but I think it should be small enough that running this command has negligible performance impact. So I'm thinking:

1. On API startup, create a worker that periodically call dbHash command.
2. Store the md5 value in-memory.
3. When rendering meta, just fetch the current in-memory md5 value.

[RaphaelDDL]

How about calculating checksum of the data? Mongodb has dbHash command that can be used for this purpose: https://docs.mongodb.com/manual/reference/command/dbHash. I'm not sure how big the mongo data is, but I think it should be small enough that running this command has negligible performance impact. So I'm thinking:

1. On API startup, create a worker that periodically call dbHash command.
2. Store the md5 value in-memory.
3. When rendering meta, just fetch the current in-memory md5 value.

That is awesome and I just did the code to support this but after coding everything and running to test, I was getting CMD_NOT_ALLOWED: dbHash. I thought was an issue with my user since I was using a read-only so I configured the permissions but still same thing. Then I discovered Atlas does not support dbHash on free tier db (M0, the one I use) and even on paid shared ones :'( https://docs.atlas.mongodb.com/reference/unsupported-commands/#unsupported-commands

For that feature, I'd have to pay M10 dedicated, which is "starting from $0.08/hr".. That would be at least $57.60 if consider 24h for 30days. Not possible.

I do have a mongodb installed on the VPS where the apps run because I was testing some stuff, but the lack of a GUI dashboard like Atlas provided made me stick to Atlas. Dang..

[Permagate]

That's unfortunate.

Well, instead of dbHash, I think a similar approach would be calculating the md5 checksum ourselves. So rather than calling dbHash command, the server can instead fetch all the collections and calculate the md5 checksum using nodejs native crypto package.

If you are cool or just curious if the above approach work, I can try to whip something up. I'm just wondering if the schema and/or data are available anywhere.