arrowheadapps/strapi-connector-firestore

Prepare for stable release

brettwillis opened this issue · 6 comments

brettwillis commented

I bootstrapped this codebase from the official strapi-connector-mongoose package at version 3.0.0, because I figured that, because Mongoose is also a NoSQL database, that package would be the best starting point.

https://github.com/strapi/strapi/tree/v3.0.0/packages/strapi-connector-mongoose

Simple operations and relations seem to be working, but most features (particularly components and more complicated relations) have either been only lightly tested, or not tested at all.

To do: see project.

Contributions and pull requests welcome, because I have limited capacity to complete and maintain this package.

See also strapi/strapi#530 and strapi/strapi#5529.

rrubio commented

@brettwillis this is a brilliant project, thank you. Is there anything particular you need help on? or do we just choose from the board?

brettwillis commented

@rrubio thanks, it would be great to have your support! The end goal is getting the tests to pass. Most of the failures are related to #23 and #15, which require a bit of a deep dive I haven't had time for. If you review the test failures there may be some smaller issues hiding in there, particularly in the search and filtering tests (look for a test suite that mostly passes but has a small number of failures).

  • 👍1
rrubio commented

@brettwillis i'll be able to do some testing this coming weekend. Let you know how i go.

rrubio commented

Load & Response Times (CRUD)

Test case included the following -

  1. created 2 new content-type with 3 fields of string value
  2. enter new records + edit + delete
  3. navigating between menu options

Response times for each -

  1. New Content-Type (after save reload) - 10+ seconds
  2. Collection types (load) - 10+ seconds
  3. New record entry (after save reload) - 20 seconds
  4. Edit (after save reload) - 20 seconds
  5. Switching between menues - 10+ seconds for first paint

Observations -

Cloud Firestore metrics show that after each save of new record/edit, on reload it equates to 100 reads.

brettwillis commented

Thanks for the data.

Yes... Strapi itself is designed in a way that causes a huge amount Firestore operations, especially when operating Strapi's front-end admin dashboard. This does not suit Firestore's usage-based pricing model.

Here are some of my observations:

  • The admin front-end counts the number of documents every time a collection is listed. This means Firestore will read every single document in the collection - disasterous for large collections.
  • The users-permissions plugin introduces a huge amount of overhead, because it fetches users, permissions, and roles on every operation. There are CRUD permissions for every single data type, so that can be a huge amount of permissions.
  • There are a large amount of operations when the backend server starts up. So it's best to use a relatively persistent server like Cloud Run or App Engine, not Cloud Functions.

Personally, I think using users-permissions plugin with Firestore in production is a bad idea, because of the overhead it introduces. Best to replace it with something that uses Firebase Auth, and eliminate all the roles and permissions, if you can. This is what I'm trying to do.

Also, regarding the latency, I found it drastically improves performance when the Strapi backend server is running local to the Firestore database (i.e. in the same GCP location). When you run the server on your local machine, the performance is poor because of the many round trips to Firestore database on every operation.

What do you suggest? Some warnings in the documentation at least... PRs to Strapi itself?

brettwillis commented

I opened a separate issue for the counting problem - #33