xebia-functional/nine-cards-v2

Backend Redis Cache: Non-Blocking Client.

Closed this issue · 9 comments

diesalbla commented

The Nine Cards Backend stores some data in a Redis cache, for which it needs a client library. At present, it is using scala-redis, but this is a blocking library, which does not fit well with the design of the backend.

The goal of this ticket is to switch the backend to use a non-blocking library, such as scredis or rediscala.

diesalbla commented

Local Quality Assurance

The goal of developer's local machine QA is to verify that the new implementation is correct, and that both the processes for consulting new packages, recording errors, and resolving pending packages work as expected.

The functionality tested in the QA is the API endpoint /applications/details, and the pending package resolution actor.

Edited For this tests, we need a pair <Android_Id>,<Google_Play_Token> that we know to be accepted in the Android Market API. To run the tests, you should execute the SQL script in /assets/postman/setup.sql, to record the ann user, and then, inside a psql console with the ninecards_user user, execute the following command
insert into installations(userid,devicetoken,androidid) values ( (select id from users where sessiontoken='anntok'), 'toktoktok', '<Android_Id>') ;

1 Fetching and storing data from Android Market API.

Clear the cache using the flushall command inside a redis-cli session. Run the command
sbt -Dconfig.file="modules/api/src/main/resources/localhost.conf" "api/run" to start the server. In another command line, execute the following command (Edited):

curl -X POST \
  -H "X-Session-Token: anntok"\
   -H "X-Auth-Token: dds"\
   -H "X-Android-ID: <Android_Id>"\
   -H "X-Google-Play-Token: <Google_Play_Token>"\
   -H "Accept: application/json" -H "Content-Type: application/json" \
  -d '{  "items" : [ "flipboard.app", "does.not.exist"  ] }' \
  "http://localhost:8080/applications/details"

To check the results, open a redis-cli session in another terminal and inspects the contents of the cache with the keys * command. There should be a key flipboard.app:Resolved, whose contents is a full card (encoded in JSON), and a key does.not.exist:Error key, and nothing more.

2 Storing pending packages in the cache (Edited)

Clear the cache again. To force packages to be added to the pending queue, we run the previous curl command but replacing the value of the X-Google-Play-Token with any mock value.

Expectations: (Edited) In the redis cache, there should only be a pending_packages key, whose type, which can be seen with the command type pending_packages, should indicate that it is a set.
To see the elements in the set, execute the command smembers pending_packages. The result should include both package names.

Note: that the resolveInterval should be large enough so that the resolver actor does not kick in too early. At present, the application.conf sets it to 1 day.

3 Automatic Package Resolution from cache

(Edited). Clear the cache. In a redis-cli session, insert several packages into the cache by running the following command:

sadd pending_packages "flipboard.app" "com.twitter.android" "com.facebook.katana" "does.not.exist"

You may also add any other packages to the queue. To make the resolution process quicker and to observe it step by step, add the following lines to the localhost.conf file:

ninecards.google.play.resolveInterval = 10 seconds
ninecards.google.play.resolveBatchSize = 1

Start the server with the modified file. The resolving actor should be solving one package once every ten seconds. After each execution, pause the server app (Ctrl-Z), and inspect the contents of the cache. Packages resolved to a card or to an error should appear in a new key, and the name of the package should not appear now in the pending_packages set.

4 Resolution of Error or Pending Packages (Edited)

Clear the cache, and insert into the cache a pending element and an error key, using the following commands:

sadd pending_packages "flipboard.app" 
lpush "com.twitter.android:Error" "170101120001000"

Start the backend server using an unmodified localhost.conf, and perform the command from the first tests, except that the request object should now be the following one:

    "items" : [ "flipboard.app",  "com.twitter.android  ]

After running the command, the cache should only contain two resolved entries, for the package names flipboard.app and com.twitter.android. The error entry should have been removed, and the pending_packages key should have disappeared.

diesalbla commented

@javipacheco ¿Could you perform the checks outlined above?

javipacheco commented

I have had a lot of problems in this QA

First, the localhost.conf in github doesn't have the information about Google Play and the packages can't be resolved. I had added google node to localhost.conf from application.conf and the packages are resolved. We should review that for QAing

I have had error in every step:

1 Fetching and storing data from Android Market API.

The does.not.exist:Error key wasn't created. This package was added to pending_packages list

2 Storing pending packages in the cache

I can't change -H "X-Android-ID: foo" because the authentification doesn't work. I have had to use my Android-Id and then, it works

The does.not.exist:Error key wasn't created. The 2 packages have been added to pending_packages list

3 Automatic Package Resolution from cache

I have used sadd command and only one key was created

127.0.0.1:6379> keys *
1) "flipboard.app"

I don't know if that is correct

After 10 seconds, the actor have been executed with this message

The server returned 0 packages to the pending queue

I have stopped and I didn't try the last step

javipacheco commented

I have tried the code in master and the errors are generated

127.0.0.1:6379> keys *
1) "does.not.exist:Error"
2) "flipboard.app:Resolved"
diesalbla commented

@javipacheco I have checked again the process of the QA. There were several mistakes written.
I have edited the process in the following manner:

  • I have added the database setup step, to be run before the first test.
  • In the test 1), in the script, I have modified the sessionToken header, to the one created in the setup.
  • In the test 2), the one with the unauthorized credential, It is not necessary to change the AndroidId, only the X-Google-Play-Token header. I have changed the expectations, since the default behavior for this case is to mark all packages as pending.
  • In test 3), I have corrected the Redis-cli command to set the pending queue.
  • In test 4), I have extended the scope of the test, clarified the setup, and added some more details.

¿Could you run the QA again?

javipacheco commented

I have an error in the 4 step

I have called to the endpoint and I have receive that:

{
  "errors": ["flipboard.app", "com.twitter.android"],
  "items": []
}

But the keys have been resolved in REDIS:

1) "flipboard.app:Resolved"
2) "com.twitter.android:Resolved"

Then, we are resolving the packages but we are sending errors to the user

The second time that I have called to the endpoint, I have received the correct JSON

javipacheco commented

My bad... The 4 step works fine

LGTM!

diesalbla commented

The previous QA steps were only considering the operations and endpoints with the cache of applications. However, this ticket has also modified the operation of the cache of rankings.
In what follows, we describe the QA for the endpoints involving application rankings.

Setup

The endpoints require us a user's sessionToken and androidId which are stored in the database. The setup.sql described above, can be used to introduced anntok and anndroidid for these.

We need to use a big enough data set for the ranking cache. For that, we are going to duplicate into our local cache the current cache in the backend's Heroku.

Get Access Credentials to Heroku Redis Cache.

In your local machine, install the [Heroku Command Line Interfacehttps://devcenter.heroku.com/articles/heroku-cli). In Ubuntu, this is sudo apt-get install heroku. Login into yout account using the heroku:login command.

Using the Redis Plugin for the Heroku CLI, we obtain the account's credentials for the Redis cache, with the command heroku redis:credentials --app APP, where APP is the name of the APP in Heroku.
The answer should be of the form redis://{user}:{CRED}@{HOST}:{PORT}.

These credentials can be used to start a local client against the Redis Server in Heroku, by doing the following: redis-cli -h HOST -p PORT -a CRED.

Replicate the Remote Server into your local Redis Server.

Open a local redis-cli client that access to your local server. Clear the data using FLUSHALL, and then make your local server into a mirror of the remote one with the slaveof command, as follows:

  1. Connect your local server to the remote server using the slaveof HOST PORT command.
  2. Authorize your local server into the remote server, using the config set masterauth CRED command.

Your local server should at this point start copying all data from the remote into its own. After a while

  1. Run the command keys * to see the contents of your local cache.
  2. If satisfactory, run the command slaveof no one to disconnect your local server from the remote one. The data will remain.

Tests

The postman collection contains several examples for each endpoint. You should try running the following ones:

  • GET /rankings/world
  • GET /rankings/countries/ES
  • POST /applications/rank
  • POST /widgets/rank.
javipacheco commented

Good work. All steps work as expected. LGTM!