Extending UI with custom feature module

Question

Extending UI with custom feature module

Closed this issue 4 years ago · 26 comments

Hi,

I created a custom feature module that assigns each image a random score(one float number). I additionally created a boolean slider in the vitrivr-ng config file, to query all images that are in a specific range. How can I specify which feature to query for in the vitrivr-ng config file, i.e. what name should I use to reference the feature module I created?
Specifically, I created the feature module RandomAestheticScore and extracted the generated value for each image to the db. Furthermore, I added a new field in the cineast.json file referencing the feature module, under the features property called aestheticscore(see attachment) and added a slider in the vitrivr-ng config file under the boolean property(see attachment).

Best,

Ribin
vitrivr-config-files.zip

Answer 1 · 2020-09-21T08:39:49.000Z

Hey Ribin,

So there is two ways to tackle this problem, depending on what your concrete use case looks like. Currently, you're trying to mix them, which doesn't work.

If your score is calculated based on some query (i.e. user input), then creating a feature module, as you did, is the correct way of doing it. However, in order to use that feature you must create some UI component for the user to make their input since usually such features come with a specific way of query formulation.
There is no way to configure this - this involves coding of your own. You can have a look at ImageQueryTermComponent for how this can be done. Apart from the custom user input, you must specify the query category, which corresponds to the category you configured in cineast.json
If your score is basically constant per media object (i.e. not depending on any user input), then you simply store it in some additional entity in Cottontail DB (e.g. features_yourfeaturetable) and use boolean queries. To do so, you create the slider in vitrivr-ng as you did. Additionally, you must add an entry to cineast.json in the category boolean. For example, if you want to add a range query for the attribute score in features_yourfeaturetable, you create a JSON structure like this:

  {
    "feature": "RangeBooleanRetriever", "weight": 1.0,
    "properties": {
      "entity": "features_yourfeaturetable",
      "attribute": "your_score_attribute"
    }
  }
]

Let me know if you run into problems.

Best,
Ralph

Answer 2 · 2020-09-21T10:54:03.000Z

Hi Ralph,

In my case, I have a machine learning algorithm, which generates a json file containing all the images/videos, with the corresponding scores. So if per media object, I read the score data from a json file, I can go with the second option?

Best,

Ribin

Answer 3 · 2020-09-21T11:50:02.000Z

Yes!

Then you only need to prepare an entity with two columns: id, which holds the id of the segment, and score, which holds the actual score, and import your data. You can do this directly in Cottontail DB or via Cineast. Either way, it will involve some coding.

Afterwards, you should be able to configure the RangeBooleanRetriever feature as describe above and then use it with the range slider you've already setup in the UI.

Best,
Ralph

Answer 4 · 2020-09-21T12:31:30.000Z

Hey Ralph,

So I have added the RangeBooleanRetriever to cineast, but I get the following error in cineast, when I perform the query:

[qt-stage0-BOOLEAN] ERROR o.v.c.c.u.ReflectionHelper - java.lang.NoSuchMethodException: org.vitrivr.cineast.core.features.RangeBooleanRetriever.(java.util.LinkedHashMap)
at java.base/java.lang.Class.getConstructor0(Class.java:3349)
at java.base/java.lang.Class.getConstructor(Class.java:2151)
at org.vitrivr.cineast.core.util.ReflectionHelper.instanciate(ReflectionHelper.java:292)
at org.vitrivr.cineast.core.util.ReflectionHelper.instanciate(ReflectionHelper.java:278)
at org.vitrivr.cineast.standalone.config.RetrievalRuntimeConfig.getRetrieversByCategory(RetrievalRuntimeConfig.java:142)
at org.vitrivr.cineast.standalone.util.ContinuousRetrievalLogic.retrieve(ContinuousRetrievalLogic.java:37)
at org.vitrivr.cineast.api.websocket.handlers.queries.TemporalQueryMessageHandler.lambda$execute$4(TemporalQueryMessageHandler.java:99)
at java.base/java.lang.Thread.run(Thread.java:834)

I have seen that the constructors for the RangeBooleanRetriever class are protected and not public. Could this cause an issue?

Answer 5 · 2020-09-22T15:01:59.000Z

Hey,

Did you have any updates on this issue?

Best,

Ribin

Answer 6 · 2020-09-22T17:49:56.000Z

Hey, could you post the relevant part of the config you are using? For an example of how the RangeBooleanRetriever can be used, you can also have a look here: https://github.com/vitrivr/cineast/blob/lsc20/cineast.json#L109

Answer 7 · 2020-09-23T13:33:14.000Z

Hey Luca,

The example you provided me with was from the lsc20 branch, whereas I'm using cineast from the dev branch. Could that make a difference when I execute the code? Furthermore, the example references the entity features_table_lsc20meta. Is this a class under feature_modules? Where can I find it?

Best,

Ribin

Answer 8 · 2020-09-23T14:07:37.000Z

The branch should not matter for what you are trying to do, as there shouldn't be any changes in between them which affect the relevant logic. The entity does not refer to any class, but to the entity (table) in the database.

Answer 9 · 2020-09-23T14:09:30.000Z

Hey @cribin i think this is something which looks like it was fixed on the lsc20 branch but the fix was not yet merged to dev. Please give it another try.

Answer 10 · 2020-09-23T14:32:19.000Z

Thanks Silvan, now the error has disappeared. Now for the usage of the RangeBooleanRetriever I have two questions:

In cottontaildb-data, I have folder called entity_features_randomaestheticscore. Therefore, should the entity property in cineast.json and the vitrivr-ng config file both reference entity_features_randomaestheticscore?
What should the attribute property reference to?

In attachments I have included the two config files, as well as the new feature module I created.
config-files.zip

Answer 11 · 2020-09-23T15:27:45.000Z

1: In cineast.json you can remove all features which you have not extracted data for (probably all categories except boolean). the configs should both reference features_randomaestheticscore, cottontail simply prefixes the folder with entity.
2: The attribute property should reference the column you have stored your score in. To see the data stored in cottontaildb, write list in the cottontail CLI to find out under which schema your table is stored, and then write table cineast features_randomaestheticscore (assuming your table is in the cineast schema).

If you want to use cineast to extract your features, please use the new branch primitive-extraction-support and change your processSegment code to:

    if (shot.getMostRepresentativeFrame() == VideoFrame.EMPTY_VIDEO_FRAME) {
      return;
    }

    if (!phandler.idExists(shot.getId())) {
      float randomScore = getRandomAestheticScore();
      String shotId = shot.getId();
      persist(shotId, PrimitiveTypeProvider.fromObject(randomScore));
      System.out.println("Aesthetic Score for shotId: " + shotId + "score: " + randomScore);
    }

This will store your random scores in the feature column. If you import your data directly into cottontail, you do not need the RandomAestheticScore feature but can simply use the RangeBooleanRetriever

Answer 12 · 2020-09-24T07:22:37.000Z

Hey,

I did as you suggested: Both the UI and cineast config files reference the name features_randomaestheticscore. The column of the randomaestheticscore schema is simply called feature, so I added it to the property field. Furthermore, I checked out the primitive_extraction_support branch and modified the code as mentioned above. However, when I query from the ui, I don't receive any results and receive the following message in cineast(see below).
Do you know what the issue could be? Cottontaildb should be used from the master branch and vitrivr-ng from the dev branch correct? Here are the updated config files and the job file I used:
config-files.zip

DEBUG o.v.c.s.r.RetrievalTask - starting RangeBooleanRetriever 2020-09-24 09:11:19.486 [pool-5-thread-1-RangeBooleanRetriever] DEBUG o.v.c.c.f.a.BooleanRetriever - No relevant expressions in RangeBooleanRetriever for query BooleanQueryContainer[expressions=[BooleanExpression[attribute=features_randomaestheticscore,operator=BETWEEN,values=[IntProviderImpl [value=3], IntProviderImpl [value=7]]]],weight=1.0,id=<null>,superId=<null>,containerId=0] 2020-09-24 09:11:19.486 [pool-5-thread-1-RangeBooleanRetriever] DEBUG o.v.c.s.r.RetrievalTask - RangeBooleanRetriever.getSimilar() done in 1 ms, 0 results 2020-09-24 09:11:19.506 [qt-stage0-BOOLEAN] WARN o.v.c.a.w.h.q.TemporalQueryMessageHandler - No results found for category boolean and qt BOOLEAN in stage with id 0. Full compoment: org.vitrivr.cineast.api.messages.query.QueryStage@68aed516[ terms=[org.vitrivr.cineast.api.messages.query.QueryTerm@6c915cdb[ categories=[boolean] type=BOOLEAN data=data:application/json;base64,W3siYXR0cmlidXRlIjoiZmVhdHVyZXNfcmFuZG9tYWVzdGhldGljc2NvcmUiLCJvcGVyYXRvciI6IkJFVFdFRU4iLCJ2YWx1ZXMiOlszLDddfV0= cachedQueryContainer=BooleanQueryContainer[expressions=[BooleanExpression[attribute=features_randomaestheticscore,operator=BETWEEN,values=[IntProviderImpl [value=3], IntProviderImpl [value=7]]]],weight=1.0,id=<null>,superId=<null>,containerId=0] ]] config=<null> ] 2020-09-24 09:11:19.519 [query-msg-handler-223] WARN o.v.c.a.w.h.q.TemporalQueryMessageHandler - No relevant segments anymore, aborting staged querying 2020-09-24 09:11:19.519 [query-msg-handler-223] DEBUG o.v.c.a.w.h.q.TemporalQueryMessageHandler - Query executed in 35 ms

Best,

Ribin

Answer 13 · 2020-09-24T09:26:53.000Z

Ah sorry my bad, the frontend config should be ["Aesthetic Score", "RANGE", "features_randomaestheticscore.feature", 2, 8]
Apologies for the underdocumented code...

Answer 14 · 2020-09-24T09:44:01.000Z

I did the modication but still received the same message in cineast:

2020-09-24 11:42:15.598 [pool-4-thread-2-RangeBooleanRetriever] DEBUG o.v.c.s.r.RetrievalTask - starting RangeBooleanRetriever 2020-09-24 11:42:15.602 [pool-4-thread-2-RangeBooleanRetriever] DEBUG o.v.c.s.r.RetrievalTask - RangeBooleanRetriever.getSimilar() done in 4 ms, 0 results 2020-09-24 11:42:15.619 [qt-stage0-BOOLEAN] WARN o.v.c.a.w.h.q.TemporalQueryMessageHandler - No results found for category boolean and qt BOOLEAN in stage with id 0. Full compoment: org.vitrivr.cineast.api.messages.query.QueryStage@5daff06a[ terms=[org.vitrivr.cineast.api.messages.query.QueryTerm@59907f07[ categories=[boolean] type=BOOLEAN data=data:application/json;base64,W3siYXR0cmlidXRlIjoiZmVhdHVyZXNfcmFuZG9tYWVzdGhldGljc2NvcmUuZmVhdHVyZSIsIm9wZXJhdG9yIjoiQkVUV0VFTiIsInZhbHVlcyI6WzMsN119XQ== cachedQueryContainer=BooleanQueryContainer[expressions=[BooleanExpression[attribute=features_randomaestheticscore.feature,operator=BETWEEN,values=[IntProviderImpl [value=3], IntProviderImpl [value=7]]]],weight=1.0,id=<null>,superId=<null>,containerId=0] ]] config=<null> ] 2020-09-24 11:42:15.635 [query-msg-handler-6ef] WARN o.v.c.a.w.h.q.TemporalQueryMessageHandler - No relevant segments anymore, aborting staged querying 2020-09-24 11:42:15.636 [query-msg-handler-6ef] DEBUG o.v.c.a.w.h.q.TemporalQueryMessageHandler - Query executed in 38 ms

Answer 15 · 2020-09-24T09:56:27.000Z

Can you copy the first two rows of your feature table from cottontail? (see #44 (comment)) This just means no results were found

Answer 16 · 2020-09-24T09:59:41.000Z

You mean with the table cineast features_randomaestheticscore command? Cottontail does not recognize the table command( under help, this command does not appear). I receive the following error:

table cineast features_randomaestheticscore com.github.ajalt.clikt.core.UsageError: Got unexpected extra arguments (table cineast features_randomaestheticscore) at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:132) at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:14) at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:252) at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:249) at org.vitrivr.cottontail.cli.Cli.loop(Cli.kt:124) at org.vitrivr.cottontail.CottontailKt.startDB(Cottontail.kt:56) at org.vitrivr.cottontail.CottontailKt.main(Cottontail.kt:38)

Answer 17 · 2020-09-24T11:21:37.000Z

Ok, I used the preview command of cottontail, which should show me the first 10 elements of the table. When I run the command, I receive 0 results, is therefore the features_randomaestheticscore table empty? If yes, could it be a problem of my RandomAestheticScore class?
modified-code.zip

Answer 18 · 2020-09-24T15:16:12.000Z

Try pulling cottontail again on master, the table command was only added recently.
And yes, if there is no data in your table, then obviously no results are returned. Have a look at https://github.com/vitrivr/cottontaildb-examples/ to see how you can import data into cottontail.

Answer 19 · 2020-09-25T06:46:15.000Z

Hey Silvan,

Is writing directly to cottontail better practice than writing the data through a cineast extractor?

For our project I have the following idea to combine our Machine Learning Code with the vitrivr-stack:

The ML Project, which takes a folder of images/videos as input, predicts features on each of them, and finally outputs a JSON file, where each image/video is annotated with the predicted feature(A feature could be the aesthetic score of an image, the detected emotion and if people are visible, etc.).
In cineast I would add a Featuremodule for each feature the ML-Project predicts.
Cineast would pass the input folder containing the images and videos to my ML-Project, wait until the prediction is finished, and pass the annotated values from the output JSON to the corresponding FeatureModule classes. In this way, the entire interaction with the ML- Project would be implemented in cineast.

Is this the best approach to integrate my ML-Algorithm with the vitrivr-stack? If no, could you provide me with a better alternative?

Best,

Ribin

Answer 20 · 2020-09-25T07:34:34.000Z

I noticed, that the table is written to when I used ReadableFloatVector, but when I use PrimitiveTypeProvider.fromObject, the table remains empty. Should I open a new issue in the cottontaildb repository?

Answer 21 · 2020-09-25T08:55:50.000Z

Your way of getting the extracted information into vitrivr is a bit complicated. Wouldn't it be possible to call your feature extraction directly from within cineast? This would also enable you to store your results to the database directly without having to pass json files back and forth.
Regarding the problem with storing stuff to cottontail: every column in an entity is strongly typed. If you created an entity with a column for a vector, you won't be able to store any primitive types in that column. So if you want to change the data type of the column, you would need to drop the entity and re-create it with the types you actually need.

Answer 22 · 2020-09-25T09:09:48.000Z

Hey Luca,

Calling feature extraction directly from cineast would certainly be an option. Could you further clarify, what you mean by directly storing to the database? Do I create a new class in cineast that writes the extracted features to the db or is this already supported by cineast?
Where do I specify the type of the column vector? Currently, I set up the database simply by running the setup and extract method with cineast and the job.json file.

Answer 23 · 2020-09-25T09:36:46.000Z

When you implement the Extractor interface in your class, you can register it to be used during the feature extraction process. Cineast will then call the processSegment method for every element for which feature information can be extracted, in your case for every image. Once your computation is done, you should call the persist method which stores whatever you extract to the specified storage layer. You can configure the storage layer to be used in the cineast.json file.
To specify how the entity in the storage layer is to be structured, there is a method called initalizePersistentLayer which by default generates an entity with an id and a feature column where the feature column is of type vector. If you need a different storage layout, you need to overwrite this method with the appropriate settings for your feature.

Answer 24 · 2020-09-25T09:52:43.000Z

Is the functionality of defining a storage layout documented somewhere or can you provide me with an example

Answer 25 · 2020-09-25T11:23:33.000Z

It's not documented outside of the actual code as of yet. If you have a look at the other feature modules, you'll find several which use a custom layout, for example https://github.com/vitrivr/cineast/blob/master/cineast-core/src/main/java/org/vitrivr/cineast/core/features/AverageColorRaster.java#L300.

This issue has gone quite far of track now, the discussion has nothing to do with the issue any more. I'll close this for now, feel free to open another one related to any new problems you may encounter.

Answer 26 · 2020-09-28T08:39:46.000Z

Is the functionality of defining a storage layout documented somewhere or can you provide me with an example

Check out https://github.com/vitrivr/cineast/blob/primitive-extraction-support/cineast-core/src/main/java/org/vitrivr/cineast/core/features/example/PrimitiveFeatureExample.java as an example