Cannot create seperate records per update or sth tables on MongoDB.

[YamacTan]

Hi,

I just started using FIWARE and I'm having trouble with the data I want to persist on MongoDB. I want to use STH Comet in minimal mode. I am using a Python script that creates my context data with a specific id and then updates certain attributes of this data. While I can't see any problem when I observe the JSON data from this script, and HTTP requests but MongoDB keeps only the last updated version of this data in the related collection. In addition, the "sth" labeled collections does not show up. However I can see the "credate" and "moddate" metadatas in attributes. But I need different records for every update.

Here is the data that I posted to Orion manually by Postman or by the Python Script that I mentioned:

{"id": "urn:ngsi-ld:entity:001", 
"type": "Log",
"a": {"type": "Datetime", "value": "17/09/2021"},
"b": {"type": "Datetime", "value": "12:00:18.0"}, 
"c": {"type": "Integer", "value": 49.51}, 
"d": {"type": "Integer", "value": 175.35}, 
"e": {"type": "Integer", "value": 24.25}, 
"f": {"type": "Integer", "value": 999.1}, 
"g": {"type": "Integer", "value": 85.87}, 
"h": {"type": "Integer", "value": -0.01}, 
"i": {"type": "Integer", "value": 37.41}, 
"j": {"type": "Integer", "value": 60.65}}

And here is the record on MongoDB when I update my data:

{
  "_id": {
    "id": "urn:ngsi-ld:entity:001",
    "type": "Log",
    "servicePath": "/"
  },
  "attrNames": [
    "a",
    "b",
    "c",
    "d",
    "e",
    "f",
    "g",
    "h",
    "i",
    "j"
  ],
  "attrs": {
    "a": {
      "type": "Datetime",
      "creDate": 1645514416,
      "modDate": 1645514416,
      "value": "17/09/2021",
      "mdNames": []
    },
    "b": {
      "value": "12:00:21.0",
      "type": "Datetime",
      "mdNames": [],
      "creDate": 1645514416,
      "modDate": 1645514597
    },
    "c": {
      "value": 666.47,
      "type": "Integer",
      "mdNames": [],
      "creDate": 1645514416,
      "modDate": 1645514640
    },
    "d": {
      "value": 175.55,
      "type": "Integer",
      "mdNames": [],
      "creDate": 1645514416,
      "modDate": 1645514597
    },
    "e": {
      "value": 24.27,
      "type": "Integer",
      "mdNames": [],
      "creDate": 1645514416,
      "modDate": 1645514597
    },
    "f": {
      "value": 999.28,
      "type": "Integer",
      "mdNames": [],
      "creDate": 1645514416,
      "modDate": 1645514597
    },
    "g": {
      "type": "Integer",
      "creDate": 1645514416,
      "modDate": 1645514416,
      "value": 85.87,
      "mdNames": []
    },
    "h": {
      "type": "Integer",
      "creDate": 1645514416,
      "modDate": 1645514416,
      "value": -0.01,
      "mdNames": []
    },
    "i": {
      "type": "Integer",
      "creDate": 1645514416,
      "modDate": 1645514416,
      "value": 37.41,
      "mdNames": []
    },
    "j": {
      "value": 1111.47,
      "type": "Integer",
      "mdNames": [],
      "creDate": 1645514416,
      "modDate": 1645514640
    }
  },
  "creDate": 1645514416,
  "modDate": 1645514640,
  "lastCorrelator": "679e1cba-93b0-11ec-be0f-0242ac120102" 

I've gone through all the docs but either couldn't find the solution or probably I am missing something in the logic. I need to create different records as a tabular format as a technical requirement and use 3rd party OSSs like Grafana. How should I go about the issue?

[AlvaroVega]

Hi, if you want to use STH Comet an option is to use fiware-cygnus with sth-sink. This way, after insert data into orion a notification will invoke cygnus that will create in mongo a full historic (not just the last inserted value) that will be used by STH Comet.
https://fiware-cygnus.readthedocs.io/en/latest/cygnus-ngsi/flume_extensions_catalogue/ngsi_sth_sink/index.html

[fgalan]

@YamacTan actually I think this is more a question about integration of componentes and functional behaviour than an issue in the software. Thus, I think it should be posted in StackOverflow https://stackoverflow.com (use fiware-orion and fiware-sth-comet labels).

If you do so, please add a link to the question as comment in this issue. Then, close the issue.

[YamacTan]

@AlvaroVega Thanks a lot for quick response. When I use Draco instead of Cygnus, I had the same issue but I will try Cygnus today.

@fgalan Thanks for your interest. Actually I had a StackOverflow post which I created when I had a problem with Draco. I updated the post from scratch and here is the link:
https://stackoverflow.com/questions/71218728/how-to-create-different-record-for-every-update-in-fiware-sth-comet

Thank you all for your cooperation.
BR,
Y.