Non-existing attribute is returned and causes float conversion error

Question

Non-existing attribute is returned and causes float conversion error

Opened this issue 9 months ago · 13 comments

Describe the bug

For this entity:

{
    _id: {
        id: 'conCenFan_oveAct_u',
        type: 'Actuator',
        servicePath: '/ALPaperStage3'
    },
    attrNames: [
        'conCenFan_oveAct_u_status',
        'conCenFan_oveAct_u_info',
        'TimeInstant'
    ],
    attrs: {
        conCenFan_oveAct_u_status: {
            type: 'commandStatus',
            creDate: 1692798690.8602214,
            modDate: 1693224142.4930723,
            value: 'OK',
            mdNames: [
                'TimeInstant'
            ],
            md: {
                TimeInstant: {
                    type: 'DateTime',
                    value: 1693224142.4899998
                }
            }
        },
        conCenFan_oveAct_u_info: {
            type: 'commandResult',
            creDate: 1692798690.8602214,
            modDate: 1693224142.4930823,
            value: 9.99729039,
            mdNames: [
                'TimeInstant'
            ],
            md: {
                TimeInstant: {
                    type: 'DateTime',
                    value: 1693224142.4899998
                }
            }
        },
        TimeInstant: {
            type: 'DateTime',
            creDate: 1692798690.8602214,
            modDate: 1696261274.039428,
            value: 1696261274.0340004,
            mdNames: []
        }
    },
    creDate: 1692798690.8602214,
    modDate: 1696261274.0394897,
    lastCorrelator: '10a74dbf-2355-4470-be58-a1638b7868b1'
}

I get the following attributes when calling get_attributes in ngsi_v2:

{
  'TimeInstant': {
    'type': 'DateTime',
    'value': '2023-10-02T15:41:14.034Z',
    'metadata': {}
  },
  'conCenFan_oveAct_u_info': {
    'type': 'commandResult',
    'value': 9.99729039,
    'metadata': {
      'TimeInstant': {
        'type': 'DateTime',
        'value': '2023-08-28T12:02:22.490Z'
      }
    }
  },
  'conCenFan_oveAct_u_status': {
    'type': 'commandStatus',
    'value': 'OK',
    'metadata': {
      'TimeInstant': {
        'type': 'DateTime',
        'value': '2023-08-28T12:02:22.490Z'
      }
    }
  },
  'conCenFan_oveAct_u': {
    'type': 'Number',
    'value': '',
    'metadata': {}
  }
}

The attribute conCenFan_oveAct_u does not exist, and as the value is an empty string but the type a Number, I get a float conversion error.

Answer 1 · 2023-10-05T13:09:02.000Z

It seems to me that conCenFan_oveAct_u is a command in IoTAent, and the entity attribute is also created by IoTAent.

If that is the case, I will suggest to use command as the type. For example:

  {
    "protocol": "IoTA-JSON",
    "transport": "MQTT",
    "explicitAttrs": false,
    "device_id":  ???,
    "entity_name": "conCenFan_oveAct_u",
    "entity_type": "Actuator",
    "commands": [
      {
        "name": "conCenFan_oveAct_u",
        "type": "command"
      }
    ],
    "attributes": [
      ...
    ],
    "ngsiVersion": "v2"
  }

Answer 2 · 2023-10-05T13:56:05.000Z

That is correct. But the type of the command is a Number, not "command". And we never changed this implementation, it was working some months ago.

Answer 3 · 2023-10-05T15:35:20.000Z

It works specifying the type="command", but I don't get why this only occurs now. We use the development fiware cluster, did you update the iotagent / orion?

Answer 4 · 2023-10-05T15:38:34.000Z

As far as I know, IoTagent will always create such an attribute in Orion. So I think the creation of conCenFan_oveAct_u is not a bug.

And still, I recommend using command as the data type, because it will be then easier to recognize the command in context broker, and this kind of attribute will not have any meaningful value anyway. The value is stored in conCenFan_oveAct_u_info

Apart from that, are you expecting filip to ignore such a data type error?

{
  "conCenFan_oveAct_u": {
    "type": "Number",
    "value": ""
  }
}

Answer 5 · 2023-10-05T15:40:49.000Z

ok, thanks! Yes, filip should avoid this error. The attr is not even listed in mongo-express. As the error only occurs due to the conversion from "" to float, this should be catched using some sort of float(value) if value else None

Answer 6 · 2023-10-05T15:58:49.000Z

I agree. I will have a look

Answer 7 · 2023-10-05T15:59:06.000Z

Branch 203-Non-existing-attribute-is-returned-and-causes-float-conversion-error created!

Answer 8 · 2023-10-17T19:41:26.000Z

@FWuellhorst @djs0109 I just faced the exact same issue. I looked into it and the trouble occurs in the ContextAttribute creation and validation.

Let's start at the beginning:

You cannot see the attribute in the database because does not live in the ContextBroker database. It comes from a registraion that the IoTA-Agent creates for it. For more details check on the registration docs
When calling the entity the contextbroker fowards the request to the iotagent and receives the data. Maybe they changed the default value, if the command is empty.
ToFix: We need to adjust filip.models.ngsi_v2.base.BaseValueAttribute.validate_value_type to handle EmptyStrings. Otherwise we need to use the the command type so far, which I probably would recommend anyways. If you want to add additional info about the type maybe the metadata field works by now for commands. The latter would require to adjust the command model which would be a minor issue filip.models.ngsi_v2.iot.DeviceCommand. Let's check on the docs! Here we go: https://iotagent-node-lib.readthedocs.io/en/latest/api.html
From the docs commands should always have the type command. Probably I followed that in the original implementation but forgot to fix the type in the CommandModel.

I'm happy to help out here :) It is true that this is annoying. However, I will not be free until November. Maybe @FWuellhorst @djs0109 you can already try to realize the second option.

Answer 9 · 2023-10-17T20:05:37.000Z

Oh, btw @FWuellhorst. If you use command you can access them directly using the api of the ContextEntity ;) same for relationships :)

Answer 10 · 2023-10-18T07:04:53.000Z

@tstorek Thanks for your suggestions! I saw that if the data type is command, the value will not be validated in filip.models.ngsi_v2.base.BaseValueAttribute.validate_value_type. That is to say, if using command as type, empty string will not cause any problem.

For adding the metadata in DeviceModel, I check the docs. It is actually contradictory... some part suggests that only attribute can have metadata, and the other part says that command can also have metadata.
So I test it with LibVersion 3.3, JSON-version 2.3:

Metadata in command will cause syntax error

    "commands": [
      {
        "name": "com1",
        "type": "command",
        "metadata": {
            "unitCode": { "type": "Text", "value": "CAL" }
            }
      }
    ]

But metadata in attributes can work

    "attributes": [
      {
        "name": "attr1",
        "type": "Integer",
        "metadata": {
            "unitCode": { "type": "Text", "value": "CAL" }
            }
      }  
    ],

Answer 11 · 2023-10-18T07:14:39.000Z

@djs0109 @FWuellhorst According to the previous test, we should do the two things:

Raise an issue in IoTA-Json for metadata
Agree on a conversation-convention in filip

What Do you think? Shall we allow any types favoring the missing metadata object? Mostly likely it would be useful also for the other conversions, right?

Answer 12 · 2023-11-06T10:45:36.000Z

Okay, thanks for checking! I would suggest to fix the validate_value_type function for this issue, the rest would be another issue, or?

Answer 13 · 2023-11-06T15:15:32.000Z

I think I would add an if statement to every data type, i.e. to explicitly allow missing values in the form of None, and "". float/int/str(value) if value else None will be too risky, because it will ignore for example 0