youtype/mypy_boto3_builder

Are NotRequired marking accurate?

Opened this issue · 6 comments

takeda commented

Not sure if this a bug, or not, but want to understand it better.

So I tried to use aioboto3, in types package for aiobotocore for sqs I found the following types for the received message:

MessageTypeDef = TypedDict(
    "MessageTypeDef",
    {
        "MessageId": NotRequired[str],
        "ReceiptHandle": NotRequired[str],
        "MD5OfBody": NotRequired[str],
        "Body": NotRequired[str],
        "Attributes": NotRequired[Dict[MessageSystemAttributeNameType, str]],
        "MD5OfMessageAttributes": NotRequired[str],
        "MessageAttributes": NotRequired[Dict[str, MessageAttributeValueOutputTypeDef]],
    },
)

Now my attention was grabbed because mypy wasn't showing an error with when I used message["ReceiptHandle"] but pyright did, I first thought pyright was wrong, but when I found the type above, it looks like it is right. So my question is, is this correct?

The documentation implies that this is always set: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs/client/receive_message.html#receive-message you also need the handle to signal that it was received. Similarly with Body I would imagine it is always present as well.

vemel commented

Hello! A proper totality check is still not implemented in mypy: python/mypy#12094

There is a workaround with MyTypedDict = Union[MyTypedDictRequiredOnly, MyTypedDictOptionalOnly], but it is not reliable. I think it should ve fixed on mypy side.

Let me know if you have any ideas, or if you can make it work.

takeda commented

I see, but is "ReceiptHandle": NotRequired[str], or "Body": NotRequired[str], actually correct? I would expect them to be always present.

vemel commented

Yes. Unfortunately, ther eis no real way to tell if some field is always present, I can only rely on botocore shapes. THese fields are not marked as required in Message shape.

However, if a shape is an output (e.g., is a method response shape), all keys should be present. let me check.

takeda commented

I'm referring to this. According to https://youtype.github.io/boto3_stubs_docs/mypy_boto3_sqs/type_defs/#messagetypedef, AWS could just respond with [{}] (empty dictionary) and that would be a valid response?

Surely Body and ReceipientHandle need always to be present as I don't know what the correct behavior would be if they are not.

vemel commented

I will take a look. Could be a huge fix :)

ReceiveMessageResultTypeDef = TypedDict(
    "ReceiveMessageResultTypeDef",
    {
        "Messages": List[MessageTypeDef],
        "ResponseMetadata": ResponseMetadataTypeDef,
    },
)

As you see, MessageTypeDef is not total, however probably it should be.

I mark only direct response shape fields as required, but nested response shapes follow the same rules as argument shapes. As I said, there is no real way to tell if all keys in response form should be present or not, so I use botocore shape required fields.