DynamoDB TypeDeserializer support encoding by aws step functions operations

[johnclarke94]

Describe the feature

The TypeDeserializer can't currently decode all items returned from arn:aws:states:::aws-sdk:dynamodb:scan by AWS step functions (and presumably other database operations).

This is caused by non-standard encoding where NULL is Nul and BOOL is Bool within a nested map. Top level attributes do however use the more standard NULL and BOOL.

Use Case

To be able to handle this form of encoding provided by AWS without a custom wrapper which will need to be added by many users. It can easily lead to unexpected errors when this special encoding is not known by a developer. It would be great to have this abstracted by boto3.

The current implementation returns this exception or similar for Bool

[ERROR] TypeError: Dynamodb type Nul is not supported
Traceback (most recent call last):
  File "/var/lang/lib/python3.12/site-packages/boto3/dynamodb/types.py", line 280, in deserialize
    return deserializer(value[dynamodb_type])
  File "/var/lang/lib/python3.12/site-packages/boto3/dynamodb/types.py", line 310, in _deserialize_m
    return {k: self.deserialize(v) for k, v in value.items()}
  File "/var/lang/lib/python3.12/site-packages/boto3/dynamodb/types.py", line 279, in deserialize
    raise TypeError(f'Dynamodb type {dynamodb_type} is not supported')

Proposed Solution

I believe a minor update to the TypeDeserializer to understand these two additional cases.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

1.34.107

Environment details (OS name and version, etc.)

Python 3.12 on AWS Lambda

[tim-finnigan]

Thanks for reaching out. It looks like similar issues have been reported in other AWS SDK repositories, for example: aws/aws-sdk-js#4294, aws/aws-sdk-js-v3#5423, aws/aws-sdk-js-v3#4948

Based on this comment it sounds like the Step Functions behavior is expected, but has forced workarounds like the one mentioned here in another comment. Since this issue is upstream with Step Functions API behavior, it probably needs to either be addressed by them or further clarified in their documentation. I'll try reaching out internally to get more feedback on this.

[tim-finnigan]

I heard back from the Step Functions team, who wanted to highlight the updated wording on parameters in their documentation:

Parameters in Step Functions are expressed in PascalCase, even if the native service API is in camelCase.

They also created a backlog item for tracking potential improvements to this workflow. But any future changes would need to be backwards compatible in order to not break customers who rely on the current behavior.

Ideally the Step Functions team can provide a solution to improve integration with DynamoDB connectors (along with any other services that might be affected.) If this were to be addressed somehow on the SDK side, then that would require coordination with other AWS SDKs, along with the Step Functions team, in order to provide consistent behavior. For now it should be a straightforward workaround to convert the types manually. In the long term, we probably need to wait for the Step Functions team to research this further and propose alternative approaches.