System.OutOfMemoryException: Cannot allocate a buffer of size 2147488223 during migration data from Cosmos DB to JSON

[yalmaty]

When using the Cosmos DB source, a System.OutOfMemoryException is thrown when reading from the database. It seems like the code is trying to load the entire data into memory before running the import, and throwing the exception. The data set size I am trying to download is much bigger than 2Gb

Platform: MacBook Pro M1
Release: 2.1.1

fail: Cosmos.DataTransfer.Core.RunCommand.CommandHandler[0]
Data transfer failed
System.OutOfMemoryException: Cannot allocate a buffer of size 2147488223.
at System.Text.Json.ThrowHelper.ThrowOutOfMemoryException(UInt32 capacity)
at System.Text.Json.Utf8JsonWriter.Grow(Int32 requiredSize)
at System.Text.Json.Utf8JsonWriter.WriteStringMinimized(ReadOnlySpan1 escapedPropertyName, ReadOnlySpan1 escapedValue)
at System.Text.Json.Utf8JsonWriter.WriteStringEscapePropertyOrValue(ReadOnlySpan1 propertyName, ReadOnlySpan1 value, Int32 firstEscapeIndexProp, Int32 firstEscapeIndexVal)
at System.Text.Json.Utf8JsonWriter.WriteString(String propertyName, String value)
at Cosmos.DataTransfer.Interfaces.DataItemJsonConverter.WriteFieldValue(Utf8JsonWriter writer, String fieldName, Object fieldValue, Boolean includeNullFields) in /data-migration-desktop-tool/Interfaces/Cosmos.DataTransfer.Interfaces/DataItemJsonConverter.cs:line 98
at Cosmos.DataTransfer.Interfaces.DataItemJsonConverter.WriteDataItem(Utf8JsonWriter writer, IDataItem item, Boolean includeNullFields, String objectName) in /data-migration-desktop-tool/Interfaces/Cosmos.DataTransfer.Interfaces/DataItemJsonConverter.cs:line 35
at Cosmos.DataTransfer.Interfaces.DataItemJsonConverter.WriteFieldValue(Utf8JsonWriter writer, String fieldName, Object fieldValue, Boolean includeNullFields) in /data-migration-desktop-tool/Interfaces/Cosmos.DataTransfer.Interfaces/DataItemJsonConverter.cs:line 63
at Cosmos.DataTransfer.Interfaces.DataItemJsonConverter.WriteDataItem(Utf8JsonWriter writer, IDataItem item, Boolean includeNullFields, String objectName) in /data-migration-desktop-tool/Interfaces/Cosmos.DataTransfer.Interfaces/DataItemJsonConverter.cs:line 35
at Cosmos.DataTransfer.JsonExtension.JsonDataSinkExtension.SaveFile(IAsyncEnumerable1 dataItems, JsonSinkSettings settings, CancellationToken cancellationToken) in /data-migration-desktop-tool/Extensions/Json/Cosmos.DataTransfer.JsonExtension/JsonDataSinkExtension.cs:line 40 at Cosmos.DataTransfer.JsonExtension.JsonDataSinkExtension.SaveFile(IAsyncEnumerable1 dataItems, JsonSinkSettings settings, CancellationToken cancellationToken) in /data-migration-desktop-tool/Extensions/Json/Cosmos.DataTransfer.JsonExtension/JsonDataSinkExtension.cs:line 38
at Cosmos.DataTransfer.JsonExtension.JsonDataSinkExtension.SaveFile(IAsyncEnumerable1 dataItems, JsonSinkSettings settings, CancellationToken cancellationToken) in /data-migration-desktop-tool/Extensions/Json/Cosmos.DataTransfer.JsonExtension/JsonDataSinkExtension.cs:line 43 at Cosmos.DataTransfer.JsonExtension.JsonDataSinkExtension.SaveFile(IAsyncEnumerable1 dataItems, JsonSinkSettings settings, CancellationToken cancellationToken) in /data-migration-desktop-tool/Extensions/Json/Cosmos.DataTransfer.JsonExtension/JsonDataSinkExtension.cs:line 43
at Cosmos.DataTransfer.JsonExtension.JsonDataSinkExtension.WriteAsync(IAsyncEnumerable`1 dataItems, IConfiguration config, IDataSourceExtension dataSource, ILogger logger, CancellationToken cancellationToken) in /data-migration-desktop-tool/Extensions/Json/Cosmos.DataTransfer.JsonExtension/JsonDataSinkExtension.cs:line 23
at Cosmos.DataTransfer.Core.RunCommand.CommandHandler.ExecuteDataTransferOperation(IDataSourceExtension source, IConfiguration sourceConfig, IDataSinkExtension sink, IConfiguration sinkConfig, CancellationToken cancellationToken) in /data-migration-desktop-tool/Core/Cosmos.DataTransfer.Core/RunCommand.cs:line 168