JSON to parquet - System.MissingMethodException: Constructor on type 'System.String' not found
Opened this issue · 3 comments
jsn-m commented
Processing JSON and writing out parquet I get the following error:
System.MissingMethodException: Constructor on type 'System.String' not found.
[2023-07-28T15:21:06.332Z] at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
[2023-07-28T15:21:06.333Z] at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
[2023-07-28T15:21:06.335Z] at System.Activator.CreateInstance(Type type, Object[] args)
[2023-07-28T15:21:06.337Z] at ChoETL.ChoActivator.CreateInstanceAndInit(Type objType, Object[] args)
[2023-07-28T15:21:06.339Z] at ChoETL.ChoConvert.ChangeType(Object value, Type conversionType)
[2023-07-28T15:21:06.341Z] at ChoETL.ChoUtility.Cast(Array array, Type elementType)
[2023-07-28T15:21:06.345Z] at ChoETL.ChoParquetRecordWriter.WriteAllRecords(StreamWriter sw)
[2023-07-28T15:21:06.351Z] at ChoETL.ChoParquetRecordWriter.Dispose(StreamWriter sw)
[2023-07-28T15:21:06.353Z] at ChoETL.ChoParquetWriter`1.Dispose(Boolean finalize)
[2023-07-28T15:21:06.355Z] at ChoETL.ChoParquetWriter`1.Dispose()
The code that makes this error:
using (var jsonReader = new ChoJSONReader(new StringReader(restResponse.Content), new ChoJSONRecordConfiguration() { ThrowAndStopOnMissingField = false, FlattenNode = true, TypeConverterFormatSpec = new ChoTypeConverterFormatSpec() { DateTimeFormat = "yyyy-MM-ddTHH:mm:ss.fffffff" } }))
{
using (var parser = new ChoParquetWriter("C:\\TEMP\\data1.parquet", new ChoParquetRecordConfiguration() { TreatDateTimeAsDateTimeOffset = true, IgnoreDictionaryFieldPrefix = true }))
{
parser.Write(jsonReader);
}
}
Packages: ChoETL.Parquet and ChoETL.JSON.NETStandard
using System.Net;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using RestSharp;
using Azure.Storage.Files.DataLake;
using ChoETL;
Cinchoo commented
can u pls share the versions of the library? also would be nice to share sample. thx
jsn-m commented
Hi Cinchoo,
I was using ChoETL version 1.2.1.60 and ChoETL.Parquet 1.0.1.27.
jsn-m commented
I eventually got it to work by removing IgnoreDictionaryFieldPrefix = true
using (var parser = new ChoParquetWriter("C:\\TEMP\\data1.parquet", new ChoParquetRecordConfiguration() { TreatDateTimeAsDateTimeOffset = true }))
{
parser.Write(jsonReader);
}