Read dynamic Files (PoC)

This project has an objective to show how can we build a code to perform when we don't know the files behaviors, that we receive.

About the Files and Map

First, we have a model class that we use to map file content into this class:

public class UserModel
{
    public string Name { get; init; }

    public string Email { get; init; }

    public int ProductCode { get; init; }

    public DateTime PaymentDate { get; init; }

    public decimal PaymentValue { get; init; }
}

And we have these files:

File with Position

For this kind of file, we need to know what position starts the field and the length.

File with Separator

For this kind of file, we need just map the header to know the fields.

Stack

.NET 7.0
Message Broker: RabbitMQ with Masstransit v8.0.15
Database: Postgres

Benchmark for Read Performance

Besides showing how we can read a file dynamically, here, we try to show how we can use 2 kinds of read files:

Stream Reader: traditional way to read files
Pipeline Reader: the way designed to make it easier to do high-performance I/O

PS: For this example, we show the benchmark references about position strategy

About benchmark:

Steps: Read the file, parse the data, and send it to the database or message broker.

10 Lines - 1 KB

#	Strategy	Mean	Error	Op/s
Send to DB	Stream	275.1 ms	NA	3.635
Send to DB	Pipe	259.6 ms	NA	3.852
Send to Broker	Stream	436.7 ms	NA	2.290
Send to Broker	Pipe	419.8 ms	NA	2.382

100 Lines - 7 KB

#	Strategy	Mean	Error	Op/s
Send to DB	Stream	375.8 ms	NA	2.661
Send to DB	Pipe	362.5 ms	NA	2.759
Send to Broker	Stream	457.3 ms	NA	2.187
Send to Broker	Pipe	447.9 ms	NA	2.232

1.000 Lines - 66 KB

#	Strategy	Mean	Error	Op/s
Send to DB	Stream	1.575 s	NA	0.6350
Send to DB	Pipe	1.499 s	NA	0.6671
Send to Broker	Stream	960.3 ms	NA	1.041
Send to Broker	Pipe	860.9 ms	NA	1.162

10.000 Lines - 655 KB

#	Strategy	Mean	Error	Op/s
Send to DB	Stream	13.01 s	NA	0.0769
Send to DB	Pipe	12.62 s	NA	0.0793
Send to Broker	Stream	3.795 s	NA	0.2635
Send to Broker	Pipe	3.819 s	NA	0.2619

100.000 Lines - 6.543 KB

#	Strategy	Mean	Error	Op/s
Send to DB	Stream	125.2 s	NA	0.0080
Send to DB	Pipe	148.7 s	NA	0.0067
Send to Broker	Stream	35.59 s	NA	0.0281
Send to Broker	Pipe	40.20 s	NA	0.0249

1.000.000 Lines - 65.430 KB