Dasync/AsyncEnumerable

Memory leak

Ttxman opened this issue · 2 comments

Some "OnEnumerationComplete" tasks are not cleaned up when AsyncEnumerable is not Enumerated to the end.

I have run 4 iterations of code below and let it run for another 100 seconds:
aeleak

Each iteration will add one "Sheduled" task + 20 objects that are stuck on heap.
aemem

I have found that the AsyncEnumeratorWithState.OnEnumerationComplete(...) throws silent NullReferenceException on "_yield", but where the problem starts is beyond me.

else if (task.IsCanceled)
{
      enumerator._yield.SetCanceled();
}
Example code (.NET 4.7 console application with AsyncEnumerable 2.1.0.0 from nuget):

static void Main(string[] args)
    {
        //initialize heap ... 
        Enumerate().Wait();//this will not create "Scheduled" Task
        while (true)
        {
            GC.Collect();
            Console.ReadLine();
            Evilize().Wait();//this will add one "Scheduled" task that does not run
            GC.Collect();
            //Take memory snapshot
        }
    }

    public static async Task Enumerate()
    {
        var allEvil = await Evil().GetAsyncEnumeratorAsync();
        while (await allEvil.MoveNextAsync()) { } //enumerating to end does not leak tasks
    }

    public static async Task Evilize()
    {
        var allEvil = await Evil().GetAsyncEnumeratorAsync();
        var oneEvil = allEvil.FirstAsync();
    }

    public static IAsyncEnumerable<string> Evil()
    {
        var input = new AsyncEnumerable<string>(async yield =>
        {
            await Task.Run(async () => await Task.Delay(1));
            await yield.ReturnAsync("Evil");
        });
        return new AsyncEnumerable<string>(async yield =>
        {
            var source = await input.GetAsyncEnumeratorAsync(yield.CancellationToken);
            while (await source.MoveNextAsync(yield.CancellationToken))
            {
                await yield.ReturnAsync("more" + source.Current);
            }
        });
    }

@Ttxman, the first problem I see is that the async enumerator never gets disposed after calling GetAsyncEnumeratorAsync. Then, while debugging, the framework has internal static collection of all incomplete tasks in the process. Two things combined most likely result in the memory leak and won't reproduce when you run the code without debugger attached to the process.
Thanks, I'm aware of the NullReferenceException - that's not a big deal at the moment (mostly annoying), but will be fixed soon.

I'm closing this issue as per comment above. We've been using this library in production for more over than a year in a cloud environment with very heavy loads that rely on AsyncEnumerator, and have never seen any memory leaks problems related to it.