Memory leak

[jlcd]

Hi, I'm creating a project that will run a couple hundred thousand operations, and started getting memory issues.

Drilling down I found the culprit (LazyQuery.php:47):

    private function getLoadedQuery(): QueryInterface
    {
        if (!isset($this->loadedQuery)) {
            $this->loadedQuery = $this->loadQuery();
        }

        return $this->loadedQuery;
    }

The $this->loadedQuery = $this->loadQuery(); portion is adding 2 MB to my memory footprint every now and then (didn't find the exact cause yet).

Did you experience anything similar? Any fixes you my think could work?

Edit:

I was about to sleep but kept going and found the issue is actually on Remorhaz\UniLex\Parser\LL1:(loadLookupTable):

    public function loadLookupTable(string $fileName): void
    {
        /** @noinspection PhpIncludeInspection */
        $data = @include $fileName;
        if (false === $data) {
            throw new Exception("Failed to load lookup table from file {$fileName}");
        }
        $table = new Table();
        $table->importMap($data);
        $this->lookupTable = $table;
    }

Specificaly here: $data = @include $fileName;.

$data is not being freed and for some reason the includes are staking. I'll send a pull request to fix this with a simple unset.

Best

[remorhaz]

Hi, thank you for the bug report and investigation! Looks like memory allocated with include is managed exotically. I will investigate that later more thoroughly, but for now I'll just accept your fix.

[remorhaz]

I've investigated remorhaz/php-unilex#1 and failed to find any leaks there. I've also wrote this test to check if query compilation leaks - but it also works, the memory gets cleaned after each query.

The $this->loadedQuery = $this->loadQuery(); portion is adding 2 MB to my memory footprint

The query in my test leaves footprint of ~80k. Are you using very complex queries? Can you give an example, please?

every now and then

Every query is compiled only once and then it can be used repeatedly with no compilation overhead; it's managed internally by LazyQuery. Are you using this feature in your code?

[jlcd]

Thanks for running the test.

I'm still investiganting whats going on, will update here once I find anything of relevance.

[jlcd]

Don't have a fix yet, but just to point out this eval here is eating 5MB+ memory on a 2000 selectOne run:

From Remorhaz\JSON\Path\Query:
$callback = eval($this->getCallbackCode());

On a 10000 selectOne run, this same portion eats 27MB+ memory, so it is stacking.

[jlcd]

Relevant: https://bugs.php.net/bug.php?id=69615

[remorhaz]

If you don't make all 10000 of queries exist simultaneously, the relevant effect should not affect you. Can you please provide a short, minimalistic script that reproduces the problem?

[jlcd]

Yes, it does affect me even in a synchronous process.

Even a simple php script like the one below gives us issues due to eval:

<?php

echo round((memory_get_peak_usage()/1024/1024), 2).PHP_EOL;
for ($i=0; $i < 100000;$i++) {
$a = eval('return function () { return 1; };');
}
echo round((memory_get_peak_usage()/1024/1024), 2).PHP_EOL;

output:

Interactive shell

php > echo round((memory_get_peak_usage()/1024/1024), 2).PHP_EOL;
0.41
php > for ($i=0; $i < 100000;$i++) {
php { $a = eval('return function () { return 1; };');
php { }
php > echo round((memory_get_peak_usage()/1024/1024), 2).PHP_EOL;
65.64

[remorhaz]

That's not eval that leaks, it's including the closure that leaks:
https://bugs.php.net/bug.php?id=76982

Let's discuss the issue here: #23

[remorhaz]

Seems to get fixed in 0.7.5.