Delay in checking status file makes RunTwiceCompareStats unreliable

[andrew-stevenson-sociomantic]

RunTwiceCompareStats runs the test suite twice and verifies that the vsize of the tested application has not increased.

turtle/src/turtle/runner/actions/RunTwiceCompareStats.d

Lines 37 to 63 in fa079f3

	public bool runTwiceCompareStats ( ref RunnerConfig config,
	ref Context context, void delegate() reset, istring[] disabled )
	{
	// run twice and compare peak stats
	auto app = cast(TestedDaemonApplication) context.app;
	enforce(app !is null);
	bool result1 = runAll(config, context, reset, disabled);
	if (!result1)
	return result1;
	auto vsize1 = app.getPeakStats().vsize;
	log.info("Peak virtual memory after first run: {}", vsize1);
	log.trace("");
	log.trace("-----------------------------------------");
	log.trace("----------- END OR FIRST RUN ------------");
	log.trace("-----------------------------------------");
	log.trace("");
	bool result2 = runAll(config, context, reset, disabled);
	auto vsize2 = app.getPeakStats().vsize;
	log.info("Peak virtual memory after second run: {}", vsize2);
	enforce!(">=")(vsize1, vsize2);
	return result2;
	}

The problem is that vsize isn't checked when queried but returned from a cache.

turtle/src/turtle/application/TestedDaemonApplication.d

Lines 172 to 175 in fa079f3

	public PeakStats getPeakStats ( )
	{
	return this.stats_grabber.peak_stats;
	}

The cache is updated using a timer.

turtle/src/turtle/application/TestedDaemonApplication.d

Line 148 in fa079f3

this.stats_grabber.set(0, 1, 0, 10);

If memory is allocated in the last test of the first run then it is quite possible that the timer doesn't fire before (I am supposing) the epoll loop is exited. That means vsize1 is assigned an out of date value.

turtle/src/turtle/runner/actions/RunTwiceCompareStats.d

Line 48 in fa079f3

auto vsize1 = app.getPeakStats().vsize;

That in turn means that, the last allocation of the first run, is thought to occur during the second run and so the RunTwiceCompareStats test fails.

As we need "final" stats in the case of RunTwiceCompareStats then I don't think the cache should be used.

A fix that works for me is below. I'm not sure it is nice to call handle_ though so I hope someone else will chime in on a correct fix. I'm also not sure how useful the cache is - how often does it get queried?

--- a/src/turtle/application/TestedDaemonApplication.d
+++ b/src/turtle/application/TestedDaemonApplication.d
@@ -164,13 +164,20 @@ class TestedDaemonApplication : TestedApplicationBase
 
     /***************************************************************************
 
+        Params:
+            use_cache = if stats should be returned from the cache or should
+                be queried live
         Returns:
             stats for peak resource usage by the application
 
     ***************************************************************************/
 
-    public PeakStats getPeakStats ( )
+    public PeakStats getPeakStats (bool use_cache = true )
     {
+        if (! use_cache)
+        {
+            this.stats_grabber.handle_(0);
+        }
         return this.stats_grabber.peak_stats;
     }
 }

diff --git a/src/turtle/runner/actions/RunTwiceCompareStats.d b/src/turtle/runner/actions/RunTwiceCompareStats.d
index 5f74bf6..a46b06a 100644
--- a/src/turtle/runner/actions/RunTwiceCompareStats.d
+++ b/src/turtle/runner/actions/RunTwiceCompareStats.d
@@ -45,7 +45,7 @@ public bool runTwiceCompareStats ( ref RunnerConfig config,
     if (!result1)
         return result1;
 
-    auto vsize1 = app.getPeakStats().vsize;
+    auto vsize1 = app.getPeakStats(false).vsize;
     log.info("Peak virtual memory after first run: {}", vsize1);
     log.trace("");
     log.trace("-----------------------------------------");
@@ -54,7 +54,7 @@ public bool runTwiceCompareStats ( ref RunnerConfig config,
     log.trace("");
 
     bool result2 = runAll(config, context, reset, disabled);
-    auto vsize2 = app.getPeakStats().vsize;
+    auto vsize2 = app.getPeakStats(false).vsize;
     log.info("Peak virtual memory after second run: {}", vsize2);
 
     enforce!(">=")(vsize1, vsize2);

[mihails-strasuns]

#60