omeka-s-modules/Scripto

Jobs may run out of memory for large instances

Closed this issue · 0 comments

The problem seems to come from ScriptoJob::getProjectItemIds() and ScriptoJob::getProjectMediaIds() when fetching the results using $query->getResult(). Since the queries are returning all rows, Doctrine's processing needs plus the resulting data structures can require enough memory to reach PHP's limit. We need to find a way to reduce memory usage. Things to try:

  • Use getScalarResult() instead of getResult() Note: While it does marginally reduce memory usage, getScalarResult() does not have a great enough effect to solve the problem;
  • Execute the queries in DBAL using $conn->prepare() (see example diff below).
diff --git a/src/Job/ScriptoJob.php b/src/Job/ScriptoJob.php
index 7b2093c..b2643df 100644
--- a/src/Job/ScriptoJob.php
+++ b/src/Job/ScriptoJob.php
@@ -22,8 +22,16 @@ abstract class ScriptoJob extends AbstractJob
             JOIN si.item i
             JOIN si.scriptoProject sp
             WHERE sp.id = :scripto_project_id'
-        )->setParameter('scripto_project_id', $project->getId());
-        return array_column($query->getResult(), 'item_id', 'scripto_item_id');
+        );
+        $conn = $em->getConnection();
+        $stmt = $conn->prepare($query->getSQL());
+        $stmt->bindValue(1, $project->getId());
+        $stmt->execute();
+        $results = [];
+        foreach ($stmt as $row) {
+            $results[$row['id_0']] = $row['id_1'];
+        }
+        return $results;
     }
 
     /**
@@ -42,8 +50,16 @@ abstract class ScriptoJob extends AbstractJob
             JOIN sm.scriptoItem si
             JOIN si.scriptoProject sp
             WHERE sp.id = :scripto_project_id'
-        )->setParameter('scripto_project_id', $project->getId());
-        return array_column($query->getResult(), 'media_id', 'scripto_media_id');
+        );
+        $conn = $em->getConnection();
+        $stmt = $conn->prepare($query->getSQL());
+        $stmt->bindValue(1, $project->getId());
+        $stmt->execute();
+        $results = [];
+        foreach ($stmt as $row) {
+            $results[$row['id_0']] = $row['id_1'];
+        }
+        return $results;
     }
 
     /**