Chalarangelo/30-seconds-web

Handle limit exceeding git queries

Chalarangelo opened this issue · 1 comments

As soon as we add enough content, we will hit a soft limit, where we'll start getting weird errors such as:

/bin/sh: fork: Resource temporarily unavailable

This stems from the following code (src/blocks/parsers/text/index.js:30):

if (withMetadata) {
  promises.push(
    new Promise(rsl =>
      exec(
        `cd ${dirPath}; git log --diff-filter=A --pretty=format:%at -- ${fileName} | head -1`,
        (error, stdout) => rsl(stdout.toString().replace('\n', ''))
      )
    )
  );
  promises.push(
    new Promise(rsl =>
      exec(
        `cd ${dirPath}; git log -n 1 --pretty=format:%at -- ${fileName} | head -1`,
        (error, stdout) => rsl(stdout.toString().replace('\n', ''))
      )
    )
  );
}

We can put a bandaid on it for the time being, as we need the data, but we'll have to tackle this at some point. The upcoming changes will introduce a far more efficient version of this system, using a single git command:

if (withMetadata) {
  promises.push(
    new Promise(rsl =>
      exec(
        `cd ${dirPath}; git log --pretty=format:%at -- ${fileName}`,
        (error, stdout) => {
          const dates = stdout.toString().split('\n');
          rsl([dates[0], dates.slice(-1)]);
        }
      )
    )
  );
}

The gain from this is that we'll have to more or less double our content volume to hit this limit (which we are fast approaching with the current version - hit it locally when testing the git repo addition), but we still might have to address this with a more resilient solution.


A proposal is to manually add both dates in the snippet metadata. We can create a quick mod to update all existing content and speed up parsing by quite a lot using that. The benefit of this is that scheduling content is going to be significantly easier (no git trickery involved) and we'll have more fine-tuned control of snippet update dates (typos should not count for example). The obvious issue is that we'll have to manually update anything we edit, which can get a bit tiresome over time, so we might have to invent a utility to update those dates for us if we need to do that every once in a while.

We're starting work on this issue as part of #345. We'll probably create and run a codemod for all repos, then update guidelines etc before testing out. Individual branches will be merged alongside this repo's PR.