Source Footer Data From a Single Location

[scissorsneedfoodtoo]

Continuing the discussion from #218, we want to move to a model of having all the footer data for each locale in a single location (probably the CDN) as YAML, and use that to build the footers for News, Learn, the forum, and so on.

11ty can't source YAML by default, but we should be able to fetch the correct YAML file at the beginning of the build process, convert it into a JS object with the JS-YAML package, and then use that object to build the footer for each page.

This should also allow us to clean up the footer template here, as escaping the translated strings is a bit messy because of the way I implemented the translation function / shortcode.

We could do this as a collection or as global data, though I'm leaning toward global data as that's how we're sourcing data from Ghost. It'll just be a matter of adding a file to the _data directory and implementing the feature there.

[ojeytonwilliams]

@scissorsneedfoodtoo if we're going to download the footer data before starting the 11ty build, i18next-fs-backend should be able to handle yaml without any extra work.

[scissorsneedfoodtoo]

Oh wow, I didn't realize i18next-fs-backend could handle that. I just assumed that it only works with local files.

I'll give it a shot now. Loading the footer YAML with i18next-fs-backend will make things easier, and will mean that the footer code for News and Learn can remain pretty similar.

[ojeytonwilliams]

I just assumed that it only works with local files.

That's my understanding, too. I'm not sure it makes sense to use i18next-fs-backend when we're having to pull the data from the CDN. However, if we made it a two step process 1) download from CDN to files 2) source with i18next-fs-backend, then the second step is fine with yaml.

[scissorsneedfoodtoo]

That makes sense.

If we don't want to download the files for each build, and don't mind adding a couple more dev packages, it seems like we can use i18next-chained-backend and i18next-http-backend to do something like this:

...

i18next.use(chainedBackend).init({
  lng: currentLocale_i18nISOCode,
  fallbackLng: 'en',
  initImmediate: false,
  preload: readdirSync(join(__dirname, './locales')).filter(fileName => {
    const joinedPath = join(join(__dirname, './locales'), fileName);
    const isDirectory = lstatSync(joinedPath).isDirectory();
    return isDirectory;
  }),
  ns: ['translations', 'trending', 'meta-tags', 'links'],
  defaultNS: 'translations',
  backend: {
    backends: [
      fsBackend,
      HTTPBackend
    ],
    backendOptions: [
      {
        loadPath: join(__dirname, `./locales/${currentLocale_i18n}/{{ ns }}.json`) // main fs backend
      },
      {
        loadPath: `https://raw.githubusercontent.com/freeCodeCamp/news/main/i18n/locales/${currentLocale_i18n}/{{ ns }}.json` // falls back to HTTP backend
      }
    ]
  }
});

module.exports = i18next;

The second instance of loadPath will need to be replaced with URLs for the CDN served YAML files, but I think that would do it.

The only downside seems to be that we'll need to update the directory structure and filenames on the CDN to build/universal/lang/trending.yaml for this to work. It seems like the filename needs to match one of the i18next namespaces, and you can't declare a namespace for a particular plugin / loadPath.

Edit: Gah, i18next-http-backend doesn't seem to work with YAML out of the box. Will need to think about this a bit more.

[ojeytonwilliams]

I'm not 100% convinced we want a fallback. If the download fails for whatever reason, that'll just collapse the pipeline. That's annoying, but we can just run it again.

If we had a fallback, we'd not realise that the data was stale and I don't think that's what we want.

[scissorsneedfoodtoo]

That's true. I don't think it's a good idea to move all of the files to the CDN, just trending.

I guess fallback isn't the best way to describe it. It is a fallback of sorts, but I think it's really just for i18next, where if it doesn't find trending as a local file, it tries to use the other backend to fetch the file. If that fails as well, then it shows the keys instead.

Either way, the code above wouldn't work, as the HTTP backend only works with JSON.

It's a bit messy, but we could do this:

...
const fetch = require('node-fetch');
const yaml = require('js-yaml');
...

// existing i18next config

(async () => {
  const url = `https://cdn.freecodecamp.org/universal/trending/${currentLocale_i18n}.yaml`;
  const currentLocaleYAML = await fetch(url)
    .then(res => res.text())
    .then(res => res)
    .catch(err => console.log(err));

  i18next.addResources(currentLocale_i18nISOCode, 'trending', yaml.load(currentLocaleYAML));
})();

Then we wouldn't need to download and write the YAML file to disk before the build. Though it would still require the JS-YAML package.

But it's no problem to write the trending.yaml files to disk before the build starts -- currently we're doing some cleanup and other things before the build. 11ty makes it easy enough.

I'm not sure what would be best on the Learn side of things, though. News should be flexible enough for either approach.

[ojeytonwilliams]

I might be missing something, but I feel like either we want the cdn to be the single source of truth for the trending guides or we just want to make it easy to copy.

If we go the cdn as truth route, then the build pipelines should all read from that source and fail if they can't reach it. Ideally after a number of tries, to avoid flakiness.

Either way, it seems like fallbacks would be unhelpful, since they just mean the builds can fail silently.

I'm not sure what would be best on the Learn side of things, though.

The same as news, I think. Instead of using client/i18n/locales/english/trending.json it would have to pull from the cdn like the code you shared.

[scissorsneedfoodtoo]

That all makes sense. That's along the lines of what I'm thinking, too.

I played around with this a bit more today and think I've got a working solution that should work here and on Learn. I'll go ahead and open a PR now.

[scissorsneedfoodtoo]

Thanks again for all the great discussion here @ojeytonwilliams :) Hopefully this method of sourcing data for the footer will help with future updates.

If you think this looks good, I can work on porting this implementation to Learn.

Will close this issue for now (via #229).