/landscapeapp

🌄Upstream landscape generation application

Primary LanguageJavaScriptApache License 2.0Apache-2.0

Landscapeapp

CII Best Practices npm version Dependency Status Netlify Status

The landscapeapp is an upstream NPM module that supports building interactive landscape websites such as the CNCF Cloud Native Landscape (source) and the LF Artificial Intelligence Landscape (source). The application is managed by Dan Kohn of CNCF and is under active development by Andrey Kozlov (who did most of the development to date) and Jordi Noguera.

In addition to creating fully interactive sites, the landscapeapp builds static images on each update. See examples in ADOPTERS.md. All current Linux Foundation landscapes are listed in landscapes.yml.

Adding and managing landscape entries

When creating new entries, the only 4 required fields are name, homepage_url, logo, and crunchbase.

- item:
  name: <entry name>
  homepage_url: <website for entry>
  # filename in hosted_logos folder. Put the svg file into the hosted_logos
  folder and reference its name.
  logo: <logo for entry> 
  crunchbase: <twitter for entry>

Additional keys that can be set are defined below:

  # url for the Twitter account; Only add if the value in Crunchbase is incorrect
  twitter: 
  # url to the repo for the project; will fetch stats if it starts with https://github.com/. If you add a `repo_url` the card will be white instead of grey. 
  repo_url: 
  # url to the GitHub organization for the project; when using `repo_url`, `project_org` can be set pointing to an organization on GitHub, this will have the effect of pulling the information for all the repos belonging to that organization but using `repo_url` for information regarding license and best practices.
  project_org: 
  # additional repos for the project; will fetch stats if they start with https://github.com/
  additional_repos: 
  # Stock Ticker for the organization of the project/entry; normally pulls from Crunchbase but can be overriden here. For delisted and many foreign countries, you'll need to add `stock_ticker` with the value to look up on Yahoo Finance to find the market cap.
  stock_ticker: 
  # description of the entry; if not set pulls from the GitHub repo description
  description: 
  # default branch to reference if not the main one for the repo
  branch: 
  # if the entry is a project hosted by the project, let's you set the maturity level. Should be a value in relations.values.children.id in settings.yml
  project: 
  # url for the CII Best Practices entry if it's not directly mapped to the repo_url
  url_for_bestpractices: 
  # set to false if a repo_url is given but the entry is a project that isn't open source
  open_source: 
  # allows mulitple entries with the same repo_url; set for each instance
  allow_duplicate_repo: 
  # set to true if you are using an anonymous organization. You will also need anonymous_organization set in settings.yml
  unnamed_organization:

For some of the key, there is some guidance as listed below.

Logos

The most challenging part of creating a new landscape is finding SVG images for all projects and companies. These landscapes represent a valuable resource to a community in assembling all related projects, creating a taxonomy, and providing the up-to-date logos, and unfortunately, there are no shortcuts.

Do not try to convert PNGs to SVGs. You can't automatically go from a low-res to a high-res format, and you'll just waste time and come up with a substandard result. Instead, invest your time finding SVGs and then (when necessary) having a graphic designer recreate images when high res ones are not available.

Tips for finding high quality images:

  • Google images is often the best way to find a good version of the logo (but ensure it's the up-to-date version). Search for grpc logo filetype:svg but substitute your project or product name for grpc.
  • Wikipedia also is a good source for high quality logos ( search in either the main Wikipedia or Wikipedia Commons ).
  • VectorLogoZone ( https://www.vectorlogo.zone/ )
  • Also search for 'svg' in the GitHub for the project, as sometimes projects will embed them there.

For new landscapes of any size, you will probably need a graphic artist to rebuild some of the logos for you.

If the project is hosted/sponsored by an organization but doesn't have a logo, best practice is to use that organization's logo with the title of the project underneath ( example ). You can use a tool such as Inkscape to add the text.

If you get an error with the image that it has a PNG embeded, you will need to find a different SVG that doesn't include a PNG or work with a graphic artist to rebuild the logo.

SVGs Can't Include Text

SVGs need to not rely on external fonts so that they will render correctly in any web browser, whether or not the correct fonts are installed. That means that all embedded text and tspan elements need to be converted to objects. Use of SVGs with embedded text will fail with an error. You can convert the SVGs as using one of the tools below.

CloudConvert
  1. Go to https://cloudconvert.com/, and click 'Select File' and select the SVG file.
  2. Next to 'Convert to', click the dropdown and select 'SVG'
  3. There will be wrench icon that appears. Click that.
  4. For the option 'Text To Path', select 'Yes' and then click 'Okay'
  5. Click 'Convert' to do the conversion and the download the converted file.
Adobe Illustrator
  1. Select all text
  2. With the text selected, go to Object > Expand in the top menu
  3. Export file by going to File > Export > Export As in top menu
  4. Select SVG from the format drop down and make sure that "Use Artboards" is checked
  5. This will open a SVG options box, make sure to set Decimal to 5 (that is the highest possible, so to ensure that sufficient detail is preserved)
  6. Click Okay to export
Inkscape
  1. Select the text
  2. Ctrl+K (path combine)
  3. Ctrl+J (dynamic offset)
  4. Save

Crunchbase Requirement

We require all landscape entries to include a Crunchbase url. We use the Crunchbase API to fetch the backing organization and headquarters location and (if they exist), Twitter, LinkedIn, funding, parent organization, and stock ticker. For open source, non-affiliated projects, we will just create a nonprofit organization representing the project (if one doesn't already exist), and set the location to the lead developer.

Using an external source for this info saves effort in most cases, because most organizations are already listed. Going forward, the data is being independently maintained and updated over time.

Overriding industries from Crunchbase

To override industries returned from Crunchbase for a specific Crunchbase entry, add it to an crunchbase_overrides top-level entry on landscape.yml. For instance, the following will set industries for Linux Foundation to Linux and Cloud Computing:

crunchbase_overrides:
  https://www.crunchbase.com/organization/linux-foundation:
    industries:
      - Linux
      - Cloud Computing

crunchbase_overrides must be a top-level key on landscape.yml, so it should be a sibling of landscape. That's to prevent having to override multiple items that share the same Crunchbase URL.

External Data

The canonical source for all data is landscape.yml. Once a day, the landscapeapp update_server pulls data for projects and companies from the following sources:

  • Project info from GitHub
  • Funding info from Crunchbase
  • Market cap data from Yahoo Finance
  • CII Best Practices Badge data

The update server enhances the source data with the fetched data and saves the result in processed_landscape.yml and as data.json, the latter of which is what the app loads to display data.

Creating a New Landscape

If you want to create an interactive landscape for your project or organization:

  1. Note ahead of time that the hardest part of building a landscape is getting hi-res images for every project. You cannot convert from a PNG or JPEG into an SVG. You need to get an SVG, AI, or EPS file. When those aren't available, you will need a graphic designer to recreate several images. Don't just use an auto-tracer to try to convert PNG to SVG because there is some artistry involved in making it look good. Please review this primer on image formats.
  2. Create a repo youracronym-landscape so it's distinct from other landscapes stored in the same directory. From inside your new directory, copy over files from a simpler landscape like https://github.com/graphql/graphql-landscape with cp -r ../graphql-landscape/* ../graphql-landscape/.github ../graphql-landscape/.gitignore ../graphql-landscape/.npmrc ../graphql-landscape/.nvmrc ..
  3. If you're working with the LF, give admin privileges to the new repo to dankohn and write privleges to AndreyKozlov1984, jordinl83, and CNCF-Bot and ping Dan after creating an account at slack.cncf.io. Alex Contini and Dan are available there to help you recreate SVGs based on a PNG of the company's logo, if necessary, and to fix other problems.
  4. Set the repo to only support merge commits and turn off DCO support, since it doesn't work well with the GitHub web interface: image
  5. Edit settings.yml and landscape.yml for your topic.
  6. Generate a QR code, setting colors to black. Save as SVG and overwrite images/qr.svg.
  7. Run y reset-tweet-count to start the count of tweets mentioning your landscape at zero.
  8. Edit landscapes.yml to add your project.

API Keys

You want to add the following to your ~/.bash_profile. If you're with the LF, ask Dan Kohn on CNCF Slack for the Crunchbase and Twitter keys.

For the GitHub key, please go to https://github.com/settings/tokens and create a key (you can call it personal landscape) with no permissions. That is, don't click any checkboxes, because you only need to access public repos.

export CRUNCHBASE_KEY_4="key-here"
export TWITTER_KEYS=keys-here
export GITHUB_KEY=key-here

Installing Locally

You can administer a landscape without ever needing to install the software locally. However, a local install is helpful for rapid development, as it reduces the 5 minute build time on Netlify to 10 seconds or less locally. In particular, you want a local install when you're reconfiguring the layout. We recommend installing one or more landscapes as sibling directories to the landscapeapp. Then, you want to install the npm modules for landscapeapp but not for any of the landscapes. Here are the install directions.

So, if you're in a directory called dev, you would do:

dev$ git clone git@github.com:cncf/landscapeapp.git
dev$ git clone git@github.com:cdfoundation/cdf-landscape.git
dev$ cd landscapeapp
dev$ npm install -g yarn@latest
dev$ yarn

Now, to use the local landscapeapp you can add the following to your ~/.bash_profile or .zshrc:

function y { export PROJECT_PATH=`pwd` && (cd ../landscapeapp && yarn run "$@")}
export -f y
# yf does a normal build and full test run
alias yf='y fetch'
alias yl='y check-links'
alias yq='y remove-quotes'
# yp does a build and then opens up the landscape in your browser ( can view the PDF and PNG files )
alias yp='y build && y open:dist'
# yo does a quick build and opens up the landscape in your browser
alias yo='y open:src'
alias a='for lpath in /Users/your-username/dev/{landscapeapp,cdf-landscape,lfai-landscape}; do echo $lpath; git -C $lpath pull -p; done; (cd /Users/your-username/dev/landscapeapp && yarn);'

Reload with . ~/.bash_profile and then use yo, yf, etc. to run functions on the landscape in your landscape directory. a will do a git pull on each of the project directories you specify and install any necessary node modules for landscapeapp.

Adding to a google search console

Go to the google search console, add a new property, enter the url of the given project, for example, https://landscape.cncf.io

Next, google will want to verify that it is your site, thus you need to choose an html tag verification option and copy a secret code from it and put it to the settings.yml of a given landscape project. Then commit the change to the default branch and wait till Netlify deploys the default branch. The key is named google_site_veryfication and it is somewhere around line 14 in settings.yml. After netlify succesfully deploys that dashbaord, verify the html tag in a google console. Do not forget to add Dan@linuxfoundation.org as someone who has a full access from a Settings menu for a given search console.

Vulnerability reporting

Please open an issue or, for sensitive information, email info@cncf.io.

Continuous Integration and NPM Publishing

We have a sophisticated build system. We build this landscapeapp repo together with every landscape after each commit to the landscapeapp. A list of landscapes is stored in the landscapes.yml An individual landscape is built on a PR to that landscape.

Details about building a repo on netlify:

Building an individual landscape

To build an individual landscape, we use Netlify. Netlify has certain issues with the performance and their caching algorithm is ineffective, thus in order to produce the fastest build, these steps are done

Note, that script netlify/landscape.js from THIS repo is used to run an individual build on every landscape.

A file netlify.toml specifies which commands are used and how to make a build. We start from the netlify folder and then download the landscape.js script from the default branch of a landscapeapp repo and then run a node netlify/landscape.js script because otherwise, Netlify will run an unnecessary npm install In order to make a build as fast as possible, we designed a way to run it on our own build server. The problem is that Netlify uses very slow and cheap amazon virtual machines, while our build server has a lot of CPUs and enough of RAM, that allows further parallelization during build steps.

Running "remotely" on our build server (fast and by default)

When an environment variable BUILD_SERVER is set, the following steps will occur:

  • the interactive-landscape package of the latest version is downloaded from npm
  • a current checkout of an individual landscape with a landscapeapp in a package folder is rsynced and sent to our build server
  • we use a hash of .nvmrc + package.json + npm-shrinkwrap.json from the landscapeapp repo as a key to cache node_modules, ~/.nvm and ~/.npm folders, this way if the hash has not changed - we reuse existing node_modules without any setup
  • if a hash is different, we install node_modules and cache ~/.nvm, ~/.npm and node_modules for further usage
  • finally, we run a build on our remote server via ssh, and when the build is done, the output is returned via rsync

Those extra steps allow us to run a build faster because we avoid an npm install step almost every time and extra RAM and CPU allow running npm tasks renderLandscape, checkLandscape and jest in parallel.

Still, if for certain reasons, remote solution stopped to work and we need to restore the Netlify build process as soon as possible, BUILD_SERVER variable should be set to empty in either a given landscape or in a shared variables section. Usually, the build will fail for all the landscapes, thus renaming the variable to BUILD_SERVER_1 in shared variables is the most efficient way.

One of the possible issues why remote builds would stop to work, although let's hope that will never change, would be that a cache folder is broken, therefore ssh root@${BUILD_SERVER} and then calling rm -rf /root/build on our build server will clear all the caches used for node_modules. Then you need to trigger a Netlify build again.

Running "locally" on Netlify instances (if the remote server is broken)

Without BUILD_SERVER variable, the following steps are done, from a file netlify/netlify.sh

  • the interactive-landscape package of the latest version is downloaded from npm
  • we go to that folder
  • we install node_modules via npm install
  • we run PROJECT_PATH=.. npm run build from the interactive-landscape package

Building this repo, landscapeapp on a Netlify

We want to ensure that we are making builds of all the landscapes, defined in landscapes.yml Netlify parameters are stored in the notilfy.toml file, and it runs the node netlify/landscapeapp.js from the netlify folder.

First, we check if the hash of .nvmrc, package.json and npm-shrinkwrap file already exists as a key of our cache on our remote server. If it does exist, it means we can use this folder for node_modules, .npm and .nvm folders for every individual landscape. Then we use rsync to send the current checkout of a repo to our remote server Then for every individual landscape, we run a build.sh file on a remote server, in each own docker container for every landscape. That is done in parallel. The file build.sh checks out the default branch of a given landscape and then runs npm run build with a PROJECT_PATH pointed to the given landscape

When all builds had been finished, the output is returned to the dist/${landscape.name} subfolder and logs are shown. Then _redirects and _headers files are generated to allow us to view individual landscapes from a Netlify build.

This repo is built only on our build server because Netlify has a 30 minutes timeout and we can not build individual landscapes there in parallel. Still, if every build fails and there are no obvious reasons, it may help to clear a node_modules cache: ssh root@${BUILD_SERVER} and then calling rm -rf /root/build and then running a new build on Netlify again

Setting up our build server to speed up Netlify builds

If for some reasons our current server is lost or wiped, or we have to rent a different build server, these are required steps

  1. Install docker on a new server. Just the latest docker, nothing else is required
  2. Generate a new pair of ssh keys, and add a public key to the /root/.ssh/authorized_keys file
  3. Take a private key without first and last lines, replace \n with space, and add as a BUILDBOT_KEY variable to the shared variable on a Netlify website
  4. Update the BUILD_SERVER shared variable on a Netlify website and provide the IP address of the new build server

To just check that all is fine, go to the netlify folder on your computer, checkout any branch you want or even make local changes, and run node landscapeapp.js, do not forget to set all required variables, including the BUILDBOT_KEY and BUILD_SERVER. The build should finish with the success and copy generated files and folders to the dist folder in the root of the repo checkout

Keeping Project Up to Date

We have an issue #75, where we update all out packages. This is how an update is usually done:

  1. Create a new folder like 75-update-2019-10-16
  2. Run ncu -u which is same as npm-check-updates -u, do not forget to install npm install -g npm-check-updates
  3. Run npm install , commit and push and make a PR
  4. Check that everything runs locally, i.e. npm run open:src should still work well
  5. Check that there are no layout issues on generated landscapes
  6. Do not forget to read README about those npm packages, which are mentioned in a red color, i.e. have a major update. They may require to implement certain changes in our code.

Embed landscape in a web site

You can embed the landscape in a website in a few different ways...

  • If you want just a full static image of the landscape in landscape mode, you can do:
<!-- Embed ASWF landscape as a PNG -->
<img src="https://landscape.aswf.io/images/landscape.png" alt="Academy Software Foundation Landscape Image">
  • If you want to embed the card mode for listing a category of entries ( for example members in a foundation or entries in a certain program ), you can do:
<!-- Embed list of all Open Mainframe Project members -->  
<iframe src="https://landscape.openmainframeproject.org/category=open-mainframe-project-member-company&amp;format=logo-mode&amp;grouping=category&amp;embed=yes" frameborder="0" id="landscape" scrolling="no" style="width: 1px; min-width: 100%; opacity: 1; visibility: visible; overflow: hidden; height: 1717px;"></iframe>
<script src="https://landscape.openmainframeproject.org/iframeResizer.js"></script>