CatalystCode/project-fortis-pipeline

Project Fortis is a data ingestion, analysis and visualization pipeline.

This repository is outdated and was migrated to project-fortis.

---

---

---

deploy fortis pipeline

Deploy your own Fortis pipeline to an azure subscription through a single click.

Pipeline Architecture

A fully containerized realtime spark pipeline powered off Kubernetes.

Background

Project Fortis is a data ingestion, analysis and visualization pipeline. The Fortis pipeline collects social media conversations and postings from the public web and darknet data sources.

Related Repositories

• Spark-based Pipeline: https://github.com/CatalystCode/project-fortis-spark
• Dashboard Interface: https://github.com/CatalystCode/project-fortis-interfaces
• Graphql Services: https://github.com/CatalystCode/project-fortis-services
• Kubernetes Helm Charts: https://github.com/CatalystCode/charts

Documentation

• Deploy to Azure Steps
• Deploy to Local Machine Steps
• Upgrading the Fortis Spark Pipeline
• Upgrading the Fortis Interface

Demo Videos

• Dashboard Walkthrough in Spanish

Deployment Prerequisites

• First and foremost, you'll need an existing azure subscription. You can create one for free here.
• Generate a Public / Private ssh key pair following these instructions. The contents from the generated MyKey.pub file will be used for the SSh Public Key field.
• You'll need an existing azure service principal. You can follow these instructions if you need to generate a new service principal. Your service principles Application ID will be used for the Service Principal App ID field, and the Authentication Key will be used for the Service Principal App Key.

Fortis Monitored Data Sources

• Public Web - Bing
• Reddit
• Twitter
• Facebook
• Instagram
• Radio Broadcasts
• ACLED

Site Types

• The site type selection drives which default topics, public sites and facebook pages are auto-generated for your site as part of the deployment process.
• Available site types
  • Humanitarian
  • Climate Change
  • Health

Post Deployment Instructions

• Grab a large cup of coffee as the deployment can take north of an hour to complete.

• Once the deployment has finished, click on the Manage your resources (highlighted below).

• Select the Tags tab in the Azure Portal (highlighted below), point your browser to the site at the FORTIS_ADMIN_INTERFACE_URL (also highlighted below).

• In the Fortis admin portal, you can now finalize the setup of your Fortis deployment; once you've completed all the admin configuration, your deployment is ready to be used.