et_upload-cookbook

A cookbook to provision an SFTP server which a collection of chroot jailed users. Primarily aimed at situations where end users need simple, but limited, SCP/SFTP access to provide data (e.g., automated importing).

Supported Platforms

Ubuntu 12.04

Attributes

Key	Type	Description	Default
`['openssh']['server']['port']`	Array	Ports OpenSSH listens on	`%w(22 43827)`
`['openssh']['server']['permit_root_login']`	String	Allow remote root logins	`'no'`
`['openssh']['server']['password_authentication']`	String	Allow password logins	`'yes'`
`['openssh']['server']['subsystem']`	String	Set a subsystem for OpenSSH	`'sftp /usr/lib/sftp-server'`
`['openssh']['server']['match']`	Hash	Provide a match config for OpenSSH	see below

set['openssh']['server']['match'] = {
  'Group uploadonly' => {
    'chroot_directory' => '%h',
    'force_command' => 'internal-sftp',
    'allow_tcp_forwarding' => 'no'
  }
}

Files

recipe/scripts

Performs several setup steps that are required for processing uploads and exports. This script does the following:

Setup the rotation schedule for /var/log/process_uploads.log and /var/log/process_scheduled_exports.log
Create /opt/evertrue/config.xml
Setup cron schedules for processing uploads and exports and removing old upload and export files

recipe/users

Manages folder structure of all upload users. This script does the following:

Load the upload.json content from the users databag
For each user do the following:
Determine the home folder of the user. For most users the folder will be /mnt/dev0/evertrue/users/{username}. However, if the parent_user and partner values have been set for the user then the home folder will be a subfolder under the parent_user named for the partner.
If the user's action is remove then delete the user's home directory and all files, otherwise do the following:
Create the OS user and set the home folder
Create the user's home folder
If the user is using SSH keys then store those in the .ssh folder
Create the uploads and exports folders under the home folder

An example user object from the upload.json file:

  "givingtreedemo": {
    "uid": 10150,
    "ssh_keys": [],
    "comment": "Chelsea Leavitt GivingTree Demo",
    "password": "abc"
  }

An example parter user object:

  "givingtreedemo-hustle": {
    "uid": 10663,
    "ssh_keys": [],
    "parent_user": "givingtreedemo",
    "partner": "hustle",
    "comment": "GivingTree Demo Hustle Access",
    "password": "abc"
  }

files/default/process_uploads

Sends files from the uploads directory for import processing. This script does the following:

Load /opt/evertrue/config.xml
For each value in unames, do the following:
Get all file names stored under any uploads folder under the uname home folder
Ignore any upload file that does not have a csv, gz or zip extension.
Determine the type of import from the name of the upload file
Copy the upload file to the org's S3 folder
Check that a mapping exists for the upload file's headers
Post the upload file to the importer
If a mapping exists then queue the import file for processing if ET.Importer.IngestionMode = AutoIngest
If a mapping does not exist then notify support and set ET.Importer.IngestionMode = NotifyOnly

Logs for this script are saved to /var/log/process_uploads.log

files/default/process_scheduled_exports

Copies scheduled export files to the org's SFTP folder for download. This script does the following:

Load /opt/evertrue/config.xml
For each value in unames, do the following:
Get the oid from the uname value. Note that not all unames map directly to an org slug
Get the latest scheduled exports for the oid via /contacts/v2/exports/latest-scheduled
Download each export file via /contacts/v2/exports
Save the export file to the uname home exports folder

Logs for this script are saved to /var/log/process_scheduled_exports.log

Usage

et_upload::default

Include et_upload in your node's run_list:

{
  "run_list": [
    "recipe[et_upload::default]"
  ]
}

For testing purposes, the users upload data bag item exists. The password for each user is password, salted & encrypted to best resemble a real password & allow for logging in via SFTP to do manual testing of SFTP functionality.

Contributing

Fork the repository on Github
Create a named feature branch (i.e. add-new-recipe)
Write you change
Write tests for your change (if applicable)
Run the tests, ensuring they all pass
Submit a Pull Request

License and Authors

Author:: EverTrue, Inc. (jeff@evertrue.com)

evertrue/et_upload-cookbook