yukon component framework

NOTE: This is still very much an active repo. It powers the directv.com entertainment site. We just haven't needed to change anything in a while. Any feature requests, issues or inquiries will be answered promptly.

yukon is a component-based, datasource-agnostic framework for serving web content. It extends the nodulejs component framework - to include back-end data gathering, standardized slots for app-defined middleware and template management.

A really simple yukon component (using the parallel-api plugin) looks like this:

module.exports = function(app) {
  return {
  
    route: '/home', 
    
    templateName: 'homePage.jade',

    apiCalls: [
      {path: '/api/cms/home'},
      {path: '/api/data/homedata'}
    ],
    
    preProcessor: function(req, res) {
      // pre-API(s) business logic goes here
    },
    
    postProcessor: function(req, res) {
      // post-API(s) business logic goes here
    }
  };
};

Just save this as a .js file in the default nodules directory, or more likely - in a directory you specify. The framework will do the rest when node boots.

Back-end data-gathering is achieved through plugins. Currently the only fully-fleshed out plugin makes 0-n REST API calls in parallel - as that was our need. I have made some starts on mysql and solr plugins, but would like a real - world implementation to battle-test them on. So by all means if you stumble across this repository, shoot me an email and I will work with you to get it up and running for your needs.

FYI - we're considering an alternate approach to non-REST data sources. See To Do section below.

Installation

$ npm install yukon

Usage

require('yukon')(app, config);

app = express instance.
config = any custom properties you want to add or defaults you want to override. See the demoApp for an example of a working yukon app. See the Config section below for more details.

If so then some of the terms that follow may be unfamilar. The good news is that the yukon framework is designed to handle a lot of the low level node "plumbing" that a node expert would typically be needed for on a large project. We've found this framework to be incredibly intuitive for front-end devs, often with zero node experience, to pick up and start cranking out web components. And again, we're still looking for more real world implementations to solidify the framework.

What is a yukon nodule?

A nodule is a self-discovering, self-registering web component tied to one or more express routes. With each incoming request, a nodule instance propagates throughout the express middleware chain as req.nodule.

A yukon nodule extends the base nodule behavior to include data gathering, stub-handling and template-rendering. It also allows custom app-defined middleware to be declared between each step of the request/response chain. Yukon attaches data returned as the res.yukon object, and sends res.yukon.renderData to the template or straight back to the client as JSON.

Nodulejs was split off from yukon to separate out the core self-discovery and initialization features, which can potentially be a building block for a wide variety of node applications or frameworks.

A nodule is analogous to a JSP or PHP page in those worlds. Unlike PHP/JSP behavior however, a nodule's route is declared and not tied by default to the filename or folder structure. So you are free to re-organize nodules without upsetting urls. More importantly, because nodules are self-discovering, there are no onerous config files to maintain (IE - Spring). This system allows a much more scalable architecture on large sites--as there are no config or other shared files which grow to enormous sizes as the site grows, and nodules can be re-organized with zero impact.

Motivation

From a feature-development point of view, we wanted to give developers the flexibility of component-based architecture as much as possible, but still keep system-wide control over the middleware chain. On a small site with a small development team the latter might not be an issue. But on a large site with devs scattered all over the globe, some kind of middleware sandbox was a necessity.

Our feature devs spend 80-90% of their effort in jade templates or on the client side. For them, node components are often mostly a pass-through to our back-end API(s)--with some business logic applied to the request on the way in, and API data on the way out. Ideally they should have to learn as little as possible of the vagaries/plumbing/whatever-your-favorite-metaphor-for-framework-stuff of node. Creating a new node component should be as easy for them as creating a new JSP - but again, without the framework losing control of the middleware chain.

From a framework-development point of view, we knew that as requirements evolved, we would constantly need to add default properties to each component, while hopefully causing as little disruption as possible to existing components. This is easily accomplished by adding a default property to the base config, then specifying the property only in the nodules that need the new property.

We also knew we'd need to add slices of business logic globally or semi-globally at any point in the request chain. By keeping control of the middleware chain we are able to do this with ease.

This diagram might make the concept a little more clear:

Config

Yukon config is broken into 3 sections:

Nodule-specific properties
Data plugin-specific properties
App-defined middleware functions and global settings

Note: You may occasionally see "MAGIC ALERT" below. This is for the handful of times where the framework does some convenience method that isn't immediately obvious, but comes up so much we felt the code saving was worth the loss in conceptual clarity.

Nodule-specific properties (config.noduleDefaults)

Yukon inherits the 4 core nodulejs defaults:

route: (REQUIRED) one or more express routes - can be a string, RegExp, or array of either
routeVerb: (OPTIONAL, default=get) get, post, put, del
routeIndex: (OPTIONAL, default=0) use to match express routes before or after others, can be negative, like z-index
middlewares: (OPTIONAL) define this in your nodule to override the entire yukon request chain. See 404.js from the demoApp for example.

Yukon also adds the following optional nodule properties:

templateName: MAGIC ALERT: if null the framework looks for a template in the same folder and of the same name as the nodule filename + templateExt. So if you have myPage.jade in the same folder as myPage.js, there is no need to specify template name.
templateExt: default template extension
contentType: 'html' and 'json' are the only current values
preProcessor: use this function to manipulate query params or other business logic before back-end data-gathering
postProcessor: use this function to process data returned from back-end data-gathering, before calling the template or sending the renderData back to the client as JSON
error: set to a string or an Error() instance to get the framework to call next(error)
apiCalls: array of API calls to made in parallel for this nodule, see the section below for details what constitutes an API call. this property is added to nodule defaults by the parallel-api plugin

NOTE: global or semi-global calls like getProfile, getGlobalNav, etc. can be added to this array in the preData middleware.

API-specific properties added by the parallel-api plugin (plugins/parallel-api/index.js - config.apiDefaults)

The yukon parallel-api plugin defines the following properties for each API call. It is important to understand that these exist in a one-to-many relationship with nodules.

path: path to API (not including server).
MAGIC ALERT: if the API path ends with a slash(/), the framework automatically tries to append req.params.id from the express :id wildcard. For us at least this is a very common REST paradigm.
params: params to send to API server. If the api verb is 'post', this can be a deep json object (bodyType=json) or a shallow object of name value pairs (bodyType=form).
verb: get, post, put, del
bodyType: valid values: json, form
host: path to the API server, can be set in app-defined middleware or overridden at the nodule level.
customHeaders: custom headers to sent with API call
timeout: (numeric) - max API return time in ms
useStub: set true to force framework to use stub instead of API
stubPath: can contain path or just name if in same folder
MAGIC ALERT: if not specified, app looks for [nodule name].stub.json in nodule folder

The parallel-api plugin also allows 2 optional app-defined functions, which are executed before and after every API call. It's important to understand that there can be several API calls per express request. So these functions are not in the standard middleware chain, although the api callback does make use of the middleware paradigm.

apiCallBefore: a synchronous function executed before of every api call. Do any common API pre-processing here.
apiCallback: an asynchronous function executed after every api call, must execute next() if defined. Do error handling and other common post-API processing here. To do: consider moving error handling to framework and making this call synchronous.

App-defined middlware

An app can create and use 4 optional express middleware functions, which splice in between the built-in yukon middleware (see yukon.js for more detail):

start: called at start of middleware, before nodule.preProcessor
preData: called after nodule.preProcessor, before data-gathering
getData: middleware which gets all data (Note: if specified in the app config, this function will bypass all plugin behavior)
postData: called after data gathering, before nodule.postProcessor
finish: called after nodule.postProcessor, before res.send() or res.render()

Global properties

There are also 3 global config properties inherited from nodulejs:

dirs: (OPTIONAL, default='/nodules') path(s) to look for your nodules, exclude property can be full or partal match
example: [{ path: '/app', exclude: ['demoApp.js', '.test.js', '/shared/'] }, { path: '/lib/nodules', exclude: ['.test.js'] }]
debugToConsole: (OPTIONAL, default=false) set to true to see nodulejs debug output in the console
customDebug: (OPTIONAL) custom debug function
example: function(identifier) { return function(msg){... your debug function here ...} }

To Run Node Tests

Download yukon - https://github.com/jackspaniel/yukon/archive/master.zip
$ npm install
$ make test

To Run Demo App as Standalone

$ node demoServer

To Do

Consider using apiSim approach for any non REST data gathering. IE - node wraps any request to say Mongo, in just another nodule whose route is API call. So node is using itself as the API server and the nodule is going out to mongo or whatever async data source desired. This would be huge for code clarity, as the yukon app would never have to worry about connecting to anything other than a REST API. Also this would make it trivially simple to split the data gathtering client and web client onto different servers - as the api server url would just be a config property. Big question - is there a lot of performance overheard to node making an REST http call to itself? Is the perf hit worth it for code clarity and flexiblity?
Reconsider stub behavior for parallel-api. Should all stubs move to apiSim behavior? What about brand new nodules where nothing is known about the API yet?

Features for future consideration

Flesh out more plugins. Currently only the paralle-api plugin is fully operational. I need real-world sites to test this out on. (Free consulting!)
API error handling for parallel-api plugin. It seems that there can be a huge variation in error behavior, and even in what constitutes an API error (status code-based?), from web-app to web-app. So for now I've punted on advanced API error handling, and let the app deal with it in the API callback. But if something like a standard is more or less agreed-upon, I will be happy to add flexible error handling.

Examples:

HTML response

(homePage.js from the demoApp)

module.exports = function(app) {
  return {
  
    route: ['/', '/home', '/special'], // multiple routes

    apiCalls: [
      {path: '/api/cms/home'},
      {path: '/api/data/homeslices'}
    ],
    
    // pre-API(s) business logic
    preProcessor: function(req, res) {
      this.debug('preProcessor called');

      if (req.path.indexOf('special') > -1) {
        this.apiCalls[1].params = {isSpecial: true}; // setting api params at request-time
        this.templateName = 'altHomePage.jade'; // using alternate template
      }
    },
    
    // post-API(s) business logic
    postProcessor: function(req, res) {
      this.debug('postProcessor called');

      var clientMsg = res.yukon.data2.specialMsg || res.yukon.data2.msg;

      res.yukon.renderData = {
        globalNav: res.yukon.globalNav,
        cmsData: res.yukon.data1,
        myData: res.yukon.data2,
        clientMsg: clientMsg
      };
    }
  };
};

JSON response

(getData.js from the demoApp)

module.exports = function(app) {
  return {
    
    route : '/json/getData/:id',       

    apiCalls: [
      {path: '/api/getdata/'}, // :id is tacked on by the framework automatically
    ],

    preProcessor: function(req, res) {
      this.debug('preProcessor called');

      this.apiCalls[0].params = {myParam: req.query.myParam};
   },
    
    postProcessor: function(req, res) {
      this.debug('postProcessor called');

       res.yukon.renderData = {
         systemMsg: res.yukon.data1.systemMsg,
         data: res.yukon.data1
      };
    }
  };
};

Form submit

(submitForm.js from the demoApp)

var _ = require('lodash');

module.exports = function(app) {
  return {
 
    route : '/json/submitForm',  

    routeVerb: 'post', // default = get       
    
    apiCalls: [{
      path: '/api/submitform',
      verb: 'post', // post to API
      bodyType: 'form', // default = 'json'
    }],

    preProcessor: function(req, res) {
      this.debug('preProcessor called');
      
      if (!_.isEmpty(req.body)) {
        this.apiCalls[0].bodyType = 'json';
        this.apiCalls[0].params = req.body; // JSON body
      }
      else {
        this.apiCalls[0].params = req.query; // url-encoded
      }
    },

    postProcessor: function(req, res) {
      this.debug('postProcessor called');

      res.yukon.renderData = {
        response: res.yukon.data1
      };
    }
  };
};

App-defined middleware

(from demoApp.js)

function demoStart(req, res, next) {
  debug("demoStart called");

  res.locals.pretty = true; // jade pretty setting - turn off at the component level if necessary

  // example of setting nodule property globally
  if (req.nodule.contentType !== 'html' && req.path.indexOf('/json/') === 0)
    req.nodule.contentType = 'json'; 

  // example of app-level logic - simple device detection (used throughout demoApp)
  if (req.headers['user-agent'].match(/android/i))
    req.deviceType = 'Android';
  else if (req.headers['user-agent'].match(/iphone/i))
    req.deviceType = 'iPhone';
  else if (req.headers['user-agent'].match(/ipad/i))
    req.deviceType = 'iPad';
  else 
    req.deviceType = 'web';

  next();
}

Global API callback and error handling