site-update
site-update
is a script that you can configure to take apart a
zipfile with upstream sources for your site (html, css, js, etc), and
integrate it into your site. This is particularly useful if you
develop pages using design tools like Webflow but want to host
them yourself with modifications.
If you modify the upstream files (e.g., split pages into reusable
components), re-importing a newer version is a horrible error-prone
task. Rather than doing it by hand, site-update
allows you to
describe the changes, so that importing a new zipfile is as easy
as running the command again. In fact, I have my laptop set-up to
detect when I download a new zipfile, and it automatically kicks off
an update.
See the sample folder for a complete config file. Below is a description of the sections and how they work.
Installation
npm install site-update
It's best to install locally (without -g), because that allows the script to find your project's toplevel folder regardless of your current directory.
If you do install globally, cd into your project to run
site-update
.
Usage
After you've created a config file (see section below), you just need to run it on a zipfile containing the sources you want integrated:
path-to-project/node_modules/.bin/site-update ~/Downloads/mysite.zip
You can put it in your PATH so you don't need to specify the
folder. Or if you installed with -g
:
cd path-to-project
site-update ~/Downloads/mysite.zip
Config file
The config file must be at the top-level of your project folder, and
must be called site-update.json
.
The config file has two main sections, "type_defaults", which contains default actions by file type, and "sources", which lets you override those defaults based on file names/paths.
Type Defaults
Type defaults describe the defaults for a particular file type. Here's a complete example for HTML files:
"text/html": {
"kind": "view",
"destdir": "views",
"destname": "{basename}.hbs",
"action": "transform",
"transform": [],
"alt_kinds": {
"layout": {
"destdir": "views/layouts"
},
"partial": {
"destdir": "views/partials"
}
}
},
text/html
The property name is the mime type of the file these settings apply
to. It can be a glob, such as image/*
to set defaults for all
images.
Note that since the type defaults are not evaluated in any particular
order, if two globs that match the same type, there is no guarantee
which one will be used. However, an exact match will always be tried
first. e.g., image/png
will be attempted before image/*
is
matched.
kind & alt_kinds
'Kinds' allow you to specify multiple variants of a default. For HTML files, a good example is (as above) views, layouts, and partials. All are essentially the same, and are split from the same HTML files--but each gets placed into its own folder.
In this example, only destdir
is being overridden, but each
alt_kind
section may override any of the defaults.
destdir
Where you want files to end up, relative to the top of your project.
destname
What you want files to be named. The example above uses a variable that gets substituted for the file being processed. See the Substitution Variables section below for a complete list.
action
What you want site-update
to do. There are three built-in possible
values:
ignore
: does nothing.copy
: copies the file as-is.transform
: parses the file as HTML and runs transform rules before writing out the file.
However, it is possible to augment this list by supplying your own javascript action. See User-Defined Actions below.
transform
Array containing transform rules to be applied on HTML files before
writing them out. Only applies when action
is set to
transform
.
See Transform Actions below for more information on the format and what you can do.
Sources
This section is an array which allows you to match files based
on filename or path. Rules are matched in order and only the first
rule to match is used, so place more generic rules (e.g., catch-alls
like [**/*html]
or [**]
) below more specific rules. Each rule can
override any settings in the type defaults for that file. Here's an
example:
{
"match": ["**/index.html"],
"type": "text/html",
"action": "transform",
"outputs": [
{
"kind": "view",
"destdir": "views",
"destname": "webflow-{basename}.hbs",
"action": "transform",
"transform": [
["body > .navbar", "remove"],
["body > .footer", "remove"],
["body > div", "removeElse"]
]
},
{
"kind": "partial",
"destname": "navbar.hbs",
"transform": [
["body > .navbar", "removeElse"]
]
}
}
match
An anymatch expression to match the file(s) you wish to act on. In this case, it will match 'index.html'.
type
optional
Allows overriding the default mime type for a file, causing a different set of type defaults to be used.
outputs
An array of objects, each corresponding to a file being output. In
this case, index.html is being split into two files:
webflow-index.hbs
(a view), and navbar.hbs
(a
partial). You can split an input file into as many files as you wish.
Each object has the same options that type defaults have, see Type Defaults above for more information on each one.
You can declare an empty outputs section to ignore matched files, even
though the type defaults might specify to copy or transform
it. However, a better way to achieve this is to set action
to
ignore
, bypassing the transform
action altogether.
Substitution Variables
Config settings are allowed to contain a few variables which get substituted. This is useful for setting defaults that apply to several files. Current set of variables is:
fullname
The full filename (without the path). For example, given the file
/foo/bar/index.html
, the {fullname} variable will expand to
index.html
.
extname
The file extension, with leading dot. For example, given the file
/foo/bar/index.html
, the {extname} variable will expand to
.html
.
basename
The filename minus the extension. For example, given the file
/foo/bar/index.html
, the {basename} variable will expand to
index
.
Transform Actions
Transform rules are only valid for HTML files, and are directly based on Cheerio. Currently, only a small subset is implemented, however (only what I've needed so far). See below for a list.
Rules are arrays with two or three items:
- Transform action
- Selector to act on
- (Optionally) Additional data
Perhaps some examples will make it clear. From the index.html
example above:
"transform": [
["body > .navbar", "remove"],
["body > .footer", "remove"],
["body > div", "removeElse"]
]
These rules do the following:
- Remove element(s) with the
navbar
class that are insidebody
. - Remove element(s) with the
footer
class that are insidebody
. - Keep
div
elements currently insidebody
, but delete everything else.
So if you started with something like:
<html>
<head>
...
</head>
<body>
<div class="navbar"></div>
<div><h1>some content</h1></div>
</body>
</html>
You would end up with:
<div><h1>some content</h1></div>
Neat!
In addition to removal, you can also add and replace content. In that case, the transform array needs a third or fourth item, depending on the specific change. The additional item can be a string with the content to be added, or an array of strings, each to be written out as a separate line.
I find it particularly useful to replace elements with partial
includes. For example, split scripts into a separate file called
upstream-scripts.hbs
, then replace them with {{> scripts}}
. Then I can maintain scripts.hbs
myself, and
include upstream-scripts
or not as I wish.
Here's an example that uses the multiline syntax:
{
"kind": "layout",
"destname": "webflow-layout.hbs",
"transform": [
["body", "text", [
"",
"{{> navbar }}",
"{{{ body }}}",
"{{> footer }}",
"{{> endscripts }}",
""]]
]
},
This output rule keeps everything outside <body>
intact, but
replaces the contents of <body>
with
{{> navbar }}
{{{ body }}}
{{> footer }}
{{> endscripts }}
The additional empty strings at the beginning and end of the array are extra newlines just for stylistic purposes, to avoid
<body>{{> navbar }}
{{{ body }}}
{{> footer }}
{{> endscripts }}</body>
You may of course omit them if you wish.
Below are the Cheerio methods currently implemented. Check the Cheerio docs for details about each one (except for a few additional methods which aren't part of cheerio--those have a description here). Note that we only support methods that change the DOM, not the ones that extract data or test values (since that wouldn't make any sense in thei context).
Methods that take no arguments
["selector", "method"]
- remove
- empty
- val
Not part of Cheerio API:
- removeElse
This will remove everything other than the matched elements. You can see it in action in the example above. Great for pulling out a chunk of a document and saving it as a partial.
Methods that take one argument
["selector", "method", "arg1"]
- removeAttr
- removeClass
- append
- prepend
- after
- before
- replaceWith
- html
- text
Not part of Cheerio API:
- copyTo
Copies anything matched by selector into arg1, which must be a selector (string) reprensenting the destination. If arg1 matches multiple elements, all of them will have the matched content copied into them.
- copyTo
Convenience method. Exactly like copyTo, but also runs remove() on the source selector.
Methods that take two arguments
["selector", "method", "arg1", "arg2"]
- attr
- data
- toggleClass
- css
User-Defined Actions
It's possible (and easy!) to add an output action for your own
project. You can also replace the built-in copy
or transform
actions with this method.
- Set
plugindir
at the top of your config file, pointing to where you'll keep your action JS files. Path is relative to your project folder. - Write a plugin, here's an example:
var path = require('path'),
fs = require('q-io/fs'),
globals = {};
exports.name = "copy";
exports.init = function(globalvars) {
globals = globalvars;
}
exports.run = function(srcfile, config) {
var destname = config("destname"),
destdir = config("destdir"),
resolved = path.resolve(globals.sitedir, destdir),
destpath = path.join(resolved, destname);
return fs.makeTree(resolved)
.then(fs.copy.bind(fs, srcfile, destpath));
}
name
String
The action name you'd use in your config file. The above example is the "copy" action.
init
Function
Return value: ignored
Arguments: Object
with strings (see below)
Executed at plugin load time. Is passed an object with app
globals. Currently the only global is sitedir
, the project top-level
folder.
run
Function
Return value: promise
Arguments: String
, Function
(see below)
Executed for each output block that specifies your action name. Here's where you do whatever you need to do, it's expected that at the end you'll return a promise for the output block to be written out. You'll be supplied with two arguments:
srcfile
The full path to the source file. It'll be pointing to a temporary location where the unzipped sources are.
config
This is a closure that will supply you with config settings, whether they come from the output block, a kind default, or a type default. Any variables in the returned string (if it's a string) will be interpolated for you.
If you want to interpolate a string of your choosing, you can get access to the interpolator function:
var interpolated = config.interpolate(some_string);
(where config
is the second argument to the run
function).
As you can see, the above simple example just creates the destination
path, then copies the source file there. Check out the transform.js
action to see a more involved example.
If you write an action that you think would be widely useful to others, shoot me a message / pull request!
Copyright & License
This work is distributed under an MIT style license.
Copyright (c) 2014 56 degrees LLC
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.