Static web page generator created by the Czech Python community.
Freezeyt is a static webpage freezer. It takes a Python web application and turns it into a set of files that can be served by a simple server like GitHub Pages or Python's http.server.
Freezeyt is compatible with all Python web frameworks that use the common Web Server Gatweay Interface (WSGI)
Freezeyt requires Python 3.6 or above.
It is highly recommended to create and activate a separate virtual
environment for this project.
You can use venv
, virtualenv
, Conda, containers or any other kind
of virtual environment.
The tool can be installed using:
$ python -m pip install .
To use freezeyt, you need a Python web application. You can use the example Flask app.
Your WSGI application should be named app
.
Both the application and Freezeyt must be importable (installed)
in your environment.
Run freezeyt with the name of your application and the output directory. For example:
$ python -m freezeyt my_app _build
Freezeyt may overwrite the build directory (here, _build
),
removing all existing files from it.
For more options, see Configuration below.
Freezeyt also has a Python API, the freeze
function
that takes an application to freeze and a configuration dict:
from freezeyt import freeze
freeze(app, config)
The config
should be a dict as if read from a YAML configuration
file (see Configuration below).
While common options can be given on the command line,
you can have full control over the freezing process with a YAML
configuration file or a variable with the configuration.
You can specify a config file using the -c/--config
option,
for example:
$ python -m freezeyt my_app _build -c freezeyt.yaml
The configuration variable should be a dictionary.
To pass the config variable, use the -C/--import-config
option,
for example:
$ python -m freezeyt my_app _build -C my_app:freezeyt_config
The following options are configurable:
The module that contains the application must be given on the
command line.
In it, Freezyt looks for the variable app
.
A different variable can be specified using :
.
Examples:
application
or
folder1.folder2.application
or
my_app:wsgi_application
To outupt the frozen website to a directory, specify the directory name:
output: ./_build/
Or use the full form – using the dir
saver:
output:
type: dir
dir: ./_build/
If output is not specified in the configuration file, you must specify the oputput directory on the command line. Specifying it both on the command line and in the config file is an error.
If there is any existing content in the output directory, freezeyt will either remove it (if the content looks like a previously frozen website) or raise an error. Best practice is to remove the output directory before freezing.
For testing, Freezeyt can also output to a dictionary. This can be configured with:
output:
type: dict
This is not useful in the CLI, as the return value is lost.
The URL where the application will be deployed can be specified with:
prefix: http://localhost:8000/
or
prefix: https://mysite.example.com/subpage/
Freezeyt will freeze all pages starting with prefix
that
it finds.
The prefix can also be specified on thecommand line with e.g.:
--prefix=http://localhost:8000/
.
The CLI argument has priority over the config file.
A list of URLs to “extra” pages within the application can be given using:
extra_pages:
- /extra/
- /extra2.html
Freezeyt will freeze these pages in addition to those it finds by following links.
Extra pages may also be give on the command line,
e.g. --extra-page /extra/ --extra-page /extra2.html
.
The lists from CLI and the config file are merged together.
You can also specify extra pages using a Python generator, specified using a module name and function name as follows:
extra_pages:
- generator: my_app:generate_extra_pages
The generate_extra_pages
function should take the application
as argument and return an iterable of URLs.
When using the Python API, a generator for extra pages can be specified directly as a Python object, for example:
config = {
...
'extra_pages': [{'generator': my_generator_function}],
}
another_config = {
...
'extra_pages': [my_generator_function],
}
Extra files to be included in the output can be specified, along with their content.
This is useful for configuration of your static server. (For pages that are part of your website, we recommend adding them to your application rather than as extra files.)
For example, the following config will add 3 files to the output:
extra_files:
CNAME: mysite.example.com
".nojekyll": ''
config/xyz: abc
You can also specify extra files using Base64 encoding or a file path, like so:
extra_files:
config.dat:
base64: "YWJjZAASNA=="
config2.dat:
copy_from: included/config2.dat
Extra files cannot be specified on the CLI.
Freezeyt checks whether the file extensions in its output correspond to the MIME types served by the app. If there's a mismatch, freezeyt fails, because this means a server wouldn't be able to serve the page correctly.
It is possible to specify the MIME type used for files without an extension. For example, if your server of static pages defaults to plain text files, use:
default_mimetype=text/plain
It is possible to register hooks, functions that are called when specific events happen in the freezing process.
For example, if mymodule
defines functions start
and page_frozen
,
you can make freezeyt call them using this configuration:
hooks:
start:
mymodule:start
page_frozen:
mymodule:page_frozen
When using the Python API, a function can be used instead of a name
like mymodule:start
.
The function will be called when the freezing process starts, before any other hooks.
It is passed a FreezeInfo
object as argument.
The object has the following method:
add_url(url, reason=None)
: Add the URL to the set of pages to be frozen. If that URL was frozen already, or is outside theprefix
, does nothing. If you add areason
string, it will be used in error messages as the reason why the added URL is being handled.
The function will be called whenever a page is saved.
It is passed a TaskInfo
object as argument.
The object has the following attributes:
get_a_url()
: returns a URL of the page, includingprefix
. Note that a page may be reachable via several URLs; this function returns an arbitrary one.path
: the relative path the content is saved to.freeze_info
: aFreezeInfo
object. See thestart
hook for details.
The status_handlers
option defines the way to handle HTTP statuses.
For example, the default reaction to a 404 NOT FOUND
status is to quit with an error,
but you can customize freezeyt
to ignore the default error.
status_handlers:
'404': ignore
freezeyt
includes a few pre-defined handlers:
'warn'
: will save the content and send warn message to stdout'save'
:freezeyt
will save the body of the redirect page, as if the response was200
.'follow'
:freezeyt
will save content from the redirected location (this requires aLocation
header, which is usually added for redirects –3xx
statuses).'ignore'
:freezeyt
will not save any content for the page'error'
: abort with an error
The user can also define a custom handler as:
- a string in the form
'my_module:custom_handler'
, which names a handler function to call, - a Python function (if configuring
freezeyt
from Python rather than from YAML).
The handler function takes one argument, task
(TaskInfo): information about the freezing task.
See the TaskInfo
hook for a description.
A custom handler should call one of the pre-defined handlers (e.g. freezeyt.status_handlers.follow
) and return the return value from it.
Freezeyt's default functions, like follow
, can be imported from freezeyt.status_handlers
and used
from the custom handlers.
For example, status_handlers
would be specified as:
status_handlers:
'202': warn
'301': follow
'404': ignore
'418': my_module:custom_handler
'429': ignore
'5xx': error
Note that the status code must be a string, so it needs to be quoted in the YAML file.
A range of statuses can be specified as a number (1-5
) followed by lowercase xx
.
freezeyt
discovers pages in the application it freezes by scanning
the application's pages for links.
By default, HTML and CSS pages are scanned this way.
It is possible to customize the scanning or turn it off by setting
url_finders
in the configuration.
For example, the default scanners would be specified as:
url_finders:
text/html: get_html_links
text/css: get_css_links
Keys in the url_finders
dict are MIME types; the values can be:
- strings in the form
"module:function"
, which name the scanning function to call, - strings like
get_html_links
, which name a function from thefreezeyt.url_finders
module, or - Python functions (if configuring
freezeyt
from Python rather than YAML).
A scanning function gets these arguments:
- an open file with the page's contents (call
read
to get the contents as bytes), - the URL of the page, as a string, and
- the HTTP headers, as a list of (name, value) tuples (as in WSGI).
The function should return an iterator of URLs (as strings) found in the page's contents. These URLs can be relative.
The freezeyt.url_finders
module includes the functions get_html_links
and get_css_links
, which you can call (for example, as fallbacks).
It is possible to customize the filenames that URLs are saved under
using the path_to_url
configuration key, for example:
path_to_url: my_module:path_to_url
The value can be:
- a strings in the form
"module:function"
, which names the function to call (the function can be omitted along with the colon, and defaults tourl_to_path
), or - a Python function (if configuring
freezeyt
from Python rather than YAML).
The function receives the path of the URL to save, relative to the prefix
,
and should return a path to the saved file, relative to the build directory.
The default function, available as freezeyt.url_to_path
, adds index.html
if the URL path ends with /
.
$ python -m freezeyt my_app _build/
$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/
$ python -m freezeyt my_app _build/ -c config.yaml
$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ --extra-page /extra1/ --extra-page /extra2/
$ python -m freezeyt my_app _build/ --prefix https://pyladies.cz/ --extra-page /extra1/ --extra-page /extra2/ --config path/to/config.yaml
Are you interested in this project? Awesome! Anyone who wants to be part of this project and who's willing to help us is very welcome. Just started with Python? Good news! We're trying to target mainly the beginner Pythonistas who are seeking opportunities to contribute to (ideally open source) projects and who would like to be part of an open source community which could give them a head start in their (hopefully open source :)) programming careers.
Soo, what if you already have some solid Python-fu? First, there's always something new to learn, and second, we'd appreciate if you could guide the “rookies” and pass on some of the knowledge onto them.
Contributions, issues and feature requests are welcome. Feel free to check out the issues page if you'd like to contribute.
- Clone this repository to your local computer:
$ git clone https://github.com/encukou/freezeyt
- Then fork this repo to your GitHub account
- Add your forked repo as a new remote to your local computer:
$ git remote add <name_your_remote> https://github.com/<your_username>/freezeyt
- Create new branch at your local computer
$ git branch <name_new_branch>
- Switch to your new branch
$ git switch <name_new_branch>
- Make some awesome changes in code
- Push changes to your forked repo on GitHub
$ git push <your_remote> <your_new_branch>
-
Finally make a pull request from your GitHub account to origin
-
Repeat this process until we will have done amazing freezer
-
Set
PYTHONPATH
to the directory withfreezeyt
, for example:- Unix:
export PYTHONPATH="/home/name/freezeyt"
- Windows:
set PYTHONPATH=C:\Users\Name\freezeyt
- Unix:
-
Install the web application you want to freeze. Either:
- install the application using
pip
, if possible, or - install the application's dependencies and
cd
to the app's directory.
- install the application using
-
Run freezeyt, for example:
python -m freezeyt demo_app_url_for _build --prefix http://freezeyt.test/foo/
For testing the project it's necessary to install additional requirements:
$ python -m pip install .[dev]
To run tests in your current environment, use pytest:
$ python -m pytest
To run tests with multiple Python versions (if you have them installed),
install tox
using python -m pip install tox
and run it:
$ tox
Some test scenarios compare freezeyt's results with expected output.
When the files with expected output don't exist yet,
they can be created by setting the environment variable
TEST_CREATE_EXPECTED_OUTPUT
to 1
:
Unix
$ export TEST_CREATE_EXPECTED_OUTPUT=1
Windows
> set TEST_CREATE_EXPECTED_OUTPUT=1
If you set the variable to any different value or leave it unset then the files will not be recreated (tests will fail if the files are not up to date).
When output changes, you need to first delete the expected output,
regenerate it by running tests with TEST_CREATE_EXPECTED_OUTPUT=1
,
and check that the difference is correct.
Unfortunately our progress of development can be watched only in Czech language.
Watch the progress:
Other communication channels and info can be found here:
We keep a blog about the development of Freezeyt. It is available here.
Be warned: some of it is in the Czech language.
The blog was tested on Python version 3.8.
The blog is a Flask application.
To run it, install additional dependecies with
python -m pip install .[blog]
.
Then, set the environment variable FLASK_APP
to the path of the
blog app.
Also set FLASK_ENV
to "development" for easier debugging.
Then, run the Flask server.
- On Microsoft Windows:
> python -m pip install .[blog]
> set FLASK_APP=freezeyt_blog/app.py
> set FLASK_ENV=development
> flask run
- On UNIX:
$ python -m pip install .[blog]
$ export FLASK_APP=freezeyt_blog/app.py
$ export FLASK_ENV=development
$ flask run
The URL where your blog is running will be printed on the terminal.
Once you're satisfied with how the blog looks, you can freeze it with:
$ python -m freezeyt freezeyt_blog.app freezeyt_blog/build
Articles are writen in the Markdown
language.
Article - save to directory ../freezeyt/freezeyt_blog/articles
Images to articles - save to directory ../freezeyt/freezeyt_blog/static/images
If te files are saved elsewhere, the blog will not work correctly.
The Czech Python community uses a lot of static web pages that are generated from a web application for community purposes. For example, organizing and announcing workshops, courses, or meetups.
The community has been so far relying on Frozen Flask and elsa in order to generate the static web content. The new freezer ought to be used with any arbitrary Python Web application framework (Flask, Django, Tornado, etc.). So the community won't be limited by one web app technology for generating static pages anymore.
See GitHub history for all contributors.
This project is licensed under the MIT License. May it serve you well.