tailwindui-crawler

This script will crawl the tailwindui.com website and download all the component HTML to the ./output folder.

🛠 How to use

To install, clone this repo and run yarn to pull down the dependencies.

Then create a .env file with your email, password, and optional output folder.

EMAIL=youremail
PASSWORD=yourpassword
OUTPUT=/path/to/output # optional, defaults to ./output
HTMLMODE=alpine|comments # save HTML with alpine (default) or comments
TRANSFORMERS=... # comma-delimited list of transformers (see below)
BUILDINDEX=(0 | 1)  # generate index file to view components offline

✨ New in v2.3

The Tailwind UI components have removed the Alpine.js code from the HTML. The crawler now includes a HTMLMODE setting to include either the Alpine.js code (alpine default) or download with HTML comments (comments).

There are also new transformers to change the color and logo of the HTML components.

NOTE: The tool uses dotenv-expand to support variable expansion like $HOME/path/to/output so if your password or any other value includes a $, make sure you add a \ (backslash) to escape the $. For example, PASSWORD=p@\$\$w0rd

Then finally, run yarn start

The script will login to tailwindui.com with your credentials, and download all the components as individual HTML files in the ./output folder.

🤔 What's it for?

The benefit of pulling down all the components is that you can commit them to a local or private repo, and by running this periodically, you can see exactly which files were added or changed. Hopefully, some time in the future, they will open up a private repo for those that have purchased the library.

🚀 New v2.0

The crawler has been re-written to make it easier to update the processing pipeline by simply adding a new transformer function.

It also adds the ability to generate an index page that emulates the tailwindui.com website so you can browse components offline.

🔄 Transformers

Each transformer is simply a JavaScript file (in ./transformers folder) that exports a function to be called from the processing pipeline. Each transformer will take a cheerio instance (basically a jQuery-like interface) which enables the transformer to update the generated HTML. The crawler will call each transformer in turn, then writes the final HTML file.

To add a new transformer, update the TRANSFORMERS key in the .env file. This is a comma-delimited list of transformers. The crawler will call each transformer in the specified order.

The following transformers are availble:

Transformer	Description
`addTailwindCss`	Adds link to tailwindui.css Use `ADDTAILWINDCSS_URL` to specify URL to css (defaults to tailwindui CDN)
`prefixSrc`	Adds `https://tailwindui.com` to any img src attribute that needs it
`useInter`	Adds link to Inter font css and styles
`convertVue`	Converts HTML component into a Vue component Use `.env` key `VUE_OUTPUT` to specify which folder to save the Vue files to (defaults to currently configured `OUTPUT`) This transformer will create a Vue component for each Tailwind UI component. This components may need some adjustements, but should in most cases be ready to go. An index script is not included, so the components can't be viewed in the browser yet.
`convertReact`	Converts HTML component into React/JSX-compatible syntax Use `.env` key `CONVERTREACT_OUTPUT` to specify which folder to save the React files to (defaults to currently configured `OUTPUT`). This transformer will create a folder for each component with an `index.js` file (this is the React component) and an `index.html` page which is a simple wrapper that will load the React component to verify it is working. NOTE: The transformer does not currently update the `alpine.js` code, so the component does not support state or interactivity. However, that is on the TODO list. You can use Parcel to test the React component. Based on gist by @RobinMalfait
`stripAlpine`	Removes all the Alpine.js attributes from the markup Use `.env` key `STRIPALPINE_OUTPUT` to specify which folder to save the files to. You must specify the output folder to ensure the raw HTML files are not overwritten
✨v2.3 `changeColor`	Changes the default color from `indigo` to value in `CHANGECOLOR_TO`
✨v2.3 `changeLogo`	Changes the logo image from the generic to URL in `CHANGELOGO_URL`
✨v2.4 `prefixClasses`	Adds prefix specified in `PREFIXCLASSES_PREFIX` to all Tailwind classes NOTE: You will want to use the `addTailwindCss` transformer and specify the URL to your custom css in `ADDTAILWINDCSS_URL`

🗂 Index page

You can set the .env key BUILDINDEX=1 to have the crawler generate an index file similar to the components page on tailwindui.com. Install and run the serve package to view the index.

yarn global add serve
cd $OUTPUT # change to your OUTPUT folder
serve

NOTE: In order to see the components styled, you probably will want to use TRANSFORMERS=addTailwindCss,prefixSrc,useInter.

Main index page

Component section page

Click button to view syntax highlighted code

You can view each component and the highlighted code. Currently resizing and copying code is not supported.

⚙️ Example `.env` file

EMAIL=******
PASSWORD=******
OUTPUT=$HOME/Projects/oss/tailwindui
HTMLMODE=alpine # apline | comments
BUILDINDEX=1    # 0 | 1

TRANSFORMERS=addTailwindCss,prefixSrc,useInter,changeColor,changeLogo,prefixClasses,convertReact,stripAlpine

# addTailwindCss
ADDTAILWINDCSS_URL=http://localhost/path/to/css # defaults to twui CDN

# convertVue
VUE_OUTPUT=$OUTPUT/vue  # path to save Vue files (defaults to $OUTPUT)

# convertReact
CONVERTREACT_OUTPUT=$OUTPUT/react # path to save React files (default to $OUTPUT)

# stripAlpine
STRIPALPINE_OUTPUT=$OUTPUT/no-alpine # path to save stripped HTML files (REQUIRED)

# changeColor
CHANGECOLOR_TO=red # name of color to change from indigo

# changeLogo
CHANGELOGO_URL=http://localhost/path/to/logo # URL of logo (defaults to generic tailwind logo)

# prefixClasses
PREFIXCLASSES_PREFIX=tw- # adds prefix to all tailwind classes

Automatically keep a private GitHub Repository up-to-date

You can automatically keep a private GitHub repository up-to-date with component changes from TailwindUI by using this tool with GitHub Actions.

Create a private GitHub repository.
Add TAILWINDUI_EMAIL and TAILWINDUI_PASSWORD secrets to the GitHub repository.
Optionally create a .env file with additional settings for the crawler.

Create a new file .github/workflows/default.yml:

name: Update
on:
  schedule:
    - cron: '0 0 * * *' # Every day at midnight

jobs:
  update:
    name: Update
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Run crawler
        uses: gregbrimble/tailwindui-crawler-action@v1.0.0
        with:
          email: ${{ secrets.TAILWINDUI_EMAIL }}
          password: ${{ secrets.TAILWINDUI_PASSWORD }}

Read more about the schedule cron syntax in the official GitHub Actions documentation.

Email Notifications

To be emailed whenever there is a change to a component, simply setup GitHub Notifications on your repository.

🚦 Upgrading to v2.

Since the transformers can make a lot of changes to the files, I would recommend you run the current crawler first to generate the diffs and commit those. Then upgrade and run with transformers enabled. This way you're not mixing up changes.

NOTE: Since this script is essentially screen scraping, there's the potential of it breaking if the HTML structure changes. I will do my best to keep it in sync with the website.