🕵️‍♂️🔍 WebQL

WebQL is an automated JavaScript analysis engine and workflow orchestration framework for modern web application analysis. It combines the power of static analysis tools like CodeQL with dynamic scanning capabilities to provide comprehensive security insights for web applications.

Check out the blog post WebQL: Using CodeQL To Conduct JavaScript Security Analysis Against Modern Web Applications for more details on how it works.

🌟 Features

🔍 URL scanning and JavaScript file extraction
🧹 Automatic JavaScript beautification
- 🔪📦 ~~Webcrack~~ & Wakaru coming soon!
🗃️ CodeQL database generation
🔬 Vulnerability analysis using CodeQL queries
📊 Results parsing and presentation
🚀 Easy-to-use CLI interface

🛠️ Installation

The easiest way to get up and running is with pipx:

pipx install git+https://github.com/queencitycyber/webql.git

You also need webcrack:

npm install -g webcrack

To install WebQL and it's dependencies using Poetry, which is useful for development:

git clone https://github.com/queencitycyber/webql
cd webql
poetry install

Activate the virtual environment created by Poetry:

poetry shell

Alternatively, you can run WebQL commands without activating the virtual environment by prefixing them with poetry run:

poetry run webql scan https://example.com

If you don't have or don't want to use Poetry, you can use pipx:

git clone https://github.com/queencitycyber/webql
cd webql
pipx install .

Docker installation:

... not just yet, but see Microsoft's codeql-container? Untested but passes sniff test!

🚀 Usage

After installation, you can run WebQL using:

webql

Or:

python -m webql

Help Menu

Usage: webql [OPTIONS] COMMAND [ARGS]...

  WebQL: An automated JavaScript analysis engine and workflow orchestration
  framework.

Options:
  --debug        Enable debug logging
  --config PATH  Path to config file
  --help         Show this message and exit.

Commands:
  full-analysis  Perform a full analysis on a given URL.
  generate       Generate a CodeQL database for JavaScript analysis.
  parse          Parse files using CodeQL.
  results        Parse and display vulnerability results from a SARIF file.
  scan           Scan URLs or files for JavaScript + webpack & sourcemaps.
  secrets        JS secret and juicy bit scanning.

WebQL provides several commands for different stages of analysis.

Scanning a URL

This command will scan the specified URL, extract JavaScript files, beautify them, generate a CodeQL database, and run CodeQL analysis.

webql scan https://example.com

Scanning a single URL with aggresive mode (not usually recommended) and specifying an output directory:

Aggressive Mode pulled directly from zb3's getfrontend found below. HUGE shoutout!

webql scan https://example.com --output-dir ./output --aggressive

Generating a CodeQL Database

This command creates a CodeQL database from the JavaScript files in the specified directory.

webql generate ./output --db-name my_analysis

Analyzing the Database

This command runs CodeQL analysis on the generated database and outputs the results in SARIF format.

webql parse ./output/my_analysis --output-file results.sarif

Viewing Results

This command parses and displays the vulnerability results from the SARIF file.

webql results results.sarif

Check out SARIF Explorer for a better view of the results: SARIF Explorer

Full Analysis

To perform a full analysis in one go:

webql full-analysis https://example.com

This command will scan the URL, generate a CodeQL database, perform analysis, and display the results.

👨🏼‍⚖️ Trial Run - Full Analysis

Full trial run of OWASP's Juice Shop (excuse the loquacious output, you can turn off in the code if you'd like:) :

webql full-analysis https://juice-shop.herokuapp.com

🧪 Examples and Testing

To ensure WebQL continues to function correctly as we develop and maintain it, we've included a set of example websites and JavaScript files that can be used for testing. These examples cover various scenarios and edge cases that WebQL should handle.

Using the Examples

Scan a Known Website

This should successfully scan the example.com website and save JavaScript files to the example_scan directory.

webql scan https://example.com --output-dir ./example_scan

Analyze Local JavaScript Files

This sequence of commands should analyze the sample JavaScript files provided in the test fixtures, generate a CodeQL database, perform the analysis, and display the results.

webql generate ./webql/vulnerable_examples/ --db-name sample_db
webql parse ./sample_db --output-file sample_results.sarif
webql results sample_results.sarif

Scanning JavaScript URLs from a File

WebQL now supports scanning multiple JavaScript URLs provided in a text file. This feature allows for batch processing of JavaScript files from various sources.

To use this feature:

Create a text file (e.g., js_urls.txt) containing one JavaScript URL per line.
Run the scan command with the --url-file option: webql scan --url-file js_urls.txt --output-dir ./output

This command will:

Read JavaScript URLs from the specified file
Download and process each JavaScript file
Beautify the downloaded JavaScript
Run Webcrack analysis on each file
Save the results in the specified output directory

After scanning, you can analyze the downloaded files using the following commands:

webql generate ./output --db-name js_analysis webql parse ./output/js_analysis --output-file results.sarif webql results results.sarif

This feature enhances WebQL's capability to handle multiple JavaScript sources efficiently, making it easier to perform bulk analysis of JavaScript files from various web applications.

Full Analysis of a Test Website

This command runs a full analysis on the OWASP Juice Shop, a purposefully vulnerable web application. It's a good test case for WebQL's ability to detect various vulnerabilities.

webql full-analysis https://juice-shop.herokuapp.com
webql full-analysis https://public-firing-range.appspot.com/

WebQL Vulnerable Examples

WebQL comes with a set of vulnerable JavaScript examples and scripts to test against them. These examples are crucial for demonstrating WebQL's capabilities and for testing purposes.

As WebQL evolves, it's important to keep these examples up-to-date and add new ones as needed:

Regularly run the example commands to ensure they still work as expected.
If a website used in an example changes or becomes unavailable, update the example with a new, suitable website.
Add new examples when implementing new features or handling new edge cases.
Include examples that demonstrate both successful scans/analyses and how WebQL handles errors or edge cases.

📁 Directory Structure

webql/
├── vulnerable_examples/
│   ├── test.js
│   ├── xss_vulnerable.js
│   ├── sql_injection.js
│   └── ... (other vulnerable JS files)
└── scripts/
    └── analyze_examples.py
    └── full_analysis.py

The vulnerable_examples/ directory contains JavaScript files with known vulnerabilities. These files serve as test cases for WebQL's analysis capabilities. Some examples include:

test.js: A basic file with multiple vulnerability types.
xss_vulnerable.js: Demonstrates Cross-Site Scripting (XSS) vulnerabilities.
sql_injection.js: Shows SQL injection vulnerabilities.

🧪 Testing with Vulnerable Examples

Using the `analyze_examples.py` Script

The analyze_examples.py script in the scripts/ directory automates the process of analyzing all vulnerable examples.

To run the script:

python webql/scripts/analyze_examples.py

This script will:

Scan each JavaScript file in the vulnerable_examples/ directory.
Generate a CodeQL database for each file.
Analyze the database using predefined queries.
Display the results

If you find a website or create a JavaScript sample that would make a good test case for WebQL:

For websites, add the URL and a brief description to the README in the "Examples and Testing" section.
For JavaScript files, add them to the webql/vulnerable_examples/ or something appropriate directory.
Create a new test in the appropriate test file under the tests/ directory.
Update this README section if necessary to include any new usage examples.

By regularly using and updating these examples, we can ensure that WebQL remains robust and effective across a wide range of scenarios.

🔮 Future Features

We're constantly working to improve WebQL. Here are some features we're planning to implement:

🔪 ~~Webcrack~~ & Wakaru coming soon!

🌐 Support for additional JavaScript frameworks and libraries

🔧 Custom CodeQL query support

📈 Enhanced reporting capabilities with graphical output

🔄 Continuous monitoring mode for real-time analysis

🔌 Plugin system for extending functionality

🔒 Integration with additional security tools and APIs

🖥️ Web interface for easier interaction and result visualization

queencitycyber/webql