BitWeb

This is a cntinuation of the P2PWeb project, using the bittorrent technology. In a few words, this is a proxy software that you can plug your browser with, and that will let you access torrents as if they were websites.

For example, when you point yout browser to http://dbbda49df0642ebc45f142059e60552a70cc3c5b.bitweb/index.html, the proxy will download the torrent that has the info_hash dbbda49df0642ebc45f142059e60552a70cc3c5b, will look at the file index.html and will serve it over http to your browser. This is the same technology as the magnet links for torrents.

The purpose of this is to have:

be able to put a website online without having to buy for hardware. This reduce costs to create a website and do not force website authors to use advertisement to pay for these costs. The end user benefits from it as he is not tracked by the advertisement companies.
promote static websites that don't rely on server software. Here, we just can't. If you need a server-specific behaviour, you are creating a web application. You could distribute the frontend using bitweb and let the user choose the backend.
decentralize completely the web. Don't separate the client from the server. Having a distributed system where anyone can add a server is great, but is not enough. People will always want to buy the servers from someone else because this is more convenient. And you have big centralized services all over again.
bring back privacy on the web. With no server to keep track of you requests, it is much more difficult to track your activity.
make it much more difficult for central authority to block websites.

Features (not all there yet)

Share static content.
Access torrent content through HTTP url.
Dynamic content can be generated using javascript. Client-side doesn't change compared to current websites
Files that didn't changed between versions are shared and don't need to be re-downloaded

Future (to do):

Identical files that didn't changed between versions don't take twice the disk space
Possibility to add new versions (in the form of a new torrent)
New versions are discovered automatically
A server-side API would need to be standardised for the client side to access dynamic content (such as comments)

Description and installation

The software is both a SOCKS4A proxy (that can reply to HTTP requests for .bitweb domains) and a bittorrent client.

In order to have your browser redirect to this proxy, create a proxy.js file containing:

function FindProxyForURL(url, host) {
    isp = "PROXY ip_address:port; DIRECT";
    tor = "SOCKS 127.0.0.1:9050";
    bit = "SOCKS 127.0.0.1:8878";
    if (shExpMatch(host, "*.onion")) return tor;
    if (shExpMatch(host, "*.bitweb")) return bit;
    return "DIRECT";
}

Put the url of this file (using the file: protocol) in the proxy settings of your browser for the autoconfiguration URL.

Usage

To start the proxy server on the default port (8878), run:

bitweb -D

To create a torrent suitable for websites, use the command line:

bitweb -Ct mywebsite.torrent -k privatekey.p8.der /path/to/my/website

You can add metadata to your pages:

bitweb -Ut mywebsite.torrent -k privatekey.p8.der -p index.html -H "content-type:text/html; charset=utf-8"

You can visualize your torrent:

bitweb -St mywebsite.torrent

You can seed your torrent:

info_hash=$(bitweb -St mywebsite.torrent | grep 'info hash' | cut -d' ' -f3)
torrent_name="$(bitweb -St mywebsite.torrent | grep 'info hash' | cut -d' ' -f3-)"

cp mywebsite.torrent ~/.cache/bitweb/$info_hash.torrent
mkdir ~/.cache/bitweb/$info_hash
cp -R /path/to/my/website "~/.cache/bitweb/$info_hash/$torrent_name"

(and restart your server, HUP signal doesn't work yet)

Future usage (using ed25519)

Generate a key pair:

$ bitweb -K
Public Key: c015b1438a5dd7b18d203758b576719ac768247e89f92444bd1e8b8e77565eca
File: c015b1438a5dd7b18d203758b576719ac768247e89f92444bd1e8b8e77565eca.key

Copy the generated .key file to ~/.local/share/bitweb

Further work, Bittorrent extensions

Padding files (implemented)

In order to be able to share pieces between different versions of a torrent, add padding files in the torrent to make sure that every legitimate file is aligned with a piece boundary. Use the libtorrent approach.

Torrent signature (implemented)

A info dict can contain a RSA public key and a signature that validates the owner of the corresponding private key authored the info dict. The signature is computed using the bencoding of the info dict (with the signature field removed).

For a request to a specific info_hash, the rule is to trust the key included in that info_hash when requesting future revisions.

TODO: Use BEP 35, signing instead? The problem is that the signature is outsie the info dict and may not be transmitted when initializing through magnet links. Or completely forget about this part as it will be taken care of by the torrent revisions

File metadata (implemented)

Add HTTP header information for each file in a torrent (the content-type for example).

Torrent versions

A torrent can reference previous versions. When referencing a previous version, a torrent must also reference recursively all the previous versions. See below for the implementation details.

TODO: maybe we might want to make this more general than bitweb. Avoid using the string "bitweb" when implementing that feature. maybe it could be proposed as a BEP.

Backlinks

A torrent can reference other torrents (or a specific file in other torrents). A link is composed of a source (torent info hash and filename), a relation type (uri that describes what relation the source has to the destination) and a destination (info hash of a torrent and a filename).

Forward links should be stored in the torrent itself and backlinks should be available in the DHT.

Encryption

torrents (info dict and content itself) should be able to be encrypted using a password (that can be embedded in HTML links). like freenet, this would allow anyone to share space and bandwidth for websites you can't know anything about. Notes would advertise that they are willing to share disk space or bandwidth and would download automatically such torrents.

Reference

Hacking

Small tasks:

Auto detect file mime type on torrent creation: Have a --autodetect-content-type when creating a torrent file.
Honor XDG basedir specification
Command line to seed a site: bitweb -A|--add -t TORRENTFILE --copy|--hardlink DIR would copy (--reflink if possible) or hardlink DIR to ~/.local/share/bitweb.
HTTP server: add mode to listen to a HTTP port (default would be 80). Use the HTTP/1.1 Host request header to detect the info_hash of the requested site, or the name of a torrent that is in ~/.local/share/bitweb (not in cache). Default to a specified info hash if no torrent found. Add ability to specify a correspondance table between a domain name and a info hash.
use libtorrent share mode
port code to libtorrent 1.x (I am using 0.x because ArchLinux package is not yet up to date).

Handle torrent versions using DHT store extension

Implementation note: session::dht_get_item() and session::dht_put_item()

In the info dict, add a "bitweb" entry containing a dict with:

public key: ed25519 public key (32 bytes string)
id: any string that, together with the public key) will uniquely identify the website.
seq: the revision of the website. Starts at 1. (integer)
parents: a list containing the revisions of the website. The first item being the first revision (having seq equal to 1). For each revision the 20 bytes of the info hash is stored as a string. The list must contain seq - 1 items. No less, no more.
parent signatures: a list containing the signatures for the parents.

New revisions are made by incrementing the seq number and adding the parent hash info to the parents list.

Revisions are advertised using the DHT store extension. A mutable item is stored with:

k: the public key value defined above
salt: a 27 bytes string. The first 7 bytes being "bitweb:" and the last 20 bytes being the sha1 hash of the id value defined above.
seq: the seq value defined above
v: the info hash of the torrent
auto update: a boolean true (integer 1) or false (integer 0). If not specified, true is assumed.

The cas field is not present, the id, sig and token fields are computed as defined in the DHT store extension.

To allow anyone to republish a version of the torrent, the torrent should contain outside of the info dict the signature required to republish the value. It cannot be included inside the info dict because the signature depends on the info hash value.

TODO: this signature should be obtained by nodes when they use magnet links to obtain the torrent. The client can check that the signature obtained is valid by checking it against the public key and the site id stored inside the info hash.

Other idea: instead of looking up websites by the info hash, look them up against sha1(info_hash + signature) and have this information stored as an immutable message through the DHT store extension. The problem with that is that it adds an indirection layer and will slow down downloading a torrent for the first time, and initial HTTP connection.

When obtained, and if a torrent file is written to disk, the signature should be included in a bitweb signature key at the same level of the info dict. Every bitweb client that share a website should regularly republish the latest signature.

URL conventions:

http://INFOHASH=.bitweb: the website at the revision of INFOHASH exactly
http://INFOHASH=N.bitweb: the website at the Nth revision as specified in the INFOHASH torrent
http://INFOHASH+.bitweb: the website at the latest possible revision
http://INFOHASH+N.bitweb: the website at the Nth revision as specified in the latest possible revision of INFOHASH
http://INFOHASH.bitweb: either the INFOHASH=.bitweb or INFOHASH+.bitweb depending on the auto update value specified in the bitweb dict.

TODO: be notified in real time when a site is updated and do not require pooling frequently for updates.

Hardening code

Each socks client (source IP and port) in a separate process (in case the process crash). Avoid too much processes (one process per connection) and avoid crashing other users or the main daemon.

Have a separate process for the torrent node as well.

Simple related tasks includes:

Add exception handlers
Add SIGSEGV handler that would print debug info and exec to a new instance.

Code issues

Crypto++

Can't load DER encoded key? Check it is in PKCS#8 format

openssl pkcs8 -in privkey.pem -out privkey.p8 -topk8 -nocrypt -outform der

See: http://crypto-users.996303.n3.nabble.com/Load-Private-RSA-key-td2851.html And: http://www.cryptopp.com/fom-serve/cache/62.html And: http://stackoverflow.com/questions/9869261/crypto-load-rsa-pkcs1-private-key-from-memory And: http://www.cryptopp.com/wiki/Keys_and_Formats

See: http://stackoverflow.com/questions/25441918/crypto-cant-der-encode-and-ber-decode-rsa-public-key-ber-decode-error

mildred/bitweb