urlproxy is an HTTP reverse proxy that allows for dynamic routing without the need for pre-defined rules like traditional reverse proxies such as Caddy. It accomplishes this by encoding the target website's address into the request URL. In addition, it can also chain to an upstream SOCKS5 proxy.
Install to $GOPATH/bin
:
go install github.com/zjx20/urlproxy@latest
Compiling from source code:
go build -o urlproxy main.go
$ ./urlproxy -h
Usage of ./urlproxy:
-bind string
Address to bind (default "0.0.0.0:8765")
-debug
Verbose logs
-file-root string
Root path for the file scheme
-socks string
Upstream socks5 proxy, e.g. 127.0.0.1:1080
-socks-uds string
Path of unix domain socket for upstream socks5 proxy
Simply run ./urlproxy
, and urlproxy will listen on 8765 by default.
Assume that the target URL is:
http://website.com/some/path?param1=val1¶m2=val2
Then the request URL to urlproxy should be:
http://127.0.0.1:8765/website.com/some/path?param1=val1¶m2=val2
That's it. Headers are forwarded to the target server as well, and urlproxy follows redirects automatically.
There are some special url parameters that can further control the proxy behavior.
-
uOptScheme
: specify the scheme for the target URL, e.g.https
.$ curl "http://127.0.0.1:8765/httpbin.org/headers?uOptScheme=https" { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip", "Host": "httpbin.org", "User-Agent": "curl/7.79.1", "X-Amzn-Trace-Id": "Root=1-63b853d8-2ec483150e898d8470823f62" } }
file
scheme is also supported. But only files inside the-file-root
folder are allowed to read. This turnsurlproxy
into an HTTP server for hosting static files. For example:$ echo hello > /tmp/hello.txt $ ./urlproxy -file-root /tmp & $ curl "http://localhost:8765/hello.txt?uOptScheme=file" hello
Another special scheme is
tpl
, see the Template Rendering section for more details. -
uOptHeader
: add extra headers to the proxied request.$ curl "http://127.0.0.1:8765/httpbin.org/headers?uOptHeader=CustomHeader1:value1&uOptHeader=CustomHeader2:value2" { "headers": { "Accept": "*/*", "Accept-Encoding": "gzip", "Customheader1": "value1", "Customheader2": "value2", "Host": "httpbin.org", "User-Agent": "curl/7.79.1", "X-Amzn-Trace-Id": "Root=1-63b8548b-78020fa9171479c72b1a422c" } }
-
uOptRespHeader
: add extra headers to the response.$ curl -v "http://127.0.0.1:8765/httpbin.org/headers?uOptRespHeader=CustomHeader1:value1&uOptRespHeader=CustomHeader2:value2" > /dev/null Trying 127.0.0.1:8765... * Connected to 127.0.0.1 (127.0.0.1) port 8765 (#0) > GET /httpbin.org/headers?uOptRespHeader=CustomHeader1:value1&uOptRespHeader=CustomHeader2:value2 HTTP/1.1 > Host: 127.0.0.1:8765 > User-Agent: curl/7.79.1 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Access-Control-Allow-Credentials: true < Access-Control-Allow-Origin: * < Connection: keep-alive < Content-Length: 297 < Content-Type: application/json < Customheader1: value1 < Customheader2: value2 < Date: Fri, 08 Sep 2023 13:38:13 GMT < Server: gunicorn/19.9.0 < * Connection #0 to host 127.0.0.1 left intact
-
uOptSocks
: specify the upstream socks5 proxy for this request.$ curl "http://127.0.0.1:8765/httpbin.org/get?uOptSocks=127.0.0.1:1081"
In particular,
"uOptSocks=off"
means that this request does not use the socks proxy. -
uOptDns
: specify the DNS server used in this request.$ curl "http://127.0.0.1:8765/httpbin.org/get?uOptDns=8.8.8.8:53"
-
uOptIp
: specify the IP address of the target server for this request.$ curl "http://127.0.0.1:8765/httpbin.org/get?uOptIp=3.229.200.44"
-
uOptTimeoutMs
: specify timeout for this request, the time is including internal retries.$ curl "http://127.0.0.1:8765/httpbin.org/delay/5?uOptTimeoutMs=1000"
-
uOptRetriesNon2xx
: number of retries for non-2xx response. Note that it only supports retries for requests that use theGET
,HEAD
,OPTIONS
orTRACE
methods.$ curl "http://127.0.0.1:8765/httpbin.org/status/500?uOptRetriesNon2xx=3"
-
uOptRetriesError
: number of retries for errors (e.g. connection timeout).$ curl "http://127.0.0.1:8765/httpbin.org:1234/get?uOptRetriesError=3"
-
uOptAntiCaching
: anti-caching by adding__t=<current time in nanoseconds>
parameter to the request URL.$ curl "http://127.0.0.1:8765/httpbin.org/get?uOptAntiCaching=true"
-
uOptRaceMode
: in race mode, urlproxy will simultaneously send several identical requests to the target server, and the first response will be used to reply to the client. The value of this parameter is a number that indicates the parallelism of the request and takes a maximum value of 5. Similar touOptRetriesNon2xx
, only certain http methods can use this mode.$ curl "http://127.0.0.1:8765/httpbin.org/get?uOptRaceMode=2"
-
uOptRewriteRedirect
:urlproxy
does not automatically follow redirects (such as 301 and 302 status codes), but returns the http status code andLocation
to the client. When redirecting, the client may not send request tourlproxy
, but directly connect to the target address. This option can rewrite the redirect, changing theLocation
to an address pointing tourlproxy
.$ curl "http://127.0.0.1:8765/httpbin.org/redirect-to?url=http://google.com&status_code=302&uOptRewriteRedirect=true"
-
uOptPipe
: the content of this parameter is a shell script. When the proxy request is successful (such as the response code is 200),urlproxy
will execute this script through/bin/sh
, and use the body of the proxy response as the stdin ofexec.Cmd
, and then forward the stdout ofexec.Cmd
to the http client.$ curl "http://127.0.0.1:8765/httpbin.org/get?uOptPipe=sed%20%27s%2Fhttpbin.org%2Fgro.nibptth%2Fg%27" # The command is equivalent to: # curl -s "http://127.0.0.1:8765/httpbin.org/get" | sed 's/httpbin.org/gro.nibptth/g'
Note: This feature is powerful but also very dangerous. Therefore, the
uOptPipe
option will only take effect when the-enable-uoptpipe
command-line parameter was added to starturlproxy
. Please make sure not to deploy this feature to the public network. -
uOptQueryParams
: add extra query parameters to the proxied request. It's useful for passinguOpt*
to the proxied request.$ curl "http://127.0.0.1:8765/httpbin.org/get?foo=bar&uOptQueryParams=hello%3Dworld"
Options can be placed in the path with the format of /uOptXXX=XXX/
. This can be useful in some cases.
$ curl "http://127.0.0.1:8765/uOptScheme=https/uOptHeader=User-Agent:MyClient/httpbin.org/headers"
{
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Host": "httpbin.org",
"User-Agent": "MyClient",
"X-Amzn-Trace-Id": "Root=1-63b99e95-439b4d3a6725f13a04464629"
}
}
urlproxy can also act as a regular HTTP proxy, this allows it to function as an HTTP-to-SOCKS proxy.
export http_proxy=http://127.0.0.1:8765
curl -v http://httpbin.org/get
urlproxy
treats Go Template as a programming language for handling http requests (similar to PHP), which allows for some complex data processing. This is equivalent to implementing func ServeHTTP(w http.ResponseWriter, r *http.Request)
with Go Template, so the http.Request
and http.ResponseWriter
objects of the current request are available in the template context. Request data such as query parameters can be retrieved by using the http.Request
object. For the response, the status code, headers and body can be set using the http.ResponseWriter
object. The render result of the template will also be appended to the response body.
When starting urlproxy
, there is a -tpl-root
parameter that specifies the directory where the template files are located so that they can be referenced in subsequent requests. If the request path points to the template file (relative to -tpl-root
) and the uOptScheme=tpl
parameter is given, urlproxy
will handle the http request using the specified template.
The following is an example template. It first retrieves the value of the "value"
parameter from the request, constructs an http request, retrieves data from httpbin.org, then parses the returned data into a map, iterates over key-value pairs for output, and finally sets the status code to 500 and appends a header named "Marker"
.
{{- /* make a request */ -}}
{{- $myheader := httpHeader "MyHeader" (.Request.URL.Query.Get "value") -}}
{{- $resp := httpReq nil "GET" "https://httpbin.org/headers" "" $myheader 10 -}}
{{- /* parse the response from httpbin.org, and then render it, as the response body */ -}}
{{- $respObj := $resp.Text | fromJson -}}
{{ range $key, $value := $respObj.headers }}
{{- printf "%s: %v" $key $value }}
{{ end }}
{{- /* It can also use .ResponseWriter.WriteString to write data to the response body.
However, it's not recommended to mix the usage of WriteString and the text
rendering, because their output order are not determined. */ -}}
{{- .ResponseWriter.WriteString "Output with .ResponseWriter.WriteString.\n" }}
{{ range $key, $value := $respObj.headers }}
{{- .ResponseWriter.WriteString (printf "%s: %v\n" $key $value) }}
{{ end }}
{{- /* control the response header */ -}}
{{- $_ := .ResponseWriter.Header.Set "Marker" "handled by Go Template" -}}
{{- $_ := .ResponseWriter.WriteHeader 500 -}}
Save the above template to ./tplfiles/addheader.tpl
and run the following command.
# start urlproxy
$ ./urlproxy -tpl-root ./tplfiles &
# invoke addheader.tpl
$ curl -v "http://127.0.0.1:8765/addheader.tpl?value=hello&uOptScheme=tpl"
* Trying 127.0.0.1:8765...
* Connected to 127.0.0.1 (127.0.0.1) port 8765 (#0)
> GET /addheader.tpl?uOptScheme=tpl&value=hello HTTP/1.1
> Host: 127.0.0.1:8765
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Marker: handled by Go Template
< Date: Sun, 10 Sep 2023 10:34:48 GMT
< Content-Length: 145
< Content-Type: text/plain; charset=utf-8
<
Accept-Encoding: gzip
Host: httpbin.org
Myheader: hello
User-Agent: Go-http-client/2.0
X-Amzn-Trace-Id: Root=1-64fd9bc7-2c13ea207781cdee55c3865e
* Connection #0 to host 127.0.0.1 left intact
Besides referencing a .tpl
file, the template can also be inlined in the "inline"
parameter. For example:
# the inlined template is `sha1sum of "hello" is {{ sha1sum "hello" }}`
$ curl "http://127.0.0.1:8765/?uOptScheme=tpl&inline=sha1sum%20of%20%22hello%22%20is%20%7B%7B%20sha1sum%20%22hello%22%20%7D%7D"
# the output is
sha1sum of "hello" is aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d
All functions from slim-sprig are available in the template context. timeoutSec
can be 0, which means no timeout. urlproxy
also provides other useful functions.
-
httpReq(ctx, method, url, body, headers, timeoutSec)
- Initiates an http request. Returns aResponseWrapper
object.{{- $resp := httpReq nil "GET" "https://httpbin.org/get" "" nil 10 -}} {{- $resp := httpReq nil "POST" "https://httpbin.org/post" "hello" nil 10 -}} {{- $resp := httpReq nil "GET" "https://httpbin.org/headers" "" (httpHeader "HeaderName" "value") 10 -}} {{- /* uses the ctx from .Request, httpReq will be interrupted if the client disconnects. */ -}} {{- $resp :=httpReq .Request.Context "GET" "https://httpbin.org/delay/10" "" nil 60 -}}
The
ResponseWrapper
object embeds ahttp.Response
, so all the fields in thehttp.Response
can be accessed by theResponseWrapper.Response.XXX
.ResponseWrapper
also comes with additional methods to handle the http response.{{- /* Text(): it returns the whole response body as a string */ -}} {{- $resp := httpReq nil "GET" "https://httpbin.org/get" "" nil 10 -}} StatusCode: {{ $resp.Response.StatusCode }} Response: {{ $resp.Text }} {{- /* LastLocation(): httpReq automatically follows redirects, so LastLocation() can be used to get the last location */ -}} {{- $url := "https://httpbin.org/redirect-to?url=https://httpbin.org/base64/aGVsbG8K" -}} {{- $resp := httpReq nil "GET" $url "" nil 10 -}} Url: {{ $url }} Last Location: {{ $resp.LastLocation }}
-
tryHttpReq(ctx, method, url, body, headers, timeoutSec)
- It is the same ashttpReq
except that it does not abort the execution of the template in case of an error. If it does,ResponseWrapper.Response
will benil
and the error can be retrieved usingResponseWrapper.Error
.{{- /* will hit a "deadline exceeded" error */ -}} {{- $resp := tryHttpReq nil "GET" "https://httpbin.org/delay/10" "" nil 1 -}} {{- if $resp.Error -}} {{ printf "request failed, err: %s" $resp.Error }} {{- end -}}
-
httpHeader(v ...string)
- Creates aHeaderWrapper
(mainly used forhttpReq
) with an even number of arguments.{{- httpHeader "Header1" "value1" "Header2" "value2" -}}
-
parseUrl(url)
- Parses the url and returns aurl.URL
object.{{- (parseUrl "https://httpbin.org/").Host -}}
-
urlproxiedUrl(url, urlproxyBaseUrl, uopts ...string)
- Converts to a proxied url (then used withhttpReq
).{{- urlproxiedUrl "https://httpbin.org/" "http://127.0.0.1:8765/" "uOptScheme" "http" "uOptTimeoutMs" "3000" -}}
The address of the current
urlproxy
instance can be obtained via the template variable.ExtraValues.urlproxyBaseUrl
.{{- urlproxiedUrl "https://httpbin.org/" .ExtraValues.urlproxyBaseUrl -}}
-
save(namespace, key, value)
- Stores a key value in the persistent store. The location of the store can be specified with the-kvstore-dir
flag when startingurlproxy
. Note thatsave
may fail for various reasons, the return value indicates success or not.{{- $ok := save "example" "foo" "bar" -}}
-
mustSave(namespace, key, value)
- The same assave
except that it aborts the execution if an error occurs.{{- mustSave "example" "foo" "bar" -}}
-
load(namespace, key)
- Loads the previously saved value. Returns an empty string if the key is not found or an error has occurred.{{- load "example" "foo" -}}
-
mustLoad(namespace, key)
- The same asload
except that it aborts the execution if an error occurs.{{- mustLoad "example" "foo" -}}
-
execTemplate(name, data)
- It executes the sub-template defined by thedefine
action and captures the render result. The idea comes from this SO answer.{{- define "greet" -}} {{- printf "Hello %s!" . -}} {{- end -}} {{- /* $msg should be "Hello Bob!" */ -}} {{- $msg := execTemplate "greet" "Bob" -}}
-
debug(format, ...any)
- It writes a message to the urlproxy log with DEBUG level. The-debug
parameter should be set when startingurlproxy
to see these logs.{{- $_ := debug "Hello %s!" "Bob" -}}
urlproxy
can recognize HLS (HTTP Live Streaming) content and accelerate it, the technology is called HLSBoost
. Typically, HLS players download segments in a conservative manner, such as downloading segments one by one in sequence with a single thread. In poor network conditions, this can easily lead to playback stuttering. HLSBoost
works between the HLS player and server, intercepts and caches segment files by sniffing and hijacking the m3u8 playlist. It employs an aggressive prefetch strategy that tracks the client's playback progress to cache required content ahead of time, thereby reducing playback stuttering. Additionally, when servers have TCP connection-level speed limits, multi-threaded slicing download technology used by HLSBoost
can effectively increase download speeds.
Suppose there is an HLS playback link:
http://example.com/hls/index.m3u8
Convert it to urlproxy
style link with uOptHLSBoost=true
parameter included:
http://127.0.0.1:8765/uOptHLSBoost=true/example.com/hls/index.m3u8
Then give the proxied link to the player for accelerated performance!
-
uOptHLSPrefetches
: Specifies the number of concurrent downloads for segments; default value is 3 and set to 0 to disable prefetching. -
uOptAntConcurrentPieces
: Specifies the number of threads used for multi-threaded downloads; setting this value to 1 disables multi-threaded downloads; default value is 5. -
uOptAntPieceSize
: If number of download threads are greater than 1, specifies how many bytes per thread will be downloaded at once; default value is 524288 (512KB). -
uOptHLSTimeoutMs
: Timeout for fetchingm3u8
playlist or segments; default value is 5000 (5 seconds). -
uOptHLSShortUrl
:HLSBoost
proxies and modifies the m3u8 playlist, rewriting the segment urls to add some tracking information. Some software cannot handle urls that are too long, such astvheadend
, which truncates long urls. In this case, you can enable the short url feature by passing theuOptHLSShortUrl=true
parameter andHLSBoost
will generate shorter segment urls to avoid this problem. If the short url feature is not enabled, the generated urls will contain the original address information of the segment, so even if the client requests an expired segment, it can still be served using this original address information. Conversely, if the short url feature is enabled, expired segments cannot be served.
MIT