xdev-developer/scala-uri

Simple scala library for building and parsing URIs

scala-uri

scala-uri is a small Scala library that helps you work with URIs. It has the following features:

• A DSL for building URIs
• A parser to parse URIs from Strings.
• Can be used outside a servlet environment as it has zero dependencies on the servlet spec or existing web frameworks.
• Ability to replace and remove query string parameters
• Support for custom encoding such as encoding spaces as pluses
• Support for protocol relative urls
• Support for user information e.g. ftp://user:password@mysite.com
• Support for matrix parameters

To include it in your SBT project from maven central:

"com.netaporter" %% "scala-uri" % "0.4.1"

Building URIs with the DSL

Query Strings

import com.netaporter.uri.dsl._

val uri = "http://theon.github.com/scala-uri" ? ("p1" -> "one") & ("p2" -> 2) & ("p3" -> true)
uri.toString //This is: http://theon.github.com/scala-uri?p1=one&p2=2&p3=true

val uri2 = "http://theon.github.com/scala-uri" ? ("param1" -> Some("1")) & ("param2" -> None)
uri2.toString //This is: http://theon.github.com/scala-uri?param1=1

To add query string parameters, use either the ? or & method and pass a Tuple2 as an argument. The first value in the Tuple is a name of the query string parameter, the second is the value. If a parameter value is an Option, it will only be rendered provided it is not None.

Shortcoming with the DSL

Note there is currently a shortcoming the the DSL when defining a path with both the / operator and the ? operator. A URI defined like so:

"http://host" / "path" / "to" / "resource" ? ("a" -> "1" ) & ("b" -> "2")

Leads to an unexpected result:

res1: com.netaporter.uri.Uri = http://host/path/to/%2Fresource%3Fa%3D1?b=2

To work around, you must use parentheses like so:

("http://host" / "path" / "to" / "resource") ? ("a" -> "1" ) & ("b" -> "2")

res2: com.netaporter.uri.Uri = http://host/path/to/resource?a=1&b=2

There is a plan to fix this in the future.

Adding multiple query parameters

import com.netaporter.uri.dsl._
val p = ("key", true) :: ("key2", false) :: Nil
val uri = "http://example.com".addParams(p)
uri.toString //This is: http://example.com/?key=true&key2=false

Paths

import com.netaporter.uri.dsl._

val uri = "http://theon.github.com" / "scala-uri"
uri.toString //This is: http://theon.github.com/scala-uri

To add path segments, use the / method

Fragments

To set the fragment, use the `#` method:

import com.netaporter.uri.dsl._
val uri = "http://theon.github.com/scala-uri" `#` "fragments"

uri.toString //This is: http://theon.github.com/scala-uri#fragments

Parsing URIs

Provided you have the import com.netaporter.uri.dsl._, Strings will be implicitly parsed into Uri instances:

import com.netaporter.uri.dsl._
val uri: Uri = "http://theon.github.com/scala-uri?param1=1&param2=2"

However, if you prefer, you can call Uri.parse() explicitly:

import com.netaporter.uri.Uri.parse
val uri = parse("http://theon.github.com/scala-uri?param1=1&param2=2")

Transforming URIs

map

The mapQuery method will transform the Query String of a URI by applying the specified Function to each Query String Parameter

val uri = "/scala-uri" ? ("p1" -> "one") & ("p2" -> 2) & ("p3" -> true)

//Results in /scala-uri?p1_map=one_map&p2_map=2_map&p3_map=true_map
uri.mapQuery {
  case (n, v) => (n + "_map", v + "_map")
}

uri.mapQuery(_.swap) //Results in /scala-uri?one=p1&2=p2&true=p3

The mapQueryNames and mapQueryValues provide a more convenient way to transform just Query Parameter names or values

val uri = "/scala-uri" ? ("p1" -> "one") & ("p2" -> 2) & ("p3" -> true)

uri.mapQueryNames(_.toUpperCase) //Results in /scala-uri?P1_map=one&P2=2&P3=true

uri.mapQueryValues(_.replace("true", "false")) //Results in /scala-uri?p1=one&p2=2&p3=false

filter

The filterQuery method will remove any Query String Parameters for which the provided Function returns false

val uri = "/scala-uri" ? ("p1" -> "one") & ("p2" -> 2) & ("p3" -> true)

//Results in /scala-uri?p2=2
uri.filterQuery {
  case (n, v) => n.contains("2") && v.contains("2")
}

uri.filterQuery(_._2 == "one") //Results in /scala-uri?p1=one

The filterQueryNames and filterQueryValues provide a more convenient way to filter just by Query Parameter name or value

val uri = "/scala-uri" ? ("p1" -> "one") & ("p2" -> 2) & ("p3" -> true)

uri.filterQueryNames(_ > "p1") //Results in /scala-uri?p2=2&p3=true

uri.filterQueryValues(_.length == 1) //Results in /scala-uri?p2=2

URL Percent Encoding

By Default, scala-uri will URL percent encode paths and query string parameters. To prevent this, you can call the uri.toStringRaw method:

import com.netaporter.uri.dsl._
val uri = "http://example.com/path with space" ? ("param" -> "üri")

uri.toString //This is: http://example.com/path%20with%20space?param=%C3%BCri

uri.toStringRaw //This is: http://example.com/path with space?param=üri

The characters that scala-uri will percent encode by default can be found here. You can modify which characters are percent encoded like so:

Only percent encode the hash character:

import com.netaporter.uri.encoding._
implicit val config = UriConfig(encoder = percentEncode('#'))

Percent encode all the default chars, except the plus character:

import com.netaporter.uri.encoding._
implicit val config = UriConfig(encoder = percentEncode -- '+')

Encode all the default chars, and also encode the letters a and b:

import com.netaporter.uri.encoding._
implicit val config = UriConfig(encoder = percentEncode ++ ('a', 'b'))

Encoding spaces as pluses

The default behaviour with scala uri, is to encode spaces as %20, however if you instead wish them to be encoded as the + symbol, then simply add the following implicit val to your code:

import com.netaporter.uri.dsl._
import com.netaporter.uri.encoding._
implicit val config = UriConfig(encoder = percentEncode + spaceAsPlus)

val uri: Uri = "http://theon.github.com/uri with space"
uri.toString //This is http://theon.github.com/uri+with+space

Custom encoding

If you would like to do some custom encoding for specific characters, you can use the encodeCharAs encoder.

import com.netaporter.uri.dsl._
import com.netaporter.uri.encoding._
implicit val config = UriConfig(encoder = percentEncode + encodeCharAs(' ', "_"))

val uri: Uri = "http://theon.github.com/uri with space"
uri.toString //This is http://theon.github.com/uri_with_space

URL Percent Decoding

By Default, scala-uri will URL percent decode paths and query string parameters during parsing:

import com.netaporter.uri.dsl._
val uri: Uri = "http://example.com/i-have-%25been%25-percent-encoded"

uri.toString //This is: http://example.com/i-have-%25been%25-percent-encoded

uri.toStringRaw //This is: http://example.com/i-have-%been%-percent-encoded

To prevent this, you can bring the following implicit into scope:

import com.netaporter.uri.dsl._
implicit val c = UriConfig(decoder = NoopDecoder)
val uri: Uri = "http://example.com/i-havent-%been%-percent-encoded"

uri.toString //This is: http://example.com/i-havent-%25been%25-percent-encoded

uri.toStringRaw //This is: http://example.com/i-havent-%been%-percent-encoded

Replacing Query String Parameters

If you wish to replace all existing query string parameters with a given name, you can use the uri.replaceParams() method:

import com.netaporter.uri.dsl._
val uri = "http://example.com/path" ? ("param" -> "1")
val newUri = uri.replaceParams("param", "2")

newUri.toString //This is: http://example.com/path?param=2

Removing Query String Parameters

If you wish to remove all existing query string parameters with a given name, you can use the uri.removeParams() method:

import com.netaporter.uri.dsl._
val uri = "http://example.com/path" ? ("param" -> "1") & ("param2" -> "2")
val newUri = uri.removeParams("param")

newUri.toString //This is: http://example.com/path?param2=2

Get query string parameters

To get the query string parameters as a Map[String,List[String]] you can do the following:

import com.netaporter.uri.dsl._
val uri = "http://example.com/path" ? ("param" -> "1") & ("param2" -> 2)
uri.query.paramMap //This is: Map(param -> List(1), param2 -> List(2))

User Information

scala-uri supports user information (username and password) encoded in URLs.

Parsing URLs with user information:

val uri = "http://user:pass@host.com"
uri.user //This is Some("user")
uri.password //This is Some("pass")

Modifying user information:

import com.netaporter.uri.dsl._
val mailto = "mailto://user@host.com"
mailto.withUser("jack") //URL is now jack@host.com

import com.netaporter.uri.dsl._
val uri = "http://user:pass@host.com"
uri.withPassword("secret") //URL is now http://user:secret@host.com

Note: that using clear text passwords in URLs is ill advised

Protocol Relative URLs

Protocol Relative URLs are supported in scala-uri. A Uri object with a protocol of None, but a host of Some(x) will be considered a protocol relative URL.

import com.netaporter.uri.dsl._
val uri: Uri = "//example.com/path"
uri.scheme //This is: None
uri.host //This is: Some("example.com")

Matrix Parameters

Matrix Parameters are supported in scala-uri.

import com.netaporter.uri.dsl._
val uri = "http://example.com/path;paramOne=value;paramTwo=value2/pathTwo;paramThree=value3"

//Get parameters at the end of the path
uri.matrixParams //This is Vector("paramThree" -> "value3")

//Add parameters to end of path
val uri2 = uri.addMatrixParam("paramFour", "value4")
uri2.toString //This is http://example.com/path;paramOne=value;paramTwo=value2/pathTwo;paramThree=value3;paramFour=value4

//Get parameters for mid path segment
uri.pathPart("pathTwo").params //This is Vector("paramOne" -> "value", "paramTwo" -> "value2")

//Add parameters for mid path segment
val uri3 = uri.addMatrixParam("pathTwo", "paramFour", "value4")

Character Sets

By default scala-uri uses UTF-8 charset encoding:

val uri = "http://theon.github.com/uris-in-scala.html" ? ("chinese" -> "网址")
uri.toString //This is http://theon.github.com/uris-in-scala.html?chinese=%E7%BD%91%E5%9D%80

This can be changed like so:

implicit val conf = UriConfig(charset = "GB2312")
val uri = "http://theon.github.com/uris-in-scala.html" ? ("chinese" -> "网址")
uri.toString //This is http://theon.github.com/uris-in-scala.html?chinese=%CD%F8%D6%B7

Including scala-uri your project

scala-uri 0.4.x is currently built with support for scala 2.10.x (and will support 2.11.x)

For 2.9.x support use scala-uri 0.3.x

Release builds are available in maven central. For SBT users just add the following dependency:

"com.netaporter" %% "scala-uri" % "0.4.1"

For maven users you should use (for 2.10.x):

<dependency>
    <groupId>com.netaporter</groupId>
    <artifactId>scala-uri_2.10</artifactId>
    <version>0.4.1</version>
</dependency>

Latest snapshot builds

For the latest snapshot builds, add the Sonatype OSS repo to your SBT build configuration:

resolvers += "Sonatype OSS" at "http://oss.sonatype.org/content/repositories/snapshots"

Add the following dependency:

"com.netaporter" %% "scala-uri" % "0.4.1-SNAPSHOT"

Contributions

Contributions to scala-uri are always welcome. Good ways to contribute include:

• Raising bugs and feature requests
• Fixing bugs and developing new features (I will attempt to merge in pull requests ASAP)
• Improving the performance of scala-uri. See the Performance Tests project for details of how to run the scala-uri performance benchmarks.

Building scala-uri

Unit Tests

The unit tests can be run from the sbt console by running the test command! Checking the unit tests all pass before sending pull requests will be much appreciated.

Generate code coverage reports from the sbt console by running the scct:test command. The HTML reports should be generated at target/scala-2.10/coverage-report/index.html. Ideally pull requests shouldn't significantly decrease code coverage, but it's not the end of the world if they do. Contributions with no tests are better than no contributions :)

Performance Tests

For the scala-uri performance tests head to the scala-uri-benchmarks github project

Migration guide from 0.3.x

• Package changes / import changes
• All code moved from com.github.theon package to com.netaporter package
• scala-uri has been organised into the following packages: encoding, decoding, config and dsl. You will need to update import statments.
• Name changes
• PermissiveDecoder renamed to PermissivePercentDecoder
• QueryString and MatrixParams constructor argument parameters shortened to params
• Uri.parseUri renamed to Uri.parse
• protocol constructor arg in Uri renamed to scheme
• Querystring renamed to QueryString
• Query String constructor argument parameters changed type from Map[String, List[String]] to Seq[(String,String)]
• Uri constructor argument pathParts changed type from List to Vector
• Uri method to add query string parameters renamed from params to addParams. Same with matrixParams -> addMatrixParams
• PercentEncoderDefaults object renamed to PercentEncoder companion object.
• Copy methods user/password/port/host/scheme now all prefixed with with, e.g. withHost
• New UriConfig case class used to specify encoders, decoders and charset to be used. See examples in Custom encoding, URL Percent Decoding and Character Sets

License

scala-uri is open source software released under the Apache 2 License.