Pantomime, a Library For Working With MIME Types In Clojure

Pantomime is a Clojure interface to Apache Tika.

Originally created as a library that deals with MIME types (Internet media types, sometimes referred to as "content types"), it now also supports extraction of document metadata and text content.

Maven Artifacts

Pantomime artifacts are released to Clojars. If you are using Maven, add the following repository definition to your

pom.xml:

<repository>
  <id>clojars.org</id>
  <url>http://clojars.org/repo</url>
</repository>

Latest Release

With Leiningen:

[com.novemberain/pantomime "2.5.0"]

With Maven:

<dependency>
  <groupId>com.novemberain</groupId>
  <artifactId>pantomime</artifactId>
  <version>2.5.0</version>
</dependency>

Supported Clojure versions

Pantomime requires Clojure 1.6+. The most recent stable release is highly recommended.

Usage

Detecting MIME type

pantomime.mime/mime-type-of function accepts content as byte arrays, java.io.InputStream and java.net.URL instances as well as filenames as strings and java.io.File instances, and returns MIME type as a string or "application/octet-stream" if detection failed.

An example:

(ns your.app.namespace
  (:require [pantomime.mime :refer [mime-type-of]]))

;; by content (as byte array)
(mime-type-of (.getBytes "filename.pdf"))
;; by file extension
(mime-type-of "filename.pdf")
;; by file content (as java.io.File)
(mime-type-of (File. "some/file/without/extension"))
;; by content (as java.net.URL)
(mime-type-of (URL. "http://domain.com/some/url/path.pdf"))

Pantomime has a variation of mime-type-of function that is suitable for cases when content was fetched from the Web and HTTP headers are also available:

(ns your.app.namespace
  (:require [pantomime.web :refer [mime-type-of]]))

;; body is a string or input stream, headers is a map of lowercased headers.
;; Ring and clj-http both use this format for headers, for example.
(mime-type-of body headers)

In this case, Pantomime will try to detect content type from response body first (because there are applications, frameworks and servers that report content type incorrectly, for example, serve PDFs as text/html) and if it fails, will use content type header.

HTTP headers map must contain "content-type" key for content type header to be used. Most Clojure HTTP client, for instance, clj-http, use lowercase strings for header names so Pantomime follows this convention.

Extension Recommendation

Pantomime can recommend an extension (one of the well known ones) for a MIME type:

(require [pantomime.mime :as pm])

(pm/extension-for-name "application/vnd.ms-excel")
;= ".xls"
(pm/extension-for-name "image/jpeg")
;= ".jpg"
(pm/extension-for-name "application/octet-stream")
;= ".bin"

Parsing and Recognizing Media Types

(ns your.app.namespace
  (:require [pantomime.media :as mt]))

(mt/parse "application/json")

(mt/base-type "text/html; charset=UTF-8") ;; => media type of "text/html"

(mt/application? "application/json")
(mt/application? "application/xhtml+xml")
(mt/application? "application/pdf")
(mt/application? "application/vnd.ms-excel")
(mt/application? (mt/parse "application/json"))

(mt/image? "image/jpeg")
(mt/audio? "audio/mp3")
(mt/video? "video/quicktime")
(mt/text?  "text/plain")
(mt/has-parameters? "text/html; charset=UTF-8") ;; => true
(mt/has-parameters? "text/html") ;; => false
(mt/parameters-of "text/html; charset=UTF-8") ;; => {"charset" "UTF-8"}
(mt/charset-of "text/html; charset=UTF-8") ;; => "UTF-8"

Language Detection

pantomime.languages is a new that provides functions for detecting natural languages:

(require [pantomime.languages :as pl])

(pl/detect-language "this is English, it should not be hard to detect")
;= "en"

(pl/detect-language "parlez-vous Français")
;= "fr"

Note that Tika (and, in turn, Pantomime) supports detection of a limited number of languages. To get the list of supported languages, use the pantomime.languages/supported-languages var.

Metadata and Text Extraction

pantomime.extract provides the single function parse for extracting metadata and text content from byte arrays, java.io.InputStream and java.net.URL instances as well as filenames as strings and java.io.File instances.

An example:

(require [clojure.java.io :as io]
         [pantomime.extract :as extract])

(pprint (extract/parse "test/resources/pdf/qrl.pdf"))

;= {:producer ("GNU Ghostscript 7.05"),
;=  :pdf:pdfversion ("1.2"),
;=  :dc:title ("main.dvi"),
;=  :dc:format ("application/pdf; version=1.2"),
;=  :xmp:creatortool ("dvips(k) 5.86 Copyright 1999 Radical Eye Software"),
;=  :pdf:encrypted ("false"),
;=  ...
;=  :text "\nQuickly Reacquirable Locks∗\n\nDave Dice Mark Moir ... "
;= }

If extraction fails, extract.parse will return the following:

{:text "",
 :content-type ("application/octet-stream"),
 :x-parsed-by ("org.apache.tika.parser.EmptyParser")}

extract/parse is a simple interface to Tika's own Parser.parse method.

Community

Pantomime has a mailing list. Feel free to join it and ask any questions you may have.

To subscribe for announcements of releases, important changes and so on, please follow @ClojureWerkz on Twitter.

Pantomime Is a ClojureWerkz Project

Pantomime is part of the group of libraries known as ClojureWerkz, together with Monger, Langohr, Neocons, Elastisch, Quartzite and several others.

Continuous Integration

CI is hosted by travis-ci.org

Development

Pantomime uses Leiningen 2. Make sure you have it installed and then run tests against all supported Clojure versions using

lein all test

Then create a branch and make your changes on it. Once you are done with your changes and all tests pass, submit a pull request on Github.

License

Distributed under the Eclipse Public License, the same as Clojure.

jboverfelt/pantomime