catching SSLErrors

[lucasluitjes]

First off thanks for the great work on spidr! Second, I noticed that spidr kinda dies on sites that completely fail at SSL. For instance https://36pizza.com causes spidr to crash with an ssl error. This can be resolved by adding OpenSSL::SSL::SSLError to the rescue clause on lib/agent.rb:684. Right now I don't have time to fork and make a pull request so I'm monkeypatching but I thought you should know.

[postmodern]

In order to catch OpenSSL::SSL::SSLError I have to require 'openssl' by default, or have an Array of exceptions we want to catch. I think requiring 'openssl' by default is reasonable in this age of https:// everywhere.

[postmodern]

Implemented in 17756db.