Bad redirect location result

Question

Bad redirect location result

Closed this issue 6 years ago · 9 comments

Hi,
I'm having a problem redirecting, getting a wrong redirect location
URL:

https://ad.doubleclick.net/ddm/trackclk/N296811.1960500FACEBOOKAD/B20703495.216783479;dc_trk_aid=415612097;dc_trk_cid=97487385;dc_lat=;dc_rdid=;tag_for_child_directed_treatment=;v?https://www.nytimes.com/2018/03/26/world/europe/trump-russia-expel-european-union.html?foo=moo&kwp_0=714258

response(bad):
https://www.nytimes.com/?dclid=COCyybTAjNoCFZI64AodNYMMeA

the real redirect URL (works with curl and chrome):

https://www.nytimes.com/2018/03/26/world/europe/trump-russia-expel-european-union.html?foo=moo&kwp_0=714258&dclid=CNmy4eXJjNoCFZQX4AodeuwJwQ

my config:

user_agent = 'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
    Faraday.new do |faraday|
      faraday.use FaradayMiddleware::FollowRedirects
      faraday.use :cookie_jar
      faraday.headers[:user_agent] = user_agent
      faraday.adapter :patron do |session|
        session.max_redirects = 3
        session.timeout = 60
        session.connect_timeout = 15
      end
    end

invoking:
response = client.get(URI.parse(url))
Thanks

Answer 1 · 2018-03-27T17:32:46.000Z

Hi, thanks for reporting and providing detailed steps to reproduce.
I’ll review as soon as possible and get back to you

Answer 2 · 2018-04-04T15:16:24.000Z

Hi @kerenbe4, apologies for the delay in my response, had a long Easter break.
I've tested the following:

Faraday: 0.14.0
Faraday Middleware: 0.12.2

require 'faraday'
require 'faraday_middleware'
require 'faraday-cookie_jar'

url = "https://ad.doubleclick.net/ddm/trackclk/N296811.1960500FACEBOOKAD/B20703495.216783479;dc_trk_aid=415612097;dc_trk_cid=97487385;dc_lat=;dc_rdid=;tag_for_child_directed_treatment=;v?https://www.nytimes.com/2018/03/26/world/europe/trump-russia-expel-european-union.html?foo=moo&kwp_0=714258"

user_agent = 'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'

client = Faraday.new do |faraday|
  faraday.use FaradayMiddleware::FollowRedirects
  faraday.use :cookie_jar
  faraday.headers[:user_agent] = user_agent
  faraday.adapter :patron do |session|
    session.max_redirects = 3
    session.timeout = 60
    session.connect_timeout = 15
  end
end

response = client.get(URI.parse(url))

And it works as expected. I receive a 200 OK response with the HTML code from nytimes.com.
Please check you're using latest version of both Gems as I showed above.
Also, I don't think you need to use FaradayMiddleware::FollowRedirects, as Patron already follows redirects. The following also works for me:

require 'faraday'
require 'faraday-cookie_jar'

url = "https://ad.doubleclick.net/ddm/trackclk/N296811.1960500FACEBOOKAD/B20703495.216783479;dc_trk_aid=415612097;dc_trk_cid=97487385;dc_lat=;dc_rdid=;tag_for_child_directed_treatment=;v?https://www.nytimes.com/2018/03/26/world/europe/trump-russia-expel-european-union.html?foo=moo&kwp_0=714258"

user_agent = 'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'

client = Faraday.new do |faraday|
  faraday.use :cookie_jar
  faraday.headers[:user_agent] = user_agent
  faraday.adapter :patron do |session|
    session.max_redirects = 3
    session.timeout = 60
    session.connect_timeout = 15
  end
end

response = client.get(URI.parse(url))

Answer 3 · 2018-04-07T18:05:40.000Z

Hi @iMacTia ,
Thanks a lot for looking into it

checking our gemfile.lock we have the following versions

    faraday (0.9.1)
      multipart-post (>= 1.2, < 3)
    faraday-cookie_jar (0.0.6)
      faraday (>= 0.7.4)
      http-cookie (~> 1.0.0)
    faraday_middleware (0.9.1)
      faraday (>= 0.7.4, < 0.10)

maybe updating to the latest will resolve the issue.

You mentioned that you receive a 200 OK response with the HTML code from nytimes.com. To which url were you redirected?
I also got 200 ok and html code from nytimes, but the redirect needs to transfer to a specific article and not to the top level domain.
Thanks again

Answer 4 · 2018-04-08T17:42:19.000Z

I tried again after upgrading to Faraday: 0.14.0, Faraday Middleware: 0.12.2
and still didn't get the correct location

response.to_hash[:response_headers]['location']
=> "https://www.nytimes.com/?dclid=CMOsxcuaq9oCFdY74AodbbMAbQ"

Answer 5 · 2018-04-11T13:45:47.000Z

@kerenbe4 It's quite hard to say as I just see a bunch of HTML code 😅
However response.to_hash[:response_headers]['location'] and also response.headers["Location"] both return nil to me, so I suppose I'm on the correct one?
I'll try to analyse the HTML to understand which page it is

Answer 6 · 2018-04-11T13:52:55.000Z

So, it looks like I got the homepage back, not sure that's correct.
However, if I open the following on my browser, I also get redirected to the homepage:

https://ad.doubleclick.net/ddm/trackclk/N296811.1960500FACEBOOKAD/B20703495.216783479;dc_trk_aid=415612097;dc_trk_cid=97487385;dc_lat=;dc_rdid=;tag_for_child_directed_treatment=;v?https%3A%2F%2Fwww.nytimes.com%2F2018%2F03%2F26%2Fworld%2Feurope%2Ftrump-russia-expel-european-union.html%3Ffoo=moo&kwp_0=714258

Answer 7 · 2018-04-11T13:57:46.000Z

I think I see it now:

response.env.url
#=> https://ad.doubleclick.net/ddm/trackclk/N296811.1960500FACEBOOKAD/B20703495.216783479;dc_trk_aid=415612097;dc_trk_cid=97487385;dc_lat=;dc_rdid=;tag_for_child_directed_treatment=;v?https%3A%2F%2Fwww.nytimes.com%2F2018%2F03%2F26%2Fworld%2Feurope%2Ftrump-russia-expel-european-union.html%3Ffoo=moo&kwp_0=714258

Notice the URL in the query param is escaped.
Instead of being

https://www.nytimes.com/2018/03/26/world/europe/trump-russia-expel-european-union.html?foo=moo&kwp_0=714258

it's

https%3A%2F%2Fwww.nytimes.com%2F2018%2F03%2F26%2Fworld%2Feurope%2Ftrump-russia-expel-european-union.html%3Ffoo=moo&kwp_0=714258

Answer 8 · 2018-04-11T15:34:53.000Z

@kerenbe4 I believe the issue has nothing to do with FaradayMiddleware and FollowRedirect, can you please try to define the following module before your connection:

module NullParamsEncoder
  def self.encode(params)
    params[:params]
  end

  def self.decode(params)
    { params: params }
  end
end

and then pass it to your connection like this:

client = Faraday.new(request: { params_encoder: NullParamsEncoder }) do |faraday|
  ...
end

Seems like this is working for me. If it works for you as well, we need to move this into Faraday 😅

Answer 9 · 2018-06-11T10:00:24.000Z

@kerenbe4 I haven't heard back from you in a while.
Were you able to give a quick test to the above or to fix your issue?