Eliminate duplicates that are not in query strings

Question

Eliminate duplicates that are not in query strings

Opened this issue 4 years ago · 8 comments

Deleted user commented 4 years ago

Hi,
thanks a lot for this tool, it is very useful!

I was wondering if it would be possible to implement also a dedupe functionality for this kind of URL:

/product/1/buy/1
/product/1/buy/2
/product/1/
/product/2/

This should results just in:

/product/1/buy/1
/product/1/

It seems to me that at this time this is not taken in consideration.

I would really like to contribute on this by myself but my C++ knowledge are really rusty :)

Thanks again!

Answer 1 · 2020-06-06T16:23:49.000Z

This functionality would probably need to be added as a switch, since, if we consider somesite.com/product/1 and somesite.com/product/2 to be the same, we would also need to consider
somesite.com/something/login and somesite.com/something/profile to be the same.
Anything else would need some heuristic for determining if the difference in the path is relevant or not, which would be complicated, error-prone and (probably) slow.

I'd propse adding a switch to the program to tell it which part of the path is irrelevant, like so:
-i 2 would make ex.com/product/1/whatever the same as ex.com/product/x/whatever
-i 2 -i 3 would make ex.com/product/1/2 the same as ex.com/product/x/y

Answer 2 · 2020-06-06T16:53:58.000Z

Yeah, I can understand.

I think it could be feasible if it is checked only in case there is a digit.

1 is likely similar to 2 so one of them could be discarded.

Two different strings have probably different meaning so they can be kept.

Answer 3 · 2020-06-06T18:06:41.000Z

Thanks for the suggestion @simonebovi! I wanted to release this in the initial release but didn’t have much time to get it done. I have started working on this already actually - I’m focusing on integer differences only, as anything else would be very hard without context and most likely cause more issues than it would solve unless done extremely well

Answer 4 · 2020-06-07T08:47:05.000Z

Just added a PR to account for this. It checks for common assets to ignore (images, fonts), as well as integers in URLs. Tested it out and seems to be working well. @larskraemer, if you want to take a look: #9

Answer 5 · 2020-06-07T17:57:36.000Z

PR Merged, feel free to give it a test @simonebovi!

Answer 6 · 2020-06-07T18:05:49.000Z

Thanks a lot @ameenmaali,
I really love open source projects!

It seems to work way better now.

I just did some tests and I have found that this can be improved again to me.

For example, these URLs are still maintained:

product/1/buy/1
product/1/buy/2
product/1/buy/3

These are basically the same so I think only one of them should be kept.

However a URL like that should be kept as well:

product/1/sell/1

Not sure if this is feasible though.

What do you think?

Answer 7 · 2020-06-07T18:24:31.000Z

Thank you, I will check this out shortly. This should already be accounted for - may be a bug

Answer 8 · 2020-06-07T19:18:53.000Z

Hey @simonebovi, just tried to test this out, and it seems to be working as expected. Can you try again to verify? Make sure to enable the -s flag when running. One thing I do know is an issue is a lack of checking for ports in the URL so that may cause 2 of the same (with different or missing port numbers) to show up. I will add that in a future update (#10)