threatexpress/domainhunter

MXToolbox reputation checking is broken

andrewchiles opened this issue · 1 comments

The service has removed the original endpoints used to query SPAM and Google Safe Browsing lists. There is a new API that returns JSON objects and is heavily reliant on JS to format the requests correctly. Any malformed request results in an IP block that requires CAPTCHA completion to remove.

All HTTP GETs

  1. https://mxtoolbox.com/domain/apples.com/ -> Initial request to the service
  2. https://mxtoolbox.com/api/v1/user -> Returns JSON that is subsequently used as cookie parameter and also an HTTP header in the GET. Need to parse and manually update requests session cookie jar and create custom HTTP headers
  3. https://mxtoolbox.com/api/v1/lookup/blacklist/apples.com -> Returns JSON with blacklist lookup results. All we really need to check is presence of records in "Failed" or "Warnings"
{
  "UID": null,
  "ArgumentType": "hostname",
  "Command": "blacklist",
  "IsTransitioned": false,
  "CommandArgument": "nwk-aaemail-lapp01.apple.com",
  "TimeRecorded": "2020-02-05T15:54:56.7293086-06:00",
  "ReportingNameServer": null,
  "TimeToComplete": "328",
  "RelatedIP": "17.151.62.66",
  "ResourceRecordType": 0,
  "IsEmptySubDomain": false,
  "IsEndpoint": true,
  "HasSubscriptions": false,
  "AlertgroupSubscriptionId": null,
  "Failed": [],
  "Warnings": [],
  "Passed": [
    {
      "ID": 333,
      "Name": "BSB Domain",
      "Url": "https://mxtoolbox.com/Problem/blacklist/BSB-Domain?page=prob_blacklist&showlogin=1&hidetoc=1&action=blacklist:nwk-aaemail-lapp01.apple.com",
      "PublicDescription": null,
      "BlacklistResponseTime": "0",
      "IsExcludedByUser": false
    },
    SNIP
  ],
  "Errors": [],
  "IsError": false,
  "Information": [
    {
      "DNS Resolution": "nwk-aaemail-lapp01.apple.com was resolved to 17.151.62.66."
    }
  ],
  "MultiInformation": [],
  "IsBruteForce": false,
  "Transcript": [
    {
      "Transcript": "DNS - Load Balancers\r\nLookupServer 328ms\r\n"
    }
  ],
  "MxRep": 100,
  "EmailServiceProvider": null,
  "DnsServiceProvider": null,
  "DnsServiceProviderIdentifier": null,
  "RelatedLookups": [
    {
      "Name": "dns lookup",
      "URL": "https://mxtoolbox.com/api/v1/lookup/a/nwk-aaemail-lapp01.apple.com",
      "Command": "a",
      "CommandArgument": "nwk-aaemail-lapp01.apple.com"
    },
    {
      "Name": "smtp diag",
      "URL": "https://mxtoolbox.com/api/v1/lookup/smtp/nwk-aaemail-lapp01.apple.com",
      "Command": "smtp",
      "CommandArgument": "nwk-aaemail-lapp01.apple.com"
    },
    {
      "Name": "http test",
      "URL": "https://mxtoolbox.com/api/v1/lookup/http/nwk-aaemail-lapp01.apple.com",
      "Command": "http",
      "CommandArgument": "nwk-aaemail-lapp01.apple.com"
    }
  ]
}

All we really need to check is presence of records in "Failed" or "Warnings"

it looks like checkMXToolbox() is checking both Google safe browsing and Phish tank. What would be the expected return value for checkMXToolbox() if there were records in the failed or warning?