HTML::RobotsMETA parses HTML fails and look for robots directive inside META tags perldoc HTML::RobotsMETA for more info