multiple request/issues

Question

multiple request/issues

Linear-bot opened this issue 4 years ago · 20 comments

hi, hope ur enjoying ur holiday.btw I have no experience in code
.ebuyer the UK seems to not be supported anymore "Error: Store currently not supported." test link https://www.ebuyer.com/670425-logitech-g402-hyperion-fury-gaming-mouse-910-004068
.https://www.overclockers.co.uk/ had a significant influx of GPU stock in the past days wondering if you could add that to the supported stores
.can you restart the code automatically when there is an error as I kept getting it every 1hr while on 6 seconds refresh on curry's UK I know that might be a low interval
.proxies are not working have I done it wrong?

and within that notepad I have a list formatted like this these are an example, not my actual ones just in case I did it wrong so u can help correct it)
215.325.323.222:1111
111.222.333.111:2222
the error

Answer 1 · 2021-04-12T13:42:46.000Z

Hello.

Double check the link you're adding because I can add it without a problem.
Post your requests to the discussions so I can keep track of them or else it will get lost.
Follow this issue to see a temporary way to restart the script.
Your format for proxies seems fine but can you elaborate more on what you mean by, "proxies are not working"?
The Amazon error was fixed. You have an older version. When did you download? Try downloading the new version.
I can't say anything about the Currys error unless I see the Currys.log file.

Hopefully I covered everything.

Answer 2 · 2021-04-12T14:20:03.000Z

hi thanks for the quick response, sorry if I was vague
. whenever I turn on the proxies option it gives me errors (shown in the second image in my previous comment) no matter how high my refresh rate is or what products on whatever supported site it was tracking
.and here is the Currys log hope it helps
Currys.log
.i have added the request in discussions
.I checked it also with other links but still ebuyer not working on my side
.regarding the amazon error I would like to update it anyway, do u want me reinstalling the whole code? if yes where are the links for the site retained/saved as I have a lot of links being tracked and I don't want them to be lost

Answer 3 · 2021-04-12T22:17:12.000Z

Your Ebuyer and Amazon issue might be occurring for the same reason: you need to update to the latest version. Just download the like you originally did and you can copy over your current config.json to the new version folder to keep all your URLs.

I'm not currently sure why the Currys issue is happening, why don't we focus on that once we sort out the other issues? It might be related to the proxies you have or they might have updated the site on their end to not allow proxies.

Answer 4 · 2021-04-13T18:30:20.000Z

sorry for my late response
I installed the latest version (3.5) after replacing the old config.json with my previous one even though everything was there from before after running the command npm run settings, I noticed that the issues with amazon, Ebuyer, and proxies were still present. So I then reinstalled the code ver3.5 without changing anything e.g. not replacing the new config.json with the old one the problems with ebuyer links have disappeared, however, the amazon and proxies issues still are present why would this be?

Answer 5 · 2021-04-13T22:21:17.000Z

Send me your amazon.js file located in the stores folder.
How many proxies do you have? Where did you get them from? Are they free or paid?

Answer 6 · 2021-04-13T22:56:20.000Z

i use webshare (free) just updated the proxies to 10 fresh ones which status seems to active and not blocked
file format not supported so here:

import { fileURLToPath } from "url";
import { ALARM, AMAZON_MERCHANT_ID, PROXIES, PROXY_LIST, OPEN_URL, USER_AGENTS } from '../main.js'
import threeBeeps from "../utils/notification/beep.js"
import sendAlerts from "../utils/notification/alerts.js"
import writeErrorToFile from "../utils/writeToFile.js"
import open from "open"
import axios from "axios";
import moment from "moment"
import DomParser from "dom-parser";     // https://www.npmjs.com/package/dom-parser
import HttpsProxyAgent from 'https-proxy-agent'


if (process.argv[1] === fileURLToPath(import.meta.url)) {
    let interval = {
        unit: 'seconds',    // seconds, m: minutes, h: hours
        value: 25           // Amazon detects bots if too low, do > 10 seconds
    }
    let url = 'https://www.amazon.com/Coredy-Super-Strong-Automatic-Self-Charging-Medium-Pile/dp/B07NPNN57S'
    amazon(url, interval, interval.value, true, false, () => null);
}


const store = 'Amazon'
let badProxies = new Set()
export default async function amazon(url, interval, originalIntervalValue, firstRun, urlOpened, resolve) {
    let res = null, html = null, proxy = null

    try {
        let options = null

        // Setup proxies
        if(PROXIES && PROXY_LIST.length > 0) {
            if (badProxies.size == PROXY_LIST.length) {   // If all proxies are used, start over
                console.info(moment().format('LTS') + ': Tried all proxies in proxies.txt. Will try them again. Consider getting more proxies.')
                badProxies = new Set()
            }
            do {
                proxy = 'http://' + PROXY_LIST[Math.floor(Math.random() * PROXY_LIST.length)];
            } while(badProxies.has(proxy))

            let agent = new HttpsProxyAgent(proxy);
            options = { 
                httpsAgent: agent, 
                headers: {
                    'User-Agent': USER_AGENTS[Math.floor(Math.random() * USER_AGENTS.length)]
                }
            }
        }
        else options = { headers: { 'User-Agent': USER_AGENTS[Math.floor(Math.random() * USER_AGENTS.length)] } }
        

        // Get Page
        let newUrl;
        if (AMAZON_MERCHANT_ID !== "None") {
            newUrl = url + '?m=' + AMAZON_MERCHANT_ID  // Add Amazon's seller ID
        } else newUrl = url;

        res = await axios.get(newUrl, options)
            .catch(async function (error) {
                if (error.response.status == 503) {
                    console.error(moment().format('LTS') + ': ' + store + ' 503 (service unavailable) Error for ' + url + '. Nothing to worry about unless you get this all the time!')
                    if(PROXIES) { 
                        console.error('Proxy', proxy, 'might be banned from ' + store + '. Adding it to the bad list.')
                        badProxies.add(proxy)
                    }
                }
                else writeErrorToFile(store, error);
            });

        
        // Extract Information
        if (res && res.status == 200) {
            html = res.data

            // If bot Detected
            if (html.includes("we just need to make sure you're not a robot")) {
                let message = moment().format('LTS') + ': ' + store + ' bot detected. '
                if(PROXIES) {
                    message += 'For proxy: ' + proxy + '. Consider lowering interval.'
                    badProxies.add(proxy)
                }
                else message += 'Consider using proxies or lowering interval.'
                console.error(message)
                resolve({interval: Math.floor(interval.value + Math.random() * originalIntervalValue), urlOpened: urlOpened})
                return
            }

            let parser = new DomParser();
            let doc = parser.parseFromString(html, 'text/html');
            let title = doc.getElementById('productTitle').innerHTML.trim().slice(0, 150)
            let inventory = doc.getElementById('add-to-cart-button')
            let image = doc.getElementById('landingImage').getAttribute('data-old-hires')
            
            if (inventory != null) inventory = inventory.getAttribute('value')
            if (inventory == null && firstRun) {
                console.info(moment().format('LTS') + ': "' + title + '" not in stock at ' + store + '.' + ' Will keep retrying in background every', interval.value, interval.unit)
            }
            else if (inventory != null) {
                if (ALARM) threeBeeps();
                if (!urlOpened) { 
                    if (OPEN_URL) open(url) 
                    sendAlerts(url, title, image, store)
                    urlOpened = true; 
                }
                console.info(moment().format('LTS') + ': ***** In Stock at ' + store + ' *****: ', title);
                console.info(url);
            }
            resolve({interval: interval.value, urlOpened: urlOpened});
        }
        else resolve({interval: Math.floor(interval.value + Math.random() * originalIntervalValue), urlOpened: urlOpened})

    } catch (e) {
        writeErrorToFile(store, e, html, res ? res.status : undefined)
    }
};

Answer 7 · 2021-04-13T23:42:03.000Z

Awesome. Seems like we got only the Amazon issue left. Does the error still happen? If so, can you share a new screenshot?

Answer 8 · 2021-04-14T00:44:05.000Z

it seems I have missed out a major part ("i use webshare (free) just updated the proxies to 10 fresh ones which status seems to active and not blocked") this was supposed to inform you on the proxies current status on webshare the proxies error is still occurring

i will update you with a scrn shot of amazon soon

Answer 9 · 2021-04-14T02:11:27.000Z

Could you repost the error(s) please?

Answer 10 · 2021-04-15T00:08:37.000Z

btw I looked into the way you could temporary way to restart the script (first.bat)(second.bat) both worked flawlessly however I'm looking for a more permanent solution into restarting the command lines as I have a lot of links across various sites being tracked (approx 170) causing crash every approx 3-7hr.

Answer 11 · 2021-04-15T00:25:22.000Z

Are the reason(s) for the crash the error your initially posted?

Answer 12 · 2021-04-15T00:25:43.000Z

this error runs non stop when I turn on proxies
"Error occurred (for the site being tracked). Written to logs/(for the site being tracked).log
This is usually not a problem but if this error appears frequently, please report the error (and the log) to GitHub."

Answer 13 · 2021-04-15T00:26:50.000Z

"Are the reason(s) for the crash the error your initially posted?" meaning?
I'm not sure what you exactly meant by that but I can give you more detail on what environment this code is running on I have a refresh interval of 10secs and delay between urls are 10 secs too, to just hopefully keep it stable (wifi speed is good/stable) links are 170 approx as I mentioned earlier, still cant wrap my head around why it keeps failing

also getting new errors in ebuyer here is a log
TypeError: Cannot read property 'textContent' of undefined
at Timeout.ebuyer [as _onTimeout] (file:///C:/Users/Downloads/StockAlertBot-3.5/StockAlertBot-3.5/stores/ebuyer.js:73:61)
at runMicrotasks ()
at processTicksAndRejections (internal/process/task_queues.js:93:5)

Answer 14 · 2021-04-15T01:54:02.000Z

Could you post all the log files you have?
Edit: The last error should be fixed in acf53f9 so try the new update!

Answer 15 · 2021-04-15T15:38:57.000Z

Awesome. Seems like we got only the Amazon issue left. Does the error still happen? If so, can you share a new screenshot?

and interval of refresh number continue to rise beyond the shown example

Answer 16 · 2021-04-15T15:47:52.000Z

amazon:
TypeError: Cannot read property 'status' of undefined
at file:///C:/Users/Downloads/StockAlertBot-3.5/StockAlertBot-3.5/stores/amazon.js:60:36
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async amazonItem.amazon [as storeFunc] (file:///C:/Users/Downloads/StockAlertBot-3.5/StockAlertBot-3.5/stores/amazon.js:58:15)

ebuyer:
Error: getaddrinfo ENOTFOUND www.ebuyer.com
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26) {
errno: -3008,
code: 'ENOTFOUND',
syscall: 'getaddrinfo',
hostname: 'www.ebuyer.com',
config: {
url: 'https://www.ebuyer.com/1121308-gigabyte-geforce-rtx-3080-10gb-gddr6x-gaming-oc-ampere-graphics-card-gv-n3080gaming-oc-10gd',
method: 'get',
headers: {
Accept: 'application/json, text/plain, /',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36'
},
transformRequest: [ [Function: transformRequest] ],
transformResponse: [ [Function: transformResponse] ],
timeout: 0,
adapter: [Function: httpAdapter],
xsrfCookieName: 'XSRF-TOKEN',
xsrfHeaderName: 'X-XSRF-TOKEN',
maxContentLength: -1,
maxBodyLength: -1,
validateStatus: [Function: validateStatus],
data: undefined
},
request: <ref *1> Writable {
_writableState: WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: false,
decodeStrings: true,
defaultEncoding: 'utf8',
length: 0,
writing: false,
corked: 0,
sync: true,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
afterWriteTickInfo: null,
buffered: [],
bufferedIndex: 0,
allBuffers: true,
allNoop: true,
pendingcb: 0,
prefinished: false,
errorEmitted: false,
emitClose: true,
autoDestroy: true,
errored: null,
closed: false
},
_events: [Object: null prototype] {
response: [Function: handleResponse],
error: [Function: handleRequestError]
},
_eventsCount: 2,
_maxListeners: undefined,
_options: {
maxRedirects: 21,
maxBodyLength: 10485760,
protocol: 'https:',
path: '/1121308-gigabyte-geforce-rtx-3080-10gb-gddr6x-gaming-oc-ampere-graphics-card-gv-n3080gaming-oc-10gd',
method: 'GET',
headers: [Object],
agent: undefined,
agents: [Object],
auth: undefined,
hostname: 'www.ebuyer.com',
port: null,
nativeProtocols: [Object],
pathname: '/1121308-gigabyte-geforce-rtx-3080-10gb-gddr6x-gaming-oc-ampere-graphics-card-gv-n3080gaming-oc-10gd'
},
_ended: true,
_ending: true,
_redirectCount: 0,
_redirects: [],
_requestBodyLength: 0,
_requestBodyBuffers: [],
_onNativeResponse: [Function (anonymous)],
_currentRequest: ClientRequest {
_events: [Object: null prototype],
_eventsCount: 7,
_maxListeners: undefined,
outputData: [],
outputSize: 0,
writable: true,
destroyed: false,
_last: true,
chunkedEncoding: false,
shouldKeepAlive: false,
_defaultKeepAlive: true,
useChunkedEncodingByDefault: false,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
_contentLength: 0,
_hasBody: true,
_trailer: '',
finished: true,
_headerSent: true,
socket: [TLSSocket],
_header: 'GET /1121308-gigabyte-geforce-rtx-3080-10gb-gddr6x-gaming-oc-ampere-graphics-card-gv-n3080gaming-oc-10gd HTTP/1.1\r\n' +
'Accept: application/json, text/plain, /\r\n' +
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36\r\n' +
'Host: www.ebuyer.com\r\n' +
'Connection: close\r\n' +
'\r\n',
_keepAliveTimeout: 0,
_onPendingData: [Function: noopPendingOutput],
agent: [Agent],
socketPath: undefined,
method: 'GET',
maxHeaderSize: undefined,
insecureHTTPParser: undefined,
path: '/1121308-gigabyte-geforce-rtx-3080-10gb-gddr6x-gaming-oc-ampere-graphics-card-gv-n3080gaming-oc-10gd',
_ended: false,
res: null,
aborted: false,
timeoutCb: null,
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
reusedSocket: false,
host: 'www.ebuyer.com',
protocol: 'https:',
_redirectable: [Circular *1],
[Symbol(kCapture)]: false,
[Symbol(kNeedDrain)]: false,
[Symbol(corked)]: 0,
[Symbol(kOutHeaders)]: [Object: null prototype]
},
_currentUrl: 'https://www.ebuyer.com/1121308-gigabyte-geforce-rtx-3080-10gb-gddr6x-gaming-oc-ampere-graphics-card-gv-n3080gaming-oc-10gd',
[Symbol(kCapture)]: false
},
response: undefined,
isAxiosError: true,
toJSON: [Function: toJSON]
}

curry:
Error: getaddrinfo ENOTFOUND www.currys.co.uk
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:67:26) {
errno: -3008,
code: 'ENOTFOUND',
syscall: 'getaddrinfo',
hostname: 'www.currys.co.uk',
config: {
url: 'https://www.currys.co.uk/gbuk/computing-accessories/components-upgrades/graphics-cards/gigabyte-geforce-rtx-3080-10-gb-eagle-oc-graphics-card-10214430-pdt.html',
method: 'get',
headers: {
Accept: 'application/json, text/plain, /',
'User-Agent': 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.10; rv:75.0) Gecko/20100101 Firefox/75.0'
},
transformRequest: [ [Function: transformRequest] ],
transformResponse: [ [Function: transformResponse] ],
timeout: 0,
adapter: [Function: httpAdapter],
xsrfCookieName: 'XSRF-TOKEN',
xsrfHeaderName: 'X-XSRF-TOKEN',
maxContentLength: -1,
maxBodyLength: -1,
validateStatus: [Function: validateStatus],
data: undefined
},
request: <ref *1> Writable {
_writableState: WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: false,
decodeStrings: true,
defaultEncoding: 'utf8',
length: 0,
writing: false,
corked: 0,
sync: true,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
afterWriteTickInfo: null,
buffered: [],
bufferedIndex: 0,
allBuffers: true,
allNoop: true,
pendingcb: 0,
prefinished: false,
errorEmitted: false,
emitClose: true,
autoDestroy: true,
errored: null,
closed: false
},
_events: [Object: null prototype] {
response: [Function: handleResponse],
error: [Function: handleRequestError]
},
_eventsCount: 2,
_maxListeners: undefined,
_options: {
maxRedirects: 21,
maxBodyLength: 10485760,
protocol: 'https:',
path: '/gbuk/computing-accessories/components-upgrades/graphics-cards/gigabyte-geforce-rtx-3080-10-gb-eagle-oc-graphics-card-10214430-pdt.html',
method: 'GET',
headers: [Object],
agent: undefined,
agents: [Object],
auth: undefined,
hostname: 'www.currys.co.uk',
port: null,
nativeProtocols: [Object],
pathname: '/gbuk/computing-accessories/components-upgrades/graphics-cards/gigabyte-geforce-rtx-3080-10-gb-eagle-oc-graphics-card-10214430-pdt.html'
},
_ended: true,
_ending: true,
_redirectCount: 0,
_redirects: [],
_requestBodyLength: 0,
_requestBodyBuffers: [],
_onNativeResponse: [Function (anonymous)],
_currentRequest: ClientRequest {
_events: [Object: null prototype],
_eventsCount: 7,
_maxListeners: undefined,
outputData: [],
outputSize: 0,
writable: true,
destroyed: false,
_last: true,
chunkedEncoding: false,
shouldKeepAlive: false,
_defaultKeepAlive: true,
useChunkedEncodingByDefault: false,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
_contentLength: 0,
_hasBody: true,
_trailer: '',
finished: true,
_headerSent: true,
socket: [TLSSocket],
_header: 'GET /gbuk/computing-accessories/components-upgrades/graphics-cards/gigabyte-geforce-rtx-3080-10-gb-eagle-oc-graphics-card-10214430-pdt.html HTTP/1.1\r\n' +
'Accept: application/json, text/plain, /\r\n' +
'User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.10; rv:75.0) Gecko/20100101 Firefox/75.0\r\n' +
'Host: www.currys.co.uk\r\n' +
'Connection: close\r\n' +
'\r\n',
_keepAliveTimeout: 0,
_onPendingData: [Function: noopPendingOutput],
agent: [Agent],
socketPath: undefined,
method: 'GET',
maxHeaderSize: undefined,
insecureHTTPParser: undefined,
path: '/gbuk/computing-accessories/components-upgrades/graphics-cards/gigabyte-geforce-rtx-3080-10-gb-eagle-oc-graphics-card-10214430-pdt.html',
_ended: false,
res: null,
aborted: false,
timeoutCb: null,
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
reusedSocket: false,
host: 'www.currys.co.uk',
protocol: 'https:',
_redirectable: [Circular *1],
[Symbol(kCapture)]: false,
[Symbol(kNeedDrain)]: false,
[Symbol(corked)]: 0,
[Symbol(kOutHeaders)]: [Object: null prototype]
},
_currentUrl: 'https://www.currys.co.uk/gbuk/computing-accessories/components-upgrades/graphics-cards/gigabyte-geforce-rtx-3080-10-gb-eagle-oc-graphics-card-10214430-pdt.html',
[Symbol(kCapture)]: false
},
response: undefined,
isAxiosError: true,
toJSON: [Function: toJSON]
} isAxiosError: true,
toJSON: [Function: toJSON]
}

Answer 17 · 2021-04-15T15:54:17.000Z

Edit: The last error should be fixed in acf53f9 so try the new update!

worked great thanks ebuyer's error stopped :)
updated the rest of the stores too just in case

Answer 18 · 2021-04-15T16:08:03.000Z

Awesome, glad to hear Ebuyer is good now. The 503 Amazon error, unfortuantely, I cannot do anything about. It is them denying access because they detected the bot on the IP, thus why I programmed the interval to increase. You may try v4.0 but know that it's currently in development and not all stores are added yet. Amazon and Ebuyer are added to it though.

Answer 19 · 2021-04-15T17:01:41.000Z

in v4.0 is there Currys?
and regarding IP if I change it using mac address cloning should this amazon issue be fixed, and if so what interval of refresh and between URLs do you recommend?
AND PLS DONT FORGET TO CONSIDER ADDING Overclockers UK TO THE SUPPORTED STORES LIST ALSO FOUND IN DISCUSSIONS
THANKS FOR BEING A PATIENT PERSON WITH ME I KNOW IT WAS TIME-CONSUMING XD

Answer 20 · 2021-04-15T17:41:56.000Z

Yes. You can see here which stores are supported. I don't think your MAC address has to do with anything. The interval and delay depends on how many proxies you're using. The more proxies you have, the lower you can make the interval. 15 sec interval, 3 sec delay is what I use with 0-10 proxies. With 10-50, you can do 5 sec interval, 0-1 sec delay. Any more proxies, go crazy.

Yes, Your request has been added to the v4.0 to-do list. However, working on porting over previous stores is a higher priority than adding new ones. If you'd like to support me, please share the project and consider donating. 😄