How would one know if got-scraping has been blocked?

[JaneJeon]

Somewhat of a vague and broad question, but still would be interested in knowing even where to begin looking for this sort of thing (#34 inspired this question).

Suppose we're using got-scraping not even for scraping, but to reliably connect to arbitrary websites (my specific use case is getting the canonical URL).

From my testing, got-scraping does pretty well against real-world websites (even with well-known "cloud IP"). However, I have to assume that at certain point, it will get blocked by some anti-bot software (e.g. cloudflare), and I'd like to know whether I got blocked vs. whether the resource simply failed to load.

How would one go about detecting that you've been blocked using got-scraping (or just got in general)?

Thanks and a happy new year

[szmarczak]

Either ECONNRESET errors or 401, 403, 404, 406, 429 status codes. If it returns 200, you should check for common captcha / JavaScript challenges. You can do so in an afterResponse hook. If using streams, then you have to check this manually. It is important that those errors need to be consistent, e.g. twice in a row. Otherwise probably a coincidence.

[JaneJeon]

CloudFlare, at least, seems to try to "hide" the status by simply returning a 503:

[szmarczak]

One way to detect got-scraping would be to send malformed HTTP responses. Node.js built-in parser throws, but browsers just ignore invalid headers.

[JaneJeon]

Wait, I thought https://github.com/apify/got-scraping/blob/master/src/hooks/insecure-parser.ts took care of malformed HTTP responses?

[JaneJeon]

And wouldn't using an insecure HTTP parser subject the process to this kind of stuff? https://snyk.io/blog/node-js-release-fixes-a-critical-http-security-vulnerability/

[szmarczak]

Wait, I thought master/src/hooks/insecure-parser.ts took care of malformed HTTP responses?

It handles some of the cases, but not all of them.

And wouldn't using an insecure HTTP parser subject the process to this kind of stuff?

We are a client, not a server.

[JaneJeon]

Ah, I see. Thank you