proxyConfiguration object

[sabatale]

In the context of using Apify actors, is this the way to implement the proxyConfiguration object?

We sometimes get The proxy server rejected the request with status code 590 but got-scraping seems to accept the option nonetheless.

const proxyConfiguration = await Actor.createProxyConfiguration(proxy);

const response = await gotScraping({
        url: host,
        proxyUrl: await proxyConfiguration.newUrl()
    });

[B4nan]

Yes, that is the correct way. That error means the proxy server is probably dead, which can indeed happen, there is no proxy rotation on got-scraping level, it will use what it gets.

[B4nan]

Nope, we don't plan it right now, the use case for us is mainly the crawlee integration, which rotates proxies on a higher level.

I am sure you can implement this yourself easily, will be happy to help with that if you'd struggle.

[sabatale]

If you have a starter template, I'm sure others will find it useful as well!

[B4nan]

I don't, but this is really trivial, isn't it? You just try to make the request, and catch error, if you want to retry it, call it again with a new proxy url (which you get from the newUrl() proxy configuration method).

[sabatale]

Oh we thought you meant there was a crawlee-specific retry logic that could be used as-is. Thanks again!

[B4nan]

In crawlee the request is what gets retried by the crawler object, you get just a warning for such a failed request, it gets retried 3 times, and for known proxy issues we retry 10 times. You can use BasicCrawler here if you don't want to parse the data with cheerio, and run requests via sendRequest helper provided in the crawling context.

https://crawlee.dev/api/basic-crawler/class/BasicCrawler

The example we have there is providing all the options explicitly, but you don't need that, simple sendRequest() without params will use the request from the context automatically.