Blacklist requests that are duplicates of existing resources or bound to fail

[Popolechien]

Following openzim/zimit#113, we should think about implementing a fairly easily editable list (hosted on drive.kiwix.org?) of blacklisted sites that can not be requested on zimit, e.g.

• kiwix.org subdomains (download and library);
• very large corporate websites (e.g. Facebook, Twitter, Reddit, Youtube, etc.)
• websites that have been scraped in the past and failed.

It's probably the matter of a separate ticket, but requests for websites we already have a scraper for (wikipedia, stackoverflow, etc.) should also be soft blocked and the user offered a direct link to the zim file.

[rgaudin]

Can you move your comment to #25 and close this? This is the scraper's repo.

[Popolechien]

@rgaudin Moved it but I'd keep it open as this ticket is a little bit different.

[rgaudin]

This one's better ; closing the other one but the problem raised there remains: where do we point to for stuff that we know exists?

[Popolechien]

Is your question "in case there are several versions of the same zim" (e.g., Wikipedia mini/nopic/maxi)?

The basic assumption here is that zimit provides a copy of the real thing, so we should send them the maxi zim file.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

See also #33