Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Please use gzip/gunzip when fetching webpages #22

Open
absolutelynothinghere opened this issue Dec 26, 2022 · 0 comments
Open

Please use gzip/gunzip when fetching webpages #22

absolutelynothinghere opened this issue Dec 26, 2022 · 0 comments

Comments

@absolutelynothinghere
Copy link

More often than not I try recursively downloading a webpage using wget, only to have it download a single index.html.gz then stop. Obviously wget can't read gzipped files so it fails to find any links for recursive downloading... I ended up using this wget fork that was last updated 10 years ago and it works fine, however I find it odd that such a basic feature never made it into mainline wget.

Please add a feature for automatically detecting and uncompressing gzipped webpages before crawling them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant