-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disable search engine crawling of non-canonical forks #90
Comments
Having a |
One thing I didn't really think about when writing this is that the main website's (Similarly for forks, the This means we actually probably just go the route of adding |
It can be convenient in forks to enable deployment to GitHub Pages for the purposes of testing. However this inadvertently creates duplicate copies of the documentation accessible on the wider public internet, which means search engines have the potential to find them.
This runs the risk of polluting search results with content which is likely outdated. I believe it also runs the risk of harming the SEO of the official documentation website. (I'm no SEO expert but my understanding is Google in particular harshly penalizes websites which duplicate other websites.)
We should
generate aadd the appropriate meta tags to non-canonical copies of the docs website.robots.txt
and/orAs a semi-related aside (since you specify it in the
robots.txt
), we should also enable thesitemap.xml
generation. Looks like it just needs to be turned on.The text was updated successfully, but these errors were encountered: