use wp_no_robots on non-public blogs, on 404, and on search pages.

This resolves #1615, where I had originally written to use an option for this. Instead, based on research, it is recommended to avoid indexing these pages. See https://yoast.com/blocking-your-sites-search-results/ .

changelog.md

@@ -17,6 +17,7 @@ This release contains bug fixes for Largo 0.6.
 - Removed all Google+ profile fields in the admin interface and buttons on the front-end due to [Google+ being shut down](https://support.google.com/plus/answer/9217723#whatshappening) on April 2, 2019. [Pull request #1667](https:/INN/largo/pull/1667) for [issue #1546](https:/INN/largo/issues/1546).
 - Makes the function `largo_get_term_meta_post()` pluggable. [Pull request #1666](https:/INN/largo/pull/1666) by GitHub user [@megabulk](https:/megabulk).
 - Widget area name is now output as an HTML comment on many sidebars, to ease debugging widget presentations. [Pull request #1632](https:/INN/largo/pull/1632) by [@seanchayes](https:/seanchayes) for [issue #1492](https:/INN/largo/issues/1482).
+- Prevents search engine indexing on 404 and search results pages. This change is to keep up with SEO best practices, to preserve your crawl budget with Google, and to prevent a SEO hijacking attack whereby spammers search for their URL on your site, then get the resulting search query result page listed in search engines using your site's reputation. [Pull request #1674](https:/INN/largo/pull/1673) for [issue #1615](https:/INN/largo/issues/1615).
 
 ### Fixes
 

inc/header-footer.php

@@ -147,9 +147,16 @@ function largo_seo() {
  // if the blog is set to private wordpress already adds noindex,nofollow
  if ( get_option( 'blog_public') ) {
  if ( is_date() || ( is_archive() && of_get_option( 'noindex_archives' ) ) ) {
- echo '<meta name="robots" content="noindex,follow" />';
+ wp_no_robots();
  }
+ } else {
+ wp_no_robots();
+ }
+
+ if ( is_404() || is_search() ) {
+ wp_no_robots();
  }
+
  // single posts get a bunch of other google news specific meta tags
  if ( is_single() ) {
  if ( have_posts() ) : the_post();