Customize robots.txt in dockerized Dataverse

[ekraffmiller]

The default robots.txt installed in the Dataverse docker image disallows all access to the site. It would be helpful to be able to configure the image with a customized robots.txt that is similar to the one we use in production.
This issue was encountered in testing the SPA - we need to test that the json-ld added to the header is accessible to web crawlers.
SPA issue: IQSS/dataverse-frontend#350

[ekraffmiller]

The SPA issue that initiated this PR is related to testing beta.dataverse.org, which @pdurbin explained does not use containers https://dataverse.zulipchat.com/#narrow/stream/375812-containers/topic/how.20to.20configure.20robots.2Etxt.3F/near/441064337. So, although this may be a nice to have for container configuration, it's not needed for the SPA, so I'm removing the label.