Implementation of Robots.txt in DXA 1.7

RE: Implementation of Robots.txt in DXA 1.7

Merry Zhong — Mon, 13 Aug 2018 01:50:33 GMT

The solution appears not working in DXA 2.0. Is there a plan for DXA 2.0?

RE: Implementation of Robots.txt in DXA 1.7

Rajesh Saminathan — Mon, 17 Jul 2017 07:09:56 GMT

Quite nice one :)

RE: Implementation of Robots.txt in DXA 1.7

Nuno Linhares — Mon, 10 Jul 2017 08:18:42 GMT

I guess one way to do it would be to have logic in your template that checks if the current TargetType "IsPreviewCapable" - this will only work if you're using Web 8+ _AND_ if your staging environment is configured to use SessionPreview... but it's probably a good start. Then use this in your template to determine which Robots.txt to render.

RE: Implementation of Robots.txt in DXA 1.7

Shalivahan Kanur — Fri, 07 Jul 2017 07:53:08 GMT

In a robots.txt file with multiple user-agent directives, each disallow or allow rule only applies to the "user-agent"(s) specified in that particular line break-separated set. If the file contains a rule that applies to more than one user-agent, a crawler will only pay attention to (and follow the directives in) the most specific group of instructions and also we can do blocking a specific web crawler from a specific folder by specifying different "User-agent"s.

but even am also not pretty sure how to handle depending on different Target types like (stage and Live)... but will investigate , try to get info and post you sooner.

RE: Implementation of Robots.txt in DXA 1.7

Nuno Linhares — Fri, 07 Jul 2017 07:20:56 GMT

This seems pretty straight-forward, but what if you need to have a different robots.txt depending on target? For instance, I may want to disallow crawling on my staging site, but allow on my live site?