diff options
| author | Valentin Popov <info@valentineus.link> | 2018-05-15 14:53:48 +0300 |
|---|---|---|
| committer | Valentin Popov <info@valentineus.link> | 2018-05-15 14:53:48 +0300 |
| commit | 7aff2fd1861af7fb454cea1d502878e6062771c6 (patch) | |
| tree | f907e3d485f00cde5307221c80e60abf89595755 | |
| parent | f0e1ca0b338c61c3537b88aa65fae12b046b3752 (diff) | |
| download | popov.link-7aff2fd1861af7fb454cea1d502878e6062771c6.tar.xz popov.link-7aff2fd1861af7fb454cea1d502878e6062771c6.zip | |
Flexible exception list
Signed-off-by: Valentin Popov <info@valentineus.link>
| -rw-r--r-- | _data/robots_disallow.yml | 1 | ||||
| -rw-r--r-- | _pages/robots.txt | 7 |
2 files changed, 6 insertions, 2 deletions
diff --git a/_data/robots_disallow.yml b/_data/robots_disallow.yml new file mode 100644 index 0000000..b5fbe1d --- /dev/null +++ b/_data/robots_disallow.yml @@ -0,0 +1 @@ +- path: "/404.html" diff --git a/_pages/robots.txt b/_pages/robots.txt index c4c4d2c..7ad8e77 100644 --- a/_pages/robots.txt +++ b/_pages/robots.txt @@ -5,6 +5,9 @@ layout: null User-agent: * Allow: {{ site.baseurl | relative_url | uri_escape }} -Disallow: {{ "/404.html" | relative_url | uri_escape }} -Sitemap: {{ "sitemap.xml" | absolute_url | uri_escape }}
\ No newline at end of file +{% for link in site.data.robots_disallow %} + Disallow: {{ link.path | relative_url | uri_escape }} +{% endfor %} + +Sitemap: {{ "sitemap.xml" | absolute_url | uri_escape }} |