How do you prevent templates from being indexed/listed?

[lesliekirk]

I'm not sure if this has come up before but I've noticed that Page Templates such as CTGY are being picked up by the search engines as https://www.domain.com/CTGY.html. If you put "noindex,nofollow" in the template it will prevent every category from being listed. I suspect this started becoming an issue when we were able to start viewing the Page Templates. This particular site is not using the Marketing Feeds Sitemap, it uses an external sitemap app.The link is not in it nor is it in the SMAP page (thank goodness). It would seem Google is finding it all on it's own. So how do we stop it?

[William Davis]

I'm not sure if this has come up before but I've noticed that Page Templates such as CTGY are being picked up by the search engines as https://www.domain.com/CTGY.html. If you put "noindex,nofollow" in the template it will prevent every category from being listed. I suspect this started becoming an issue when we were able to start viewing the Page Templates. This particular site is not using the Marketing Feeds Sitemap, it uses an external sitemap app.The link is not in it nor is it in the SMAP page (thank goodness). It would seem Google is finding it all on it's own. So how do we stop it?

How about robots.txt disallow statement for the category in question file?

[lesliekirk]

How about robots.txt disallow statement for the category in question file?

It's not a category - it's the Page Template (code CTGY name Category Display). This issue seems to have started when MM10 started letting you view a Page Template just by clicking the View icon. Maybe it's been a problem all along but this time I have encountered the Template Page being listed in Google.

[Bruce - PhosphorMedia]

I think william means something like this:

disallow: https://www.domain.com/CTGY.html

in the robots.txt file.

i don't think its an issue but just to be safe, you can use Web Master Tools to verify that a category USING that page template can still be followed and indexed.

[William Davis]

I think william means something like this:

disallow: https://www.domain.com/CTGY.html

in the robots.txt file.

i don't think its an issue but just to be safe, you can use Web Master Tools to verify that a category USING that page template can still be followed and indexed.

Correct!

[lesliekirk]

I think william means something like this:

disallow: https://www.domain.com/CTGY.html

in the robots.txt file.

i don't think its an issue but just to be safe, you can use Web Master Tools to verify that a category USING that page template can still be followed and indexed.

Well that's a little scary - so I could potentially block all of the category pages by doing this?

[William Davis]

No, you only want to block one category:

Example:

Disallow: /cat-abc.html

https://www.robotstxt.org/robotstxt.html

[Kent Multer]

Maybe I'm missing something, but this seems weird to me. I don't see how a URL like https://www.domain.com/CTGY.html can even display a page. Since there is no category code, it ought to either generate a not-found error, or redirect to the storefront.

Maybe there is a form on a page that's submitting this URL and passing the Category_Code in a hidden field? If so, you can just change the form to use the store's main URL, and put the "CTGY" in a hidden field with the name "Screen."

Or maybe there's a bug in their feed, and they are submitting invalid URLs to the search engines?

I guess what I'm saying is, I don't understand why a store would use such a URL. There may be a bug somewhere, and fiddling with things like robots.txt is not a real solution. Sorry if I'm off base; no offense intended to anyone.

[lesliekirk]

Maybe I'm missing something, but this seems weird to me. I don't see how a URL like https://www.domain.com/CTGY.html can even display a page. Since there is no category code, it ought to either generate a not-found error, or redirect to the storefront.

Maybe there is a form on a page that's submitting this URL and passing the Category_Code in a hidden field? If so, you can just change the form to use the store's main URL, and put the "CTGY" in a hidden field with the name "Screen."

Or maybe there's a bug in their feed, and they are submitting invalid URLs to the search engines?

I guess what I'm saying is, I don't understand why a store would use such a URL. There may be a bug somewhere, and fiddling with things like robots.txt is not a real solution. Sorry if I'm off base; no offense intended to anyone.

No offense - I'm just trying to figure out how Google picked up the link. It is not in the sitemap for the store. If you go in an MM10 admin, go to any of the of the Page Templates, then click the "View" icon. You'll be able to see the page.

[alphabet]

Just a few ideas:

Have you looked at the URI Management settings and see if there is a rule that is creating a default page template URI's?

Or have you checked the URI tab of the CTGY page to see that there are no URI entries?

Or try a Google search with the link: operator

[Bruce - PhosphorMedia]

the /CTGY.html page is a legit page on most miva sites due to either htaccess rules or URI settings for example, default shadows has

/category-display.html

as a loadable page (kent: a call to the category page without a category code should load all to level cats.)

as for 'messing' with the robots file--that's what its there for. for example, we'll often block all urls starting with /mm5/ to force bots to index the friendly URIs. Obvoiously, that requires what we "allow" *.jpg, *.css, *.js files to be indexed.

there are some great online robot file builders and testors out there (sorry don't have one currently bookmarked) and no harm would come from a 20 minute test.

[Bruce - PhosphorMedia]

Just a few ideas:

Have you looked at the URI Management settings and see if there is a rule that is creating a default page template URI's?

Or have you checked the URI tab of the CTGY page to see that there are no URI entries?

Or try a Google search with the link: operator

if URI Management is turned on, by default it creates a URI when a page is created.

[lesliekirk]

Just a few ideas:

Have you looked at the URI Management settings and see if there is a rule that is creating a default page template URI's?

Or have you checked the URI tab of the CTGY page to see that there are no URI entries?

Or try a Google search with the link: operator

There is an entry /CTGY.html - should it be removed?

[Bruce - PhosphorMedia]

you can't leave the URI blank once its been populated.

[Kent Multer]

There is an entry /CTGY.html - should it be removed?

I would think so, yes. The category page should not have a URI, because it makes no sense to view it except when a specific category is requested. So the categories have URIs, not the page template.

When you enter that generic URL into your browser, what do you see?

When I go to the CTGY page in admin and click the View button, I get an empty category page. It has a header and footer but no content: no products or subcategories.