Dumb SEO Questions

(Entry was posted by Lewis Barrs on this post in the Dumb SEO Questions community on Facebook, 02/03/2015).

Too many pages are being indexed, is it okay?

Hi, I wonder if anyone can lend any advice here, I have this site www.frumistdotcom which is magento designed, way too many pages are being indexed and we can not seem to get to the bottom of why. Should have around 2000 indexed, it has 60, 000?

Any pointers would be very helpful here
This question begins at 04:00:00 into the clip. Did this video clip play correctly? Watch this question on YouTube commencing at 04:00:00
Video would not load
I see YouTube error message
I see static
Video clip did not start at this question

YOUR ANSWERS

Selected answers from the Dumb SEO Questions Facebook & G+ community.

  • Lewis Barrs: Hi, I wonder if anyone can lend any advice here, I have this site www.frumistdotcom which is magento designed, way too many pages are being indexed and we can not seem to get to the bottom of why. Should have around 2000 indexed, it has 60,000?

    Any pointers would be very helpful here
  • Mike Roberts: Do you have lots of posts?
  • Lewis Barrs: Not that many, the system does produce many duplicate pages for each product but the canonicals should rectify this so there are not 1000s of duplicate pages indexed. But this seems to be happening
  • Mike Roberts: Hmm, sorry. Can't help on this one. Sure someone will drop in with more savy then I have. 
  • Rajeshwar Reddy: Hey Lewis, there seems to be lots of products in your website. Are you sure there are duplicate pages. There will be a pattern in url. See which pages under a folder are not needed to be shown in Google SERPs. If you can do that, you can restrict robots from reading those pages using robots.txt.
    Suggestion: Ask your dev to remove index.php in url. It seems to be in all the urls.
  • promoz seo: +Lewis Barrs ; Are you using rel canonical tags properly on pages? It may be duplicate pages (URLs) issue, due to the CMS software.
  • Daniel Lloyd Barrett: I ran it through Xenu, I think you may need to remove the advanced search terms here: http://www.frumist.com/catalogsearch/term/popular/

    Each term is being crawled then each product in the term results are also being crawled.
  • Lewis Barrs: Ok,These should all be implemented but we are still seeing the pages being indexed

    1)  ;the host supports rewrites.
    2)  ;the .htaccess that ships with magento is in the root folder.
    3) System/Configuration/General/Web
     ; set 'Use Web Server Rewrites' to yes

    Dupe content:

     ;System/Configuration/Catalog/Frontend
     ; set 'Use Canonical Link Meta Tag For Categories' to yes
     ; set 'Use Canonical Link Meta Tag For Products' to yes

    Site speed:

     ;System/Configuration/Catalog/Frontend
     ; set 'use flat catalog category' to yes
     ; set 'use flat catalog product' to yes
     ; ;
    General

     ;System/Configuration/Catalog/Search Engine Optimisations
     ; 'use categories path for product URLs' to yes
     ; 'use categories path for product URLs' to yes
     ; ;
     ; ;
     ;the Google sitemap is set up and current.

     ;robots.txt with the sitemap.xml added.

    # Crawlers Setup
    User-agent: *
    Crawl-delay: 10

    # Allowable Index
    Allow: /*?p=
    Allow: /index.php/blog/
    Allow: /catalog/seo_sitemap/category/
    Allow: /catalogsearch/result/


    # Paths (clean URLs)
    Disallow: /index.php/
    Disallow: /catalog/product_compare/
    Disallow: /catalog/category/view/
    Disallow: /catalog/product/view/
    Disallow: /catalogsearch/
    Disallow: /checkout/
    Disallow: /control/
    Disallow: /contacts/
    Disallow: /customer/
    Disallow: /customize/
    Disallow: /newsletter/
    Disallow: /poll/
    Disallow: /review/
    Disallow: /sendfriend/
    Disallow: /tag/
    Disallow: /wishlist/

    # Files
    Disallow: /cron.php
    Disallow: /cron.sh
    Disallow: /error_log
    Disallow: /install.php
    Disallow: /LICENSE.html
    Disallow: /LICENSE.txt
    Disallow: /LICENSE_AFL.txt
    Disallow: /STATUS.txt
  • Lewis Barrs: Daniel lloyd barret can you please expand on this, 
  • Lewis Barrs: +Daniel Lloyd Barrett ;do you mean remove the page with all search terms on entirely?
  • Lewis Barrs: Also would this be auto generating these terms and if so do you know how to stop it
  • Daniel Lloyd Barrett: You can disable this in the admin area.

    I'm not sure what version of magento you are using but try the following:

    To configure Popular Search Terms:
    From the Admin panel, select System > Configuration.
    In the Configuration panel on the left, under Catalog, select Catalog. ; Then, do the following:
    Click to expand the Search Engine Optimizations section.
    Set Popular Search Terms to “Disable.”
    Click the Save Config button to save the setting.

    Download Xenu Link Sleuth and run your site through it before and after enabling the search terms. This wil give you a better insight as to where all the products are being generated.

View original question in the Dumb SEO Questions community on Facebook, 02/03/2015).