Question about indexing of large ecommerce site

Selected answers from the Dumb SEO Questions Facebook & G+ community.

• 
  Michael Martinez: Are you asking why the other pages are not being indexed, how to get them indexed, or both, or something else? How long did you wait for the site to be recrawled? Does site navigation link to the unindexed pages or are you just using XML sitemaps to handle crawling for that much content? Are you using "rel=`nofollow`" on any internal links or blocking any folders in "robots.txt"? Are you using "noindex" or "nofollow" robots meta directives on pages?
• 
  Razvan Alexa: Thanks for the answer Michael Martinez, your blog was the first I read about SEO many years ago. There is no "noindex" or "nofollow" and the pages are not orphaned, we have a section for these new products even on the homepage. This is what is baffling me, usually when working with these large sites I had a problem from having too many pages indexed, not too few. The setup was like this, for the sake of the example: we have a headlight that fits both ford f150 and ford f350. They created a generic product and the 2 pages for the f150 and f350 were canonicals to the main generic one, so only that got indexed. Since that had no keywords, being listed only as Headlight 45445, I removed the canonicals, so that the pages headlight for f150 and headlight for f350 are to be indexed. We did this about 30 days ago and updated the submitted sitemaps accordingly.This where the problem is, these 2 pages don`t get indexed. I manually submitted one of them, and it ranked in the top 10 for "f150 headlight" in a matter of days, so there is no penalty for the site. I keep on thinking that there is an internal magento setting that is wrong, but again maybe it`s just google indexing a little slower.
• 
  Michael Martinez: This is a situation where I would consider crawling the site to see what the SEO crawler finds. And I know people who follow my comments will be SHOCKED I recommended that.
• 
  Razvan Alexa: In fact I was just looking at my options for this service right now :) what would you recommend deepcrawl, oncrawl, smth else?
• 
  Dave Elliott: Razvan Alexa deepcrawl is an awesome tool but ridiculously expensive! Screaming Frog, the answer is always screaming frog.
• 
  Razvan Alexa: Hey Dave Elliott screaming frog is great but crashes on sites bigger than 300, 000 pages. No matter how much ram i allocated, still crashed. Had better results with sitebulb. Thanks for the reply
• 
  Dave Elliott: Razvan Alexa these days there is an option to use your hard drive rather than your RAM. Took me a couple of days but i managed to do a 4.5million page site a couple of weeks ago.
• 
  Razvan Alexa: Got it...thanks, I`ll have a look
• 
  Dave Elliott: there in fact
• 
  Razvan Alexa: Thanks mate
• 
  Michael Martinez: It has been a while since I worked with Magento but I think there is a settings report, is there not? If that`s the CMS I am thinking of, have you printed out the settings to see what is active/inactive?
• 
  Razvan Alexa: I did...aparently there is a problem with the sitemap content....something I asked the developer to check 10 times...thank you for the feedback