Dumb SEO Questions

(Entry was posted by Jefe Birkner on this post in the Dumb SEO Questions community on Facebook, 08/11/2016).

Screaming Frog can`t crawl my site

I`m trying to solve a puzzle, have a WP site that has 13 pages in the sitemap, all show up as status 200 when I upload a file with a list of pages, but if I have Screaming Frog just crawl the site, from the home page, it only finds one page. Any help or suggestions would be appreciated

Best I can tell, robots.txt isn`t blocking access, and I don`t think I have SF set wrong, because it crawls other sites correctly

pages are things like;
/location-hours/
/pay-bills/
/forms-policies/

and robots looks like this;

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:

# digg mirror
User-agent: duggmirror
Disallow: /

# global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category//
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Allow: /wp-content/uploads/
This question begins at 00:15:40 into the clip. Did this video clip play correctly? Watch this question on YouTube commencing at 00:15:40
Video would not load
I see YouTube error message
I see static
Video clip did not start at this question

YOUR ANSWERS

Selected answers from the Dumb SEO Questions Facebook & G+ community.

  • Jefe Birkner: I'm trying to solve a puzzle, have a WP site that has 13 pages in the sitemap, all show up as status 200 when I upload a file with a list of pages, but if I have Screaming Frog just crawl the site, from the home page, it only finds one page. Any help or suggestions would be appreciated

    Best I can tell, robots.txt isn't blocking access, and I don't think I have SF set wrong, because it crawls other sites correctly

    pages are things like;
    /location-hours/
    /pay-bills/
    /forms-policies/

    and robots looks like this;

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:

    # digg mirror
    User-agent: duggmirror
    Disallow: /

    # global
    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-content/plugins/
    Disallow: /wp-content/cache/
    Disallow: /wp-content/themes/
    Disallow: /trackback/
    Disallow: /feed/
    Disallow: /comments/
    Disallow: /category//
    Disallow: */trackback/
    Disallow: */feed/
    Disallow: */comments/
    Disallow: /*?
    Allow: /wp-content/uploads/
  • Kyle Hawk: Not too familiar with crawlers, but are these pages linked to the home page? My thought would be it's looking for links on the home page. However, if it is going based off the sitemap, then never mind! haha. Hope you find an answer, bud!
  • Jefe Birkner: Yes, all pages are linked from the home page
  • Collin Davis: Could you confirm if you are using Angular JS on the website...Screaming frog doesn't crawl Angular JS well..
    
  • Dave Elliott: Try loading the sitemap.xml via mode -> list into screaming frog and see if it crawls them then.

    Also make sure you haven't got a nofollow tag on your homepage.

View original question in the Dumb SEO Questions community on Facebook, 08/11/2016).