Under website mode crawler will crawl all the URLs from sitemap (if exists in robots.txt or in custom robots.txt and is not ignored in configuration panel) and all those URLs as well which are linked internally. Option can be found in ‘Mode’ menu.

modes menu

[the_ad id=”6392″]

Data Extraction

Website mode scraps following data from a webpage.

  • Response Code
    • Redirect Location
    • Content Type
      • HTML
      • Image
      • Java Script
      • CSS
      • Flash
      • PDF
      • Others
      • External Links
      • Internal Links
        • Links Coming to Page
        • Links Going from Page
        • Meta Description
          • Meta Description Status
            • Unique
            • Duplicate
            • Not Found
  • Meta Description Length
    • Short
    • Long
    • Recommended
    • Title
      • Title Status
        • Unique
        • Duplicate
        • Not Found
  • Title Length
    • Short
    • Long
    • Recommended
    • Heading (H1 – H6)
    • Schema
      • Schema Details
      • Schema Tree
      • Google Analytics
      • Meta Robots
      • Custom Text Search (if implemented)
      • Canonical

Other Resources


About Ahmad Ali

Ahmad is the co-founder and CEO at Webbee Inc. He’s been working as a digital marketer for past few years and has worked with some notables names across different industries. He is also the developer of Webbee SEO spider, one of the most advanced SEO spider tool on the internet.

Leave a Comment