User agent strings are identifications when connecting to a webpage on a server. Server, when giving response for a webpage request, verifies the identification and gives information as per that. So, there are possibilities that server will give different information when try to connect with two different user agent strings. Below are some popular strings […]
Spider Speed and Turbo Crawl Spider speed option lets the user to increase the speed of the crawler. By default application creates 5 threads which scrap the website data in parallel. But if more speed is required (if computer hardware and internet speed in capable of handling this) then this can be done with spider […]
Spider Configuration There can be instances when particular page types are required to be crawled e.g. if user wants to crawl only HTML pages or he do not wants robots.txt to be followed or sitemap extraction is not desired or specific directory crawling on a website is the target. In these cases, and others alike, […]
XML Sitemap helps the crawlers to identify the changes to your website. In return this helps in better rankings and indexing. Moreover, xml sitemaps contain all the pages of website that are intended to be crawled by search engines and to be ranked. Regarding sitemaps there are some features which have added in the application. […]
DEPRECATED! Webmaster is a free web application provided by the Google Inc. Webmaster, when integrated with the website, lets you know the actual health of the website. Our concern here with webmaster is to check the webpages with header status code 404. These are those pages which do not exists but they are somehow linked […]
When right clicked on any URL, a pop menu pop ups with different options showing page based information. Popup menu contain following options. Copy URL This option copies the URL to clipboard for later use. Open in Browser When clicked, selected URL will be opened in default browser of computer. Google Cache When clicked, a […]
Data Segregation is a concept to separate data view by using different filters and searches so that the data can be in much more understandable form then its normal form. In this regard, Webbee provides some awesome data filtration techniques to the user to filter the data as they want to be. There are 3 […]
There could be instances when some specific text searching in required on webpage/s. Custom search lets the user perform this task. Option can be found in Spider menu. See the snapshot. The panel for custom search looks like; There are two options for search, ‘contain, and ‘does not contain’. Whenever there is an input in […]
Note: Read Custom Robots before continuing this article. ‘Custom robots’ is a feature to crawl website with editable robots.txt to understand the behavior of actual robots.txt. If ‘Custom Robots’ is enabled; the BeeBot will crawl the website as per the directives stated in custom robots. But this behavior can be different with different user agents. […]
