Web Crawling feature can fail in two stages:
- Fails to start due to URL validation failure either due to connectivity issues or a misspelt URL, click Retry or Edit Configuration to Edit URL
- Fails after starting successful URL Validation for the given website and during web crawling.
Note: If you are attempting to crawl the same site without enabling automatic Frequent Scheduling toggle, a Duplication warning message pops up, reading “web crawling cannot be duplicated” instead try using the Crawl by schedule.