{"id":457,"date":"2021-05-13T09:03:05","date_gmt":"2021-05-13T09:03:05","guid":{"rendered":"https:\/\/multisite.korebots.com\/SearchAssist\/?p=457"},"modified":"2021-05-13T09:03:05","modified_gmt":"2021-05-13T09:03:05","slug":"manage-content-of-a-web-page","status":"publish","type":"post","link":"https:\/\/multisite.korebots.com\/SearchAssist\/source-management\/content\/manage-content-of-a-web-page\/","title":{"rendered":"Manage Content of a Web Page"},"content":{"rendered":"

Once you add content to the application, it needs to be updated as the content from websites may not be static. You can manage (schedule periodic web crawling and edit crawling) and ensure that the content is in sync with the data on the website.<\/span><\/p>\n<\/div><\/div>

<\/div><\/div>

<\/span>Schedule Web Crawling<\/span><\/h2>\n

The scheduler allows you to schedule a job to re-crawl the configured website periodically. To schedule a web crawling job, follow the below steps:<\/span><\/p>\n

    \n
  1. On the Indices page, click <\/span>Content <\/b>on the left pane.<\/span><\/li>\n
  2. On the Content list view page, select the respective source from the list.<\/span><\/li>\n
  3. On the source dialog box, click the <\/span>Configuration <\/b>tab.<\/span><\/li>\n
  4. On the Configuration tab, turn on the <\/span>Schedule <\/b>toggle.<\/span><\/li>\n
  5. Set the <\/span>Date<\/b>, <\/span>Time<\/b>, and <\/span>Frequency<\/b>.<\/span><\/li>\n
  6. Turn on the <\/span>Crawl Everything<\/b> toggle to crawl all the domains.\u00a0<\/span><\/li>\n
  7. If you wish to crawl only selected domains, then turn off the <\/span>Crawl Everything<\/b> toggle.<\/span><\/li>\n
  8. After you turn off the <\/span>Crawl Everything<\/b> toggle, the <\/span>Allow List<\/b> toggle is turned on automatically. You can enter the allowed list of URLs in the <\/span>Allow URLs<\/b> field.<\/span><\/li>\n
  9. If you wish to block URLs, then turn off the <\/span>Allow List<\/b> toggle.<\/span><\/li>\n
  10. After you turn off the <\/span>Allow List<\/b> toggle, the <\/span>Block List<\/b> toggle is turned on automatically. You can enter URLs to block in the <\/span>Block URLs<\/b> field.\u00a0<\/span><\/li>\n
  11. Select <\/span>Crawl Settings<\/b>:\u00a0<\/span><\/li>\n<\/ol>\n
      \n
    • JavaScript-rendered<\/span><\/li>\n
    • Use Cookies<\/span><\/li>\n
    • Respect robots.txt<\/span><\/li>\n<\/ul>\n
        \n
      1. Click <\/span>Save<\/b>.<\/span><\/li>\n<\/ol>\n<\/div><\/div>
        <\/div><\/div>

        <\/span>Edit Crawler Configuration<\/span><\/h2>\n

        To edit a web crawling source, follow the below steps:<\/span><\/p>\n

          \n
        1. On the Indices page, click <\/span>Content <\/b>on the left pane.<\/span><\/li>\n
        2. On the Content list view page, select the respective source from the list.<\/span><\/li>\n
        3. On the source dialog box, make the required changes.<\/span><\/li>\n
        4. Click Save<\/b>.<\/li>\n<\/ol>\n<\/div><\/div><\/div><\/div><\/div><\/div><\/section>\n","protected":false},"excerpt":{"rendered":"Once you add content to the application, it needs to be updated as the content from websites may not be static. You can manage (schedule periodic web crawling and edit crawling) and ensure that the content is in sync with the data on the website. Schedule Web Crawling The scheduler allows you to schedule a...","protected":false},"author":12,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/posts\/457"}],"collection":[{"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/comments?post=457"}],"version-history":[{"count":1,"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/posts\/457\/revisions"}],"predecessor-version":[{"id":458,"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/posts\/457\/revisions\/458"}],"wp:attachment":[{"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/media?parent=457"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/categories?post=457"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/multisite.korebots.com\/SearchAssist\/wp-json\/wp\/v2\/tags?post=457"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}