{"id":3691,"date":"2022-04-25T10:35:46","date_gmt":"2022-04-25T10:35:46","guid":{"rendered":"https:\/\/multisite.korebots.com\/SearchAssist\/?p=3691"},"modified":"2022-08-17T11:21:32","modified_gmt":"2022-08-17T11:21:32","slug":"crawling-web-pages","status":"publish","type":"post","link":"https:\/\/multisite.korebots.com\/SearchAssist\/concepts\/managing-content\/crawling-web-pages\/","title":{"rendered":"Crawling Web Pages"},"content":{"rendered":"
\n
\n
\n
\n
\n
\n
\n
\n

Organizations usually have web pages which a user can query, such as product information or process knowledge pages. You can leverage these pages by mapping your SearchAssist app to the content.<\/p>\n

SearchAssist enables you to ingest content through web crawling. For example, consider a banking website. Its pages contain information that can answer most user queries. SearchAssist can crawl the bank\u2019s website and index all the content. When a user submits a query, the SearchAssist app retrieves and displays the correct content.<\/p>\n

To maintain an up-to-date index, schedule automated web crawl sessions (e.g. time and frequency) as required. After you customize crawl settings, click Proceed at the bottom of the page.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/section>\n

\n
\n
\n
\n
\n
\n
\n
\n

<\/span><\/span>
\n<\/em>Adding Content by Web Crawling<\/span><\/h2>\n
    \n
  1. Follow these steps to crawl web domains:\n
      \n
    1. Log in to SearchAssist.<\/li>\n
    2. Go to the All Apps heading.<\/li>\n
    3. Select the correct SearchAssist app.\"\"<\/a><\/li>\n
    4. Click the Sources menu tab.<\/li>\n
    5. In the left pane, click Content.\"\"<\/a><\/li>\n
    6. Click the +Add Content drop-down menu and select Crawl Web Domain. (Or click the + Add Content button in the top right corner of the page.)\n

      \"\"<\/a><\/li>\n

    7. On the Crawl Web Domain page, fill in the Source Title and Description fields.<\/li>\n
    8. Enter the domain address in the Source URL field.\"\"<\/a><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/div>\n<\/div>\n
      \n
      \n

      <\/span><\/span>Scheduling Crawls<\/span><\/h3>\n

      Activate this optional feature to schedule regular crawls on your site. This feature is well-suited for organizations that regularly publish new website content and want to keep their search index up to date.<\/p>\n

      Note<\/b>:\u00a0If you do not schedule auto-crawls, you need to manually launch new crawls to keep the index up to date.<\/p>\n

       <\/p>\n

        \n
      1. Click the toggle switch to the ON position. The default setting is OFF.\"\"<\/a><\/li>\n
      2. Click the Select Date field.<\/li>\n
      3. Set the day and date of the first crawl on the calendar.<\/li>\n
      4. Enter the time to launch crawl. (use 24-times)<\/li>\n
      5. Click the Time Zone field and select an option. SearchAssist currently supports three time zones: IST, EST, and UTC.
        \n
        \n<\/a>        \"\"<\/a><\/li>\n
      6. \u00a0Click the Frequency field.\"\"<\/a><\/li>\n
      7. Select an option.<\/li>\n<\/ol>\n

        Select the Custom option if you want granular control over index updates and then click Ok.<\/p>\n