What Enhanced Web Crawler Does
Enhanced Web Crawler scans public website pages and adds extracted content to the selected Knowledge Base.
It can extract supported content from:
Accordions
Tabs
Lazy-loaded sections
Infinite-scroll areas
Dynamic website layouts
Public structured data
Enhanced Web Crawler only crawls public content. It does not crawl private, login-gated, paywalled, or portal-only content.
Open the Web Crawler
Go to AI Agents.
Open the AI Agent or Knowledge Base you want to train.
Click Knowledge sources.
Select the Web crawler tab.
Review Existing Crawled URLs
The Web crawler table shows each crawled page, its training status, the last updated date, and available actions. Use this screen to confirm whether a page is already trained before adding it again.
Add a Website
Click + Add website.
The Add website window opens.
Choose a crawl mode.
Enter the website URL.
Click Train this page.
Enter the URL and Train the Page
Enter the full public URL in the Enter URL field. After the URL is entered, click Train this page to crawl and train the selected page or website scope.
Crawl Mode Options
Choose the crawl mode before training.
Exact URL: Crawls one specific page.
All URLs with the Path: Crawls pages under a specific URL path.
All URLs in this Domain: Crawls pages across the selected domain.
Use the exact URL when you only want one page trained. Use all URLs with the Path when the content is grouped under a specific folder or section. Use all URLs in this Domain when the AI Agent needs broader public website knowledge.
What the Crawler Can Capture
Enhanced Web Crawler can capture public website content such as:
Hero section text
Service descriptions
Product descriptions
Testimonials
Team bios
Pricing details
Contact details
Business hours
FAQ content
Structured data
What the Crawler Avoids
Enhanced Web Crawler avoids risky website actions.
It does not:
Submit forms
Click the checkout buttons
Change filters
Log in to private areas
Access private portals
Extract paywalled content
Advanced Link Discovery
Enhanced Web Crawler can discover public links from multiple sources.
It can use:
HTML parsing
JavaScript evaluation
Sitemap crawling
Compressed sitemap files
Interaction-based discovery
This helps Kyrios find public pages that may not appear in a simple static page scan.
Common Issues and Fixes
The crawler did not find a page
Confirm the page is public and does not require login access. If the page is inside a specific path, use All URLs with the Path.
The AI Agent is missing website information
Confirm the correct URL was entered and trained. The AI Agent only uses content from pages that have been crawled and trained.
The crawler does not access private content
Enhanced Web Crawler only works with publicly accessible website content. Content behind a login, paywall, or private portal will not be extracted.
FAQ
What does smarter content discovery mean?
Smarter content discovery means the crawler can detect supported public content inside dynamic sections such as accordions, tabs, and lazy-loaded areas.
Do I need to configure anything to extract hidden public sections?
No. Enhanced Web Crawler automatically attempts to discover and extract supported public dynamic content.
Can Enhanced Web Crawler read interactive or hidden content?
Yes. It can read supported public content inside accordions, tabs, and lazy-loaded sections.
Will Enhanced Web Crawler submit forms or click checkout buttons?
No. Enhanced Web Crawler avoids risky actions such as form submissions, checkout clicks, and filter changes.
What happens if content is behind a login?
Login-gated content will not be crawled. Only publicly accessible content can be extracted.
Can I choose which crawled pages train the AI Agent?
Yes. Use the crawl mode and URL selection to control which public website content is trained into the Knowledge Base.