Ingest Data from Content Sources

Content Sources enable Search AI to ingest data from websites, documents, and third-party applications, creating a unified knowledge base for generating accurate answers. When content is ingested, Search AI automatically initiates training to create the answer index using configured extraction strategies.

​

Websites

Crawl and index web pages from any public or authenticated website. Navigate to Content > Websites to manage web sources.

​

Adding a Web Crawler

Click +Web Crawl and configure the following:

​

Advanced Crawl Options

​

URL Filtering

Control which pages are crawled using rules:

• Crawl everything — All discovered URLs
• Crawl everything except — Block URLs matching specified conditions
• Crawl only specific URLs — Allow only URLs matching specified conditions

Conditions include: equals, not equal, contains, does not contain, begins with, ends with.

​

Authentication

Search AI supports two authentication methods for protected websites: Basic HTTP Authentication

• Username/email and password
• Optional authorization fields (headers, payload, query string, path parameters)
• Authentication URL (may differ from source URL)
• Test type: Text Presence, Redirection, or Status Code

Form-based Authentication

• Form fields with key, type, and value
• Session maintained after initial authentication
• Same test options as Basic Auth

​

Managing Web Sources

​

Troubleshooting

​

Documents

Upload and index files directly to Search AI. Navigate to Content > Documents to manage uploaded files.

​

Supported Formats

PDF, DOCX, PPT, TXT (scanned PDFs and password-protected files not supported)

​

Upload Options

​

Upload Limits

Contact support to increase these limits.

​

Managing Documents

• View files: Click any directory to see its files with type and page count
• View file details: Click a file for preview, metadata, and JSON view
• Delete file: Actions > Delete (from directory or file details page)
• Delete directory: Actions > Delete (removes directory and all files)
• Search: Use search bar on Directory page or within directory details

​

Connectors

Pre-built integrations for 60+ third-party applications. Navigate to Content > Connectors to configure integrations.

​

Connector Setup Workflow

1. Authentication — Provide OAuth credentials, API keys, or tokens
2. Ingestion — Select content types and apply filters
3. Field Mapping — Align source fields with Search AI schema (optional)
4. Permissions — Configure RACL settings

​

Configuration Options

Content Types: Most connectors support multiple content types (pages, articles, tasks, tickets, documents). Select which types to ingest under the Ingestion section. Filters: Some connectors support selective ingestion by timeframe, category, user assignment, or other criteria. Field Mapping: Customize how source fields map to Search AI’s schema using custom scripts for data transformation.

​

Sync Operations

​

Managing Connectors

• Enable/Disable: Toggle connector without deleting data (disables sync temporarily)
• View content: Content tab lists ingested items with URLs and timestamps
• Remove connector: Connection Settings > Remove this content source (deletes all indexed data)

​

Custom Connector

For applications without a pre-built connector, the Custom Connector uses REST APIs via a middleware service. Setup Steps:

1. Download the reference implementation from Search AI
2. Configure config.json with application details and content fields
3. Set authentication in .env file
4. Host the service and configure endpoint in Search AI
5. Add headers (Authorization required for default implementation)

The service processes content in batches of 30 documents. RACL support available via sys_racl field in config.

​

JSON Connector

Upload structured data in JSON format for indexing.

• Upload up to 10 files at once
• Maximum 15 MB per file
• Add to existing source or create new

​

Supported Connectors

Search AI provides out-of-the-box support for ingesting data from a range of third-party repositories. If you want to use a repository not listed below, contact us. RACL: Auto — permissions synced automatically from source; Manual — RACL available, manual entity management required; No — not supported. ¹ LumApps supports attachment ingestion. For any connectors not listed above, use the Custom Connector or contact Kore.ai.

​

Role-based Access Control (RACL)

RACL ensures users only see content they’re authorized to access by synchronizing permissions from source applications.

​

How RACL Works

1. Ingestion: Search AI imports permissions (users, groups, criteria) along with content
2. Storage: Access information stored in the sys_racl field of each chunk
3. Access Control: Only users whose identities appear in sys_racl can view answers from that content

​

Configuring RACL

Go to Permissions page of the connector and select:

​

Permission Entities

Permission entities represent groups or user criteria from source applications. Search AI handles these in two ways: Automatic Resolution (supported by some connectors):

• Fetches group membership data from source
• Maintains up-to-date user-group mappings
• Associates users with permission entities automatically

Manual Resolution (other connectors):

• Use Permission Entity APIs to manage user associations
• Required for connectors without automatic resolution support

​

Verifying Permissions

1. Go to Content page
2. Open JSON view for any file
3. Check the sys_racl field contents

The same information appears in individual chunks via Content Browser.

​

Permission Sync Scheduling

Access permissions often change more frequently than content. Configure separate sync schedules:

1. Open Permissions section of connector
2. Enable Permissions Sync Scheduled
3. Set time and frequency

The Sync Scope column shows “Permission Sync” for RACL-specific operations.

​

RACL Limitations

• Supported for connector content only (not websites or uploaded documents)
• Switching from Restricted to Public Access automatically disables RACL

​

Quick Reference

---