Importing knowledge sources for an advanced AI agent

What's my plan?

Add-on

AI agents - Advanced

The UltimateGPT settings page is the place to manage knowledge sources for your UltimateGPT AI agent.

On the UltimateGPT page, you can import articles from a Zendesk help center or with a CSV file. UltimateGPT has its search engine made specifically for the use-case and UltimateGPT doesn't fetch data live from the knowledge base but instead imports the articles into the search engine.

UltimateGPT allows you to add multiple knowledge sources to your AI agent. It means that you can import articles from multiple Zendesk Help Centers, CSV files, or a combination of both.

By default, all articles from all your knowledge sources will be used by the AI agent to find an answer - but you can also build rules based on conditional logic which we cover here.

This article contains the following topics:

• Importing a Zendesk help center
• Importing a Salesforce help center
• Importing a Freshdesk help center
• Importing a CSV file
• Using the web crawler as a knowledge source
• Search rules for knowledge sources
• Examples

Importing a Zendesk help center

To import a Zendesk help center, choose "Zendesk" as the source and type in the help center URL. The URL is the full URL to the help center or a part of the help center. You can specify the locale in the URL path to specify which language articles to import. For example ultimateai.zendesk.com/hc/en-us

Private articles can be imported from your help center by specifying an email address for a user account that has access to the pages and an API access token to your Zendesk (switch on "Import private articles" first).

The API access token can be generated by following these steps.

1. Go to your Zendesk admin console
2. Select “Apps and integrations”
3. Select “Zendesk API”
4. Enable “Token access” if not enabled yet
5. Click “Add API token”
6. Add a name for the token
7. Copy the token value to “API Access Token” field to the AI agents - Advanced dashboard

One caveat of importing private pages is that Zendesk doesn't give any hints if the credentials are wrong but instead just exports all of the public articles successfully. Therefore, you won't see any errors if the email or the token were mistyped. It's highly recommended to double-check the values before importing and test that the AI agent can answer questions with the private pages after the import has finished.

Zendesk help centers can also be set up for regular reimporting to keep the AI agent up to date. Simply select the import frequency or select "Never" if you want to import the help center only once.

Importing a Salesforce help center

To import a Salesforce help center, choose "Salesforce" as the source.

Before typing in the help center URL you want to add as a knowledge source, you need to log in to your Salesforce environment. This is necessary for authentication purposes and to provide you with access.

Once authenticated, you can add the URL to your help center and import.

Importing a Freshdesk help center

To import a Freshdesk help center, choose "Freshdesk" as the source and type in the help center URL. The URL is the full URL to the help center or a part of the help center.

You can then define an import frequency as "daily", "weekly", or "monthly".

To finalize the import, an API token is needed. You can generate this token in your Freshdesk environment.

Importing a CSV file

CSV files are a solution for importing all the knowledge bases and sources that aren't natively supported by AI agents - Advanced. The CSV file is a simple with "title" and "content" columns for each article, one per row. Optionally each row can also have "labels", "locale" and "article_url" fields for organizing the articles. Labels is a list of label names separated by space and the values can be anything you like. Locale is for organizing articles by language and market and while the values can be anything, it's recommended to follow standard practices of locales and use values such as "en-US" or "fi-FI". article_url is the external web address where the article can be found and is used in the sources and insights. The file format has a comma `,` as a column separator, and a double quote `"` as the string quote character. The first row is for the header.

The contents can contain HTML tags so there's no need to strip the HTML tags out. And indeed the tags can help as they give structure to the articles and that structure helps the search understand the article sections.

The CSV files cannot be reimported automatically so if your content changes, you need to create a new CSV and import that.

Note: If any of the cells contain Markdown, the Markdown must be valid or the content of that cell will not be imported.

Using the web crawler as a knowledge source

“Web Crawler” is an available Knowledge Source option when setting up an UltimateGPT AI agent. It means that UltimateGPT AI agents can retrieve information from a web page. 

To set it up click “Add Source” and select “Web Crawler”. 

Choose a name for your new Knowledge Source. It is how you will identify it in our dashboard.

Insert the URL you want the Web Crawler to go through. The Web Crawler will only collect information from pages listed in the Start URL and their sub-pages. By activating the Crawl exact URL the Web Cawler will only collect information from pages listed in the Start URL’s section without the sub-pages. If disabled, the Web Crawler crawler will apply a maximum crawling depth of 15 sub-pages.

The Web Crawler is best suited to get information from a knowledge base, a FAQ, or a product description page. It is less suited to e-commerce web shops. For e-commerce pages, we recommend our users build an integration able to retrieve the relevant product information.

Note: Currently, you cannot use the web crawler on a site protected by single sign-on (SSO). Instead, use a CSV export.

Using the Include URLs and Exlude URLs settings allows you to customize the crawling scope. Note that this affects only the links found on the pages, when crawling sub-pages. If you want to crawl a page, make sure to specify its URL in the Start URLs field.

Include URLs example

If the URL structure is inconsistent as in the example below.

Start URL: https://support.example.com/en/support/home

Article URL: https://support.example.com/en/support/solutions/articles/… 

You can add the following URL in the Include field:

https://support.example.com/en/support/** 

This way the Web Crawler will include all of the articles even though their path is different from the starting URL.

Exclude URLs example

Start URL: https://www.example.com/en

This page is very broad and includes irrelevant pages (*e.g. careers page). 

To exclude these irrelevant pages you would need to add the following URL in the Exclude field: https://wwwexample.com/en/careers/**

This way crawler will exclude all of the content from the careers page and its sub-pages.

Alongside plaintext, Globs have a set of control (or special) characters that allow them to become far more powerful than just plain text. 

For example: https://support.example.com/** let the crawler access all URLs starting with https://support.example.com/

For example: https://{store,docs}.example.com/** let the crawler access all URLs starting with https://store.example.com/ or https://docs.example.com/ 

https://example.com/**/*\?*foo=* allows the crawler to access all URLs that contain foo query parameters with any value.

Learn more about globs and test them here.

Search rules for knowledge sources

It can be helpful to create search rules to control which knowledge sources are used to find answers to a user’s question. For example, you could create search rules that use different knowledge sources based on the visitor’s locale or other session parameters to customise the experience for different visitor segments.

To configure search rules for your AI agent's knowledge sources, go to the “Search Rules” tab on the UltimateGPT AI agent settings page.

The default rule to use all data sources will already be there. You can not delete this default rule but you can edit it if you want to configure a different default behaviour ie: to use one specific knowledge source as the default.

To create a new search rule, follow these steps:

1. Click on the Create Rule button.
2. Enter a name and optionally add a description for the rule.
3. Select the knowledge sources that you want to include in the rule.
4. In the Advanced Settings section, you can add conditions to each knowledge source. For example, you could add a condition that only includes articles that match a specific URL, locale or have a specific label.
5. Click on the Save button.

Once you have created a search rule, you can use it in the dialogue builder by following these steps:

1. Select the UltimateGPT Node that you want to edit.
2. Turn off the “Use default search rule” toggle.
3. Select the rule you want to use.
4. Click on the Publish Changes button.

We recommend using the search rules in combination with conditional blocks in the dialogue builder. That way you can ensure that the AI agent provides the most relevant and accurate answers to your user’s queries based on parameters in the conversation session.

Examples 

Multilingual Knowledge Sources

1. Note the URL the language code for the versions of your knowledge base/ For example, we have en-us at the end of our URL for the American English version - ultimateai.zendesk.com/hc/en-us
2. Create a search rule for each - "English KB" "Turkish KB" "Hebrew KB"
3. In the GPT Reply of each language - define the correct KB
4. You can also use Conditional Blocks based on active_language parameters so if active_language: heb ---> use "Hebrew KB"

Labeled Content

The label option within the search rules allows you to show or hide content based on the labels you apply to the articles. It could be you have member-specific pages or product-specific pages that you only want to be used if the user is identified. So within your dialogue flow, after you have done CRM, API, or metadata collection to verify a user, you can then show them gated content with the rule.

In the screenshot below I could use the ugpt label that is applied to this article and have a rule with conditional blocks that only shows this to people who are identified via our CRM as being on a package that contains UltimateGPT.