Duaer

Guides

Search websites and help centres with Duaer

Connect product docs, a help centre or an intranet site to the Duaer knowledge base. People ask about what the site says and the answer links to the page; it is read again every day and everyone in the organization finds it.

What does Duaer find on a website?

From the start address, Duaer reads the pages under that address's folder and the PDF, Word, Excel, PowerPoint and text files they link to, up to the page limit (500 by default, 5,000 at most). Of a page it keeps the main text, not the navigation, headers, footers or forms.

People can ask “How many days do we have to request a return?”. Duaer finds the answer in the help centre and cites it, and the citation opens the page.

How do you connect a website to Duaer?

  1. Choose the start address: to read a whole help centre, enter its home, such as https://help.example.com/docs/; Duaer reads only the pages under that folder.
  2. As an owner or admin, open Data sources in the sidebar, select Add source, choose Website under Files and documents (a search for website finds it), enter the start address and the most pages, and select Add site.
  3. A site is added only when its start page opens. Reading starts at once; the panel shows the pages read, the last crawl, and which pages were not indexed and why.
  4. An intranet site: first add a gateway under On-premises data gateway with the site's address in ALLOWED_TARGETS (such as http://wiki.intranet), then add the site.

Which pages are not read?

  • Pages outside the start address's folder, and pages of other sites.
  • Pages robots.txt closes: Duaer keeps robots.txt as DuaerBot, with DuaerBot's own rules when there are any, else those for *; Crawl-delay is kept up to 10 seconds.
  • Pages marked noindex; links on a page marked nofollow are not followed.
  • Pages behind a sign-in; images, video and archives; pages over 5 MB and files over 20 MB.

Who finds the pages, and how fresh are they?

Everyone in the organization: a website is for whoever can see it. Duaer reads each site again every day: a page that has not changed (same ETag or Last-Modified) is not downloaded again, a changed one is read again and a deleted one leaves the knowledge base. Index now on the panel reads a site again at once.

To see how Duaer puts knowledge search and follow-through together, open the Duaer home page.

Questions

Does crawling put load on the website?

Very little. Duaer sends one request at a time to a site, at least 0.5 seconds apart, longer when robots.txt's Crawl-delay asks; an unchanged page is only asked whether it changed.

Can Duaer read a site that needs a sign-in?

Not yet. Pages behind a sign-in are not read; use that system's own connection, or publish the content at an address that needs none.

Can Duaer leave parts of a site out?

Yes. The start address covers only its folder, and the site's robots.txt can give DuaerBot Disallow rules.

In this section

Knowledge base

Turn SharePoint and Google Drive into a Q&A knowledge base with Duaer

Connect SharePoint, Google Drive, Notion, Jira, Salesforce, Zendesk and more to the Duaer knowledge base. Files stay put; answers cite the source.

Knowledge base

Search files in S3 and Alibaba Cloud OSS buckets with Duaer

Connect the files under a prefix of an S3 bucket, or of Alibaba Cloud OSS, Tencent COS or MinIO, to the Duaer knowledge base. People ask about the policies, manuals and reports in it; everyone in the organization finds them.

Knowledge base

Permission-aware enterprise AI search: how Duaer keeps document access intact

Permission-aware enterprise AI search filters results by what each person may open in the source system, before any answer is written. Duaer follows the source’s permissions: a document shows up only for people who can open it where it lives, and answers use only those documents.

Knowledge base

AI answers with citations and no hallucination: how Duaer does it

To get AI answers with citations and without made-up facts, do three things: answer only from documents the person may read, mark which file each statement comes from, and give no answer when there is no source. The Duaer knowledge base answers this way by default.

Knowledge base

How to build an AI assistant on your company documents with Duaer (no code)

You can build an AI assistant on your company documents without code. In Duaer, connect the places your documents already live and let the knowledge base index them. People then ask in plain words, and each answer cites its sources and uses only what that person may open.