Content Search & Query

There are several different ways to search the Content on your site, on both the dotCMS front end and back end. This document provides an overview of the different types of Content search, so you can select the tools that best suit your needs.

Front-end vs. Back-end Searches

The distinction between front-end searches and back-end searches in dotCMS is important.

Front-end searches use the dotCMS Site Search feature to enable your front-end users to search within search indexes you specifically create for the purpose of front-end searches.

• Front-end searches are not enabled by default.
  • You must both create a SiteSearch index and add Velocity code to allow them.
• Front-end searches only query content which is visible on your front-end site.
  • Specifically, front-end queries only find content which is displayed on pages which are included in the specific Site Search index being queries (either directly or via a URL Map).

Back-end searches use the built-in dotCMS site indexes to allow you to perform searches of all your dotCMS content.

• Back-end searches are enabled by default.
  • And the back-end content index is automatically updated with all new changes (you do not need to create or update the back-end content indexes under normal circumstances).
• Back-end searches include all content on your site, whether or not it is visible from a page.

For more information on front-end and back-end search, please see the appropriate sections below.

Front-end Search

You can allow your front-end users to search your site using the Site Search feature. This requires both that you create a SiteSearch index and that you use the SiteSearch Viewtool in your Velocity code to perform a search against the index using the user's entered search terms.

Site Search Indices

You can create site search indices to facilitate front-end searches on your site. You can create multiple indices to allow searches over different portions of your site content; for example, you could create one index to allow users to search the documentation section of your site, and another index to allow users to search the products section of your site.

For more information on creating and managing Site Search indices, please see the Site Search documentation.

Performing Searches

Once you have created your Site Search indexes, you can use the SiteSearch Viewtool in your Velocity code on any page to provide search functionality to front-end users of your site. You can also use the SiteSearch Tool on the dotCMS backend to test searches against specific SiteSearch indexes.

Although users may enter any search terms they wish, your code must submit queries to the SiteSearch Viewtool using valid Elasticsearch query syntax. This means that you may wish to allow users to enter Lucene search terms, or you may wish to pre-process user-entered search terms into full ElasticSearch JSON syntax to allow users to perform more complicated searches on your site.

For more information on how to generate queries for use with the SiteSearch Viewtool, please see Site Search Tool in the Back-end Search section, below.

Back-end Search

The following tools provide different ways of searching dotCMS Content from the dotCMS back-end. Each of the tools accepts different types and levels of search syntax, so the best tool to use will depend on your needs.

Site Search Tool

Although the Site Search feature is designed to enable front-end users to search the content on your site, you may also perform searches against your Site Search indexes on the dotCMS back-end, to test and verify the content that will be returned to your front-end users when they perform searches from different areas of your site..

The Site Search tool allows you to query your Site Search Indexes from the dotCMS back-end using simple text strings (similar to what front-end users would enter), allowing you to query the content repository without the more sophisticated search terms dotCMS uses when presenting content. This is useful when you wish to perform simple queries of your content indexes to ensure your content has been indexed properly, and understand what content will be discovered by front end users using the Site Search feature.

To open the Site Search tool, select Dev Tools -> Site Search and click the Search tab.

In the Site Search tool, you can select which Site Search Index to use for the search and enter a text string and/or search terms. When you press the Search button, the tool automatically returns all results within the selected index which match your query.

Note:

• If you enter a simple string to search for, all results are returned which contain that string in any field (regardless of which field within the content the text field was discovered in).
• When you perform a search using the Site Search tool, the screen displays the query syntax used to fetch the displayed results.
  • This provides a way to determine the full Elasticsearch query syntax used for a simple query.
  • However the syntax provided can not be copied and pasted directly into the ES Search tool.
    • Certain terms (such as "default_field") are not recognized by the ES Search tool, so you must remove some terms from the query before you can use it in the ES Search tool.

Query Tool

The Query Tool enables you to perform queries against your dotCMS Content, using the Elasticsearch Lucene query syntax. To open the Query Tool, select Dev Tools -> Query Tool.

You may enter any valid Lucene syntax into the Lucene Query field, and you may find the Lucene syntax for Content searches using the Show Query popup.

If you enter simple search terms, they will be searched across all searchable fields in your content. By default, if you enter multiple text strings, the Query Tool will return all results which match any of the terms (e.g., using a logical OR). If you wish to search only for Content which includes all search terms, join each search term with a logical AND operator (&& - see the Content Search Syntax documentation for more information).

You may specify an Offest and Limit for the returned results. This helps you identify what results will be returned if you use a paginated content pull in your Velocity code. You may also specify which field(s) to Sort by.

If you check the Reindex Contents box, the contents returned by the search will be marked to be reindexed as soon as possible.

ES Search Tool

The ES Search tool allows you to perform full Elasticsearch queries from the dotCMS backend. To open the ES Search tool select Dev Tools -> ES Search.

The ES Search tool requires that you specify queries using Elasticsearch JSON query syntax. You may limit the results of your query to only display Content which is Live on your site by checking the Live box.

Note:

• The Show Query popup can not be used to determine the full Elasticsearch syntax.
  • For examples of full Elasticsearch query syntax, use the ES Tool Help button or see the ES Search Portlet documentation.
• However if you use an Elasticsearch query which accepts a “query” string, you may use the Show Query popup to find the Lucene syntax for the “query” string (only).

Help Button

The ES Search tool includes a Help button to aid you in creating valid Elasticsearch queries. When you press the Help button, a popup displays with a number of examples you can use as a basis to create your own ES Queries. You may copy any of the queries in the Help popup directly into the ES Query field of the ES Search tool, and you may modify the terms of the query appropriately to customize these queries for your search.

Searching from Velocity Code

There are several different ways to search your site content from your Velocity code. As with the back-end search tools, which Velocity tool you use depends on your needs and the sophistication of the queries you wish to perform.

• The Site Search Viewtool allows you to perform front-end Content searches from your Velocity code, using Elasticsearch queries.
• The various different content pulls in dotCMS use Lucene queries to perform back-end searches which retrieve and access Content from within your Velocity code.
• The ES Viewtool allows you to perform complex back-end Elasticsearch queries using full Elasticsearch query syntax within your Velocity code.
  • As with the ES Search tool, you can use the examples provided by the ES Search tool Help button to create valid queries to use with the ES Viewtool.

Show Query Popup

The Show Query option allows you to find the Lucene query syntax to retrieve any content which you can display in a back-end dotCMS listing. For example, if you open the Content tab and perform a Content search, you may use the Show Query option to find the Lucene query syntax you will need to retrieve the exact same list of Content displayed in the Content tab based on your search.

The Show Query popup displays the syntax and code needed to reproduce the appropriate Content list in all of the following formats:

• Velocity Content Pull
• Lucene Query (for use with the backend Site Search tool
• REST API

For a full list of all the tools and places in dotCMS where Lucene syntax can be used, please see the Content Search Syntax documentation.

Example

The following image shows a list of all content on the Content -> Search screen of the dotCMS demo site of the “Blog” Content Type with the word “today” in the Content:

From the Content Search screen, you can click the Search button and select Show Query to display the following popup, which displays the query syntax required to perform the same search in different ways:

Troubleshooting

Important The query tools all perform queries against your site indexes, not directly against the Content in your site.

This means that if you have changed the Content of your site since the indexes were last updated, the search will return results which do not match your current Content.

Therefore, please keep the following in mind when testing and troubleshooting search results:

• It is important to ensure that you set up jobs to regularly index your content.
  • For many sites, it is best to reindex at least once per day.
  • However if you have a large amount of Content that changes frequently and it is important that your searches return information updated more recently, you may wish to reindex your Content more frequently.
• If your searches on the backend are not producing the results you expect, try the following:
  • Perform an equivalent search in the Content tab.
    • Content Search within the Contents tab is always performed against your current content, not your indexes, so it will produce more accurate results if your content changes frequently.
  • Manually reindex your Content before testing your search results.