While Search Platforms help to drive sites faster, they are also prone to stability issues. The chances of failures are directly proportional to the dependencies & customizations involved.
By default, when an exception is encountered during the indexing of a computed index field, the corresponding field will be excluded from the indexed document, thereby leading to invalid results in the front-end. In regards to Custom Crawler, whenever an exception/issue is encountered and not thrown, by default the entire index would be cleared. Hence it is essential to improve the SOLR index resiliency against these kinds of failures for a consistent user experience. This blog explains the approach to retain the previously indexed data in SOLR when the indexing operation fails or experiences any issue.
Prerequisities: Ensure Rebuild Core is setup for the SOLR index to support swapping of indexes during the rebuild, https://doc.sitecore.com/en/developers/101/platform-administration-and-architecture/switch-solr-indexes.html
High-Level Approach:
STEPS TO STOP INDEXING/SWAPPING DURING COMPUTEDINDEXFIELD FAILURES:
-
- Update the ‘StopOnCrawlFieldError’ setting to true
-
- Add appropriate validations in the computedindexfield to ensure that the right data in the expected format is received from dependencies/external services; else throw an exception. Catch any exceptions encountered and stop indexing as indicated below.
-
- Create a new class for SearchIndex inheriting SwitchOnRebuildSolrSearchIndex. Resume indexing from the PerformRebuild method after the current rebuild is complete, so that the IndexingCustodian can execute the next queued job.
-
- Change the type attribute of the index node in the index’s config to refer to the above created class.
STEPS TO STOP INDEXING/SWAPPING DURING CUSTOM CRAWLER FAILURES:
-
- Sitecore handles the exception happening within custom crawler code (created by inheriting FlatDataCrawler class) and stops indexing/swapping automatically . Hence it is important that an exception shouldn’t be caught within custom code. If the exception is caught for any reason, it must be rethrown, so that Sitecore can stop indexing.
- In order to stop indexing/swapping for any error that may happen when Sitecore is processing the data to prepare index documents, the stopOnError property must be set to true within the crawler config node,
Now that the SOLR index is more resilient towards failures, it is vital to set up appropriate monitoring & alerting to get notified of the failures happening in the rebuild core to prevent stale results from being displayed on the site.
Happy Searching!