Automating Sitemap Submissions to Google & Bing using Sitecore Powershell Extensions

Keeping Search Engines apprised of the content updates(additions/modifications/removals) is crucial for businesses to amplify user acquisition and to improve user experience. While Google’s scheduled recrawl may take up to weeks for scanning certain sites again, it is recommended to notify Google of essential content updates early to drive the needed traffic to gain maximum possible value.

NOTE: If the site holds short-lived content like live events, jobs, etc., you will also be able to benefit from Google Indexing API. This blog describes how to integrate Google Indexing API with Sitecore Publish.

With Sitecore Powershell Extensions, Sitemaps can be submitted to Search Engines’ Ping services quickly and flexibly.

STEPS TO CREATE POWERSHELL MODULE
Powershell Module(Eg: ‘SEO’) can be created under /sitecore/system/Modules/PowerShell/Script Library using ‘Module Wizard’ Insert Option,

Ensure to select the ‘Content Editor’, ‘Tasks’ and ‘Shared Functions’ Integration Points while creating the module,

Remove the items under Ribbon item. Build the following tree structure under SEO module item based on templates/insert-options indicated below,

  • Content Editor
    • Ribbon
      • SEO (PowerShell Script Library)
        • Sitemap (PowerShell Script Library)
          • Submit (PowerShell Script)
  • Functions
    • Invoke-PingService (PowerShell Script)
    • Submit-Sitemap (PowerShell Script)
  • Tasks
    • Submit Sitemap (PowerShell Script)

 

Below PowerShell Scripts shall be copied into the ‘Script body’ field of above created ‘PowerShell Script’ items,

Ensure to update the live site’s sitemap URL(s) within the above Submit-Sitemap function.
Navigate to ‘PowerShell ISE’ from Sitecore LaunchPad. Select the ‘Settings’ tab and choose ‘Sync Library with Content Editor Ribbon’ from ‘Rebuild All’,

‘Submit’ option should now be available within the ‘Sitemap’ Chunk of the new ‘SEO’ tab, using which Content Authors can make on-demand Sitemap submissions to Google, Bing, Yahoo, and DuckDuckGo Search Engines.

STEPS TO CREATE POWERSHELL SCHEDULED TASK
A Powershell Scripted Task can be created under /sitecore/system/Tasks/Schedules from Insert Options to automate submissions at a scheduled frequency,

The schedule frequency can be anywhere between few hours to few days for the site depending on the frequency of content updates happening and how frequently Google crawls the site (can be identified from the Google Search Console). Ideally, the Google bot crawl rate shouldn’t affect the site performance. If needed crawl rate can be optimized as per this documentation.

Bing’s index covers Yahoo and DuckDuckGo search engines, hence they do not require dedicated sitemap submissions.

Ensure that the lastmod field for the URL(s) are up-to-date in Sitemap file, as Google uses this field to determine if an URL is modified and if it requires crawling. It may take few minutes to few hours for the Search Engine bots to crawl the site once the request is submitted. Once Search Engine starts crawling sitemap, ‘Last read’ value gets updated with the current date in the Google Search Console,

Happy Crawling!

In-Sitecore Alerts & Push Notifications: Effectively communicating Maintenance Activities to Authors & Marketers

Building a strong DevContentOps within the organization helps to keep the productivity of all teams at maximum. This module eliminates friction and enables seamless collaboration between Authors, Marketers, Developers, and Operations.

The module can be downloaded from the below links,
Sitecore Maintenance Notification-v1.0 – Sitecore Package
sitecore-maintenance-notification:1.0.0.0-1903 – Docker Image

Integrating the above image into CM/MSSQL Containers requires few changes to .env, docker-compose/override, and Dockerfiles of CM & MSSQL Containers, as indicated in this screenshot (build is required for the changes to reflect). Alternatively, Sitecore packages can also be installed in Docker as described here.

Key Benefits:
• Optimized delivery of maintenance alerts through in-sitecore alerts and push notifications(works even if browser is not open) with appropriate auto-expiration
• Introduces Opt-in/Opt-out flexibility for Maintenance Notifications from Sitecore Control Panel.
• Displays Maintenance Page during Maintenance to avoid confusion.
• Displays Maintenance Alerts in User’s Local Timezone.
• Enables smooth handling of any Schedule Changes and Cancellations.
• Sends reminders before outage & Completion Message immediately after warm-up to keep content freeze at a bare minimum.
• Permits modifying Scheduled Alert messages, Reminder & Completion messages, Reminder Duration, etc. quickly from Sitecore

NOTE:
This module is built on top of Sitecore Powershell Extensions (SPE), please ensure SPE(v5.0+) is installed. 

STEPS TO CONFIGURE THE MODULE:
After installing the module, navigate to ‘PowerShell ISE’ from Sitecore LaunchPad. Select the ‘Settings’ tab and choose ‘Sync Library with Control Panel’ from ‘Rebuild All’,

Below changes need to be introduced in web.config of CM instance. As a best practice, this transform file can be added to the solution. If there is any existing CM web.config transform, the below config can be appended to it.

  <location path="sitecore modules/Maintenance Notification/js/serviceworker.js">
    <system.webServer>
        <httpProtocol>
            <customHeaders>
                <add name="Service-Worker-Allowed" value="/" />
            </customHeaders>
        </httpProtocol>
    </system.webServer>
  </location>

Module comes with default VAPID Keys but it is highly recommended to update the defaults for security reasons. New VAPID Keys can be created using web-push npm package. Below command can be executed to generate new Public/Private VAPID key pair,

npm install web-push -g
web-push generate-vapid-keys


Public Key needs to be updated in /sitecore/system/Modules/PowerShell/Script Library/Maintenance Notification/Notification Settings item. Private Key needs to be updated in the DevOps/Windows PowerShell scripts (described in the next section).

Module includes a default Sitecore API Key(works for 9.1+). But it is necessary to clone/create a new ‘OData Item API Key’ item(with same values) under /sitecore/system/Settings/Services/API Keys of master database for v9.1+ (make sure to publish to web). Item needs to be created under /sitecore/system/Settings/Services/API Keys of core database if v9.0.

For Impersonation User, it is recommended to create a new user(Eg: sitecore\MaintenanceNotification) with just read access only to the items under /sitecore/system/Modules/PowerShell/Script Library/Maintenance Notification. Item ID of this newly created ‘OData Item API Key’ item will need to be updated in DevOps/Windows PowerShell scripts (described in the next section).

FOR RECEIVING ALERTS:
Content Authors and Marketers will have to just select ‘Subscribe to Scheduled Maintenance Notifications’ of ‘Preferences/My Settings’ section of Control Panel, from normal mode of browsers (not incognito/inprivate) to receive maintenance notifications

FOR SENDING ALERTS:
DevOps team will have two options to send notifications during deployments,
DevOps Integration(Automated):
Module comes with two Powershell scripts,

The scripts require four variables. Since $sitecoreHostUrl, $vapidPrivateKey and $sitecoreAPIKey will be consistent across releases, they can be created as pipeline variables. VAPID Private Key and Sitecore API Key Item ID obtained in the above section needs to be assigned to $vapidPrivateKey & $sitecoreAPIKey. $sitecoreHostUrl will be the CM Domain Url. $maintenanceStartDateTime will need to be supplied as input for every release (Format: yyyy/MM/dd HH:mm EST  Ex:2021/01/18 10:00 EST). Script can be tweaked to accept deployment start date/time in another timezone if needed.

This script works with all the DevOps tools that support Powershell tasks. The initial demo uses Azure DevOps self-hosted agent to deploy to a windows machine.

Windows PowerShell(Manual):
Module includes a Windows PowerShell script (SendMaintenanceNotification(Optimized for Manual Notification).ps1), which can be used for sending/clearing notifications anytime without using DevOps tool.
Script requires deployment start date/time as input for every release, to send notifications accordingly. Sitecore CM Domain Url and VAPID Private Key & Sitecore API Key Item ID (created in the above section), needs to be updated in this script directly for $sitecoreHostUrl, $vapidPrivateKey, $sitecoreAPIKey variables.
This Script will automatically send scheduled alerts, reminders, and completion messages based on the input date/time to all subscribed users. Script will send notifications based on the availability of the site, hence do not close this powershell window till end of maintenance, else reminders and completion messages might be skipped.

The module configuration is now complete!

BACKGROUND:
This module leverages the following popular libraries/modules,

  • Service Worker – will be installed in the browser with on click of Subscribe button for receiving notifications. Service Worker is supported in the recent versions of all latest browsers
  • Web-push NPM Package – used for sending optimized Push Notifications.
  • IDB-KeyVal library served via cdnjs – a tiny js imported by Service Worker into users’ browsers for easy storing/retrieving of values in/from Browser IndexedDB (helps to avoid loss of data upon browser/system restart)

This module also relies on the below Sitecore services to keep the details centralized and editable from Sitecore without impacting security,

  • Sitecore ItemService – for fetching Notification Settings and storing User Subscription information received from Service Worker
  • Sitecore OData Item Service – for reading the Subscriptions from Sitecore during DevOps/Windows PowerShell script execution

TIPS:

  • In the event of maintenance cancellation, running the Manual PowerShell script without specifying any input will automatically clear notifications for all subscribed users
  • If the maintenance time or details change, sending the updated time or details through Windows/DevOps PowerShell will overwrite the previously sent details for all users
  • There are chances that a user might have removed the service worker while clearing browser data. In such cases, PowerShell script will throw subscription expired error with failed user’s subscription endpoint url. This won’t affect notifications for other users, but it is recommended to manually remove the respective subscription json(along with date) from /sitecore/system/Modules/PowerShell/Script Library/Maintenance Notification/Push Subscriptions item.
  • If authors/marketers will need more lead time, adding Agentless jobs with Manual Intervention or Delay tasks will be helpful to communicate well in advance and will also free up agents for other tasks during the wait time. Alternatively, manual approach can be used just for initial communication.
  • Service Worker will require localhost or https. In case if the implementation doesn’t meet this requirement, Invoke-AddWebFeatureSSLTask powershell command might help to get a quick self-signed certificate.
  • Notification Settings can be updated if/as needed in /sitecore/system/Modules/PowerShell/Script Library/Maintenance Notification/Notification Settings,

Source Code for this module is available in Github.

Voila!