Push Publishing Best Practices and Troubleshooting

Push Publishing is a sophisticated feature, and while very sophisticated Push Publishing architectures can be used, the more complex your Push Publishing architecture, the more potential there is for Push Publishing conflicts.

This document provides a number of recommended best practices when configuring Push Publishing. Especially if you are new to dotCMS or Push Publishing, it is recommended that you read and follow these practices initially, and consider deviating from them only after you've gotten your Push Publishing connection working well.

Best Practices

Push Publishing facilitates user-acceptance testing (UAT) arrangements by accommodating separate servers for content creation and consumption — i.e., authoring and production. As the public-facing version, the production server's state and content should be guarded against unexpected behaviors or state changes. Enforcing strict procedures on unidirectional, univocal, and intentional data movement can help to harden a production server against such outcomes. Care should also be taken with security and technical considerations.

The following guidelines aim to substantially address these concerns.

1. Allow Push Publishing in One Direction Only

Push Publishing is simplest and easiest to manage when servers which can push content can not also receive content from the server(s) they send to. Allowing content to be pushed both directions creates complications, both because it makes it possible for users on each server to overwrite changes which were pushed to the server, and makes it difficult to track which content is the “source” content and which is the copy.

More sophisticated Push Publishing configurations are possible. For example, it is possible to have multiple servers, each of which acts as the “source” for certain types of content and objects, but receives other types of content and objects from the other servers. This configuration is much more challenging to set up and maintain, however, and more prone to having both integrity conflicts and accidental content overwrites. If you wish to create a more sophisticated configuration of this type, it is recommended that you:

• Contact dotCMS Support for recommendations, and
• Consider using Push Publishing Filters to enforce the rules for which content is allowed to be pushed from each server to the other servers.

2. Disallow Direct Changes to Content on the Receiver

One of the greatest sources of Push Publishing conflicts is when content which has been pushed from a sending server is edited on the receiving server. This both creates versions of content that exist on only the receiving server, and makes it very easy for these versions to be overwritten from the sending server, and to cause integrity conflicts between the servers.

To avoid these problems, it is recommended that whenever possible, you do not allow content which is sent from one server to another to be edited on the receiving server. This includes additions, changes to, removal, or archiving of content or objects.

Key considerations:

• You should perform all changes to the receiving server only by pushing from the sending server.
• The simplest way to do this is to strictly limit access to the receiving server.
  • Ensure that the user list on the production server is extremely restricted.
  • Users with access to edit content on the sender should not automatically have access to edit content on the receiver.
  • Do not implement SSO on the receiving/production server.
• Some access to the back end on a production server is still required for upper-level administrators.
  • This is purely for the sake of system maintenance, which is independent of content.
  • Content should still not be edited directly on the receiver, even by administrators.

3. Use Push Publishing Filters to Prevent Accidental Pushes

Push Publishing is intentionally designed to automatically send objects which you do not explicitly include in the push. For example, when you send a Page, and the Template the Page uses has been updated, the Template will also automatically be included as a dependency, to ensure that the Page displays the same way on the receiver that it displays on the sender.

There may be times, however, when you do not want dependencies to be included in the push. For example, if you have users performing both development and content editing on the same sender, you may wish to prevent a push of content (managed by the content team) from also causing development objects (managed by the development team) to be pushed.

You can use Push Publishing Filters to prevent pushes of one type of content from causing other types of content or objects to be pushed — such as preventing a push of normal content from causing development objects from being pushed.

This can be done by:

• Creating filters which exclude certain types of content or objects from being pushed (such as Content Types, Templates, and Containers).
• Setting permissions on the filters so that users who should not be able to push these types of content and object only have access to the filters which exclude them.

4. Monitor Cache Eviction Data

Building and pushing large bundles requires adequate server resources. Having a healthy server with enough RAM and CPU processing power, and ensuring cache region sizes are tuned well, will help prevent problems with Push Publishing reliability and performance.

It is recommended that you monitor the eviction data in the Cache tab of the Control Panel → Maintenance tool regularly, especially when first setting up your Push Publishing connection between two servers, or when the size or volume of your pushes increases.

5. Synchronize Categories Before a Push

When pushing content which contains Category fields which may have changed, it is recommended that, before you push the content, you push Categories from the Category Manager.

When you push individual content items which have Category values, the specific Category values used by those content items will be pushed, but any other Category changes will not be pushed. By pushing all Categories, you ensure that any code which iterates through or displays Categories (such as pages which list Categories as search filters) are properly updated.

6. Prevent Overwrite of Workflows on the Receiver

Push Publishing always pushes Workflows from the sender to the receiver, even if only a single content item is pushed. This means that, to prevent Workflow on the receiver from being overwritten, you must choose one of two options:

• A. Do not modify Workflows or Workflow tasks on the receiver.
  • Ensure that users never create or modify Workflows or Workflow tasks on any dotCMS instance which is configured to receive pushed content from another server.
• B. Prevent all Workflows from being pushed.
  • If you wish to have different workflows on the sender and receiver, you can use Push Publishing Filter to prevent Workflows from being pushed to the receiver.
  • However for this to work, all Push Publishing Filters on the sender must include Workflow in the excludeDependencyClasses property of the filter.
    • If even one filter does not include this limitation, then a single use of that filter will cause the Workflow on the receiver to be overwritten.

7. Manage the Size of the Bundles Folder

If you push frequently, or push large amounts of content, your bundles may take up significant disk space on both the sending server and receiving server. It is important, then, to be aware of how much space your bundles are taking, and ensure that the bundles do not take up enough space to cause your disk to run low on (or run out of) space.

7a. Monitor the Bundles Folder Size

It is recommended that you monitor the size of the bundles folder during normal operation, and the available disk space on your server, to ensure that the bundles folder does not grow large enough to risk causing your server to run out of disk space. You may wish to increase the time that bundles are kept on the server when you are making significant changes to your server configuration or push publishing configuration (to enable troubleshooting if necessary), and reduce the time once your push publishing functionality is stable.

7b. Adjust the Length of Time Bundles Are Stored

Bundles are created and stored in the bundles folder on both the server and the receiver. This allows you to download and view bundles from previous pushes, and to re-try bundles which may have failed due to temporary problems (such as network connectivity issues).

By default, bundles are automatically deleted two days after the push has completed (whether the push was successful or not). You can adjust how long bundles are kept on each server by modifying the bundles.delete.older.than.milliseconds property in the dotmarketing-config.properties file.

// delete bundles older than two days
bundles.delete.older.than.milliseconds=1000 * 60 * 60 * 24 * 2

You may also disable deletion of old bundles completely by setting the bundles.delete.enabled property to false. However disabling this feature is not recommended unless you already have an automated process to manage the size of the bundles folder; the size of the bundles folder will continue to grow, and can grow large enough to cause the server to run out of disk space — which will cause the dotCMS process to stop and take all your dotCMS sites off-line.

bundles.delete.enabled=true

8. Securing Transmission Channels via WAF

Security is an everpresent concern for virtually any server. We recommend perusing our security best practices for many general pointers. These include disabling unused ports and configuring XSS prevention filters.

In the case of Push Publishing, the topic of whitelisting is of particular importance: By using a web-application firewall (WAF), you can restrict traffic to the receiving server to specific IPs and ports. If the production server will only accept modification from the authoring server, then this effectively rules out a wide variety of attempts to intrude, spoof, or otherwise interfere with your servers.

It is also possible, as a further restriction, to configure your servers to allow back-end access only through a VPN.

Note that the exact requirements of this configuration will vary according to the implementation. For example, a headless implementation will still need more permissive API-level access. For more context-aware recommendations, contact dotCMS Support.

Correcting or Reversing an Unwanted Push

If you make a mistake or wish to reverse a push of content, you can make corrections using one of the following methods (for both Dynamic Endpoints and Static Endpoints):

• Edit the object(s) and push again.
  • Note: You can revert content to a previous version from the Content History tab.
• Perform a Push to Delete to remove the object from the Endpoint.

Important Notes:

• You should NOT manually unpublish or revert the changes directly on the Endpoint.
  • Doing this can cause conflicts that will prevent push publishing from working correctly in the future.
• You can only perform a Push to Delete if the object you wish to remove from the Endpoint still exists on the sending server.
  • If you have deleted an object, there is no object to push, so you can not perform a Push to Delete.
• Using a Push to Delete removes the selected object(s) from the Endpoint.
  • If you wish to revert to the version of the content that existed before the original push, you must edit the object(s) and push again, instead of using Push to Delete.

Troubleshooting

Forcing Items to be Pushed

By default, when you push content from a sender to a receiver, dotCMS limits network bandwidth by automatically excluding any content which was previously pushed from that server to that receiver. dotCMS keeps a timestamp of when each content item and object on the sender was last pushed to that specific receiver, and will only include items which has changed since it was last pushed (the modification date of the item is more recent than the timestamp of when that item was last pushed).

In some cases — especially if you allow content to be edited on the receiver directly — this can cause the sender to think that an item was already pushed to a receiver, and not send it when you want it to (even if you explicitly select that item to be pushed).

You can force dotCMS to push content which the sender has marked as already having been pushed in 3 different ways:

1. Force Push
   • This makes dotCMS ignore the comparison between the last modification date and the last time an object was pushed — which forces dotCMS to include all the required data dependencies in the bundle.
   • This is very useful when you want to overwrite a Site, or the entire contents of a folders.
     • However, this can generate very large bundles (which requires a lot of processing power), and must be carried out very carefully when including large numbers of items (such as a whole Site).
   • Note that you can perform a “limited” Force Push by combining this feature with Push Publishing Filters.
     • For more information, please see Limiting Pushes of Unpushed Content, below.
2. Delete the Push History for the content
   • As described here: https://dotcms.com/docs/latest/push-publishing-content#ContentPushHistory you can just remove the push history of the objects you want to refresh.
   • This can be considered a smaller and more careful version of the Force Push option, because you can choose individual content items to force to be pushed, rather than allowing dotCMS to automatically force push all dependencies.
   • This can be useful when you have specific items that you want to ensure are updated on the receiver, but you don't want to do a “blanket” force push.
3. Delete the Push History for the entire system
   • Ultimately, as described here: https://dotcms.com/docs/latest/push-publishing-content#SystemPushHistory you can delete absolutely all the Push Publishing history.
   • When you do this, it returns dotCMS to the state it was when you first create your Push Publishing environment.
     • The sender will act as if no content has ever been sent to the receiver, so it will push all content and dependencies.
   • This can be used as a last resort if you have Push Publishing conflicts that you can not resolve with the other two methods, above.

Limiting Pushes of Unpushed Content

The Force Push operation may sometimes push a very large amount of content by dependency. One way you can limit the amount of content included with a Force Push is by using Push Publishing Filters to limit what content is pushed by dependency when you perform the push.

For example, when you push a Page, a number of dependencies are checked, including:

• The folder the Page is located in.
• The Template used by the Page.
• All Containers on the Page (either within the Template, or added to the Page in the Page layout).
• All content added to the Page.
• Any content which is related to the Page, or to any content which was added to the Page.

In addition, when you push the folder the Page is located in, it's dependencies will also be checked, so it's parent folder will be included by dependency, and this will continue all the way up to the Site.

This means that when you perform a Force Push of the Page, you can end up forcing a very large amount of content and objects to be pushed, many of which have nothing to do with the Page.

However you can use Push Publishing Filters to perform a more targeted Force Push. For example, you could create a filter which excludes folders and Sites, to ensure that only content directly related to the Page is included, even when you perform a Force Push.

Use Small Bundles

Using smaller bundles when running into problems with Push Publishing can help you in several ways:

The size of the bundle is smaller, so it will be faster to process it. For example, instead of pushing the whole Site, try to push one root folder at a time.

The first bundle you push (after clearing your Push Publishing History, for example) will bring in a lot of dependencies, just one time. So, when you push more information the next time, the other bundles will be smaller.

If you “split” your contents into bundles – data groups – it'll be easier for you to pinpoint the content or folder that is causing problems. This is a great way to troubleshoot and isolate problems, we use it frequently. So, you can push almost all of information of a Site and work with that, while we diagnose the specific problem.