Multivariate testing Sitecore DMS and beyond

Testing and optimizing your web site's messaging and appearance is essential to your success and there are a large number of tools available for this purpose.

When working with Sitecore DMS you furthermore have the ability to use Sitecore's built-in multi-variate testing features as a very viable alternative. Note that the purpose of Sitecore DMS is not - and has never been - to offer best-of-breed functionality in every area covered by DMS, but instead to offer solid already-integrated functionality in most areas with the potential for extending and adapting exactly to the client's needs while maintaining the transparency from having everything in Sitecore.

While the built-in Multi-variate testing is solid, it is also one of the areas where the span between DMS and the best-of-breed application is significant and therefore room for improving upon DMS standard behavior.

First of all - let's be clear on the question "Should we use DMS and customize it or find a 3rd party best-of-breed tool?".

The answer to this will depend on the context, but in most cases my recommendation would be to go with DMS or, if the client already has a preferred tool, start with this tool but migrate to DMS.

In my opinion the long term benefits of utilizing DMS for Multi-variant testing are worth the investment in customizations and should decide the approach. And - often there definitely is a need to customize DMS too meet the client's needs on this area, so let's take a look on the most common challenges XCentium have met - and helped overcome - with our clients:

• Cause-effect tracking: While Sitecore track MV testing results using the engagement value, which only allows you to see results across full visits, most of our clients want to track the performance of individual components and the direct effect (what makes the visitor click). For this we typicallt implement click tracking on top of DMS.
• Traffic distribution: To allow for controlling how much traffic is directed to individual MV test variants, if the client only wants to test on a subset of the visitor population but keep most of the traffic untouched, it is necessary to implement traffic distribution to e.g. only test on 20% of the population.
• Routing: In certain cases the client wants to control the visitor's experience throughout the visit (or even across visits) based upon the variants presented in MV tests. As Sitecore's MV testing is main oriented towards the individual page, a routing mechanism is needed to control the site's behavior past the MV test.
• MV testing across pages: In some cases the client wants to control - and see results for - combinations of tests across multiple pages. The control part is often a variant of the Routing described above and seeing the results requires customized reporting based on the logic implemented for the site.
• Segmented testing: DMS does not currently support combining MV testing and personalization, which is necessary to segment testing and e.g. only test users coming from Google or when wanting to test different variants for different segments.
• Control center: Sitecore's MV testing is page based and doesn't allow the client to get an overview of the tests currently running on the site - and how they perform. For small sites this may not be a problem, but for a 10,000 page web site with many running MV testsa custom control center app is necessary.
• Automatic Selection: To avoid having to monitor the running MV tests manually, an automated approach to selecting winners and disabling poorly performing variants is often necessary. Typically this is implemented by selecting thresholds for e.g. exposures, differences in success rates and lowest acceptable performance and automatically manage the tests to ensure the best variants are chosen and the worst are eliminated as quickly as possible.

Should you have some of the above requirements for your MV testing and don't see them covered out-of-the-box with DMS, don't just select a 3rd party tool, which will often be disconnected from DMS and the rest of your site. We have lots of experience with these kinds of issues and can help you maximize your DMS value.