CEO Corner: Analyzing the CMS Analysts: Which one is right for you?

Those of you who have been in the content management industry a while will remember the CMS Watch report. I’m not sure when I first heard of it, but I think it was at an AIIM conference in 2005. I remember thinking that if we could just get Agility into that report, we would have it made. Looking back, it’s amazing to see how many of the systems that ranked well in 2005 are either gone or no longer relevant.

A few years later, I saw an article on CMS Wire that Ektron had made it into the Gartner Magic Quadrant. I realized that it was going to be hard to complete with this level of recognition, no matter how superior Agility is.

Fast forward to 2013, and Agility is yet to be found in either of these reports. I reached out several times to CMS Watch in the early days, but we were just too small. I called Gartner in 2010 to see what it would take to get on the Magic Quadrant. They were quick to point out that Agility was probably too small, but if we paid $20,000 they would take a closer look.  Riiiight.

It struck me that it might be worthwhile to write an article on the topic to dig deeper into the options available for CMS vendors. Here are my findings.

The Competition

I decided to focus my comparison on analysts that have been active in the CMS community over the past few years. Fortunately, I was able to schedule face-to-face interviews with three of the four at CMS Expo this past May. Here is the competition in the order that I spoke with them. For more background on each, click their logo to check out their website before continuing:

Tony White    
tony 637907_ArsLogicaLogo-180 Ars Logica is an “ultra-premium, high touch firm” that offers direct access to the senior analysts. Their analysts all have at least 12 years of experience and their service is highly customizable.
     
Scott Liewehr    
scott-100 DCGlogo-100 Digital Clarity Group understands that “technology is only 1/3 of the success of a project.” They will find the appropriate agency-vendor pair that makes the most sense for your situation. They do a ton of research and writing and their 9-step selection process is the most comprehensive of the lot.
     
David Aponovich    
David-Aponovich Forrester-logo-180 Forrester understands that the buyer of technology is changing from the CIO to the CMO. They look at technology in terms of how it supports the mandate of the business and never recommend technology for technology’s sake. They are the largest of the analyst groups I talked to.
     
Irina Guseva    
irina RealStoryGroup-Logo-180 Real Story Group customers are impressed with the unparalleled level of depth and detail in their research. They are also the only analyst I talked to that has a strict vendor independence policy so that they are not unfairly influenced when they are recommending solutions to their clients.

Gartner is conspicuously absent – they did not attend CMS Expo and they declined my request to respond over email, despite a fair amount of pressing.

The Criteria

My objective was to evaluate the candidates from a vendor’s perspective – who would we want to work with? But a lot of the information I gathered – and what they all seemed most passionate about – was how they help companies decide which CMS was best for them.

Here are the five primary questions:

1) What services do you provide to CMS vendors?

2) What criteria do you evaluate vendors against?

3) What is the process for recommending a vendor to a client?

4) What are the most sought after CMS features right now?

5) What CMSs do you recommend most often?

What I Learned

First of all, I was very impressed with all of the interviews. We are fortunate to have industry analysts with incredible passion and depth of expertise.

1) What services do you provide to CMS vendors?

Favorite answer:

RSG: “None.” – I love a concise answer – but I was also disappointed because my initial purpose of this article was to focus on vendor services. However, they do come into contact with vendors via “regular updates, analyst presentations, product updates, analyst briefings, email and phone conversations on various matters, fact-check process where vendors help us make sure our evaluations are correct and industry events.”

What I learned:

There is tremendous value to CMS vendors in working with an analyst. These guys know everything about what’s going on in the industry and can help you determine where you should focus your efforts to meet the needs of customers and differentiate yourselves. Whether you’re trying to find your niche, confused about why sales have slipped, or want to make the biggest splash possible on a product launch, these guys can help.

All of the analysts, with the exception of RSG, provide these and other consulting services to vendors. Overall, I would say that Tony from Ars Logica was the most passionate about this side of the business. He’s a straight shooter who has been doing this a long time and he knows what he’s talking about. I got the sense that he really cared about the success of his vendor customers and that he would be available to talk with directly about issues and questions.

One of the things Tony mentioned which stood out is how important it is to know how to talk to an analyst – briefings are short and hard to come by – if you get the chance to do one, you want to maximize it.

2) What criteria do you evaluate vendors against?

Favorite answer:

My favorite for this one goes to David from Forrester. It seems like the Forrester vendor analysis process is the most systematic of the bunch. They split evaluation criteria into three buckets: manage, engage and measure. Manage is the basics – creating, managing and publishing content with a workflow. Engage is the more advanced stuff – personalization, contextualization, multivariate testing, and other digital experience elements. And Measure is the data side – analytics and optimizations.

What I learned:

All of the analysts talked a lot about integration. David in particular seemed to favour the best-of-breed approach over the all-in-one suite with his comment "WCM is more about integration with other tools and technology, it's not about WCM as a single suite". Whereas Scott seemed to refer to best-of-breed and all-in-one options with equal frequency.

I was also happy to hear that cloud is coming up more in the CMS space.

3) What is the process for recommending a vendor to a client?

Favorite answer:

All of the analysts were very passionate about their answers to this question. They all  have a lot of depth in this area and well thought out processes. One of the things I really liked about the DCG approach is that vendors are invited in to meet the client before the request for proposal (RFP) goes out. This gives the vendor a chance to learn about the company, the team, how they work and what their priorities are. As someone who has bid on many CMS RFPs, this has tremendous value for everyone involved. It’s hard to know how to position a proposal without a good understanding of the team and capabilities on the other side.

Another thing I liked about the DCG approach was how much they focus on the implementation and the agency who does it. They understand that the only way you’ll know if you can work well with an agency is if you try it. So, they set up a two week paid engagement with the agency to test the waters, which also makes it much easier for them to accurately scope out the main project.

What I learned:

The primary service for all of the analysts I talked to is recommending vendors and implementation teams to clients. And they are all very good at it. I was expecting one of the four to lean more heavily on the vendor side, but it is clear that all of them are acting in the best interests of their clients. RSG seems to be particularly committed to offering an unbiased opinion of the vendors they recommend. They refuse to be wined and dined by vendors for fear of being influenced and there is no way for a vendor to buy their way onto any RSG reports.

4) What are the most sought after CMS features right now?

Favorite answer:

David from Forrester perked up with this question. He seemed most in his element when talking about the new and popular features in the space, especially those around “digital experience” – a phrase he used in his Forrester blog post about the Wave Report.

What I learned:

There are a few primary things trending in CMS features. The obvious things like workflow and ease-of-use are still important. Ease of integration is becoming more of a concern with the “best-of-breed” approach. Remember that these guys all know more than just CMS – they are well versed in marketing automation, analytics, and all of the pieces that make up the digital marketing toolset. And finally, “digital experience” features like personalization, contextualization, regionalization and multi-channel publishing are trending.

5) What CMSs do you recommend most often?

Favorite answer:

Scott from DCG provided my favorite answer to this one. Whereas the others were cautious about naming names, Scott came out and told me who comes up the most. He mentioned that Sitecore and Adobe have been coming up a lot in conversations – no surprise there. On the open source side, he mentioned Hippo, Magnolia (for a best-of-breed solution) and eZ Systems (for an all-in-one solution).

What I learned:

All of the analysts track 30 or more vendors, which is more than I expected. In my experience, there are only a handful of companies that we come up against in bids on a regular basis, so this was unexpected. It just goes to show how rich our industry is and how deep the analyst knowledge goes.

The Surprises

I was surprised to find out that Irina was the first of the lot to cover Agility back in 2008 when she was at CMS Wire. I remember how excited I was to be mentioned, but I didn’t realize it was her until our interview.

I was surprised that Gartner “didn’t have time” for an interview and refused to even respond to the questionnaire over email – yet they have no problem sending their sales team after me month after month. That’s disappointing.

I’ll admit, I was a little surprised at how impressive all of the analysts were. These folks know their stuff!

My Conclusions

Honestly, you can’t go wrong working with any of the four analysts I talked to. The best fit for you will depend on your objectives, budget and geography. For Agility, my shortlist would include Ars Logica and Digital Clarity Group because I like working with smaller companies that provide the access to the senior people. I would give a slight edge to DCG because of three factors:

1) They have a presence in Toronto (where Agility is based)

2) I have known Scott for a few years and he has helped raise our profile by offering me speaking opportunities at the Gilbane and CMS Expo conference

3) I feel like they are the most involved in the community and have given so much back including the CMS Connected show, speaking at events and providing useful resources

Analysts Are Extremely Valuable

The most profound conclusion I came to was that if you are a company who is looking for a new CMS, analyst services are invaluable. There are so many variables involved in a CMS selection process and so many ways that a project can go wrong.  It is a worthwhile investment to engage these experts to help guide the way. And this is my totally unbiased opinion since Agility has yet to work with any of these companies.