Magic Quadrant Season
The Gartner Magic Quadrant (MQ) is the most influential analyst report in the UC industry. Actually in many industries. Gartner is an MQ machine (something like 700 of them). In the enterprise communications space, there are four relevant MQs: UC, UCaaS, Corporate Telephony, and Contact Center. Although arguably a continuous process, each MQ has its own cycle with an annual start and finish typically about 4-6 months each. The UC, UCaaS, and Corporate Telephony MQs are currently underway.
I’ve had some exposure to the process, so I thought I would attempt to correct a few misconceptions about MQ reports. I am not, nor ever have been, a Gartner employee.
Is the MQ Pay to Play?
Gartner has monetized the MQ in numerous and impressive ways, but not in this way. When an analyst loses his/her objectivity it is game over. Even if one analyst was biased, the internal peer-review process at Gartner would likely catch it. However, as in politics, money can buy additional exposure – Gartner clients can meet with Gartner analysts more readily than non-clients and they have access to analyst content that reveals current thinking.
Do the MQ Ratings Reflect Vendor Improvement?
There are two issues here: First, the MQ grid and reports are about relative strengths and improvements. How well a given company did in the past year is only half the battle – the other half is how well everyone else did – sometimes doing great isn’t good enough. Secondly, since products and services get better over time, the MQ adapts by evolving the evaluation criteria. A report from two years ago was based on criteria from two years ago – undoubtedly different than the criteria used today. For example, the 2014 UCaaS MQ now requires international operations. A company in in the 2013 MQ may not even be eligible for the 2014 MQ, but that doesn’t mean lack of improvement.
Are MQs Useful in Comparing Companies over Years?
It might be interesting to see how certain competitors have moved closer or further apart, but comparing MQs year-to-year can be misleading. First, as mentioned above, the criteria and weightings (which are not published) change. The bigger mistake is comparing a single company year to year (company X moved 3 mil to the left) because the scales are not consistent. Sometimes the dots end up pretty close to each other, so Gartner adjusts the zoom factor to exaggerate the differences. The zoom is subjective, and could determine a firm’s final quadrant placement.
Does Gartner Get it Right?
The thing to remember is that the MQ, like all analyst reports, is an opinion. It’s an important and informed opinion, but still an opinion. I have heard about mistakes Gartner made such as missing signs of the Nortel bankruptcy – no idea if these are true or not (but it doesn’t matter). Of course Gartner sometimes gets it wrong. However, they also get a lot of things right. We all rely on expert opinions – financial advisers, doctors, even product reviews on Amazon, etc. Expert opinions are useful, but rarely come with guarantees. It’s up to the reader to 1) understand the opinion, and 2) balance it with additional research. The bigger question is what did Gartner evaluate and weight because no solution is ideal for all situations. Gartner lists their broad criteria, and it may or may not align with yours.
Is Gartner’s Questionnaire Sufficient for this Research?
I mentioned above that there’s a 4-6 month MQ cycle. Within it, in UC, is a questionnaire that is sent to the vendors – think of it is as an RFI. The UC questionnaire was about 30 short answer questions – so short that it is difficult to be thorough. That may seem unreasonable, but there’s more to the story. First, the questionnaire is only a part of the research. Gartner analysts, like all of us, use lots of resources in their research. Secondly, the questionnaire isn’t even necessary. Gartner analysts have many venues to get the information they need – including lots of end user clients. If a Gartner analyst solicits information for the MQ, it was his/her choice to do so as it is not required by Gartner. Short responses are frustrating, but at least they are an opportunity for the vendor to participate. The key point that many vendors don’t realize is the MQ is not a 4-6 month process. The research evaluates year-round content.
Should Only “Leaders” be Considered When Searching for a New Vendor?
The MQ has four quadrants, and the most magical of them is the upper right known as the Leaders quadrant. This is the quadrant reserved for companies (note: not products) that according to Gartner have the most complete vision and ability to execute. If you ask a Gartner analyst which vendors to evaluate for a given project, the short answer is the ones in the Leaders quadrant. But short answers have their limitations. Keep in mind MQs are very broad, remember there is only one MQ per market. Since there is only one UC MQ report published each year for the world, it is highly generalized. One of the factors considered is channel effectiveness – the effectiveness of channels often vary by global region – even state. It may not help you if the vendor has a highly effective channel in every other market than yours. This reminds me of how I used to throw out resumes that had a typo. That could make sense if I was hiring a proof-reader, but as that was not the case it was a bit arbitrary. I may have skipped over the perfect candidate – that’s the risk of filters, and there’s plenty of great firms not in the Leaders quadrant.
Do MQs Only Include Gartner Clients?
Being a client has no bearing on which vendors get included or excluded. The only factor is the inclusion criteria which are specified in each report. If a vendor meets those criteria, they are included – regardless if they are clients, or even potentially against their will. The reality is that most vendors want to be in the MQ, and more likely to be arguing that they meet the inclusion criteria then trying to get out of the report. So there’s no need to drop $40k or so to become a Gartner client, unless of course you value their opinions and content.
Do the Vendors See the Report Before it is Published?
Before Gartner publishes an MQ, it gives each vendor a 5 day preview of the new graphic and the portion of the report dedicated to that vendor.
How Does Gartner Effectively Test all These Products?
That’s easy – they don’t. Gartner does not perform product evaluations. They rely on third party sources – such as customers – to determine if a product or service performs as promised. This is more thorough than one might expect, remember Gartner MQs typically have a high bar revenue requirement. It’s not hard to find enterprise customers that are anxious to talk about a vendor’s performance.
Why is the Gartner MQ so Influential?
I can’t defend it, but I can explain it. Being favorably positioned in the MQ can be worth billions. It legitimizes a vendor offering even if the customer has never heard of that vendor. It opens up doors to more deals, it can increase valuations as well as M&A activity. The MQ is a simple way for enterprise organizations to outsource learning an industry. It’s a bit ridiculous, but I guess I rarely watch movies I’ve never heard of that only have a single star on the guide.
Is the MQ based on Objective data or Subjective data?
Both. The problem is there just isn’t much objective data out there. Take something as simple as sales revenue and you can see the complexity. As the industry moves away from hardware toward recurring services, revenue is changing and difficult to compare. Licenses are hard to count due to bundling. With the MQ, they are “measuring” completeness of vision – how do you compare the vision of multiple companies? One might think ability to execute is objective based on results, but that’s not so simple either. Gartner also looks at things like customer experience and marketing effectiveness for execution.
Are there MQ Alternatives?
Yes. There are lots of analysts out there, and all of them have opinions. Several analyst firms do multi-vendor UC reports. We are working on one now at Wainhouse Research.