Monday, January 12, 2009

The tough task of operationalizing

The WebTrends blog (http://blog.webtrends.com) asked "How do you define success?" this morning.

Yes, that’s THE question, and as Deborah (the author) said, the answer is usually pretty broad, just a starting point. More traffic, more sales, more customer engagement (ugh) are typical answers.

The FOLLOWUP question is, for me, a lot more difficult. The conversation usually branches after the first question, into 1) “Okay, but what does that mean?” or 2) “Well, aside from that, what kinds of things indicate partial success, or probable future success?”. And the one that seems to be the killer: "Which (of these two measures that we've worked out in the past hour) is more important?"

Getting people to operationalize, i.e. turn their generalities into specifics that are fairly finite and objective, is challenging and requires a really experienced person in the room (hopefully it's the metrics person). For a metrics person, the task is to turn a big generality into one or more littler, more objective ones, and continue that process until something pops out that can somehow be turned into reliable data and quantified. Sometimes the "data" version is right at the top level. Sometimes it's several levels removed. Sometimes it's nowhere. And often (another topic) it leads to surveys (because it's really just opinions) or crossing to other channels.

Surprisingly often, these “define success” working sessions for web analytics are the first time that the people in the room have actually had to work through, in their own minds, what they are after and how they’ll know it when they see it. Seriously. It’s amazing to me how people can do their jobs without operationalizing their goals, but they do it every day, all day, and seem successful.

The flip side of operationalizing is living with ambiguity, which is a valuable skill in itself. The curse of a mentality that likes to operationalize is the danger of being compelled to operationalize. "What do you mean by 'I love you'?"

One reason I like being in analytics is because the metrics person is often the one who pushes others to do this kind of mental exercise. In fact, I’ve seen metrics people labeled as “pests” about it (by those who are comfortable with ambiguity) … eventually accompanied by “that session was hard but it was incredibly helpful and productive.”

It’s pretty neat when, in one of these sessions, we reach a point where the operationalized objectives start to easily turn into actions … “we need to do more of such and such because it’s clearly related to an important objective” … “we have to think about whether we should be doing such and such because its relationship to the objectives is really pretty thin; it's cool but coolness isn’t enough.” Or, simply, "we won't be able to measure that unless we do such and such." It is NOT easy to get to this point in the discussion.

I dislike, I mean REALLY dislike, multi-syllabic jargon but “operationalize” is one to keep.