Got Analytics?

Google Analytics Now Has Word Clouds!

2011-03-20T10:39:00.001-07:00

Word clouds. Still around, are they? They need to either improve or fade away. Here are my top six reasons for why this is Google pandering to a perceived need rather than acting as an analytics leader.

1. The size of type does nothing that a quantity-sorted list doesn't do

2. Adjacency means almost nothing. What good is alphabetic sorting?

3. Color (when used) means nothing; it's just for dazzle.

4. They irritate smart executives who know they are gratuitous and who lose respect for the analyst

5. They dazzle not-so-smart executives who then expect you to find tremendous insights in them

6. They frustrate good analysts who would rather use that big chunk of real estate for something truly worthy

A word cloud has several available dimensions, in other words --- size of type, location within the cloud, color, and color intensity. A good word cloud would use all of these dimensions but current word clouds use just one. Location (adjacency) could represent either similarity or co-occurrence. Color could represent a typology (problem words vs solution words, or whatever fits into your strategy). Color intensity could represent something quantitative, such as pages per visit.

I don't know of any word cloud generator that uses more than font size, do you? Please slam me if I have missed something here.

Google absolutely has the in-house talent to make word clouds into something better. What a shame they didn't bother.

What, exactly, do you consider a SEARCH ENGINE?

2011-01-30T11:09:00.000-08:00

The question above is just me being curious.

Consider this list:

Amazon
Facebook
EBay
ShopZilla
Info.com
YellowPages.com
Bizrate
Target.com
Comcast.net

Would you say the above are all "search engines"?

Do you want them in your search engines reports, or kept out?

Do you want to be paying for text ads on them as part of your paid search program, or not?

All of them have a search function. They may display results from their own site, or paid search ads, or organic results from the internet at large. Or all three.

Some of them are explicitly counted as "search engines" by SiteCatalyst, WebTrends or Google Analytics. Some of them are not counted as search engines but you and I probably would consider them as such.

In short, the search engines reports of our web analytics tools are messy because "search engine" just isn't easily definable these days.

Please think back to your thought process as you scanned my little list. Would you say you have mental rules about what is and what isn't a search engine? What are your rules?

Better yet, would you say you have mental buckets for these sites that aren't just yes/no for being a search engine? I, personally find it useful to chop up all these referring sites into a few groups. The following are the biggest groups:

Really-truly search engines, where people go to search the whole internet. I think we'd all agree that Google is one.
Shopping search engines, which also have value as price/feature comparison engines. ShopZilla and Nextag qualify for this.
Primarily within-site search engines that happen to show related paid content. Target.com is definitely in this category ... the paid results may lead you to buy something at another store, but Target seems willing to take that risk. Same for Amazon. They've probably done the cost-benefit math.
Same thing as the above, but you have to submit an xml file to get listed. Froogle comes to mind, may it rest in peace.
Sites with a whole other purpose, of which search is a tiny part. (Have you ever noticed that if you do a search on Facebook, below all the FB-related hits is a set of organic results powered by Bing?)

Getting analytics tools to give me results according to my subcategories is not easy. As far as they're concerned, a domain is a search engine, or it is not.

But I would claim that my subcategories give me more information about my audience, and my input into my marketing activities, than the simple search-engine unichotomy.

The tough task of operationalizing

2009-01-12T06:47:00.000-08:00

The WebTrends blog (http://blog.webtrends.com) asked "How do you define success?" this morning.

Yes, that’s THE question, and as Deborah (the author) said, the answer is usually pretty broad, just a starting point. More traffic, more sales, more customer engagement (ugh) are typical answers.

The FOLLOWUP question is, for me, a lot more difficult. The conversation usually branches after the first question, into 1) “Okay, but what does that mean?” or 2) “Well, aside from that, what kinds of things indicate partial success, or probable future success?”. And the one that seems to be the killer: "Which (of these two measures that we've worked out in the past hour) is more important?"

Getting people to operationalize, i.e. turn their generalities into specifics that are fairly finite and objective, is challenging and requires a really experienced person in the room (hopefully it's the metrics person). For a metrics person, the task is to turn a big generality into one or more littler, more objective ones, and continue that process until something pops out that can somehow be turned into reliable data and quantified. Sometimes the "data" version is right at the top level. Sometimes it's several levels removed. Sometimes it's nowhere. And often (another topic) it leads to surveys (because it's really just opinions) or crossing to other channels.

Surprisingly often, these “define success” working sessions for web analytics are the first time that the people in the room have actually had to work through, in their own minds, what they are after and how they’ll know it when they see it. Seriously. It’s amazing to me how people can do their jobs without operationalizing their goals, but they do it every day, all day, and seem successful.

The flip side of operationalizing is living with ambiguity, which is a valuable skill in itself. The curse of a mentality that likes to operationalize is the danger of being compelled to operationalize. "What do you mean by 'I love you'?"

One reason I like being in analytics is because the metrics person is often the one who pushes others to do this kind of mental exercise. In fact, I’ve seen metrics people labeled as “pests” about it (by those who are comfortable with ambiguity) … eventually accompanied by “that session was hard but it was incredibly helpful and productive.”

It’s pretty neat when, in one of these sessions, we reach a point where the operationalized objectives start to easily turn into actions … “we need to do more of such and such because it’s clearly related to an important objective” … “we have to think about whether we should be doing such and such because its relationship to the objectives is really pretty thin; it's cool but coolness isn’t enough.” Or, simply, "we won't be able to measure that unless we do such and such." It is NOT easy to get to this point in the discussion.

I dislike, I mean REALLY dislike, multi-syllabic jargon but “operationalize” is one to keep.

The few blogs I like

2008-10-24T06:46:00.000-07:00

I have almost a hundred analytics-related blogs in my reader and I'm realizing there are only a handful that I refer people to, usually for individual posts. So I'm going to list them here.

http://www.mymotech.com/
the MymoTech blog - I have no idea what "mymo" means but this blogger, Michael Helbling, is awfully good at telling it like it is ... when he actually posts, that is. His description of his experience with the TCO (Total Cost of Operation) of Google Analytics is one of the few done anywhere and it lays it out beautifully.

http://thebigintegration.com/
The Big Integration - one of two blogs by Jacques Warren, who consistently seems to add something to the general conversation. I give his blogs points for a fresh perspective almost every time. In other words, his writing tends to yank my leash a little bit.

http://webtrendsoutsider.com/
The WebTrends Outsider - written by a group of non-WT employees who act like product insiders. Clearly there is a ton of WT experience here, an ability to think outside the WT box, and some kind of crazy urge to help people. And a little bit of poking at WebTrends. Somebody's gotta do that.

http://semphonic.blogs.com/semangel/
Gary Angel's blog. The postings tend to be really long and multi-parted. But he's the most original thinker in analytics, IMHO. Not because he's coming up with original ideas (he does that), but because he pulls existing concepts and thoughts from other fields and gets them to work with web analytics. His fault, if any, is in not referring to those other fields and therefore maintaining our delusion that we are doing something special and unique. I think a lot of web analytics people are bottom-up --- they migrated into analytics and flourished there. I have a feeling that Gary Angel is one of those people who was already flourishing in other, bigger areas and migrated to analytics where he overlays intelligent adaptations of what he has absorbed elsewhere. It takes a special kind of mind to be able to do that. I save his blog for when I have the time to read it more than once. It makes me walk around to other people's offices at work saying "hey, you know, we oughta ..."

http://www.bruceclay.com/blog/archives/analytics/
The Bruce Clay Inc. analytics blog. When I grow up I want to work there, because they probably know everything. If I can keep up with their blogs (analytics, search stuff ...) then I feel like I have a chance of keeping up, period.

http://blog.webanalyticsdemystified.com/weblog/
Eric Peterson's blog. Eric has two things going for him that put his blog on this list. 1) He's very very smart which means he thinks and writes really well. 2) He's a true insider in the industry, possibly even "an elder statesman" and deservedly so. #1 is good enough all by itself. #2 adds the spice - early knowledge, the confidence to make some controversy (in a good way), and a great ability to see things with seasoned perspective. His recent take on Google Analytics is a perfect example.

http://www.minethatdata.blogspot.com/
Kevin Hillstrom's blog. He comes out of the catalog world and, please believe me in this, the catalog world is incredibly important, and different. And old. He knows data and selling and writes wonderfully and he doesn't pull punches. My new go-here-first blog as of February 2009.

Google Trends for web sites? No thanks

2008-08-24T10:10:00.000-07:00

Here's why I stay away from this potential source of great competitive information, at least for now: I don't trust it at all.

I compared GTfWS data to our own data for the most prominent site for which I have solid web traffic info (but not Google Analytics). I analyzed seven months of our data, looking for a week-by-week stat, any stat, that would produce a trend line that resembles what GTfWS produced for the same period, same site. Unh-unh. Nope. No way. Zip. Nada. Neither the trend line or the quantities resembled each other. No stat I tried matched, certainly not the particular stat that Google Trends says it's showing - Daily Unique Visitors (per week).

I don't know from where they get their estimates. I know I've seen other people's similar analyses that corresponded very well - but they are sites that use Google Analytics. I know that my Google Toolbar connects to its home base every time I click on a page, even though I have "Send usage statistics to Google" turned off in the settings. Maybe these are related. Don't know, don't care. For competitive data, I'm staying away.

One True Thing

2008-02-21T08:20:00.000-08:00

If you know me personally at all, you know that I have more fun on my job than a person has a right to expect. A lot of the reasons boil down to: talented and conscientious coworkers; smart and ethical management; a tool that lets me stretch my web-analytics legs; and my own quirky personality that's addicted to getting insights from quantification in general.

There's another piece - the clients and the fact that, as an agency, we have a variety of them.

This means we also have a variety of types of relationships with those clients in the analytics corner of the business. Our "deliverables" run the gamut:

setting up their reporting tool and letting them look at the tool's output (with ongoing help as needed),
delivering Excel scorecards of varying degrees of detail that allow them to skip looking at the tool's output,
delivering the above plus written interpretation or summary or recommendations,
deep-dives into a single, one-time analysis,

and my favorite:

the free-ranging relationship that has come to be known here as "One True Thing" reporting.

I want to talk about the last one because it's just too enjoyable to keep to myself.

The name "One True Thing" comes from a smart and creative client person with a limited budget who wants it all. He appreciates that the detailed scorecard is important but wants to skip the other time-consuming deliverables while still getting at important stuff that he can actually use, whether to influence decisions or impress his organization. (Hey, both of these are really important!)

He said, "If I can get just One True Thing every month out of analytics, something that's interesting and helpful, then the whole effort is worth the money we put into it."

What resulted was very simple. We have a four-person phone call every month, a few days after the detailed scorecard is delivered. By the time of the call, he's looked at the data we send and may have questions or head-scratching puzzlers. He and his site manager also talk about conundrums they're dealing with in his business (where analytics might help), or recent events and changes for which they want to measure effectiveness. On our end, we might talk about new metrics or methods we've been playing with.

He and his site manager know his business needs, I know how to express his issues as analytics problems (I hope), and in addition our engagement manager and I each have tons of web experience so we throw that into the discussion as well. There's a lot of "what the heck," "I wonder why," and "wouldn't it be cool if" going on, and by the end of the phone call we've scoped a "One True Thing" topic that our analytics team will investigate before the next phone call. And, of course, we go over the outcome from the last One True Thing question, usually with just an informal email and some graphs instead of a formal report (saves billable time). The best part is if the analysis gets them to say "That's great! Just what we needed!"

A lot of mutual education goes on. We've all gotten better at these calls over time. The two of them have become more analytics-savvy and for our part we understand their business better than ever before. They are getting a lot for their money and we are getting to poke into analytics questions that we wouldn't have thought of. Along the way we get to invent new analyses and maybe add them to our overall analytics practice. We're also proposing this kind of relationship to other clients and the One True Thing Phone Call has become a bit of a standard. Or at least we'd like it to be.

The web analytics world tends to be a little too formal, IMHO, emphasizing quantity of data and dashboards as our output when what might help most of all would be these kinds of conversations that stress questions, answers, and insights without running up a big bill. Obviously, the point of this blog entry is to suggest that more people get back to basics like this. Partly because it's a good thing in itself, and partly because it's so much fun.

Web 2.0 and usability

2008-01-19T08:43:00.000-08:00

I just discovered a piece by Jakob Nielsen, the sometimes controversial usability pundit, who wrote a provocative article on Web 2.0.

His main point: “If you focus on over-hyped technology developments, you risk diverting resources from the high-ROI design issues that really matter to your users …”

Early in the article, he provides a working definition of Web 2.0 as a structure for the rest of his article:

- "Rich" Internet Applications (RIA)
- Community features, social networks, and user-generated content
- Mashups (using other sites' services as a development platform)
- Advertising as the main or only business model

The rest of the article is about the usability risks for each of these facets of Web 2.0, in the context of five kinds of web sites – marketing, e-comm, media, intranets, web applications.

Whether you agree or disagree, it’s worth reading. My main reaction was a big hurray, simply because of the refreshing clarity with which he defined his terms and deconstructed the issue. He's a hero to me just for that reason (another writer who jazzes me in the same way is Gary Angel over at SEMphonics). If I could find more people who work within a logical structure when they write, I would be a lot less confused in general.

It's almost a footnote that I agreed with most of his conclusions.

http://www.useit.com/alertbox/web-2.html

The DoubleClick bind

2007-12-31T08:23:00.000-08:00

It's been an ongoing struggle to reconcile DoubleClick's (DFA) stats with what we get elsewhere, for example from Google Analytics, WebTrends SDC, and server log files. Most of us trust the data provided by the tagbased analytics and mistrust DoubleClick's numbers, but it would be nice to know what really causes the differences. And they are BIG differences, because DoubleClick's numbers (clicks) can be 100% higher than Google Analytics or WebTrends (page views).

It becomes an issue with people who are invested in spending money on interactive advertising. They are, justifiably, wanting to take the DoubleClick numbers at face value. It makes the clickthrough rate look a lot better.

We're trying to look into it, with a lot of help from WebTrends custom reports, server logs, SDC logs, and Google Analytics as a sort of backup.

We know this:

DFA's numbers are 'way higher than the tagbased data, as much as 100% higher.

DFA's numbers are very similar to the numbers in server logs.

The two tagbased reports, WebTrends SDC and Google Analytics, agree with each other but are usually only half the size of DFA stats and server logs.

When we carefully compare individual page view events between server logs and SDC (tagbased) logs, we can separate out the "extra" events that don't show up in the tagged logs. Close examination does NOT show a pattern of repeated User Agent or IP information. There's no obvious evidence of one or two entities doing a lot of clicking. No easy answer about bots or cottage industry clickfraud.

However, we are finding one interesting thing that points to a possible explanation, or at least a corroboration of the notion that DoubleClick's numbers contain a lot of non-humans. DoubleClick hits are, as you know, marked with extra parameters. So, a given banner destination page will have, in the logs, some hits with DC marker parameters (coming from banner clicks) and other hits without those marker parameters (coming from non-banner sources). We did separate analyses of these two groups - only analyzing hits that were also first hits in a visit.

We found that if the hits had DC marker parameters, the discrepancy between server and SDC/Google logs was a huge one - in the up-to-100%-more range. But if the hits did NOT have DC marker parameters, the discrepancy was really quite small --- server and SDC/Google more or less had the same numbers. In other words, the hits that appeared in server logs but did not appear in tag-based logs were mostly banner hits.

We all know that there are several reasons for an event to appear in server logs but not in tag logs. Two big ones are: 1) an entity that does not execute javascript will not appear in tag logs (rather, the identity of the page being requested won't, although the hit will appear as a site hit), and 2) an entity that does not request images will not appear in tag logs.

It looks to us like a big proportion of clicks on banners are by entities that don't request images or don't execute javascript - basically, bots.

Someday, I'll talk to an insider who understands what's going on and who runs these bots and why. Could be legitimate, or not.

In the meantime, I'm continuing to say that WebTrendsSDC/Google Analytics numbers for banner traffic are the ones to trust, because they are probably the human traffic. DoubleClick numbers contain vast amounts of non-human traffic.

Crazy Egg

2007-06-30T14:24:00.000-07:00

This weekend one of the other analysts and I spent some time trying to figure out Crazy Egg's output (http://http://www.crazyegg.com/). For a simple, cheap tool it's a bit on the complicated side, I have to say. We ended up with a 21-page internal document (lots of screen shots fluffed it up considerably).

Just some sample points:

1. Crazy Egg employs the term "visit" when it's talking about a view of the tested page.

2. Crazy Egg CAN track clicks in the flash portion of a page, if the page is set up right.

3. The "tracks clicks in text, div elements, and table cells" option is definitely worth doing. These are called "other" (as opposed to the main links, images, forms, and javascript click elements).

4. A lot of quirky stuff shows up, somewhat erratically, in the elements listing. We think they might be browser-dependent. We also see clicks that are double-counted, i.e. that are covered by more than one type of element on this list. We saw bits of HTML and js comments. Altogether, for a page with 38 possible images, links, etc we found lists of 50 clicked elements.

5. The Confetti view opens with all the clicks scattered all over the screen, quickly flying down to their actual locations on the screen shot. A cute animation, but it has a nice use --- if your page has a few really dense areas of clicks, you can't tell how many clicks are in the solidly-colored masses. The fly-down gives you a quick look at how many dots really are in those spots.

6. The heat map is cool-looking but really faint blushes of heat can be hard to see because the underlying screen shot can disguise them. We found a way to see the heat without the underlay - by choosing the "other" view and then unchoosing it by clicking on "other"again. In this mode, neither "live" or "other" or "both" is active --- and two new heat maps appear below the main one. The two new heat maps are the "live" one and the "other" one but they don't have the underlay.

7. The "time to click" feature in the confetti view is surprisingly interesting.

8. You can click on an individual piece of confetti and get a little report on just that click.

9. We analysts prefer the confetti view as long as there aren't too many clicks in all. Clients are dazzled by the confetti view but they think the glow view is more executive-appropriate.

10. Crazy Egg's heat maps must be viewed through Firefox, not IE. IE7 shows all kinds of errors and misplacements.

Overall, we continue to be amazed at the value of Crazy Egg. It has some issues (particularly with tracked pages that resize according to browser resolution) but it is worth several times its price. We're waiting for some vendor to gobble it up and make it unavailable to everybody else.

The life cycle of a new profession

2007-05-20T11:02:00.000-07:00

Web analytics seems pretty secure right now as a profession, with emphasis on "seems". I wonder if anyone else, besides me, has doubts.

I've been lucky enough to be intimately involved in the births of five different professional categories, over the years. Two of them "stuck" and three ended up as professional footnotes.

I also have been lucky enough to have had a lot of organizational development coursework in school, and to have had jobs that placed me within earshot of some fairly famous CEOs, boards of directors, O.D. researchers, creative geniuses, business gurus, and O.D. gurus. I didn't appreciate at the time how lucky I was to be sitting across the table from people like ... ah, no name dropping. Sorry.

And, finally, I have been lucky enough to have had some superb professional opportunities lately that have caused me to think long and hard about professional growth, both personal and, well, for the profession.

Here, based on my vast experience and knowledge, is my private mental list of signs of a startup professional specialty that is beginning to run hot. Some are essential for a profession to thrive but others contribute to an eventual fade to an interest area.

Increasing numbers of press articles and the beginning of awareness of the field, on the part of laypeople. For example, our parents now "get" what we do to the point of wanting to explaining it to their friends.
In the press coverage, a perception that [profession] is new, exciting, and possibly the source of business miracles ... and that it's pretty easy and simple.
CEO's saying "we are focusing on this" with, between the lines, "this will rescue us"
Quick expansion of, or establishment of, internal staffs to do whatever the profession is supposed to do. Accompanying this may be a senior executive who has decided to head it up, and who is likely to not know much about it except that it's cool, essential, or whatever.
A very small number of companies that really and truly are building amazing internal capabilities ... usually quietly
New names for existing methodologies and models, and wide ignorance of the fact that these are, really, just new names for proven stuff
A lot of recruitment action and job hopping among the established professionals
A glut of job postings in general
People with perfectly good "other" jobs thinking that the field looks lots more interesting and profitable and fun than what they do now, wanting to switch careers
Ditto for people in grad schools
Academic people thinking that the field is a source of funding, starting courses and maybe even institutes and "centers"
Established consultants thinking that now's the time to jump from being excellent to being a guru ... or a mogul
A very small group of true profession-builders
Hero worship of the gurus, moguls, or profession-builders, most evident at conferences
Established publishers thinking now's the time to bring out the definitive book, followed later by trying to bring out a book with a new angle, either a compatible or a diverging one
The startup of a professional association followed later by a 2nd competing one
Talk about professional certification, followed later by talk about grandfathering for said certification
National or international meetings where the same people seem to always be the speakers and where attendees focus on (and post photos of etc) the people and cool events as much as the information. (The phenomenon of the same people always being the speakers is a fascinating phenom worthy of its own list of bullet points!)
Vendors coming out of the woodwork, followed by waves of shakeouts (this, too, is very complicated and interesting and a major topic in org dev and biz studies)
Vendors bringing out new features without repairing the old ones, including before their own tech support or documentation are ready
Vendors starting their own pseudo-academic institutes
Much of the above happening during a period of economic expansion, when organizations can be open to having unproven entities around.
When the tolerance for unproven entities starts to wane due to economic tightening, the profession heats up further for a while, but this time it's a different kind of heat - it's under pressure. There will be an urgency to prove or differentiate in order to be allowed to persist. Most of the above points will become pressure-driven rather than opportunity-driven.

At the top of the list I used the phrase "beginning to run hot." The metaphor feels right to me. When you have a machine or a organism that speeds up and generates heat, you're in a process that can have a good or a bad ending. There are a few things along that way that influence the outcome, in my humble opinion, and I will follow up on this topic later.

Google Analytics vs WebTrends, flip side

2007-05-06T11:44:00.000-07:00

On the other hand, it's obvious that GA has some strong points. Aside from the obvious wonderful geography graphic and the more comprehensive conversion stats (better stats, in more reports), I'd add:

Cool dashboard-like indicators (but the length of the list is limited)
There is a Print-Report function. With WebTrends, reports are always cut off on the right side.
Exit rates per page (very simple to get FROM WebTrends but you can't get it IN WebTrends)
Pageviews per visit statistic along with page views and visits stats.
Goal Conversions
"Absolute Unique Visitors" is, I believe, more accurate than WebTrends'. It looks like each visitor is counted only once in these stats, either as a first-timer OR as a returner, even if the visitor had multiple visits during the reporting period.
Help Cards are clearer and they don't fall back on using the term to define the term.
New vs Returning report is better.
Nice drilldowns on individual referrers - data over time, to-date lifetime value, cross-segment performance i.e. broken down further by browser, campaign, keyword, content, geography, and something called "visitor type."
Campaign reports that include cost per click!
A/B testing
A drilldown based on the directory path! (we've talked about that in these forums a lot and it's impossible to get in WT without a lot of coding)

Google Analytics

2007-04-30T15:08:00.000-07:00

This topic comes up a lot because GA and its progenitor Urchin are pretty darn good tools. So why pay for anything else?

On the whole, I like Google Analytics a lot but as an analyst with a lot of experience I see much more value in WebTrends, especially for an e-commerce site where a bit of insight is worth real money. One thing that often prevents WebTrends from being fully utilized is its learning curve, which we get around by starting clients at an above-average level of sophistication in their reports that will hopefully serve them well for a long time, transferring the end-user's "thinking" part of the work to Excel where possible. But directly out of the box, report for report, WebTrends does not have an advantage over GA and certainly not if you factor in the price.

Google Analytics is an excellent tool up to a point. It does a fine job of rendering the basic level of statistics and is superior for campaigns and referrers, and visuals like its geography reports. However, it is a “report card” type of tool that is used for trending high-level statistics or side-by-side comparisons of campaigns or keywords according to a few bottom line performance indicators. And recently, GA added some A/B testing capability which adds another type of decision support. It’s certainly an exceptional value, but for managing a web site, web site marketing, or product management, I don't perceive it to be as valuable as the more flexible tools.

One major drawback of ASP tools like GA is that data cannot be re-analyzed at all. New ideas cannot be investigated by checking historical patterns and errors in setup can’t be fixed except going forward. Without re-analysis, many decisions go unsupported or are delayed until enough data has been collected.

A shortcoming of GA's feature set that is frustrating to many users is its inability to create a report confined to a visit or visitor type. Google Analytics handles segmenting by referrer or campaign but cannot report separately on groups separated by things like geography, time of day, day of week, frequency or recency of visits, visitor lifetime value, or whether a particular page was seen during the current or any past visit.

The lack of visitor history information is especially frustrating. Apparently, a campaign can score a conversion only in the original visit. Campaign latency reports aren't available.

Another has to do with its method of path analysis. Believe it or not, there are several ways of describing the path people take through a site, and Google happens to use the least accurate method. Since a carefully analyzed path analysis can be the basis for developing a typology of visit behaviors, this is a drawback for the idea of basing marketing programs or site personalization on visit behavior types.

Google blobs together all parameters in one report which is a nightmare for sites that use more than one or two parameters in one URL. Not being able to isolate them (or pairs of them) would just about drive me crazy. I suppose the solution is to dump that whole report into Excel and start writing macros. This shortcoming surprised me more than anything else when I started using GA.

Analytics people who want to cull patterns from massive amounts of data like to aggregate rather than split data. In web analytics this means treating several pages as one unit in order to know about visits that saw one or more of a certain set of pages that the analyst thinks belong together. In WebTrends and other software this is done with “content grouping.” [the following was changed 2/2/2009 thanks to a constructive challenge on e-nor's web analytics blog] Google Analytics does Content Grouping as an Advanced Custom Filter, or maybe it's called a Custom Advanced Filter, based on the whole URI. Although it doesn't allow quite as much precision as WebTrends (which treats the URI stem and query string separately when setting up a content group), it does allow regular expressions and, frankly, our categorizing this as a deficiency on GA's part has to be confined to the fact that it's so well hidden in the interface!

There are whole web sites and blogs that deal with Google Analytics and ways to hack around its shortcomings. One “discovery” that was recently widely cited had to do with tricking GA into reporting on the “actual keyword searched on” juxtaposed with the “keyword paid for” for individual visits. This kind of report is trivial to do in WebTrends using the Custom Report feature. Another area of reporting that is somewhat neglected by GA is on-site search. There can be a big payoff in being able to see what is searched for, what keywords produce no results, the page context for searches, and actions after searches. While not helpful for simple high-volume sites like some retailers, it's invaluable for information-intensive sites.

But I think GA’s greatest shortcoming is that it has no API for its results, which means it is limited in being used for more complex analysis and trending, combining with other data sources such as store sales, or using customer databases to create analyses based on CRM data. I've been heavily using the WebTrends API and an interface-building tool (DataLinks by BIData) to quickly create, in Excel, dashboards as well as detailed reports that would have been only a dream six months ago. As I said in our meeting, these spreadsheets can be refreshed quickly with new data ranges, but more importantly the analysis and presentation layers can be custom-modified by end users who know Excel.

Cost can't be ignored, though. Regarding the cost of WebTrends, there are several ways to increase value while controlling costs. Not all pages have to be tagged, and if all are tagged, not all have to be analyzed if good preprocessing or report planning is used. Buying software is a way to save money on WebTrends’ ASP model. Features can be purchased selectively, and discounts can be negotiated. Training and the more sophisticated reporting setup can be spot-farmed out.

CMS meets Analytics article

2007-02-23T10:40:00.000-08:00

My article for a general webbie audience on making CMS work with analytics and vice versa. It's a summary of things we figured out over the course of a couple of projects.

http://www.websitemagazine.com/content/blogs/posts/articles/When_Content_Managament_Systems_Meet_Data_Analytics.aspx

http://tinyurl.com/62yngo

Our general approach to measuring quality

2007-02-04T10:08:00.000-08:00

Our approach has been working especially well lately. What I mean is, some non-analytics people are getting it, resulting in credibility and patience. Patience --- that's a very valuable commodity around here when the source is the project managers and the clients. We're exposing the process a lot more and they are understanding why measuring visit quality isn't done overnight.

Here is a brief outline.

Think of it as a spreadsheet. In fact, a spreadsheet is currently our best way of keeping it organized and grokkable.

First, we try to break down the web site in a meaningful way. The most common breakdown possibilities are by topic, by audience, by function, or by business unit within the owning organization. This is the first column of the spreadsheet.

Then we spend time with the consumers of the reports to find out what they want from their web site and, to an extent, from the business. The underlying question is "define success" but we learned early that "define success" is a terrible way to ask for what we want. So we ask them indirectly: why do they have the site/feature/sites section/audience/function/topic/business unit. What do they want to happen. And so forth. It's often a messy conversation that flips from one type of breakdown to another and it's usually up to us to either keep things on track or just give up on that and restructure it later.

These high-level success measure "ideas" are the second column. We now have a lot more rows than we started out with, of course.

Next, we take that discussion and the high-level success measures and operationalize them in terms of web site behavior. Usually, every vague measure from the discussion gets from 1 to maybe half a dozen operationalized pseudo-measures. Do I need to say that this is the third column of the spreadsheet?

At this point we circle back to the consumers to check on everything. The spreadsheet format is nice for that discussion.

Next comes a partial site inventory, and the fourth column. We get reeally familiar with the site at this point and attache specific pages or events to each of the operationalized definitions, to the extent possible. Here is where we start to find holes in the site, usability problems, things that aren't tracked, and so on. We often end up with a wall-size site diagram made of screen shots and bits of string showing links and paths, plus sticky notes and markups. We have a cache of very large foamcore sheets for this purpose. The result is primitive-looking but exceptionally useful. However, these spectacular art objects are mainly a method to get to the content that goes into that fourth column.

There's also a fifth column of notes about the holes, problems, untracked things.

The sixth column contains comments on how the heck we are going to measure these, using whatever analysis tool we have. If it's WebTrends, this column talks about content groups, path analyses, filters, and so on. More rows.

The seventh column starts to build specifications or configurations for the WebTrends reports we need. If it's a content group, will it be defined with Regular Expressions, and what's the common alphanumeric string? What should be the exact name of the content group? The URLs we put in column four are consulted over and over. Sometimes we realize we might need to get the engineers to change a URL ... that goes into the comments.

The eighth column sorts all those specs into WebTrends profiles as efficiently as possible. There's a lot of overhead for each profile so we try to keep the number down, but there are rarely fewer than four and often more than twenty profiles by the time we're done.

Usually, columns seven and eight are done concurrently with actually WT setup work and trial runs because, well, you learn a lot that way.

Right at this moment I'm doing a column 7-8 thing and I really have to get back to it.

I should add that column 7-8 stuff is best done on a WEEKEND. Because it's quiet and nondistracting.

Excuse me, but ...

2007-02-03T13:18:00.000-08:00

In the development of any organization, there are many roles that make the difference between the organization maturing versus withering. A profession shares many properties of organizations. And web traffic analysis is, arguably, a profession or a potential one. Or so they say.

I love how, lately, various people are writing about all these new-looking scales, concepts, indexes and scores, even creating new names for them, I really do. It makes us web traffic analysts look smart, original and even authoritative. Certainly, it makes us look busy. Terrific.

Another thing this accomplishes is that it puts down another thin layer between the "professionals" and all those people who somehow think that web traffic analysis is something that a smart person can do with common sense and general web experience. Believe me, I'm all in favor of making those folks a tad more insecure because far too many are thinking "why does it take so long? I'm sure I could do it in a few minutes. After all, I'm on the web all the time and I make web sites and ..."

It's also nice that many of these new scales, concepts, etc are drawn more or less directly from early work in now-established niches in the social and marketing sciences. Once you realize why they seem so familiar-yet-not-familiar, it's reassuring to realize that the new things have good pedigrees, or at the very least were used a long time elsewhere with apparently satisfactory results. This is nothin' but good.

I'm looking forward, however, to the next phase. Or rather I hope the next phase comes, period. It's the phase called validation and it's a whole lot of really difficult work. It lasts a long time and it's extremely frustrating especially for the people who came up with the ideas in the first place.

I do realize that the vast majority of western business operations, at least those that have to do with humans and decisions and habits, have flourished quite well for decades without very much validation at all. In market economies, there is just as much power in persuasion, structure, and impression as there is in science or, dare I say, truth.

So I guess web traffic analysis has a decent chance of skipping the whole phase of validation and, instead, using the time-honored traditions of manipulating perceptions and expectations about the value of what we do. Psychology for example (the non-biological kind) in its entire history has come up with maybe as many as three practical and enduring theories that really do explain things. Well, maybe two. Yet many thousands of people are employed "doing" or "using" psychology of all stripes, and the economy is better off for it because the impression of making a difference is, in today's world, sometimes about as much as you need.

Where I really need to spend my time

2007-01-10T04:31:00.000-08:00

The hardest, most time-wasting part of my job is not the time spent using the tools. It's getting humans to understand what is involved. So far this month the following typical problems have surfaced:

1. Getting across a vague idea of the full spectrum of what could be in reports, to the point of helping the other person decide at what level they want to buy in and feeling confident about that decision
2. Creating credibility for the longish period of time it takes to get the tool to produce the more advanced insights
3. Convincing people that they can and perhaps should start small and cheap, but in a way that allows easy adding-on or changing of scope
4. Living with the fact that there is noise in the data ... and a couple other principles and realities that are basic to the science of forecasting
5. Living with the fact that any given statistic can probably be interpreted several ways
6. Persuading people that, even though they are regular consumers of business statistics (read: USA Today graphs) and statistics tools (read: Excel) that one day of vendor training in the tool will NOT make them a tool administrator
7. At the same time, persuading them that a day of one-on-one training on how to use the tool's output, working with their own data, is a great way to start.

While I'm at it, I'll list what I wish all tool vendors "got."

1. Sharing their strategy and getting feedback during design and development will always pay off. I'm referring to dialog between the vendor (managers down to tech support and sales) and the users.
2. Dividing the market into verticals AND horizontals is an important exercise, not just for selling but for developing.
3. The features will sell the tool. The details and documentation will make or break it once the tool has been bought, and when it's being talked about among users and potential customers.

And of course, here's what I think vendors' customers need to "get." This is the group I fall into.

1. It's insane to think these tools could be free or cheap. It's business. The market drives the price.
2. The vendor will always have the big picture that determines the course of the product. The customer has to respect this, even if they don't like the decisions that get made.
3. The customer has a responsibility to communicate to the vendor. In a constructive way. Even if the vendor doesn't listen, it's the right thing for the customer to do.

Crummy analogy

2006-12-11T04:48:00.000-08:00

Grr.

I'm going to skip the tempting crummy analogy, actually. It would be insulting to whatever I tried to compare online help communities to.

Let me just say that going into certain public technical discussion forums (with emphasis on "public") is a foray into a completely different culture that is based on the quick easy free approach to everything, with a great deal of expectation about what the world owes to the writer and how everything is a lie or a ripoff.

An example that I just now encountered: without doing any research, which would involve exactly one second of Google search, somebody decides "gclid" is evidence of an attempt at online theft and gets otherwise busy people to deal with their hysteria.

It's like watching a crazed Pekingese dog: all reaction and very little brain.

But as I said, lumping online help communities together with them would be insulting to Pekingese dogs.

Can't get around it ...

2006-11-19T13:26:00.000-08:00

Since my previous post about the "dutiful" analytics situation, I have heard the following comments about the lesson that little story might hold:

1. Being in analytics is just like whistling in the wind (the actual wind-related metaphor was a little more colorful)

2. You'll never go wrong underestimating the [insert vague personnel reference here]

3. We are NOT the Borg

4. It's never too early in the day to go home depressed

Although I think they all qualify as properly existential and therefore perfectly valid interpretations of the cosmic moment described, I have bounced back from my initial dismay. The lesson I am personally taking away is:

5. Never, never, ever let anybody run a WebTrends report by themselves. Unless they have personally wowed you in some kind of metrics-related discussion. Without that proof of worthiness, they will inevitably bring shame and harm to your profession and shorten your career and life.

Definitely NOT changing someone's world

2006-11-11T15:07:00.000-08:00

I got a call from an acquaintance yesterday needing some WebTrends help. Her organization has had WebTrends for about six months. She had been sending out Word versions of WT reports to stakeholders every month since the site redesign in June - just the Overview report, nothing else. But yesterday she decided to get fancy and do a side-by-side comparison of June and October. Unable to produce it, she called me.

Well, to make a long story short, it took about 45 seconds of research to find that she hadn't had logs to analyze since July 25. WebTrends had been dutifully turning on and analyzing nothing since July 26. And she had been dutifully creating and mailing reports covering the period June through [current month] --- 5 times since last July. The bar graph for traffic had been dutifully displaying its data, with the June-July traffic receding each month until it now displays only one bar on a long, lonely, empty x-axis.

And people had apparently been dutifully deleting her emails.

I am thinking hard about what might be the existential lesson here.

Changing someone's world

2006-09-29T07:55:00.000-07:00

Yesterday I had an opportunity to explain to our Digital Marketing department (and project managers, and an engineer) about an alternative way to get some complicated statistics out of the relatively simple ODBC output of one of our web analytics tools, WebTrends 8. Actually the statistics are fairly simple - function-based quality measures for six categories of events. But the current process of getting them from multiple WT reports and profiles and into deliverable form is tedious.

So I explained what DataLinks can do (http://www.bidata.net). Or, rather, what we can do with DataLinks. This tool creates a refreshable conduit between an Excel workbook and WebTrends. Implementing requires configuring WebTrends, creating the conduit with DataLinks, then adding programmed layers to the Excel output, with the top layer being the deliverable sheet. It's too simple, really. And it's 'way overdue in the WebTrends-user world.

Within two hours of our meeting, two people - an engineer and a Digital Marketing person - separately told me that if I could do this it would "change their whole world" and I could "name my reward." They're pretty fed up with the tedious and lengthy and, frankly, broken MS Access process they are using now to process the output from WebTrends.

There are only two slightly out of the ordinary things here that are going to "change their world." One is that there's a person involved (me) who knows WebTrends AND Excel AND their measurement objectives. The other is the DataLinks tool, which we didn't have last year.

I'm not surprised that the above two things are going to make things easier. What I'm surprised about is that WebTrends (and its competitors) just don't get that this kind of front end (made possible in my case by the indispensable DataLinks tool) is the future of their products. WebTrends sorta knows it --- it has made a few attempts in the past to shift computational and data management responsibility to Excel via WebTrends tools like SmartReports and ol' Export. But there's been no visible, on-purpose progress from WebTrends since SmartReports a couple versions ago. Then a few months ago WebTrends accidentally opened a door with version 8 which had a 75% complete ODBC driver connecting to the reports (not to the databases). DataLinks has pushed that door open a lot wider. WebTrends still doesn't get it, though.

HBX may be ahead of WebTrends with their Report Builder tool. But from what I have seen, the tool is a little on the complicated side. DataLinks can be used in a very simpified way, which is what I'm doing now. I'm using my existing Excel knowledge to do the funky parts. And if I used the more advanced abilities of DL, DL also holds my hand for more complicated things --- dropdown menus, drag and drop stuff. I'm not a HBX Report Builder user but its interface seems to ask for a lot of knowledge (and a four-hour training session at the recent HBX conference). I may come back and retract this statement later ...

got-analytics-so-what

2006-09-17T07:05:00.000-07:00

I recently had a dialog with somebody who was somewhat justifiably complaining that what she wanted was not analytics, but analytics that actually mattered. She was having trouble threshing out the latter from the chaff that all these programs produce in abundance. Or maybe that's the wrong metaphor. She had this full pantry of ingredients but didn't own a cookbook and, furthermore, didn't even know what she wanted for dinner. Or (sorry, sorry, sorry! I'm going too far with this ...) she didn't even know whether it was time for breakfast or dinner.

It sounds like a pretty typical conversation that analytics people hear a lot (or have with themselves when nobody's around). But this lady was a really smart and experienced person, although as she admitted, a bit of a princess at times. This particular got-analytics-so-what conversation was more thought-provoking than usual.

Her half of the conversation (which was sporadic over a couple of days at a conference) made it clear that she had already done a lot of good analysis that was definitely in the "matters" category. The perplexing thing for me was that a lot of this analysis didn't use site traffic data. She was doing focus groups and surveys as well as site data. A lot of what she was learning was coming from the former.

So the first thought-provoking part of the conversation was this musing: I have a survey research and focus group and interview background and the situation is familiar. A web site is an experience for human beings. Being able to ask direct questions to actual human beings is probably what most of us would instinctively gravitate towards. So surveys and groups happen.

When analyzing site traffic we are a couple of concepts removed from asking human beings about what we want to know. We may be able to come up with really smart questions (which is 90% of the talent of good researchers), but when doing site traffic analysis we have to do additional steps, namely translating those questions into other questions that can be answered by what amounts to photographs of footprints.

Getting hands dirty - log files

2006-08-27T07:00:00.000-07:00

Somebody asked me about my previous log file comment. Hands-on work directly with logs is a best practice that I feel strongly about. It will help avoid problems later in setup, help troubleshoot, and teach you about the inner workings of your analysis tool.

Here is a minor rant:

Open a log with Excel (space delimited), sessionize it by sorting on IP/UA or cookie or whatever, and look at individual visits carefully. There are a lot of things to look at, but landing page redirects, GET/POST actions, status codes, and cookie consistency are four big ones that pay off most of the time. Of course, check whether the important fields are being logged, perid.

For maximum benefit do this: go to the site yourself, perform all the important-to-know-about actions, and examine your own visit to make sure that the logs look like what you expect. How to find your own hits? You can find your own IP (use ipconfig from command window, usually) or do it this easy way: when you arrive at the site, refresh the home page --- but FIRST add something to the home page URL in the address window - like the parameter "special=chris-g-tracking-visit". Then just search for that string in the logs and you'll be able to pull out the rest of your visit using the cookie or IP field.

A variation on log-diving is creating a test log, small and 100% understood by you. It's the fastest way I know to debug the more difficult reports like campaigns. It's also the fastest way to understand the more bizarre statistics in WT like the "Most Recent" ones. (If you make one, make sure you have an extra last line that is a day later than the next-to-last line. WT won't analyze this fake line but it's critical to have it there.)

End message: garbage in, garbage out. Open that trash can and get dirty.

Back to basics ... again

2006-08-16T13:33:00.000-07:00

The best first thing you can do to understand web analytics is to open a log file and mess around. It will either help you understand your site, or it will allow you to skip over the first fifty or so stupid questions about your analytics tools.

Then when you get really good at analytics, the best next thing you can do when you get stuck is ... open a log file and mess around. You never get away from it.

Functionalism - why I like it

2006-08-09T15:08:00.000-07:00

Gary Angel is doing some writing on a concept he calls functionalism. Ordinarily I steer clear of people slapping names onto things, especially when the names are academic retreads. Yes, I guess it's better than inventing a new word. But here's the thing. Gary Angel truly is working on delineating an out-of-the-ordinary approach and the name "functionalism" even sorta fits.

In any field, an emerging construct has to have a name in order to turn into a building block. Analytics Functionalism has that kind of potential.

What Gary Angel is talking about is, at its simplest level, grouping pages by functions then looking at the traffic patterns of those groups. It's a drill-up way of doing analytics, as opposed to the drill-down catechism that drives some analytics vendors. But it's a lot more than that. First of all, the groupings or page types he has chosen to share are thought-provoking and he even has little corollaries for them here and there. Second, when I say "looking at the ... patterns" I mean he has specific analyses in mind, as opposed to just "smart eyeballing" as my friend Lou puts it. More about that from me after I do some more thinking.

You MUST read his white paper and his blog. More than once, please. Then we'll talk.

Now, to digress a bit. You may have figured out after, what, three whole posts? that I am not fresh out of school. Too true. My analytics road has gone through four different early-stage, human-factors-oriented fields to get here --- environmental psychology, white collar productivity research, facility management, office ergonomics, and now it's web site traffic analytics. I must be attracted to small fields still in the creative phase or something. I have helped start up professional associations when the referenced professions didn't even have a common name yet (I'm taking a pass on the current Web Analytics Association). I've worked on the first textbooks in these fields, have helped write federal, state, and international standards, and more. And best of all I've had the chance to hang out with the thinkers and movers in those fields - the ones exuding the above-mentioned creative juice. People I can't name because I don't want this blog to show up when people google them for term papers.

So I'm saying that I think I know structure-emerging-from-entropy when I see it, and this is the genuine article. If enough people get it and use it and take it further, we'll someday have a real profession instead of a vocation.

http://semphonic.blogs.com/semangel/2006/08/functionalism_a.html

I want my within-visit associations!

2006-08-04T02:13:00.000-07:00

Yesterday I had a brief correspondence with Robert Allison of SAS, who has been posting several really good dashboards on Dashboard Spy (http://www.dashboardspy.com). The one that I wanted to know about was this one http://dashboardspy.wordpress.com/2006/08/02/an-e-commerce-dashboard-web-marketing-analysis-with-sasgraph/ because it had to do with web analytics and because some of the stats were really cool yet simple. I wanted to know the identity of the program that had produced, from raw site activity, the underlying data that his dashboard was based on. As it turned out, he got the data from Stephen Few's book. I have that book on order so I'll check further when I get it.

Meanwhile, the point of this is my frustration with web analytics programs today, which are for the most part dressed-up tabulators with a sessionizer tacked on. You get the equivalent of cubes for data within a hit, hit data with first-hit extras (referrer etc), and some visitor history dimensions.

I want associations more than anything else. I got out of grad school umpteen years ago and this was an old idea back then.

Allison's (and presumably Few's) dashboard had, for example, these two lists:

Top 10 [pairs of] Products Purchased Together but Not Displayed Together
Top 10 [pairs of] Products Displayed Together but Not Purchased Together

Those two tables are worth a lot not only to web managers but product managers. Try getting them out of a web analytics program.

PostScript: In response to my question, Rob Allison remarked that the stats didn't seem very sophicated to him. He's right. But then, he's in R&D and let's not forget that he actually works for SAS (!) which means what is sophisticated for him is some kind of nirvana for us web analytics drones. But let's at least get web analytics out of the within-hit crosstab world.

Convergence on Excel --- who knew?

2006-08-02T18:48:00.000-07:00

Having cut my analytic teeth on multivariate analysis tools, it's kind of a shock to realize that I've ended up focusing on a spreadsheet program. And that I'm not too unhappy about it. Of course, it means tossing out just about every consideration of statistical significance, interactions, or assumptions about normal distributions. Luckily with samples the size I'm working with now --- millions --- I don't feel too guilty about TOTALLY BETRAYING EVERYTHING MY PROFESSORS TAUGHT ME. haha, they're still struggling to get samples in the low four figures.

Anyway, at AdTech last week I "taught" a little session on some of the small things I do with Excel to explore the data that fall out of web analytics programs like WebTrends, Hitbox, Omniture. The response was quite nice even though none of the tricks were earthshaking or anything. The Powerpoint is here: http://www.enlighten.com/analytics/adtech/analytics-excel.ppt

At the beginning of forever

2006-07-31T17:00:00.000-07:00

Somebody asked me how long I've been doing analytics and I answered "seems like forever." And late nights when I'm wrestling with WebTrends it REALLY seems like forever.

In truth, I've been doing analytics of one kind or another since I was quite small. I firmly believed that if I could just keep good enough records of the race results (in the Newark Star Ledger that my dad brought home every night), I could eventually predict the winners and parlay my allowance into early retirement (before age 10, I was hoping). Seemed simple enough. About a month later I had a stack of carefully printed index cards, one per horse. Quite a large stack. Not a single card in that stack had more than one entry on it. It started to dawn on me that the population of race nags was possibly infinite and that the Belmont track was a very tiny sample indeed. My first statistics concepts.

Ever the optimist, I focused on the good news --- there were a lot more beautiful horses in the world than I thought.

So here I am, still thinking that the answer is keeping track (pun intended). To the packs of index cards I've added a Ph.D., a lot of statistics training that never gets used in the business world, some really nifty past jobs and awesomely smart past co-workers, and the Internet .... where I can collect the equivalent of that pile of index cards in about .005 seconds. Ever the optimist, I say, How cool is that.