Presentation Description
This presentation discusses the importance of taxonomy testing to ensure its effectiveness before implementation, focusing on qualitative and quantitative methods, and emphasizing the need for a well-defined research plan to achieve testing goals. The presentation also covers various contextual analysis methods such as focus groups, card sorting, scenario testing, and content tagging to validate the taxonomy and gather evidence for stakeholder approval.
Type: Webinar
Venue: Information Architecture Conference, May 2, 2025, Philadelphia
Date: May 2, 2025
Broad Topic: Information Architecture
Presenter: Joseph Busch, Principal Analyst, Taxonomy Strategies
Watch Presentation
Download Presentation: PDF | PPTX | MP4
Download Slide Script: PDF | DOCX
[Slide 1: Taxonomy Testing: How to Prove that It Will Work]
Good morning. My name is Joseph Busch. I’m the founder and one of the principal analysts at Taxonomy Strategies.
In other talks I have described the painstaking process of developing a taxonomy using quantitative and qualitative methods. This includes engaging with stakeholders from the start by getting buy in to the business case, interviewing stakeholders to develop the requirements and use cases, and periodically briefing stakeholders along the way sharing interim deliverables, drafts, and prototypes. Aside from engagement with stakeholders during the taxonomy development process, what can you do to demonstrate that the taxonomy will work before it is implemented? This talk provides practical advice on how to demonstrate that the taxonomy that’s been created will work.
[Slide 2: Taxonomy Testing Lifecycle]
Taxonomy testing is a continuous process. Today, I will focus on how to get through an initial cycle, but optimizing the way that content is organized is a continuous process of refinement. Like any process, it’s important to have a plan that clearly describes your goals and the steps you plan to take to reach them. There are a variety of quantitative and qualitative methods in your taxonomy testing tool kit. You should plan to use a combination of them. You have more control over using quantitative tools such as query logs, click trails and content analytics assuming that they are available and appropriately configured. On the other hand, qualitative tools such as surveys, focus groups, and usability testing require more planning and coordination because they involve more immediate stakeholder participation. The object of all this is to create enough confidence that the taxonomy is good enough to be implemented. Will it work in real life is the ultimate test.
- Will my customers find and buy products?
- Will staff find the answer they need in a knowledgebase?
- Will a contractor be able to complete a task on a mobile app?
A taxonomy should constantly be assessed primarily by reviewing logs. Anomalies should lead to new research questions which in turn should be investigated in new studies.
[Slide 3: Research Planning]
Taxonomy testing should begin with a research plan. There’s a recent Nielsen Norman Group (NN/g) article on developing user research plans by Maria Rosala. It provides a good overview and even provides a research plan template. The research plan should
- Clearly state the testing goal,
- Describe the testing activity, and
- Identify the target audience.
This is what you need to sell to your sponsors. Once they’ve agreed in principle to the research activity, then you should specify the task details or tactics. For example, if the activity is scenario testing, then you should articulate all the scenarios you plan to test. This may be a representative sample of user tasks that were elicited during stakeholder interviews or constructed based on frequent search queries (from organic or site search logs). Depending on the target audience, other tactical information could include a questionnaire for screening the study participants, an email invitation to study participants, and so on.
[Slide 4: Testing Goals]
Let’s talk a bit more about research study goals and the tactics to achieve their objectives. There are many possible objectives of user research. Depending on the study’s objectives you may choose to use different tactics in your study.
If I am focusing on whether the taxonomy is “good enough” to implement, then the tactic may be a focus group. To achieve this objective, it may be sufficient to walk through and discuss the taxonomy in a focus group consisting of 4 to 6 product managers for key product lines. Exploring questions like:
- “What do you think of these product categories overall?”
- “Are there too few or too many?”
- “Are there any missing?”
If the focus is to gather evidence that the taxonomy will work to find certain products or content, then the tactic may be tree testing. To achieve this objective, you might have a group of representative customers navigate a taxonomy to find a representative set of products or content items. Then you could measure the success or failure to find the products or content items.
If the focus is on tagging content with taxonomy categories, then an appropriate tactic may be to tag some sample content. To achieve this objective, you might have a group of product marketing staff each tag the same 10 items. Then you could compare the tagging to determine how consistent or inconsistent the tags were.
Other testing considerations may relate to the taxonomy lifecycle. For example, if the taxonomy is in production, then you may be concerned with assessing the ability to respond to change requests, or accessing the ability to consolidate or expand categories based on category usage. (I will not focus on taxonomies that are in production in today’s talk, but it is very important to maintain your taxonomy over time so that it remains effective).
[Slide 5: Qualitative Testing Methods]
This table shows various types of qualitative testing methods, what is the testing activity, who is the target audience, what is required to do this sort of activity, and what outcome is being measured. All these types of qualitative testing methods are used throughout the taxonomy development process. In the early stages of taxonomy development, these are sanity check-ins with key stakeholders to get early feedback and be sure that you are on the right track. In later stages you are validating that the taxonomy is or will be effective. Today I will focus on methods that have been adapted from usability testing. Taxonomy Strategies was an early adopter of usability testing. I remember being inspired by people like Indie Young at early Information Architecture summits, and have been particularly influenced by her book on Mental Models which is listed on my resources slide at the end of this presentation.
[Slide 6 Digital Taxonomists] For taxonomy developers or digital taxonomists, the objective of these contextual analysis activities is to 1) reach consensus among stakeholders on the broad outline and relationship among categories, and 2) to gather empirical evidence that provides confidence that the categories will enable successful completion of tasks, that is, conversion in whatever context that outcome is measured such as find it, retrieve it, buy it, use it, save it, and so on.
[Slide 7: Quantitative Testing Methods]
This table shows various types of quantitative testing methods, testing activities, target audiences, requirements, and outcomes to be measured. Most of these methods require collections or content that has been tagged with a taxonomy or that has been curated for a specific scope of purpose. In these cases, the analysis is testing the “fit” of the taxonomy to the collection based on expected distributions or predicted categorization based on information or language statistics. The one exception is closed card sorts which is a usability technique to measure human consensus and consistency.
[Slide 8: Contextual Analysis]
In the rest of this talk, I will focus on some contextual analysis methods of taxonomy testing in more detail. These include
- Facilitated focus groups via online meetings,
- Online closed card sorts with 30-40 representative categories,
- Online scenario testing of taxonomy trees, and
- Group content tagging.
[Slide 9: Target Audience]
Regardless of the activity (focus group, card sorting, scenario testing, and others), identifying the target audience is very important to the client. The first consideration is whether the target audience will be internal or external. Sometimes clients do not want you to interact with their customers for a variety of reasons. Instead, work with staff who act as surrogates for the ultimate target audience. In some cases, clients are concerned that internal staff will not be credible substitutes for their customers. Both situations provide valid concerns, but in practice I have found that there is not much difference between internal staff surrogates and actual customers. [Click] However, the most important consideration is what target audience will have credibility with the client.
While most target audiences come from existing client lists, it is useful to know the steps to identify a target audience from scratch. This is typically done by identifying the key criteria for the target audience which are
- Demographics (such as age),
- Behavioral characteristics (such as how often you visit the site),
- Psychographics (such as how you feel about the site), and
- Needs and benefits (such as did you find you were looking for on the site).
These key criteria are turned into a set of questions used to screen participants to identify a representative sample. For example, the client may want a representative sample in terms of geography, age, role, experience, and satisfaction with the site. Customized screeners can be administered directly via “intercepts” on a website to reach actual users in real time using tools like Ethnio; or generic screeners can be used to recruit participants from existing pools of testers using tools such as Optimal Workshop.
Screening services also facilitate incentives that can be offered to participants. Incentives may be cash, gift certificate, or merchandise. In my experience, the incentives ensure a high level of participation in testing in a short time frame.
Finally, the number of participants can be a key factor in the credibility of taxonomy testing. Different types of testing activities have different participation thresholds that make a meaningful difference. One-on-one user research is the most limited in terms of the number of sessions that can feasibly be conducted in a short time frame. For taxonomy testing I sometimes conduct one-on-one tagging activities. I usually limit this to ten sessions. The results are helpful for validation with content and taxonomy editors but are ultimately not persuasive for the project sponsors who decide whether and when to go live with a taxonomy. Card sorting results tend to resolve around consensus at 20-30 sessions. Additional sessions do not generally add more clarity. Tree testing tends to resolve at 50-100 sessions. It’s no problem to run more online tests using tools like Optimal Workshop assuming you can identify testers, but more sessions do not necessarily add to more reliable results. Nevertheless, credibility may hinge on quantity rather than quality.
[Slide 10: Facilitated Focus Group]
Facilitated focus groups are small group interviews with participants who have common traits. For taxonomy validation, I convene groups of senior stakeholders such as department heads or editorial boards or groups of senior staff who function as de facto editorial boards for the organization. I pose a set of high-level questions related to the taxonomy and gauge the participants’ reactions. For example, these are typical questions that I ask:
- What do you think about these Topics overall?
- Are there any Topics that you did not understand?
- Are there too many or too few Topics?
- Are there any Topics that can be combined?
- Are there any Topics you expected to see, that you think are missing?
This is usually sufficient to get a good discussion going. The goal is to identify any problematic concepts or labels, and get advice on how to correct these problems. The aim is to gauge whether you have a good candidate taxonomy for further more detailed testing, or whether you need to do more work and come back to this group again. In effect, this is the kick-off of Stage 4: Testing and Validation of the Stage-Gate Process.
[Slide 11: Closed Sorting Exercise]
Closed card sorting helps reveal the degree to which participants agree on which cards belong under each category. In this example a variation of card sorting is used where users are asked to sort representative examples into broad content types. One of the challenges with card sorting is providing some context for the items that you want sorted, as well as the categories into which you want them to be sorted. Tools like Optimal Workshop provide various techniques to add context. These include roll-over pop-ups where descriptions or graphics can be displayed. However, in my experience, the additional context doesn’t make much difference because users tend to be lazy and don’t pay attention to the extra information. In the end, the categories need to be clear and intuitive so that users understand what they are, or can learn the frame of reference quickly.
Online tools provide real-time analysis of the co-occurring patterns. Card sorting results do not get clearer by doing large numbers of sessions. In my experience the patterns emerge after 20-30 sessions. Instead, running more card sorting scenarios can be beneficial to test changes in categories and labels.
[Slide 12: Scenario Testing]
Tree testing is best facilitated by using scenarios to trick the participant into reviewing the detailed taxonomy categories. While I invite participants to provide comments about the taxonomy and not just the specific scenario they are trying to solve, I rarely get much feedback about the taxonomy in general. I collect and validate use cases in the early stages of taxonomy development and then reuse them as the scenarios for this tree testing activity. Sometimes I call this “Find It” testing, meaning “Where would you look among these categories for XYZ?”
[Slide 13: Taxonomy Tagging]
Besides searching for content, the other major use case for taxonomy in content oriented projects is for tagging content. This might be inline tagging within text, it might be tagging large content items like reports or articles, or it might be small chunks of content like a product description. Tagging may be descriptive answering the question “What is it?” such as a report or article, or tagging may be topical answering the question “What is it about and why is it important?” such as sustainability or heart disease. One of the most difficult things about tagging is how to focus on the most important type or topic, and not all the types and topics that are mentioned in the item. Tags are not the only way that an item will be found, but they will be used to support other use cases related to content assembly, publishing, distribution, and weighting. The objective here is to train editors how to use the taxonomy to appropriately tag content, and to validate that the taxonomy is fit for this purpose.
[Slide 14: Summary]
In summary, everything I have discussed needs to be in your research plan. The purpose is to identify what you are testing, why you are testing it, who you are testing it with, and how you are testing it.
Remember that your key stakeholders must approve your plan, and that testing is a means to an end and not an end in itself. The true goal of testing is to provide credible evidence that the taxonomy is fit for purpose and is ready to be implemented. This is not about demonstrating that the taxonomy is perfect and that it will result in world peace. This is not about doing research that will be published in a scholarly journal. It is about doing practical demonstrations of how the taxonomy will function and deliver value as part of your content ecosystem.
[Slide 15: Resources]
This is a list of some commercial tools and articles that I’ve mentioned in this talk.
[Slide 16: Earlier Versions of Presentation]
These are references to some earlier versions of this talk. As I mentioned, I’ve been doing taxonomy validation testing at Taxonomy Strategies since the early 2000’s, for almost 20 years.
[Slide 16: Questions]
Please let me know any questions. I’m happy to answer any you have now, or please write to me at jbusch@taxonomystrategies.com and you can find more information on my website which is www.taxonomystrategies.com. Thank you.