Presentation Description
This presentation traces the speaker’s journey with Knowledge Organization Systems (KOS) from early vocabularies to their vital role in Generative AI. Early efforts in knowledge mapping anticipated today’s AI, though modern LLMs (Large Language Models) became possible with advancements like GPUs and transformers. Joseph Busch stresses that while LLMs are powerful, their reliability improves significantly by integrating KOS, particularly through Retrieval Augmented Generation (RAG), which injects relevant, pre-processed information. The ongoing NKOS workshops are crucial for bridging KOS and AI to achieve more precise and dependable information retrieval.
Date: June 25, 2025
Venue: ISKO VIII Brazil International Society for Knowledge Information
Location: Canela, Brazil.
Broad Topic: Information Retrieval
Presenter: Joseph Busch, Principal Analyst, Taxonomy Strategies
Watch this Presentation
Download Presentation: MP4 | PDF | PPTX
Download Slide Script: DOCX | PDF
[Slide 1]
The Struggle to Make Language Models a Human Powered Instrument: How Did We Get Here and What Can We Do About This?
Good afternoon, everyone. Thank you very much for inviting me to be the keynote speaker at the 8th Brazilian Congress on Knowledge Oraganization. In this talk I will tell the story of a life spent working on knowledge organization systems (KOS), from faceted vocabularies to generative artificial intelligence (Gen AI). Language generation has a long lineage. I will briefly discuss the Getty vocabulary program, the origins of NKOS (Networked Knowledge Organization Systems), Metacode content processing pipelines and knowledge maps, AI before language generation, what made large language models (LLMs) possible, and RAGs (Retrieval Augmented Generation). Like organic web search I think LLMs are an example of an information technology that has the potential to deeply enhance human expertise, but too often information technology is seen as a means to replace the inefficiencies of people. I will talk about the challenge of how to effectively program generative AI using natural language instead of the old programming languages. I think that KOS will play an important role in this, but we will need to define concepts and relationships in ways that can more readily be consumed by LLMs. Over the last few years, I have organized NKOS workshops on KOS in AI and AI in KOS. Finally, I will share with you what I have learned about the future of KOS and AI from these workshops.
[Slide 2]
In 1997, I organized the first NKOS workshop as part of the second ACM Conference on Digital Libraries held in Philadelphia, PA. NKOS stands for Networked Knowledge Organization Systems. At the time I was working for the Getty Art History Information Program which was the Getty foundation program that supported the development of the Art & Architecture Thesaurus known as the AAT, the Thesaurus of Geographic Names known as TGN, and the Union List of Artist Names known as ULAN – these are the so-called “Getty vocabularies.” We also supported the development of art and architectural history indexing services as well as digital archives projects related to the provenance of cultural objects and the rediscovery of antiquities during the European Renaissance.
[Slide 3]
Besides designing the means to publish and maintain the well-known Art & Architecture Thesaurus described in a chapter in an American Society for Information Science monograph titled “Studies in Multimedia.”, a major contribution to the field of knowledge organization was the so-called “Getty systems data model” described in a paper published in the proceedings of the Second International ISKO Conference which was also a primary inspiration of the CIDOC Conceptual Reference Model (CRM), an ontology for concepts and information in cultural heritage and museum documentation.
[Slide 4]
To give you a flavor of the state of the art in multimedia information systems in the early 1990’s, this is a picture of Arnold Nesseltrath, the Director of the Census of Antique Art and Architecture Known to the Renaissance database project giving a lecture using the system which included 2 videodiscs and 2 monitors integrated with a relational database running on a Unix computer. That’s me following a script where I typed in queries to retrieve data and images to illustrate Arnold’s talk. Arnold is currently Professor of Art History at the Humboldt Universität in Berlin. From 2009 until 2017 he was Director of the Art History Departments of the Vatican Museums.
[Slide 5]
From the same time period, this is the official group portrait from the OCLC NCSA (National Center for Supercomputing Applications) Metadata Workshop in 1995 which was the first Dublin Core Workshop. I didn’t attend the workshop, but the Getty sent a representative from my team. The Dublin Core became the de facto standard for metadata description of web pages, was adopted by Microsoft and Adobe, and is evident in many widely used software products to this day. The National Center for Supercomputing Applications is perhaps best known for developing the Mosaic browser which was instrumental in popularizing the World Wide Web by integrating multimedia such as text and graphics. Mosaic was the first browser to display images inline with text instead of a separate window or on separate screens like the Census multimedia system.
[Slide 6]
This was the early days of the World Wide Web and it was just beginning to open up beyond research scientific organizations like CERN (L’Organisation européenne pour la recherche nucléaire) where the Web was first invented by Time Berners-Lee, to libraries, archives, and museums. For the first NKOS Workshop at the second ACM digital libraries conference, I invited my colleagues who were doing interesting work on developing faceted indexing schemes for the Getty Bibliography of the History of Art, the Modern Language Association International Bibliography, and other resources. In retrospect, I think we had an awareness that we were on the verge of a major digital transformation in libraries, archives, and museums. But we certainly didn’t realize how dramatic and complete this change would be over the following years. Nor did we imagine that the NKOS workshop would continue to this day.
At the time computers were still very limited in terms of processing speed, memory (RAM), and storage (floppy and hard disks), like dinosaurs they were physically large but with very small brains. The main thing about the web was that it began to be possible to connect computing devices together so that local storage and processing power became less problematic. The infrastructure began to be built where you could easily look things up online at little or no cost.
One year later, in 1998 after ten years of working in the culture sector, I was recruited at the American Society for Information Science meeting in Portland, Oregon to work at a dot com start-up company. This was not a dot com that wanted to sell stuff online. It was a company that wanted to organize very large amounts of content to try to resolve very complex problems. My job title was “Vice President for Information Product Development”.
[Slide 7]
The Datafusion company was the brainchild of Joel Schatz, who had made a lot of money building a mobile telephone network in Russia after the fall of the USSR. He was a proponent of systems dynamics – “a methodology and mathematical modeling technique to frame, understand, and discuss complex issues and problems.” Joel wanted to build a tool that could visualize inputs, processes, and outputs with knowledge maps. “Knowledge mapping is a structured process that visually represents the flow, storage, and access to knowledge within an organization.” Joel wanted to develop a tool that could visualize big systems and the relationships between entities – systems related to water resources, air quality, transportation, and world peace. These sorts of systems models needed content, lots of content, and lots of that content was not structured much or at all.
[Slide 8]
To make that less structured content useful, we needed to develop a content processing pipeline that could apply language processing methods iteratively to break down the narrative into meaningful chunks and then identify and disambiguate the named entities. Finally, the content needed to be analyzed to determine what it was about and why it was important.
[Slide 9]
Once we had built a highly curated collection and tagged the hell out of it, we could generate visualizations of the relationships among the named entities within the chunks and ultimately construct many narratives. Sounds a little like generative AI. It was, except without any of the computing power that we have today. Simply put, n-gram level tokens were out of the question because it was simply impossible to hold a large enough matrix in memory to compute relationships at a very granular level.
[Slide 10]
Indexing is one of the core functionalities of information retrieval. In addition to natural language processing and entity recognition, we needed to do content tagging at scale. In 1998, a number of information retrieval engines using vector and probabilistic statistical methods were available as open source or as licensable OEM products. Latent Semantic Indexing and clustering were still primarily in research and development. We opted to implement a vector method because it generally performed better in our testing. The challenge was to build training sets. While we focused on domains where there was readily accessible well-categorized content such as medicine and law, we were constantly asked to build general purpose categorizers for common business functions, but we failed to be able to do this effectively.
[Slide 11]
At the same time web search engines were beginning to be available on the fledgling World Wide Web quickly displacing the Yahoo and dmoz topic trees for browsing the Web. AltaVista was the first web search engine to become wildly popular until it was overtaken by Google in 2000. Previously online searching was only available through expensive time share services or onsite at some research libraries, it was limited to a few published subject indexes, required training and expertise to use effectively, and only got you a bibliography, not usually the full text of items. Web search engines were accessing full text resources limited only by what was available on the web, access was free, and the user interface was extremely simple—type and go. But under the hood, the democratization of online searching was ranking search results by popularity rather than Boolean searching subject and named entity indexes to find relevant and comprehensive results. While everyone could use Google, not everyone was able to get good search results. Even with the dead simple user interface, choosing effective search terms and knowledge about Google syntax could enable better search results. For many, the new millennium seemed like the golden age for information access.
[Slide 12]
In 2000, Datafusion was renamed Metacode. Metacode was the name we gave to the unique identifiers assigned to concepts in knowledge maps. This was an early implementation of namespaces and universal resource identifiers (URIs) and one of the first instances of assigning URIs to concepts such as terms in a controlled vocabulary. While Metacodes pre-date SKOS, the SKOS data model views a knowledge organization system as a concept scheme comprised of a set of concepts. We built a vocabulary management tool for creating and managing metacodes, a content tagging pipeline for assigning metacodes to content items, implemented a metadata database for storing metacodes associated with content objects, and provided a graphical user interface for viewing and browsing metacodes in knowledge maps.
[Slide 13]
By the end of 2000, we had a few early customers and Metacode was acquired by Interwoven, a publicly traded web content management company, for the astounding price of $150 million. Interwoven rebranded the product as Metatagger emphasizing the content tagging pipeline for automating the generation of metadata to enable methods of publishing dynamic website content.
After spending two years building information product domains at Metacode, I spent two years working at Interwoven promoting tagging content with metadata. Think of this as moving from building the Art & Architecture Thesaurus ontology for material culture, to promoting the use of Dublin Core metadata for cataloging content. Interwoven was a much larger company than Metacode. I was assigned to work in field sales which means that I went on calls with sales professionals all over the world to all kinds of businesses and organizations in many different sectors who were all trying to figure out how to do business on the world wide web. My job was to promote metadata tagging using the Metatagger product as a part of the Interwoven web content management ecosystem. I had the job title of “Content Commando”. I was exposed to myriad use cases every day of the week. It was overwhelming and a crash course in business information management. The business organizational concerns about how to make money and stay out of jail may not have been as lofty as Datafusion’s concerns about water, transportation, and peace but they were more immediately actionable and measurable.
[Slide 14]
One of those sales calls was to NASA’s Jet Propulsion Laboratory (JPL) which became the first project of a new company, Taxonomy Strategies, that I would form with my former Datafusion chief scientist Ron Daniel. Taxonomy Strategies worked with JPL to develop the NASA Taxonomy from 2002-2005. In 2003, Taxonomy Strategies created the content model for the first U.S. government web portal firstgov.gov (now USA.gov) and later developed web1.0 information architectures for many U.S. government agencies. Taxonomy Strategies also developed the first online shopping taxonomy for HomeDepot (a large home improvement retailer) in 2003 and then went on to develop consumer and business-to-business online shopping taxonomies for many well-known companies like Dell, Nike, and Oracle. By applying the lessons learned at Datafusion, Metacode, and Interwoven, Taxonomy Strategies had the opportunity to work on dozens of projects in various sectors that have helped to make digital assets easy to find, use, and share.
[Slide 15]
In retrospect, the Datafusion start-up company neglected to pay enough attention to certain aspects of artificial intelligence that emerged in the late 1990’s. While we made the right decisions to focus on natural language processing, we should have taken machine learning neural network research more seriously. But we had a reluctance to embrace “black box” technology. We were more interested in automatic classification, especially using predefined knowledge organization systems such as NAICS, MeSH, and the AAT. We conducted experiments to optimize the process of building training sets. Generally, this required finding KOS with well-categorized content. But the biggest challenge was finding discrete training sets, meaning content that was about one discrete concept and a critical mass of these examples. We didn’t attempt to build any machine learning with regression and recurrent processes and processing, and to be honest we didn’t have the expertise to do that in our team.
[Slide 16]
Datafusion was five or ten years too early to be able to commercialize machine learning. The real resurgence of neural network research didn’t happen until 2005, and the graphical processing unit (GPU) only began to be used to rapidly perform the really large numbers of calculations required by machine learning neural networks in 2010. But Datafusion demonstrated the value of KOS and rich metadata in web content management even if the automation of its production was rudimentary.
While the GPU has provided the computing horsepower for the current generation of AI, the key innovation that characterizes the current generation of AI is a special type of neural network architecture called a transformer. Transformers enable neural networks to keep track of patterns in their input using “self-supervised learning” on large amounts of raw text generally taken from the Internet. This is highly effective for natural language processing and natural language generation as illustrated by Large Language Models (LLMs) like ChatGPT, Claude, Llama, and DeepSeek.
[Slide 17]
These models are trained to analyze text and estimate the most probable word following a given sequence.
Because these models are trained on enormous amounts of content (13 trillion tokens or the equivalent of 80 million books, more than twice the number of books in the largest research library imaginable), they are very good at the task of language generation.
Another remarkable aspect of the current generation of AI is that the user interface is natural language. When you provide instructions to an LLM it is very reminiscent of a web search engine. In the LLM ecosystem natural language queries are called prompts, and the generation of prompts is called prompt engineering. You can think of web search engines as examples of so-called “zero shot” prompts meaning that you type in a query and get a response, even though the prompt itself was not necessarily part of the training data for the model. Also, with a zero-shot prompt there is not necessarily an intention of iterative prompting. The choice of terminology, style, and format of the prompt is very important. Changing a single word can significantly alter the output.
[Slide 18]
Prompts need to be precise, avoid jargon, use punctuation, and be structured like a use case; meaning it should identify your perspective or role (who), what you want to know and why, and how you desire the output to be formatted. This table illustrates a “zero shot” prompt template and some example prompts.
[Slide 19]
More interesting than zero shot prompting is effectively dialoging with an LLM. This can be with a few examples, or can be through reasoning or thought generation, decomposing a complex problem, providing verification, or a combination of perspectives …and there are other techniques.
Chat user interfaces to LLMs keep a record of each session and provide an option to start a new chat. This provides an opportunity at least for immediate or near-term iteration without any programming.
[Slide 20]
An interesting development is the emergence of prompt collections and libraries. These are curated prompts grouped by broad categories, which are free and for sale.
The good news about AI is that everyone is fiddling with it. The bad news is that the results are unreliable. I think it’s similar to the experience with web search. Everyone is looking things up. The results are good enough. This search provides answers to simple questions, but it can be frustrating to get definitive answers, find a known result, or do any sort of in-depth authoritative research.
[Slide 21]
To improve the reliability of search you need to use better search terms. One way to do this is to look for and choose search terms from KOS. The same is true for improving the reliability of KOS.
I remember interviewing an art historian many years ago when I was working at the Getty Trust. I asked her: “How do you use the Art & Architecture Thesaurus?” [Click] She answered: “It helps me think.”
[Slide 22]
The approaches to integrating KOS into prompts are the same as for integrating KOS in information retrieval:
- Incorporate synonyms (and quasi-synonyms) in the prompt using a domain thesaurus
- Broaden or narrow the concepts in the prompt using a hierarchical taxonomy
- Provide specific contexts using facets such as content type, geography, language, time, process, or other contexts
While it’s possible to manually refer to KOS while writing prompts, LLMs integrate with other systems through the “context window”. The context window is the input text that the LLM can process. LLMs predict the next token based on the input sequence, and by manipulating this input, one can affect the model’s output. The ChatGPT context window is capped at 32,000 tokens. A token is roughly equivalent to 4 characters or about three quarters of a word. But processing large context windows can be computationally expensive and slow.
[Slide 23]
An application programming interface (API) allows for development of applications that can integrate additional resources such as KOS and search results into prompts. The ChatGPT context window is up to a million tokens when accessed via the API. In this context KOS can also be used to:
- Guide language generation results by identifying entities in the output, linking entities to knowledge graphs, and connecting models to knowledge bases;
- Index source content to improve document recall and precision; and
- Annotate output to improve explainability and trustworthiness.
Retrieval augmented generation (RAG) involves searching for relevant facts and inserting them into the context window to improve the model’s answers. The approach can be expanded to include various data sources beyond search engines, such as an indexed domain specific collection, databases, and calendars, making it a complex but powerful tool for enhancing LLM performance. The response may include annotations, such as source citations or bibliographic references from the domain specific collection content. A good way to think about RAG is as a method for pre-processing LLM prompts to make them more responsive to domain specific questions.
A knowledge graph is a representation of the relationships among named entities. A knowledge graph may be queried in the RAG method to improve the information retrieval with equivalent terms but especially to provide associative, typed related concepts. It can also be integrated as a post-process to an LLM response to identify possible refinements through reasoning, decomposition, verification, or other methods.
RAG applications may use information retrieval in different ways. In particular, they may not necessarily index content using KOS. They may use vector search to identify similar content statistically without any reference to KOS. A better way (to achieve more reliable and measurable better recall and precision) is to use vectors to map content to KOS, and then to use KOS to identify similar content.
[Slide 24]
The good thing about standards is that there are so many to choose from.
The same is true of KOS. BARTOC, the Basic Register of Thesauri, Ontologies & Classifications, a database of KOS, lists over 3700 resources. ISO has more than 900 terminology standards. Some of these are standards are about how to create and manage terminologies like ISO25964: Part 1: Thesauri for information retrieval; management and use. Others are standard domain-specific terminologies. It is challenging to manage terminology and vocabularies in various fields, and particularly in biomedical science. Existing systems like the Unified Medical Language System (UMLS) which has more than 105 sources, are often inconsistent and complicated.
[Slide 25]
This is a big challenge for the AI community. This is a big challenge for developers. They are simply not equipped to deal with the intricacies of assessing and selecting KOS and integrating them appropriately into general purpose applications. LLMs make it easy to get to 80% accuracy quickly, which can lead to ad hoc solutions rather than systematic approaches. This is one of the reasons that AI has generally avoided KOS. It has been left to consultants to develop custom applications for niche industries where the benefits of more precise retrieval outweigh the difficulties and costs of KOS integration and development, or to researchers.
However, I think there has always been an interest in a general purpose categorizer for common business functions, just as there is an interest in universal classification systems for organizing content collections such as the Dewey Decimal Classification (DDC), the Library of Congress Classification (LCC), the Universal Decimal Classification (UDC), Integrative Library Classification (ILC), and others. I would argue that general purpose systems would be welcome as part of the AI ecosystem assuming that they were intuitive, easy to use, kept up to date, and free.
[Slide 26]
The Networked Knowledge Organization Systems/Services Workshop or NKOS Workshop has been convened annually since 1997 in the U.S. and/or Europe and occasionally in Asia. NKOS is devoted to the discussion of functional and data models for enabling knowledge organization systems/services or KOS—such as classification systems, thesauri, and ontologies—as networked interactive services to support the description and retrieval of diverse information resources. According to Tragott Koch and my co-chair, Doug Tudhope, “Knowledge Organization Systems/Services (KOS) model the underlying semantic structure of a domain. Embodied as Web-based services, they can facilitate resource discovery and retrieval. They function as semantic road maps and make possible a common orientation by indexers and future users (whether human or machine).”
[Slide 27]
Although incorporating KOS in information retrieval applications has been a long-standing interest of NKOS, the current co-organizers Marcia Zeng (Kent State University), Douglas Tudhope (University of South Wales), and myself began focusing on KOS and AI in 2023. The workshop held in Daegu, South Korea in conjunction with the International Conference on Dublin Core Metadata Applications included two presentations. This was followed by two workshops in 2024, one in March at the biennial ISKO conference in Wuhan with four presentations, and one in October at the International Dublin Core Conference in Toronto with ten presentations. Finally, a special issue of KO (Knowledge Organization) is in preparation with five research papers. These materials cover various approaches to using LLMs as tools.
[Slide 28]
- Projects that build more consistent ontologies and taxonomies,
- Projects that enhance resource description with more complete and consistent metadata and annotations,
- Projects that are applications of KOS-based classification, and
- Projects that are conducting domain studies of LLM and KOS users and uses.
The range and breadth of this work is impressive. As a co-organizer of the NKOS workshop and co-editor of the special issue on KOS and AI, I’ve had the opportunity to read and review most of the submissions, and to listen to and moderate the workshops. I have reflected on these many and varied research projects and marveled at the inspiration and creativity and level of interest and engagement with the AI.