Images as Knowledge Repositories

Dr. Lyndon J.B. Nixon is a pioneer in the field of semantic technologies. Since the early days of online search engines, the researcher has been exploring how data on the World Wide Web can be made readable and accessible to both machines and humans. As an assistant professor at Modul University Vienna and CTO of Modul Technology, he is currently studying the topic of media understanding. He investigates how images and videos can be enriched with data that allows machines to “understand” the world of humans.

In his research projects, Dr. Nixon has worked on applications of knowledge-based technologies in the cultural sector for 20 years. As coordinator of the GreenGLAM project, he is now helping to connect digital knowledge repositories with sustainability issues.

Lyndon will co-host the cultural hackathon as a mentor and we are looking forward to his expertise. Registration is open: https://pretix.eu/OpenGLAMat/GLAMhack25/

What Are Semantic Technologies?

Computers process information in the form of numbers. Semantic technologies allow machines to understand information through knowledge-based models and to place it in a specific context. In fields like artificial intelligence, data analysis, or knowledge management, they ensure that human users find what they are looking for more quickly and help with interpretation. Semantic systems understand not only what is being searched for, but also why.

What Makes AI so Promising for GLAMs?

GLAMS such as museums, archives, galleries, and libraries often work with databases and catalogs that use text-based searches. However, for such institutions, images often harbor much more information than can be covered by textual descriptions alone. In these cases, it is more useful to enrich digital images with information, so-called semantic metadata, which a machine can read. One method of achieving this is through knowledge graphs, huge networks of knowledge that link data on objects, concepts, events, or people from the real world. This makes it possible, for example, to find artifacts, documents, or artwork in a database, even when you only know a certain visual element or don’t remember the exact title. The system understands what you mean, even based on incomplete or outdated information.

In addition, cultural works are subject to constant reinterpretation, based on factors like their historical context. National borders shift over time, an artwork or artifact can change its meaning depending on its viewers and their location in space and time. We can enrich images and photos with data that sheds light on the worldview or social background of the original creators, helping to recontextualize works and capture a cultural dialogue.

What Are the Benefits for GLAM Audiences?

As we discussed above, AI-based technologies facilitate the research work of archivists or curators, thus leaving more time for presentation and education. In the GreenGLAM project, we are going one step further, giving the public access to AI tools to help them find out more about the GLAMs themselves.

As a practical use case, we have chosen the United Nations‘ Sustainable Development Goals (SDGs). Galleries, libraries, archives, and museums often don‘t have an overview of the impact their work has in relation to the SDGs, even if they may already be actively contributing to them. GreenGLAM will use AI to link knowledge from GLAMs with public discussions in online media and connect them to the relevant SDGs. Visitors who want to know what a certain GLAM contributes to the SDGs can then explore the collected data processed in interactive data sculptures using augmented reality (AR). This gives them the information they need and at the same time lets them try out innovative technologies for themselves.

The GreenGLAM project, funded by the Austrian Research Promotion Agency and the Federal Ministry of Climate Action, Environment, Energy, Mobility, Innovation and Technology,  has just started. 20 years after his first research collaboration on this topic, Dr. Nixon is looking forward to showing how semantic metadata and knowledge graphs can further improve computational understanding of the world. The GreenGLAM project team will spend the next three years working to advance these technologies to help GLAMs and the public understand how cultural institutions contribute to sustainability issues.