Till startsida
University of Gothenburg
To content Read more about how we use cookies on gu.se

Ielka van der Sluis "The PAT project: Annotation and Evaluation of Pictures and Text"

Research profile seminar

Ielka van der Sluis "The PAT project: Annotation and Evaluation of Pictures and Text"

In this talk I will present the PAT project in which we investigate the use, effects and optimisation of documents that contain pictures and text (PAT). While the benefit of including pictures has been established, the design of pictures, text, and picture-text relation(s) has not been researched in a systematic manner. PAT aims to gain an in-depth understanding of their characteristics to augment existing theories on cognitive processing of multimodal presentations. Resulting models will be validated by implementing them in natural language generation algorithms and comparing their output to human-authored presentations.

The PAT project launches a methodical investigation of multimodal instructions (MIs) used in first-aid practices to help people in need. Currently, there are no guidelines for the design of MIs that effectively instruct people to operate an AED, place a victim in a recovery position, remove ticks etc. The huge variations in pictorial and verbal means employed in these instructions demonstrate the urgency to obtain validated guidelines based on empirical evidence collected from readers and users. Investigating multimodality in these MIs allows us to evaluate the effectiveness of combining pictures and text in a practical context focussing on e.g. attention, comprehension, recall, user judgements, and task performance.

The PAT project makes use of a annotated corpus of MIs and a workbench that has been developed for the annotation and retrieval of the MIs. The MIs are first-aid instructions that appear in Het Oranje Kruisboekje and variations of these instructions from other sources, like hospitals, health and safety organisations and the internet.

In the PAT project approaches from Information Design Research and Computational Linguistics employing corpus collection and analysis, (automatic) annotation, experimentation, and natural language generation are combined. The project will deliver theoretical results in terms of empirically validated models for effective MIs. Results of practical value include new annotated multimodal corpora, implemented taggers to automatically annotate potentially effective properties of MIs, algorithms to automatically generate effective text-picture combinations and authoring guidelines to produce good quality instructions.

Project website:

Lecturer: Ielka van der Sluis Detailed CV: https://www.rug.nl/staff/i.f.van.der.sluis/

Date: 11/20/2018

Time: 1:15 PM - 3:00 PM

Categories: Linguistics

Organizer: CLASP

Location: Department of Philosophy, Linguistics and Theory of Science
Seminar room, Dicksonsgatan 4

Event URL: Read more about Ielka van der Sluis "The PAT project: Annotation and Evaluation of Pictures and Text"

Contact person: stergios chatzikyriakidis

Page Manager: Stergios Chatzikyriakidis|Last update: 5/23/2016

The University of Gothenburg uses cookies to provide you with the best possible user experience. By continuing on this website, you approve of our use of cookies.  What are cookies?