ML and NLP approaches and the processes surrounding them often focus on finding a single “truth” or gold standard by majority vote and removing any “outliers”. From the curation of data sets (where we use measurements like inter-annotator agreement as a quality attribute) to the presentation of results (where less favorable options are regularly excluded), different perspectives and disagreements are typically excluded in ML. In reality, such a single truth often does not exist. From the legal to the medical domain, experts often have different opinions. Erasing deviating perspectives and disagreement is not only an untruthful representation of reality, but it can also negatively impact the reliability and usefulness of an ML system. In societal and political debates, having and exchanging different opinions is even one of the defining properties of democratic societies.
This PhD project aims to investigate how different opinions of and disagreements between annotators and other sources for “ground truth” can be represented in NLP and leveraged to improve the support that AI systems can provide to their users. The project will consider all steps of the process, from the gathering and annotation of data, the use of the data to train NLP models, to the representation of results.
During the project, the PhD candidate will perform research on some of the following tasks:
- Design experiments to create annotated text corpora;
- Develop new NLP models and approaches using Python and train and fine-tune existing open-source models;
- Perform intrinsic evaluation of ML models;
- Apply the insights gained to specific domains of application;
- Develop software prototypes for performing task-based evaluations.
As a doctoral student, your key responsibilities will be to manage and carry out your research project within 48 months and write a PhD thesis. You will participate in research and training activities and disseminate research in the scientific community (international conferences) and non-scientific community by outreach and public engagement. Furthermore, you will write articles for scientific peer-reviewed journals and progress reports and prepare results for publication and dissemination via public lectures, presentations, and the web.
You will also be responsible for taking courses per the research education and liaising with the other research staff, students, and partner institutions working in broad areas relevant to the research project. Participation in other department duties, such as meetings, teaching, etc., is expected to cover a maximum scope of 20%.