Internship: Relational Background Knowledge and Scene Graphs for Object Detection Regularization

Come work with us as a Research Intern

At BrainCreators, we're at the forefront of applied AI with many years of successful research internship projects that combine cutting edge science with the challenges of applying AI in the real world. The focus of this year’s AI research internship projects will be on the technical challenges at the heart of our Machine Learning platform, BrainMatter.

What we expect from you

  • A full-time commitment to the research internship project.
  • A solid background in the theoretical subjects relevant for your particular project and ML coding skills in pyTorch.
  • Good communication and presentational skills, and a willingness to learn as much as possible in this exciting year.
  • Your project will have a scientific component on which you are encouraged to work towards a publishable paper at the end of the year.
  • Your project will also have an applied component, the result of which is a functional and documented piece of cutting-edge software that can be integrated into BrainMatter.
  • Bachelor’s degree in Artificial Intelligence or related field.

What we can offer you

  • The opportunity to work in our research team as a full time member.
  • A workplace in our Prinsengracht HQ with access to our compute cluster if required.
  • Support and supervision, including a weekly personal supervision meeting and research team group meeting as well as support for integration into our software stack when needed.
  • Internal weekly workshops about scientific and industrial progress.
  • Become part of a vibrant team of AI realists that know how to get things done.
  • Our best interns will be offered a full time job opportunity after graduation.

Project overview

The frontal pose of the human face has a vertical line of symmetry, one nose above the mouth and below the two eyes. This is an obvious fact to all of us. However, most Deep Learning facial understanding never used a descriptive, explicit piece of information like this. Instead it relied only on a very large set of examples, and its implicit information encoded in the input space distribution only. Although this is currently an accepted state of affairs in Deep Learning, it might not be the best way forward in the future. In particular, if the application concerns object types for which there simply are not many examples, while explicit relational and rule-based descriptions are almost trivially available. 

This certainly applies to some of the use-cases BrainCreators currently works on in the field of visual asset inspection. The asset might, for example, be a segment of road surface, an item on a production line, or a piece of real estate. Some visual data is available, but image annotation efforts are expensive. In settings like this, the Deep Learning pipeline could potentially greatly benefit from explicit knowledge in the form of rules or other relational descriptions that would be relatively easy for a domain expert to formulate. 

The question is then: how to exploit such knowledge, and how to integrate it into the Machine Learning pipeline? There are many ways to combine symbolic information with statistical learning. For a broad overview of existing methods and design patterns, see [1]. A survey of the field of Neural-Symbolic Integration can be found in [2]. 

In this project we would like to focus on the particular approach of using explicit object descriptions as regularizers for the learning process. The central idea is that, instead of introducing bias in the form of more training data, the explicit object descriptions would be exploited to act as regularizers on the learning process. One example would be the work on Logic Tensor Networks (LTNs) for semantic image interpretations [3] that allows the formulation of logical soft-constraints to be integrated into the Deep Learning pipeline in a differential way. 

The notion of Scene Graphs is particularly useful in this regard, and constitutes an active area of research that the intern is encouraged to take into account. Examples of this are [4] and [5], that each use scene descriptions to improve the understanding of objects and their relationships. For inspiration from a cutting-edge approach that applies these ideas to video, see [6]. More examples of this active area of research can be found via [7]. 

The practical goal of the project is to work towards a software deliverable that integrates one or more methods from this field into our Machine Learning platform, BrainMatter. The research intern will set up their own experimentation pipeline to assess the strengths and weaknesses of a selection of approaches, starting with LTNs. The end-user should be able to provide object or scene descriptions that are taken into account by the automated ML pipeline based on our existing KubeFlow/Kubernetes workflows. The software should be as modular as possible to facilitate the application of the methods to a variety of object detection algorithms already present in BrainMatter. 


At the same time, the research intern is expected to formulate a relevant scientific hypothesis concerning this topic. There will be substantial freedom for the intern to consider possible academic research questions and work towards a publishable paper at the end of the project. 

[1] A Boxology of Design Patterns for Hybrid Learning and Reasoning Systems.   [2] Neural-Symbolic Learning and Reasoning: A Survey and Interpretation 

[3] Logic Tensor Networks for Semantic Image Interpretation   [4] Improving Visual Relationship Detection using Semantic Modeling of Scene Descriptions 

[5] Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships  [6] Learning Physical Graph Representations from Visual Scenes 

[7] Scene Graph Representation and Learning (ICCV2019) 


If you'd like to apply for this internship, send your CV and cover letter to our Head of Research, Maarten Stol.