KnowPic is a framework for semantic image interpretation that leverages ontological knowledge for understanding the content of pictures. KnowPic extracts structured information from images and represent it in an RDF graph (= set of triples). The RDF graph produced by KnowPic contains nodes that refer to objects detected in the picture (bounding boxes), their types and their semantic relations. Object types and object relations are semantically described by the domain ontology. The automatic generation of picture content in terms of an RDF graph, as the one proposed by KnowPic, opens the possibility to use standard well-developed techniques for semantic image processing. Some examples are:
- semantic image retrieval via SPARQL;
- automatic caption generation via RDF/OWL verbalizers;
- image semantic enrichment by matching RDF image description and linked open data.
Currently, KnowPic is implemented for interpreting images according to the part-whole relation, that is, objects are whole objects or parts and arcs represent the part-whole relation between a whole object and its parts. This implementation is called Part-Whole Clustering Algorithm (PWCA) and it combines a deep-learning object detector and a clustering algorithm that mixes semantic and numeric features. In the Resources links below you can find the ontologies and datasets that we used for the experiments.
Publication: Ivan Donadello Ontology-Based Semantic Image Interpretation, Proceedings of the Doctoral Consortium (DC) co-located with the 14th Conference of the Italian Association for Artificial Intelligence (AI*IA 2015), vol.1485 2015, pp. 19-25, Ferrara, Italy, September 23-24, 2015.
Research topics: Knowledge representation