Segmentation is an important task in computer vision where the pixels belonging to a region of interest, that often share similar characteristics, have to be separated out, and assigned a unique label. This task is relevant in both the image and video domains, and finds direct applications in a wide range of fields including but not limited to autonomous driving, robotics, image editing, and surveillance. For both images and videos, the task of segmentation has various flavours that are often highly related and equally challenging. With the advent of deep learning, modern computer vision algorithms are able to leverage large amounts of available data to achieve impressive performance for many of these segmentation problems. However, such algorithms are often catered either towards segmenting a pre-defined set of object categories, or to some specific sub-domains, and require to be adapted for out-of-domain tasks by training these methods on additional domain-specific data, that is expensive to annotate. As a result, we need algorithms that generalise well to data from unseen domains and also to new task settings in addition to having methods that can efficiently annotate data.

The first part of this thesis focusses on efficiently annotating objects using Interactive Segmentation, where the goal is to segment and refine objects in an image using user clicks. Here, I'll present two of our research works that advances the state-of-the-art in the interactive segmentation domain.  The first among these, ITIS, develops a novel iterative training strategy in which clicks are added iteratively during training based on the error regions in the network predictions. The iterative training strategy aligns the simulated user-click patterns that are used during training with the actual click patterns that the network would encounter at test time. While this strategy is effective in reducing the number of clicks required to annotate objects, ITIS can only segment one object at a time due to the limitations of its network architecture. We address this problem in our subsequent work called DynaMITe, where we formulate user clicks as spatiotemporal sequences, and develop a novel Transformer based formulation that can process such a sequence and encode them into relevant object or region descriptors. These descriptors are then used to generate the relevant instance segmentation masks. Unlike previous methods, our architecture can process clicks for multiple objects at once, and correspondingly predicts non overlapping segmentation masks without any post-processing.