Image review
Humans aren't obsolete yet! Many use-cases require reviewers to be kept "in the loop" to correct any mistakes made by the ML models and fill gaps in their classification repertoire.
Understanding the Object/Label and Tag data models
Before moving on, it's important to first understand the Object, Label, and Tags data models, which are described in the Structure, concepts, and terminology page.
Managing Labels and Tags
Project Managers are responsible for creating and maintaining the list of allowable Labels and Tags that can be applied to images in their Project. This adds an additional setup step, but it is a useful tool for keeping your image annotations clean and consistent and it offers other users some guidance on how you'd like your images reviewed.
To create, update or delete your Project's Labels and Tags, select the "Manage Labels and Tags" button from the sidebar menu. From the popup that appears, you can:
Create new Labels and Tags
Edit the the name and color of existing Labels and Tags
Note: if the Label/Tag has already been applied to Images and Objects, editing it will also update all existing instances
Delete Labels and Tags
Note: deleting a Label/Tag will:
remove it as an option to apply to your images (if this is your only goal, this can also be accomplished by "disabling", rather than deleting, the label.)
remove all instances of it from your existing images
if deleting a Label, and the Label has been validated as the correct, accurate label on objects, deleting it will remove the Label and unlock those objects, which will revert all affected images to a "not-reviewed" state
THIS ACTION CANNOT BE UNDONE
Edit the enabled/disabled state of Labels
Note: disabling a Label will prevent users from applying it to images going forward, but it will not remove existing instances of the label on your images.
Note: machine-learning-generated Labels are added to your Project's list of allowable Labels automatically if any are predicted that aren't already included in your list.
Keyboard shortcuts
We highly recommend using the WASD or arrow keys to navigate forward and backwards through images while you're reviewing them. This allows you to use the keyboard to iterate through the images with one hand, while your other hand is free to use the mouse/track-pad to validate or edit Labels, apply Tags, etc.
Undo Label editing actions with ctrl-z, and redo them with shift-ctrl-z
Reviewed vs. Not-reviewed Images
An Image is considered "reviewed" once all Objects are locked (either a user has validated an ML-predicted Label, manually added their own Label, or invalidated all of the proposed Labels thus removing the Object).
Validating / invalidating / editing Labels
If a user "validates" a suggested Label, the Object will become "locked", and the most recently validated Label will become the "winning" Label. "Invalidating" a Label will remove that Label from view (invalidated labels are still tracked in the database, however), and if it was the only non-invalidated Label suggestion associated with that Object, the whole Object will be removed from the Image. However, if there are more non-invalidated Label suggestions associated with the Object, the next non-invalidated Label will be displayed for the user to review.
Adjusting bounding boxes
While an Object is "unlocked", you can also move its bounding box by clicking and dragging its center or adjust its dimensions by clicking and dragging its corners.
There are a few ways to edit Labels that have been predicted by ML models:
Right-clicking a bounding box will open a menu of Label-editing actions:
Hovering over a bounding box will reveal Label Validation / Invalidation buttons:
Clicking on the Label will open the Label editing and selection menu and allow you to manually enter new Labels. You can exit the Label editing mode by clicking anywhere on the screen.
Adding new Objects
To add new objects, either select the Add Object button located below the image, or right-click the image and select the Add Object item from the menu. Then simply click, hold, and drag the cursor down and to the right to draw a bounding box around the animal or object you'd like to annotate. When you let up on the click, the edit Label input automatically pops up and users can enter a new Label.
If you plan on using your labeled data to train a machine learning model, it's worth reviewing these labeling best practices.
Draw tight bounding boxes
If you plan on using your annotated images for machine learning training data in the future, it's best to keep the bounding box as tight around the as possible. When it comes time to train a new model, this will allow you to crop out as much of the background as possible, which can confuse machine learning classifiers.
Deleting Objects
Technically, Objects never get deleted from the database, but if you need to remove an object from view, unlock it and invalidate all of the suggested Labels associated with it. When all Labels have been invalidated, the Object's bounding box will be removed from the Image.
Deleting Images
To permanently delete a camera trap image from the Animl platform from the image review panel, open the menu in the upper right hand corner of the panel (the icon looks like three "dots"), then select the "Delete" option.
You can also delete more than one image at a time by bulk-selecting images in the images table (to bulk-select, click the image at the start of the range of images you'd like to select, hold shift, and then select the image at the end of the range). Once your desired range of images is selected, right-click any of the selected images, and then click "Delete Images" from the menu that appears.
NOTE: This action can not be reversed at this time.
Marking an Image as "empty"
Full disclosure: this is not intuitive and we're thinking about (and open to suggestions!) ways for improving this. But currently, there are a couple things to know about how to handle images without any animals or objects of interest in them:
marking an Image as "empty" - both when a user does it manually or a machine learning model does it automatically - is essentially just adding an Object to the image with an "empty" Label that has a bounding-box that encompasses the entire image
Images do not get automatically labeled as "empty" if you remove all Objects from them. You still need to manually mark the Image as empty after invalidating all other Label suggestions on the Image.
Similarly, if an ML model has incorrectly predicted that an image is empty and you add new Objects to it, the original "empty" object will not automatically be removed. You still need to manually invalidate the "empty" object.
To mark an image as "empty", either click the "Mark Empty" button at the bottom of the Image, or right-click anywhere on the image and select "Mark Empty" from the menu.
Adding Tags
Tags, are a little different that Objects/Labels in that they are intended to describe the entire image, rather than some object or animal within an image. Some examples of Tags might include "seen", "favorite", or "predation event". To apply them to an image, click the "+ Tag" button below the image, and select the tag you'd like to apply.
Last updated
