Image review
Humans aren't obsolete yet! Many use-cases require reviewers to be kept "in the loop" to correct any mistakes made by the ML models and fill gaps in their classification repertoire.
Understanding the Object & Label data model
Before moving on, it's important to first understand the Object and Label data model, which is described in the Structure, concepts, and terminology page.
Reviewed vs. Not-reviewed Images
An Image is considered "reviewed" once all Objects are locked (either a user has validated an ML-predicted Label, manually added their own Label, or invalidated all of the proposed Labels thus removing the Object).
Validating / invalidating / editing Labels
If a user "validates" a suggested Label, the Object will become "locked", and the most recently validated Label will become the "winning" Label. "Invalidating" a Label will remove that Label from view (invalidated labels are still tracked in the database, however), and if it was the only non-invalidated Label suggestion associated with that Object, the whole Object will be removed from the Image. However, if there are more non-invalidated Label suggestions associated with the Object, the next non-invalidated Label will be displayed for the user to review.
Adjusting bounding boxes
While an Object is "unlocked", you can also move its bounding box by clicking and dragging its center or adjust its dimensions by clicking and dragging its corners.
There are a few ways to edit Labels that have been predicted by ML models:
Right-clicking a bounding box will open a menu of Label-editing actions:
Hovering over a bounding box will reveal Label Validation / Invalidation buttons:
Clicking on the Label will open the Label editing and selection menu and allow you to manually enter new Labels. You can exit the Label editing mode by clicking anywhere on the screen.
Adding new Objects
To add new objects, either select the Add Object button located below the image, or right-click the image and select the Add Object item from the menu. Then simply click, hold, and drag the cursor down and to the right to draw a bounding box around the animal or object you'd like to annotate. When you let up on the click, the edit Label input automatically pops up and users can enter a new Label.
If you plan on using your labeled data to train a machine learning model, it's worth reviewing these labeling best practices.
Draw tight bounding boxes
If you plan on using your annotated images for machine learning training data in the future, it's best to keep the bounding box as tight around the as possible. When it comes time to train a new model, this will allow you to crop out as much of the background as possible, which can confuse machine learning classifiers.
Deleting Objects
Technically, Objects never get deleted from the database, but if you need to remove an object from view, unlock it and invalidate all of the suggested Labels associated with it. When all Labels have been invalidated, the Object's bounding box will be removed from the Image.
Deleting Images
To permanently delete a camera trap image from the Animl platform from the image review panel, open the menu in the upper right hand corner of the panel (the icon looks like three "dots"), then select the "Delete" option.
You can also delete more than one image at a time by bulk-selecting images in the images table (to bulk-select, click the image at the start of the range of images you'd like to select, hold shift, and then select the image at the end of the range). Once your desired range of images is selected, right-click any of the selected images, and then click "Delete Images" from the menu that appears.
NOTE: This action can not be reversed at this time.
Marking an Image as "empty"
Full disclosure: this is not intuitive and we're thinking about (and open to suggestions!) ways for improving this. But currently, there are a couple things to know about how to handle images without any animals or objects of interest in them:
marking an Image as "empty" - both when a user does it manually or a machine learning model does it automatically - is essentially just adding an Object to the image with an "empty" Label that has a bounding-box that encompasses the entire image
Images do not get automatically labeled as "empty" if you remove all Objects from them. You still need to manually mark the Image as empty after invalidating all other Label suggestions on the Image.
Similarly, if an ML model has incorrectly predicted that an image is empty and you add new Objects to it, the original "empty" object will not automatically be removed. You still need to manually invalidate the "empty" object.
To mark an image as "empty", either click the "Mark Empty" button at the bottom of the Image, or right-click anywhere on the image and select "Mark Empty" from the menu.
Keyboard shortcuts
There are a number of keyboard shortcuts and controls that are designed to speed up the image review workflow:
Navigate forward and backwards through images with the WASD or arrow keys
Undo Label editing actions with ctrl-z, and redo them with shift-ctrl-z
Additional tips and tricks
Removing “accidental labels” - sometimes users will accidentally add a label that is misspelled or incorrect. If strange labels start to show up in your Labels Filter panel, that is likely what happened. The best way to remove them is:
un-select all label filters, then select the mis-typed label to retrieve the image(s) that contain it
Next, simply unlock the object and invalidate all labels on that object until the bounding box goes away. After that just create a new object and label it correctly. After refreshing the entire page, the offending label should be removed from the label filters list.
If you are still seeing misspelled labels in the Label Filter panel, but when you select only that label filter no images are found, try using a Custom Filter instead with the following query to find and correct all images with misspelled labels:
Last updated
