Reviewing Predicted Labels
Humans aren't obsolete yet! Many use-cases require reviewers to be kept "in the loop" to correct any mistakes made by the ML models and fill gaps in their classification repertoire.
Understanding the Object/Label and Tag data models
Before moving on, it's important to first understand the Object, Label, and Tags data models, which are described in the Structure, concepts, and terminology page. In short, a machine learning model will predict if there is an Object in an Image, and will draw a bounding box around it. The model will then attempt to classify what the Object in the bounding box is, and predict a Label for it.
Validating, Invalidating, or Editing Labels
Users can validate, invalidate, or edit predicted labels.
When you validate a predicted label, the object becomes "locked" and is considered "reviewed." The validated label becomes the assigned Label, although you can always unlock the object if you need to edit the label.
Invalidating the label simply removes the predicted label from the object. If there are multiple predicted labels, the next one will appear for review on the object. A good example of when a user invalidates a label is when something is incorrectly predicted as an animal (such as a rock or a boulder).
Editing a label is just correcting the predicted label. When you edit a label, the object becomes locked and the new label is considered the "assigned" Label.
There are multiple ways to do these actions, in various efficiencies. We will go through each of them, and at the end of the article highlight which workflows are best for speedy image review.
Reviewed vs. Not-reviewed Images
An Image is considered "reviewed" once all Objects are locked (either a user has validated an ML-predicted Label, manually added their own Label, or invalidated all of the proposed Labels thus removing the Object).
From a visual standpoint, you will know if an image is reviewed or not-reviewed by looking at the border surrounding the label or bounding box.
If an image is not-reviewed, the lines will appear dotted
If an image is reviewed, the lines will appear solid
Navigating your images
The first thing to know about easily navigating your images to review their predicted labels is that all the checkboxes in the left sidebar are filters. This makes it very easy to filter your images down to a specific predicted species, a specific camera, or filter out certain labels like "empty".
Important Keyboard Shortcuts
Click on any image to see it in an expanded view.
We highly recommend using the WASD or arrow keys to navigate forward and backwards through images while you're reviewing them. This allows you to use the keyboard to iterate through the images with one hand, while your other hand is free to use the mouse/track-pad to validate or edit Labels, apply Tags, etc.
Undo Label editing actions with ctrl-z, and redo them with shift-ctrl-z
How To Validate, Invalidate, or Edit Labels
There are three "tiers" that you can validate, invalidate, or edit labels. You can take these actions 1) directly on the bounding box of an object, 2) on the image (so it affects all objects in the image), or 3) on multiple images at once (so it affects all objects in all images selected. We'll walk through each below.
Option 1: Object level review
The first option is to take action directly on the Object.
Hover over the predicted label and two buttons will appear.
The green checkmark validates the label, while the red X invalidates the label. Clicking on the label allows you to edit it.
Note: the only Labels available to assign are those previously predicted by the machine learning models, or Labels you manually added to your project. Adding and managing your list of available Labels is covered in the Managing Labels section.
Similarly, if your right click on an Object in a bounding box, it will bring up a menu that allows you to do the same three things.
Option 2: Image level review
You can also validate, invalidate, or edit all of the predicted labels in an image at once. This means that the action will be taken on all objects in an image (for example, if there are multiple animals in one image).
The menu under an image is where you can validate labels by clicking on the checkmark, invalidate labels by clicking on the X, or edit labels by clicking on the pencil icon.
It is in this toolbar that you can also mark an image as "empty" if you care about having "empty" as an assigned and human-validated label.
Option 3: Bulk Select Image review
You can also take all these actions on multiple images at the same time. This is what allows for rapid, bulk image review.
The first thing to know is that right-clicking on an image in the thumbnail view will bring up a menu to take these actions (and more).
You can select multiple images by 1) clicking any image, 2) holding the Shift key, then 3) clicking another image. Doing this will automatically select all images between the two you clicked on.
For example in the video below, the user is clicking on the 2nd image, pressing and holding Shift, then clicking on the 8th image. You’ll see that the selection then covers all images 2 - 8.
You can use this method to select multiple images at once, even if you scrolled down the page. Once you select multiple images, right clicking will bring up the menu of actions. Any action you take will be applied to all images selected. Keep reading until the end of this article to see how you can use this feature to speed up your review process!
Tools to Aid Image Review
There are multiple tools to help you understand what is in your image.
Temporarily Hide Bounding Boxes
This tool lets you temporarily hide the bounding boxes, because they can sometimes obscure what is in the image. To do this, click on the eye icon below the image.
Click it again to bring back the bounding box
Image zoom in / zoom out
You can also zoom in on an image to help you see what's in it.
To zoom in, click the magnifying glass with the plus sign, and drag the image to your point of interest.
The other magnifying glass (with the minus sign) zooms out. To quickly revert back to the original image view, just click the "Reset Zoom" icon.
Rapid Fire Review
It’s completely up to you how you approach review and you will need to find what works best for you! Below is an example of a workflow we’ve seen speed up the process, but please continue iterating to find your ideal flow.
Step 1: Filter to your needs.
Filter out the empties, or even down to just one species that seems consistently predicted correctly.
Step 2: Review images
Start image review by advancing through your photos, either by using the arrow keys or the WASD keys.
Step 3: Correct along the way
Correct anything along the way that seems incorrect, directly on the bounding box. In the example below the user is correcting mislabeled coyotes. Note: we are not validating things that are correct, we will do this later in bulk.
Step 4: Bulk Select and Validate
When you feel you’ve gone through your desired amount of photos, or come to a photo that you’re unsure about, bulk select all the images you’ve gone through. Select those images by:
Scrolling to where you started
Pressing Shift and clicking the first photo
Once all the images are selected, just right click and validate the labels.
All of those images were just reviewed and validated! You can then continue this process how you see fit (with the same species, a different species, etc). Remember that all the filters are adjustable, and a particularly helpful one is to filter out the previously reviewed images.
Advanced:
Managing Labels
Project Managers have the ability to edit the list of allowable Labels that can be applied to their images. For instructions, see Managing Labels.
Adding new Objects
To add new objects, either select the Add Object button located below the image, or right-click the image and select the Add Object item from the menu. Then simply click, hold, and drag the cursor down and to the right to draw a bounding box around the animal or object you'd like to annotate. When you let up on the click, the edit Label input automatically pops up and users can enter a new Label.
If you plan on using your labeled data to train a machine learning model, it's worth reviewing these labeling best practices.
Draw tight bounding boxes
If you plan on using your annotated images for machine learning training data in the future, it's best to keep the bounding box as tight around the animal as possible. When it comes time to train a new model, this will allow you to crop out as much of the background as possible, which can confuse machine learning classifiers.
Deleting Objects
Technically, Objects never get deleted from the database, but if you need to remove an object from view, unlock it and invalidate all of the suggested Labels associated with it. When all Labels have been invalidated, the Object's bounding box will be removed from the Image.
Last updated