Uploading images
How to upload images to Animl in bulk from your desktop
Last updated
How to upload images to Animl in bulk from your desktop
Last updated
In addition to linking and ingesting images in real-time from wireless camera traps, Animl supports directly uploading camera trap images in bulk from your computer. The process is simple: you package your images in a Zip file, upload them to your Project via the bulk upload interface, and then Animl takes over and does the rest. Behind the scenes, Animl is marshaling hundreds of parallelized computers to make your machine learning processing jobs as fast and reliable as possible.
Animl is highly flexible and can consume images from most camera trap types, but there are a few requirements to be mindful of:
The maximum file-size of the Zip file can not exceed 50 GB. If your dataset is larger than 50 GB, you'll have to break it in smaller batches and zip and upload them individually.
Animl only processes .jpg and .png files it finds within the Zip file. It doesn't support video or any other file type.
The maximum file size for each individual image file is 8 MB.
Each image must have two fields present it its Exif metadata: a DateTimeOriginal field indicating when the image was created, and a SerialNumber field indicating what camera took the image. Nearly all images should have DateTimeOriginal fields, but some common camera trap manufactures and models do not provide SerialNumbers (see table below for list of known camera traps that do not support SerialNumbers).
Reconyx - Hyperfire
Reconyx - MR5 Microfire
Buckeye X80 / UWS
Cuddelink / Cuddeback
Browning / Prometheus
Bushnell
Exodus
Little Acorn
RidgeTec
SpyPoint
If you're trying to upload images from a camera make that is not listed below and you're not sure whether or not your images' metadata contain SerialNumbers, the best way to check is with Phil Harvey's Exiftool (https://exiftool.org/).
If you are uploading images from camera(s) without Serial Numbers, be sure to follow the instructions below on using the Serial Number Override feature.
Before uploading a batch, it's worth double checking that you Project's Automation Rules are configured appropriately for your data processing needs. All images you upload will be processed according to the Automation Rules you've configured, and if you don't have any Automation Rules configured, Animl will save the images to your Project but it will not make any machine learning predictions.
The upload and batch processing workflow involves the following steps:
Step 1: uploading file - the Zip file is uploaded from your computer to Animl's backend.
Step 2: preprocessing - the file is unzipped, the files within it are validated, the parallelized computers that will support the machine learning processing are deployed, and the images are saved to the Animl database.
Step 3: processing - the images are processed by the pipeline of machine learning models you've defined in your Automation Rules.
Step 4: cleaning up - the processing computers are shut down. Once you reach this step, your images and their predicted label are ready for your review; you don't have to wait until it's finished cleaning up!
The amount of time it takes to upload your file will depend on the speed of your internet connection. For reference, with an upload speed of 360 Mbps, a 10 GB zip file would take under 4 minutes to upload. If you're curious, you can estimate the upload time for your files with a file upload time calculator (https://www.uploadtimecalculator.com/).
Once the file has been uploaded, the remaining processing time depends on (a) how many images are in your batch, (b) how many ML models you are invoking in your processing pipeline, and (c) how many other users are also processing batches of images at the moment.
If no one else is using the bulk upload functionality and you're only requesting predictions from one model (Megadetector, for example), Animl can process ~11-12 images per second, meaning processing a batch of 100k images should take about 2.5 hours to complete.
When it's safe to close your browser window
Large bulk uploads can take many hours or even days to fully process, but you don't have to keep your browser window open and logged into Animl that whole time. Once the Zip file is fully uploaded to the backend (i.e., after Step 1 is complete), you can close the window and check back in later.
After you've Zipped up your batch of images, click on the "Bulk Upload Images" icon in the left-hand sidebar, and click "Choose File" and select your Zip file (you can also drag & drop the Zip into that input field).
If you're sure you're uploading images that contain Serial Numbers in their Exif data, initiate the upload by clicking "Upload". A green progress bar will appear to indicate the progress towards completing the file upload (Step 1, described above). If your images do not contain Serial Numbers, you'll have to follow the additional steps outlined below before uploading.
If your images do not include a camera Serial Number field in their Exif metadata, Animl has no way of knowing which Camera the images belong to, so you will have to manually provide this information by adding your own "serial number" to the the Serial Number Override input field.
"Serial number" is in quotes because it can be any string you'd like to use to uniquely identify that particular Camera. The only requirements are that the value can't contain spaces or special characters, and it must be unique to your Project. If a Camera already exists in your Project with the same "Serial Number", the images will be grouped together and associated with that existing Camera. There's nothing wrong with that per se, and in fact it may be desirable (for example, if you're uploading a fresh batch of images from a camera you've already uploaded some images from).
Additionally, it's important to note that the value you provide in the Serial Number override field will be applied to ALL images in the Zip file. So if you have images from multiple cameras, some or all of which do not have Serial Numbers, you'll have to zip up and upload images from each camera separately, each with separate & unique Serial Number Override values. This is not a requirement if all of your images are from cameras that include Serial Numbers (if that's the case, a single Zip file can contain images from multiple cameras and Animl will sort them and associate them with the correct cameras automatically).
While the Zip file is being uploaded to Animl (i.e., Step 1, as described above), a green progress bar will appear directly below the upload form, and once the database has a record of the batch upload, an entry will appear in the table below with a status of "Uploading file...":
Once Step 1 is complete, the green progress bar will disappear and the remaining processing can be monitored with the bulk upload table entry. At this stage, you also no longer have to keep the bulk upload popup or the browser window open; feel free to sign out entirely and return back any time to check in on progress.
Note: if you click off of the upload dialogue box, simply click back on the upload icon again.
Another thing to note is that during the machine learning processing stage (Step 3), Animl provides an estimate of the number of images it's processed, but this number is not always accurate (especially on batches of relatively few images), and at times it may even appear to reverse progress briefly before continuing to grow again. Despite not being perfectly accurate all of the time, it's still a very useful indicator of progress.
No need to wait around for clean up Once the "Status" reads "Finished processing images. Cleaning up...", your upload has finished processing and your images and predictions should be ready for review. Feel free to close out of the bulk upload popup and check them out!
Once your upload has finished processing, you can close out of the bulk upload model and view the images and predicted labels. If the images are not immediately present in the Animl user interface, you may have to click the "Refresh data" button at the bottom of the Filters Panel.
If the warning icon to the right of the bulk upload entry turns red, it means that one or more errors occurred while processing your images. To download a complete CSV of all of the errors that occurred, click the warning icon and open the CSV in a spreadsheet editor to view each image attempt and the reason it failed.
Errors can occur for a number of reasons, and some can be ignored (for example, Duplicate Image errors). Certain errors, however, may be remediable. For example, if the errors indicate that certain images failed to be saved because they had Unknown serial number errors, you can attempt to re-upload them using the Serial number override feature.
