Media overlays with VideoSync

Firetail allows you to import video and media data for replay in the context of tag sensor and GPS data using the VideoSync module.

Start the module via Data > Sync Video Data.

Video footage (and images) from camera traps can be synchronized with the main timeline to replay and understand acceleration data using visual evidence. This can be a powerful way to annotate sections in the data or to evaluate predicted behavioral annotations.

The module will refer to resources as videos reflecting its main use case, but images will typically work fine to overlay visual evidence at specific points in time.

Key Features

With the VideoSync module you can

• add video resources
• apply systematic offsets to UTC to video resources
• quickly fine-adjust UTC start timestamps
• save video mappings with resource notes
• load video mapping
• query timestamps from the main timeline

Prerequisites

Although not a strict prerequisite, Firetail will assume that video snippets do not overlap. Overlapping resources may not be found using “Search for overlapping media”, but should work fine otherwise.

Resources with timestamps equal up to the millisecond will be shifted by 1ms automatically, which is uncritical for most use cases. This is likely to happen, when multiple timestamps could not be auto-detected.

Linking mpv player

For video replay and to support a wide range of video codecs, Firetail relies on the freely available MPV player (https://www.mpv.io). Installation and linking the mpv player binary (exe) to the VideoSync module requires a few easy steps (no admin rights required):

• Open “Data > Sync Video Data” from the main menu
• follow the link to the MPV Player (https://www.mpv.io)
• download one of the official builds (e.g. zip files), Nightly builds will work well for Win/Mac (Linux: third-party builds available)
• extract it to any location of your choice
• in the VideoSync module, select the path to the mpv binary

Typical Workflow

1. Apply a UTC Offset: adjust the offset to UTC for captured resources

   • choose if the offset is UTC plus X or UTC minus X
   • Pre-select an offset to UTC for the video timestamps (hour/min/second)
   • these can be adjusted to the second-level to correct for systematic errors
   • 0h 0m 0s indicates your footage is UTC timestamped

2. Load Video/Audio resources

   • Add new media files. The selected UTC offset will be applied when loading.
   • Note: Firetail attempts to extract recording timestamps and durations from the files. You can always adjust timestamps/durations manually and save your work.

3. Double-clicking the “Resource Path” will start the video and jump to the corresponding position in the main timeline.

   • The video initially pauses
   • the video will play if the main timeline ‘Play’ is active
   • Note that the video will only be in sync for ‘1 second equals 1 second’

4. Adjust Offsets:

• Fine adjust timestamps by hovering it in the ‘UTC video start’ timestamp, then:

• Quick Save or Save your adjustments (quick save/export/import)

Manual timestamp edit

Double-click on a timestamp to manually set the UTC start. If the timestamp cannot be validated (invalid format or not a valid date), the cell will flash red and reset its value to the previous setting.

Adding notes to a resource

Double-click items in the ‘notes’ column to add a note for a resource. Notes will be saved along with the video mapping. A mapping is automatically saved when the VideoSync window is closed.

Clear list

Remove all resources.

Save Video Mapping

Save the current list (JSON format).

Load Video Mapping

Load one or more mappings (JSON format). If multiple video mappings are selected they are merged in the VideoSync list. This allows for a separation of resources by animal or type in individual files.

Search for Overlapping Media

The “Timestamp” field is always updated from the main timeline. Using “Search for overlapping media” will search for a video overlapping the timestamp and replay it with the appropriate offset (paused depending on main time control ‘Play’).

Forward Synchronization

The workflow is centered around the video resources in the sense that the main timeline will update when an event (seek, start, pause) is triggered from the VideoSync module. Vice versa, seek will not trigger an update in the Video viewport. The reason is simply that the main timeline control is typically too coarse to find specific sequences within a video snippet.

Note that changing the speed of either data or video replay will lead to an offset.

Keyboard shortcuts in mpv viewport

Another advantage of mpv player is its great set of keyboard shortcuts that can pause, seek or even zoom within the current viewport. Below, we list a subset that may come in handy for the work with Firetail.