Inference

Once you have trained a model you’ll likely want to run inference on new videos.

Similar to training, there are several tools for running inference:

  1. A set of high-level functions used for processing videos and creating labeled clips. You can combine these to create your own custom inference script. This is required if you used the pip package installation method.

  2. An example inference script provided in the conda from source installation method. This demonstrates how to combine the high-level functions.

Note

The steps below assume the conda from source installation method. If you did not use this installation method, see the example inference script. You can also see how video inference is handled in the example train script.

Inference with example data

To run inference with a model trained on the example dataset, run the following command from inside the lightning-pose directory (make sure you have activated your conda environment):

python scripts/predict_new_vids.py eval.hydra_paths=["YYYY-MM-DD/HH-MM-SS/"]

This overwrites the config field eval.hydra_paths, which is a list that contains the relative paths of the model folders you want to run inference with (you will need to replace “YYYY-MM-DD/HH-MM-SS/” with the timestamp of your own model).

Inference with your data

In order to use this script more generally, you need to update several config fields:

  1. eval.hydra_paths: path to models to use for prediction

  2. eval.test_videos_directory: path to a directory containing videos to run inference on

  3. eval.saved_vid_preds_dir: optional path specifying where to save prediction csv files. If null, the predictions will be saved in eval.test_videos_directory.

  4. eval.save_vids_after_training: if true, the script will also save a copy of the full video with model predictions overlaid. The labeled video is saved in eval.saved_vid_preds_dir.

As with training, you either directly edit your config file and run:

python scripts/predict_new_vids.py --config-path=<PATH/TO/YOUR/CONFIGS/DIR> --config-name=<CONFIG_NAME.yaml>

or override these arguments in the command line:

python scripts/predict_new_vids.py --config-path=<PATH/TO/YOUR/CONFIGS/DIR> --config-name=<CONFIG_NAME.yaml> eval.hydra_paths=["YYYY-MM-DD/HH-MM-SS/"] eval.test_videos_directory=/absolute/path/to/videos eval.saved_vid_preds_dir=/absolute/path/to/dir

Note

Videos must be mp4 files that use the h.264 codec; see more information in the FAQs.