WISE is a search engine for images, videos, and audio powered by multimodal AI, allowing you to quickly and easily search through large collections of audiovisual media. You can search using natural language, an uploaded image/audio file, or a combination of these modalities.
Natural language search
Use natural language to describe what you want to search for.
WISE uses a language model to understand the meaning behind your query, allowing you to flexibly describe what you are looking for. Moreover, WISE uses a vision model to understand what's being depicted in an image (i.e. it searches by image content rather than metadata such as keywords, tags, or descriptions), so the images do not need to be manually tagged or labelled with text captions.
Visual similarity search
Upload an image or paste an image link to find similar images:
Multi-modal search
Combine images and text in your query. For example, if you upload a picture of a golden retriever and enter the text "in snow", WISE will find images of golden retrievers in snow.
Various multimodal / vision-language models supported
Various models are supported including vision-language models from OpenCLIP (including OpenAI CLIP) and the Microsoft CLAP audio-language model.
Different ways to perform searches
Searches can be performed via:
- CLI
- REST API
- Web frontend
(Note: currently the search functionality in the CLI may be missing some features.)
Safety features
- Specify a list of search terms that users should be blocked from searching
- 'Report image' button allows users to report inappropriate/offensive/etc images (temporarily removed; will be added back soon)
We are planning on implementing the following features soon. Stay tuned!
-
Searching on image and audio files
Currently, WISE 2 only supports searching on video files (on both the audio and visual stream of video files). Searching on images and pure audio files is not supported yet. Please use WISE 1.x.x for now if you need to search on images. -
Cross platform easy installation
We are working on creating an easy-to-use installer which allows users to install WISE on Mac, Windows, and Linux without needing to use the command line.
The WISE open source software is developed and maintained by the Visual Geometry Group (VGG) at the University of Oxford.
Here are some documents for users and developers of WISE.
- Install : describes the process for installing WISE
- User Guide : demonstrates the usage of WISE using a sample video dataset
- Metadata : describes support for text metadata search in WISE
- Evaluation
- Developer Resources
- Data Loading: describes interface for loading media files
- Feature Extractor : guide for creating new feature extractors in WISE
- FeatureStore : describes the data structure containing the extracted features
- Frontend : describes the frontend web-based interface
- Database : describes the structure of the internal metadata database, which stores information about the source collections (i.e. input folders), media files (e.g. images, videos, or audio files), vectors, and extra metadata
- Tests : describes the software testing process for WISE
Please submit any bug reports and feature requests on the Issues page.
For any queries or feedback related to the WISE software, contact Prasanna Sridhar, Horace Lee or Abhishek Dutta.
Development and maintenance of WISE software has been supported by the following grant: Visual AI: An Open World Interpretable Visual Transformer (UKRI Grant EP/T028572/1)