A multimodal explainability framework that combines SHAP, SAM, and Vision Language Models to provide both visual and textual explanations for image classifiers.
Powerful, flexible, and easy-to-use explainability for any image classification model
Works directly with pre-trained models without any additional training. Just load and explain.
Highlight important image regions using SHAP values combined with SAM segmentation masks.
Generate human-readable natural language descriptions of model predictions using VLMs.
Supports both custom-trained and pre-trained models. Compatible with multiple architectures.
Includes optimizations for deployment on edge devices with FP16 support and quantization.
Comprehensive docstrings, examples, and API documentation for easy integration.
How LAVE combines multiple AI techniques to explain model predictions
Computes feature importance using game theory.
Meta's state-of-the-art segmentation model.
Efficient VLM for text generation.
Everything you need to get started with LAVE
Set up your development environment.
Train custom models with transfer learning.
Run predictions and explanations.
Multiple backbone architectures.
Repository organization.
Help improve LAVE.
Get started with LAVE in minutes
# Clone the repository
git clone https://github.com/Purushothaman-natarajan/VALE-Explainer.git
cd LAVE-Explainer
# Create and activate environment
conda env create -f environment.yaml
conda activate LAVE-Explainer
# Download model checkpoints
# SAM: https://github.com/facebookresearch/segment-anything# For pre-trained models
python pre-trained_model_explainer.py \
--model_name densenet121 \
--img_path path/to/image.jpg \
--sam_checkpoint path/to/sam_vit_h.pth
# For custom-trained models
python custom_model_explainer.py \
--model_path model.pth \
--img_path path/to/image.jpg \
--num_classes 10from pre_trained_model_explainer import PyTorchExplainableWrapper
explainer = PyTorchExplainableWrapper(
model_name_or_path="densenet121",
sam_checkpoint="./sam_vit_h.pth",
tiny_llava_model_path="bczhou/TinyLLaVA-3.1B"
)
explainer.run_pipeline("path/to/image.jpg")If you use LAVE in your research, please cite our paper
@article{natarajan2024vale,
title = {VALE: A Multimodal Visual and Language Explanation Framework for Image Classifiers},
author = {Natarajan, Purushothaman and Nambiar, Athira},
journal = {arXiv preprint arXiv:2408.12808},
year = {2024}
}