This repository provides the code to reproduce the main parts of the paper. In addition, it provides the code to learn interpretable representational embeddings from behavioral responses to natural images using a triplet odd-one-out task.
This repository is split into two parts:
- Code and experiments to reproduce the main results of the paper Dimensions underlying the representational alignment of deep neural networks with humans. See Main Experiments of the Paper for more details.
- Code to learn interpretable representational embeddings from behavioral responses to natural images using a triplet odd-one-out task. See Learning Representational Embeddings for more details.
This project uses Python 3.9.12 and Poetry for dependency management. Most experiments can be run using a normal desktop computer in a reasonable amount of time. However, most experiments require PyTorch and an NVIDIA GPU.
First, install Poetry:
curl -sSL https://install.python-poetry.org | python3 -
export PATH="$HOME/.local/bin:$PATH"
Clone the repository and navigate to the project root:
git clone [email protected]:florianmahner/object-dimensions.git
cd object-dimensions
Install the project dependencies using Poetry:
poetry install
poetry shell
Prior to running the experiments, download the required data from osf by executing the following command, which will download the data and extract it to the data
folder:
make data
Additionally, we use the THINGS dataset, which consists of 1,854 images of everyday objects. We provide a script to download the THINGS and THINGS+ data. To download the data, run:
make images
We have provided a separate README in the experiments folder that explains how to use the data and reproduce the experiments. We provide config files for each experiment in the configs folder.
We learn interpretable representational embeddings from behavioral responses to natural images using a triplet odd-one-out task. The choices can be obtained by:
- Running behavioral experiments with humans.
- Simulating triplet choices from any representation space (e.g., DNN activations, neural recordings).
The repository supports both the simulation of behavioral choices and the use of actual behavioral data to train an embedding model. A small example demo can be found in scripts/demo.ipynb.
Triplets can be simulated from any representation space or collected from actual behavioral responses. If you have actual behavioral responses, make sure that the data is of shape [n_samples, 3], where each row contains the indices of the triplets and the last column by definition denotes the odd one out.
Use the run_tripletization.py
script to simulate triplets. Example configurations are provided in the configs folder.
To extract triplets from DNN representations, run:
python run_tripletization.py --config "./configs/tripletize.toml"
Train the model with different hyperparameters and optimization methods. For a list of available arguments, run:
python run_optimization.py --help
You can train the model deterministically using MLE (as in SPoSE) or variationally (as in VICE) by specifying the --method
flag.
Example command:
python run_optimization.py --config "./configs/train_behavior.toml" --method "deterministic"
The model run paths are organized based on the modality:
- Behavior:
./log_path/identifier/behavior/n_samples/prior/init_dim/batch_size/beta/seed
- DNN:
./log_path/identifier/deep/model_name/module_name/n_samples/prior/init_dim/batch_size/beta/seed
Each run path includes the following files:
path
├── config.toml
├── training.log
├── params
│ └── parameters.npz
├── tboard
├── checkpoints
│ └── checkpoint_epoch_*.tar
The final model parameters are stored in parameters.npz
, along with all training configurations. For SPoSE, only the mean of the embedding is modeled, while for VICE, both the mean and variance of the Gaussian variational distribution are modeled.
For any questions or issues, please open an issue on GitHub or contact the repository maintainers.