update
This commit is contained in:
HamsterMimi
parent
2410fe9f5e
commit
8355e4c741
173
README.md
173
README.md
@ -1,143 +1,68 @@
|
||||
# Zero-Cost Operation Scoring in Differentiable Architecture Search (Zero-Cost-PT)
|
||||
Official impementation for AAAI 2023 submission:
|
||||
"**Zero-Cost Operation Scoring in Differentiable Architecture Search**".
|
||||
# MeCo: Zero-Cost Proxy for NAS Via Minimum Eigenvalue of Correlation on Feature Maps
|
||||
|
||||
## Installation
|
||||
|
||||
## Installation
|
||||
```
|
||||
Python >= 3.6
|
||||
PyTorch >= 1.7.1
|
||||
torchvision == 0.8.2
|
||||
tensorboard == 2.4.1
|
||||
scipy == 1.5.2
|
||||
gpustat
|
||||
PyTorch >= 2.0.0
|
||||
nas-bench-201
|
||||
```
|
||||
|
||||
|
||||
## Preparation
|
||||
|
||||
1. Download three datasets (CIFAR-10, CIFAR-100, ImageNet16-120) from [Google Drive](https://drive.google.com/drive/folders/1T3UIyZXUhMmIuJLOBMIYKAsJknAtrrO4), place them into the directory `./data`
|
||||
2. Download the [`data` directory](https://drive.google.com/drive/folders/18Eia6YuTE5tn5Lis_43h30HYpnF9Ynqf?usp=sharing) and save it to the root folder of this repo.
|
||||
3. Download the benchmark files of NAS-Bench-201 from [Google Drive](https://drive.google.com/file/d/1SKW0Cu0u8-gb18zDpaAGi0f74UdXeGKs/view) , put them into the directory `./data`
|
||||
4. Download the [NAS-Bench-101 dataset](https://storage.googleapis.com/nasbench/nasbench_only108.tfrecord), put it into the directory `./data`
|
||||
5. Install `zero-cost-nas`
|
||||
```bash
|
||||
cd zero-cost-nas
|
||||
pip install .
|
||||
cd ..
|
||||
```
|
||||
|
||||
## Usage/Examples
|
||||
|
||||
### Correlation Experiment
|
||||
|
||||
```bash
|
||||
cd correlation
|
||||
python NAS_Bench_101.py
|
||||
python NAS_Bench_201.py
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
### Experiments on NAS-Bench-201
|
||||
Scripts for reproducing our experiments can be found under the ```exp_scripts/``` folder.
|
||||
|
||||
#### 1. Prepare NAS-Bench-201 Data
|
||||
1. Download NAS-Bench-201 checkpoint from [NAS-Bench-201-v1_0-e61699.pth](https://drive.google.com/file/d/1SKW0Cu0u8-gb18zDpaAGi0f74UdXeGKs/view), and place it under ```./data``` folder.
|
||||
1. Run Zero-Cost-PT with appointed zero-cost proxy:
|
||||
```bash
|
||||
cd exp_scripts
|
||||
bash zerocostpt_nb201_pipline.sh --metric [metric] --batch_size [batch_size] --seed [seed]
|
||||
```
|
||||
You can choice metric from `['snip', 'fisher', 'synflow', 'grad_norm', 'grasp', 'jacob_cov','tenas', 'zico', 'meco'] `
|
||||
|
||||
#### 2. Prepare NAS-Bench-201 and Zero-Cost-NAS API
|
||||
i. Install NAS-Bench-201 api via `pip`
|
||||
```
|
||||
pip install nas-bench-201
|
||||
```
|
||||
ii. Install Zero-Cost-NAS API
|
||||
### Experiments on DARTS-CNN Space
|
||||
|
||||
Clone the code repository from [Zero-Cost-NAS](https://github.com/SamsungLabs/zero-cost-nas). Go to the root directory of the cloned repo and run:
|
||||
```
|
||||
pip install .
|
||||
#### 1. DARTS CNN Space
|
||||
|
||||
```bash
|
||||
cd exp_scripts
|
||||
bash zerocostpt_darts_pipline.sh --metric [metric] --batch_size [batch_size] --seed [seed]
|
||||
```
|
||||
|
||||
#### 3. Run Zero-Cost-PT on NAS-Bench-201
|
||||
#### 2. DARTS Subspaces S1-S4
|
||||
|
||||
````bash
|
||||
cd exp_scripts
|
||||
bash zerocostpt_darts_pipline.sh --metric [metric] --batch_size [batch_size] --seed [seed] --space [s1-s4]
|
||||
````
|
||||
|
||||
You can run our Zero-Cost-PT with the following script:
|
||||
```
|
||||
bash zerocostpt_nb201_pipeline.sh --seed [SEED]
|
||||
```
|
||||
You can specify random seeds with ``` --seed ``` for reproducibility. In our experiments we use random seeds 0, 1, 2, 3.
|
||||
|
||||
You could also run with different zero-cost proxies by specifying ```--metrics```, and different edge discretization order with ```--edge_decision```. The number of searching interations (N in our paper) is controlled by parameter ```--pool_size```, while the number of validation iterations (V in our paper) can be specified by ```--validate_rounds```. Please see Section 4.2 in our paper for more information on those parameters.
|
||||
|
||||
For example, a typical experiement setting could be:
|
||||
|
||||
```--pool_size 10 --edge_decision random --validate_rounds 100 --metrics jacob --seed 0```
|
||||
|
||||
### Experiments on NAS-Bench-1shot1
|
||||
Scripts for reproducing our experiments on NAS-Bench-1shot1 can be found under the ```nasbench1shot1``` folder.
|
||||
|
||||
#### 1. Prepare NAS-Bench-101 Data
|
||||
1. Download NAS-Bench-1shot1 dataset from [nasbench_full.tfrecord](https://storage.googleapis.com/nasbench/nasbench_full.tfrecord), and place it under ```./data``` folder.
|
||||
|
||||
#### 2. Prepare NAS-Bench-101 API
|
||||
Please refer orginal [NAS-Bench-101](https://github.com/google-research/nasbench) for details of API installation
|
||||
|
||||
#### 3. Run Zero-Cost-PT on NAS-Bench-1shot1
|
||||
|
||||
You can reproduce our Zero-Cost-PT with the following script:
|
||||
```
|
||||
cd nasbench1shot1/optimizers/darts/
|
||||
```
|
||||
i. Run the following script for search architectures with Zero-Cost-PT from different sub-search-space on NAS-Bench-1shot1
|
||||
```
|
||||
python network_proposal.py --seed [SEED] --search_space [SEARCH_SPACE]
|
||||
```
|
||||
In NAS-Bench-1shot1, it contains 3 sub-search-space which you can select from [1, 2, 3]
|
||||
|
||||
ii. Evaluated final searched model
|
||||
```
|
||||
python post_validate.py --seed [SEED] --search_exp_path [PATH_to_LAST_STEP_LOG_FOLDER]
|
||||
```
|
||||
|
||||
### Experiments on NAS-Bench-Macro
|
||||
Scripts for reproducing our experiments on NAS-Bench-1shot1 can be found under the ```nasbenchmacro``` folder.
|
||||
|
||||
#### 1. Prepare NAS-Bench-Macro Data
|
||||
1. Download NAS-Bench-Macro dataset from [nas-bench-macro_cifar10.jsonnas-bench-macro_cifar10.json](https://github.com/xiusu/NAS-Bench-Macro/tree/master/data/nas-bench-macro_cifar10.json), and place it under ```./data``` folder.
|
||||
|
||||
#### 2. Run Zero-Cost-PT on NAS-Bench-Macro
|
||||
|
||||
You can reproduce our Zero-Cost-PT with the following script:
|
||||
```
|
||||
cd nasbenchmacro/
|
||||
```
|
||||
i. Run the following script for search architectures with Zero-Cost-PT on NAS-Bench-Macro
|
||||
```
|
||||
python network_proposal.py --seed [SEED]
|
||||
```
|
||||
|
||||
### Experiments on DARTS-like Spaces
|
||||
Scripts for reproducing our experiments can be found under the ```exp_scripts/``` folder, and Zero-Cost-NAS API is also needed.
|
||||
|
||||
#### 1. For DARTS CNN space
|
||||
|
||||
Run the following script to search architectures with Zero-Cost-PT and train the searched architecture directly (with the same random seed):
|
||||
```
|
||||
bash zerocostpt_darts_pipeline.sh --seed [SEED]
|
||||
```
|
||||
Our default parameter settings are:
|
||||
|
||||
```--pool_size 10 --edge_decision random --validate_rounds 100 --metrics jacob```
|
||||
|
||||
|
||||
#### 2、For DARTS subspaces S1-S4
|
||||
|
||||
On CIFAR-10 use the following script:
|
||||
|
||||
```
|
||||
bash zerocostpt_darts_pipeline.sh --seed [SEED] --space [s1-s4]
|
||||
```
|
||||
|
||||
On CIFAR-100 and SVHN, use the following scripts:
|
||||
|
||||
```
|
||||
bash zerocostpt_darts_pipeline_svhn.sh --seed [SEED] --space [s1-s4]
|
||||
bash zerocostpt_darts_pipeline_c100.sh --seed [SEED] --space [s1-s4]
|
||||
```
|
||||
|
||||
#### 3、Directly train the searched architectures reported in our paper
|
||||
|
||||
For reproducibility we also provide training scripts for evaluation of all the reported architectures in our paper. For an architecture specified by ```[genotype_name]```, run the following scrips to train:
|
||||
|
||||
```
|
||||
bash eval.sh --arch [genotype_name] # for DARTS C10
|
||||
bash eval-c100.sh --arch [genotype_name] # for DARTS C100
|
||||
bash eval-svhn.sh --arch [genotype_name] # for DARTS SVHN
|
||||
```
|
||||
|
||||
The model genotypes are provided in ```sota/cnn/genotypes.py```. For instance, genotype `init_pt_s5_C10_0_100_N10` specifies the architecture searched by Zero-Cost-PT (with default settings as explaind above) on DARTS CNN space (S5), using 10 search iterations (N=10), 100 validation iterations (V=100), and random seed 0.
|
||||
|
||||
#
|
||||
|
||||
|
||||
### Other Experiments Reported in Appendix
|
||||
We also provide code to reproduce experiment results reported in appendix, e.g. genotypes for maximum-param and random-sampling baselines, and Zero-Cost-PT for MobileNet-like spaces.
|
||||
|
||||
|
||||
|
||||
## Reference
|
||||
Our code (Zero-Cost-PT) is based on [dart-pt](https://github.com/ruocwang/darts-pt) and [Zero-Cost-NAS](https://github.com/SamsungLabs/zero-cost-nas). For experiments on Nasbench1shot1 is based on [nasbench-1shot1](https://github.com/automl/nasbench-1shot1)
|
||||
|
||||
Our code is based on [Zero-Cost-PT](https://github.com/zerocostptnas/zerocost_operation_score) and [Zero-Cost-NAS](https://github.com/SamsungLabs/zero-cost-nas).
|
||||
|
||||
Loading…
Reference in New Issue
Block a user