diff --git a/README.md b/README.md index cee8633..59b727a 100644 --- a/README.md +++ b/README.md @@ -1,143 +1,68 @@ -# Zero-Cost Operation Scoring in Differentiable Architecture Search (Zero-Cost-PT) -Official impementation for AAAI 2023 submission: -"**Zero-Cost Operation Scoring in Differentiable Architecture Search**". +# MeCo: Zero-Cost Proxy for NAS Via Minimum Eigenvalue of Correlation on Feature Maps +## Installation -## Installation ``` Python >= 3.6 -PyTorch >= 1.7.1 -torchvision == 0.8.2 -tensorboard == 2.4.1 -scipy == 1.5.2 -gpustat +PyTorch >= 2.0.0 +nas-bench-201 ``` - + +## Preparation + +1. Download three datasets (CIFAR-10, CIFAR-100, ImageNet16-120) from [Google Drive](https://drive.google.com/drive/folders/1T3UIyZXUhMmIuJLOBMIYKAsJknAtrrO4), place them into the directory `./data` +2. Download the [`data` directory](https://drive.google.com/drive/folders/18Eia6YuTE5tn5Lis_43h30HYpnF9Ynqf?usp=sharing) and save it to the root folder of this repo. +3. Download the benchmark files of NAS-Bench-201 from [Google Drive](https://drive.google.com/file/d/1SKW0Cu0u8-gb18zDpaAGi0f74UdXeGKs/view) , put them into the directory `./data` +4. Download the [NAS-Bench-101 dataset](https://storage.googleapis.com/nasbench/nasbench_only108.tfrecord), put it into the directory `./data` +5. Install `zero-cost-nas` + ```bash + cd zero-cost-nas + pip install . + cd .. + ``` + ## Usage/Examples +### Correlation Experiment + +```bash +cd correlation +python NAS_Bench_101.py +python NAS_Bench_201.py +``` + + + + ### Experiments on NAS-Bench-201 -Scripts for reproducing our experiments can be found under the ```exp_scripts/``` folder. -#### 1. Prepare NAS-Bench-201 Data -1. Download NAS-Bench-201 checkpoint from [NAS-Bench-201-v1_0-e61699.pth](https://drive.google.com/file/d/1SKW0Cu0u8-gb18zDpaAGi0f74UdXeGKs/view), and place it under ```./data``` folder. +1. Run Zero-Cost-PT with appointed zero-cost proxy: +```bash +cd exp_scripts +bash zerocostpt_nb201_pipline.sh --metric [metric] --batch_size [batch_size] --seed [seed] +``` +You can choice metric from `['snip', 'fisher', 'synflow', 'grad_norm', 'grasp', 'jacob_cov','tenas', 'zico', 'meco'] ` -#### 2. Prepare NAS-Bench-201 and Zero-Cost-NAS API -i. Install NAS-Bench-201 api via `pip` -``` -pip install nas-bench-201 -``` -ii. Install Zero-Cost-NAS API +### Experiments on DARTS-CNN Space -Clone the code repository from [Zero-Cost-NAS](https://github.com/SamsungLabs/zero-cost-nas). Go to the root directory of the cloned repo and run: -``` -pip install . +#### 1. DARTS CNN Space + +```bash +cd exp_scripts +bash zerocostpt_darts_pipline.sh --metric [metric] --batch_size [batch_size] --seed [seed] ``` -#### 3. Run Zero-Cost-PT on NAS-Bench-201 +#### 2. DARTS Subspaces S1-S4 + +````bash +cd exp_scripts +bash zerocostpt_darts_pipline.sh --metric [metric] --batch_size [batch_size] --seed [seed] --space [s1-s4] +```` You can run our Zero-Cost-PT with the following script: ``` bash zerocostpt_nb201_pipeline.sh --seed [SEED] ``` -You can specify random seeds with ``` --seed ``` for reproducibility. In our experiments we use random seeds 0, 1, 2, 3. - -You could also run with different zero-cost proxies by specifying ```--metrics```, and different edge discretization order with ```--edge_decision```. The number of searching interations (N in our paper) is controlled by parameter ```--pool_size```, while the number of validation iterations (V in our paper) can be specified by ```--validate_rounds```. Please see Section 4.2 in our paper for more information on those parameters. - -For example, a typical experiement setting could be: - -```--pool_size 10 --edge_decision random --validate_rounds 100 --metrics jacob --seed 0``` - -### Experiments on NAS-Bench-1shot1 -Scripts for reproducing our experiments on NAS-Bench-1shot1 can be found under the ```nasbench1shot1``` folder. - -#### 1. Prepare NAS-Bench-101 Data -1. Download NAS-Bench-1shot1 dataset from [nasbench_full.tfrecord](https://storage.googleapis.com/nasbench/nasbench_full.tfrecord), and place it under ```./data``` folder. - -#### 2. Prepare NAS-Bench-101 API -Please refer orginal [NAS-Bench-101](https://github.com/google-research/nasbench) for details of API installation - -#### 3. Run Zero-Cost-PT on NAS-Bench-1shot1 - -You can reproduce our Zero-Cost-PT with the following script: -``` - cd nasbench1shot1/optimizers/darts/ -``` -i. Run the following script for search architectures with Zero-Cost-PT from different sub-search-space on NAS-Bench-1shot1 -``` - python network_proposal.py --seed [SEED] --search_space [SEARCH_SPACE] -``` -In NAS-Bench-1shot1, it contains 3 sub-search-space which you can select from [1, 2, 3] - -ii. Evaluated final searched model -``` - python post_validate.py --seed [SEED] --search_exp_path [PATH_to_LAST_STEP_LOG_FOLDER] -``` - -### Experiments on NAS-Bench-Macro -Scripts for reproducing our experiments on NAS-Bench-1shot1 can be found under the ```nasbenchmacro``` folder. - -#### 1. Prepare NAS-Bench-Macro Data -1. Download NAS-Bench-Macro dataset from [nas-bench-macro_cifar10.jsonnas-bench-macro_cifar10.json](https://github.com/xiusu/NAS-Bench-Macro/tree/master/data/nas-bench-macro_cifar10.json), and place it under ```./data``` folder. - -#### 2. Run Zero-Cost-PT on NAS-Bench-Macro - -You can reproduce our Zero-Cost-PT with the following script: -``` - cd nasbenchmacro/ -``` -i. Run the following script for search architectures with Zero-Cost-PT on NAS-Bench-Macro -``` - python network_proposal.py --seed [SEED] -``` - -### Experiments on DARTS-like Spaces -Scripts for reproducing our experiments can be found under the ```exp_scripts/``` folder, and Zero-Cost-NAS API is also needed. - -#### 1. For DARTS CNN space - -Run the following script to search architectures with Zero-Cost-PT and train the searched architecture directly (with the same random seed): -``` -bash zerocostpt_darts_pipeline.sh --seed [SEED] -``` -Our default parameter settings are: - -```--pool_size 10 --edge_decision random --validate_rounds 100 --metrics jacob``` - - -#### 2、For DARTS subspaces S1-S4 - -On CIFAR-10 use the following script: - -``` -bash zerocostpt_darts_pipeline.sh --seed [SEED] --space [s1-s4] -``` - -On CIFAR-100 and SVHN, use the following scripts: - -``` -bash zerocostpt_darts_pipeline_svhn.sh --seed [SEED] --space [s1-s4] -bash zerocostpt_darts_pipeline_c100.sh --seed [SEED] --space [s1-s4] -``` - -#### 3、Directly train the searched architectures reported in our paper - -For reproducibility we also provide training scripts for evaluation of all the reported architectures in our paper. For an architecture specified by ```[genotype_name]```, run the following scrips to train: - -``` -bash eval.sh --arch [genotype_name] # for DARTS C10 -bash eval-c100.sh --arch [genotype_name] # for DARTS C100 -bash eval-svhn.sh --arch [genotype_name] # for DARTS SVHN -``` - -The model genotypes are provided in ```sota/cnn/genotypes.py```. For instance, genotype `init_pt_s5_C10_0_100_N10` specifies the architecture searched by Zero-Cost-PT (with default settings as explaind above) on DARTS CNN space (S5), using 10 search iterations (N=10), 100 validation iterations (V=100), and random seed 0. - -# - - -### Other Experiments Reported in Appendix -We also provide code to reproduce experiment results reported in appendix, e.g. genotypes for maximum-param and random-sampling baselines, and Zero-Cost-PT for MobileNet-like spaces. - - - ## Reference -Our code (Zero-Cost-PT) is based on [dart-pt](https://github.com/ruocwang/darts-pt) and [Zero-Cost-NAS](https://github.com/SamsungLabs/zero-cost-nas). For experiments on Nasbench1shot1 is based on [nasbench-1shot1](https://github.com/automl/nasbench-1shot1) + +Our code is based on [Zero-Cost-PT](https://github.com/zerocostptnas/zerocost_operation_score) and [Zero-Cost-NAS](https://github.com/SamsungLabs/zero-cost-nas). diff --git a/correlation/NAS-Bench-101.py b/correlation/NAS_Bench_101.py similarity index 100% rename from correlation/NAS-Bench-101.py rename to correlation/NAS_Bench_101.py diff --git a/correlation/NAS-Bench-201.py b/correlation/NAS_Bench_201.py similarity index 100% rename from correlation/NAS-Bench-201.py rename to correlation/NAS_Bench_201.py