diff --git a/README.md b/README.md index dae08da..762777b 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ The datasets can also be downloaded as instructed from the NASBench-201 README: [https://github.com/D-X-Y/NAS-Bench-201](https://github.com/D-X-Y/NAS-Bench-201). -To exactly reproduce our results: +To reproduce our results: ``` conda env create -f environment.yml @@ -13,11 +13,19 @@ conda activate nas-wot ./reproduce.sh ``` -Will produce the following table: +For a quick run you can set `--n_runs 3` to get results after 3 runs: | Method | Search time (s) | CIFAR-10 (val) | CIFAR-10 (test) | CIFAR-100 (val) | CIFAR-100 (test) | ImageNet16-120 (val) | ImageNet16-120 (test) | |:-------------|------------------:|:-----------------|:------------------|:------------------|:-------------------|:-----------------------|:------------------------| | Ours (N=10) | 1.73435 | 88.99 $\pm$ 0.24 | 92.42 $\pm$ 0.33 | 67.86 $\pm$ 0.49 | 67.54 $\pm$ 0.75 | 41.16 $\pm$ 2.31 | 40.98 $\pm$ 2.72 | | Ours (N=100) | 17.4139 | 89.18 $\pm$ 0.29 | 91.76 $\pm$ 1.28 | 67.17 $\pm$ 2.79 | 67.27 $\pm$ 2.68 | 40.84 $\pm$ 5.36 | 41.33 $\pm$ 5.74 +The size of `N` is set with `--n_samples 10`. To produce the results in the paper, set `--n_runs 500`: + +| Method | Search time (s) | CIFAR-10 (val) | CIFAR-10 (test) | CIFAR-100 (val) | CIFAR-100 (test) | ImageNet16-120 (val) | ImageNet16-120 (test) | +|:-------------|------------------:|:-----------------|:------------------|:------------------|:-------------------|:-----------------------|:------------------------| +| Ours (N=10) | 1.73435 | 89.25 $\pm$ 0.08 | 92.21 $\pm$ 0.11 | 68.53 $\pm$ 0.17 | 68.40 $\pm$ 0.14 | 40.42 $\pm$ 1.15 | 40.66 $\pm$ 0.97 | +| Ours (N=100) | 17.4139 | 88.45 $\pm$ 1.46 | 91.61 $\pm$ 1.71 | 66.42 $\pm$ 3.27 | 66.56 $\pm$ 3.28 | 36.56 $\pm$ 6.70 | 36.37 $\pm$ 6.97 + + The code is licensed under the MIT licence. diff --git a/reproduce.sh b/reproduce.sh index dd0fc9b..5e63cdf 100755 --- a/reproduce.sh +++ b/reproduce.sh @@ -3,9 +3,9 @@ #python search.py --dataset cifar100 --data_loc '../datasets/cifar100' --n_runs 3 --n_samples 10 #python search.py --dataset ImageNet16-120 --data_loc '../datasets/ImageNet16' --n_runs 3 --n_samples 10 -python search.py --dataset cifar10 --data_loc '../datasets/cifar10' --n_runs 3 --n_samples 100 +python search.py --dataset cifar10 --data_loc '../datasets/cifar10' --n_runs 3 --n_samples 100 python search.py --dataset cifar10 --trainval --data_loc '../datasets/cifar10' --n_runs 3 --n_samples 100 -python search.py --dataset cifar100 --data_loc '../datasets/cifar100' --n_runs 3 --n_samples 100 -python search.py --dataset ImageNet16-120 --data_loc '../datasets/ImageNet16' --n_runs 3 --n_samples 100 +python search.py --dataset cifar100 --data_loc '../datasets/cifar100' --n_runs 3 --n_samples 100 +python search.py --dataset ImageNet16-120 --data_loc '../datasets/ImageNet16' --n_runs 3 --n_samples 100 python process_results.py --n_runs 3