iicd/naswot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Quick produce table and full run

Browse Source
This commit is contained in:
jack-willturner 2020-06-03 16:44:28 +01:00
parent 204a72f95e
commit 656e75cba8
2 changed files with 13 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
README.md
View File

@ -4,7 +4,7 @@
The datasets can also be downloaded as instructed from the NASBench-201 README: [https://github.com/D-X-Y/NAS-Bench-201](https://github.com/D-X-Y/NAS-Bench-201).
To exactly reproduce our results:
To reproduce our results:
```
conda env create -f environment.yml
@ -13,11 +13,19 @@ conda activate nas-wot
./reproduce.sh
```
Will produce the following table:
For a quick run you can set `--n_runs 3` to get results after 3 runs:
| Method | Search time (s) | CIFAR-10 (val) | CIFAR-10 (test) | CIFAR-100 (val) | CIFAR-100 (test) | ImageNet16-120 (val) | ImageNet16-120 (test) |
|:-------------|------------------:|:-----------------|:------------------|:------------------|:-------------------|:-----------------------|:------------------------|
| Ours (N=10) | 1.73435 | 88.99 $\pm$ 0.24 | 92.42 $\pm$ 0.33 | 67.86 $\pm$ 0.49 | 67.54 $\pm$ 0.75 | 41.16 $\pm$ 2.31 | 40.98 $\pm$ 2.72 |
| Ours (N=100) | 17.4139 | 89.18 $\pm$ 0.29 | 91.76 $\pm$ 1.28 | 67.17 $\pm$ 2.79 | 67.27 $\pm$ 2.68 | 40.84 $\pm$ 5.36 | 41.33 $\pm$ 5.74
The size of `N` is set with `--n_samples 10`. To produce the results in the paper, set `--n_runs 500`:
| Method | Search time (s) | CIFAR-10 (val) | CIFAR-10 (test) | CIFAR-100 (val) | CIFAR-100 (test) | ImageNet16-120 (val) | ImageNet16-120 (test) |
|:-------------|------------------:|:-----------------|:------------------|:------------------|:-------------------|:-----------------------|:------------------------|
| Ours (N=10) | 1.73435 | 89.25 $\pm$ 0.08 | 92.21 $\pm$ 0.11 | 68.53 $\pm$ 0.17 | 68.40 $\pm$ 0.14 | 40.42 $\pm$ 1.15 | 40.66 $\pm$ 0.97 |
| Ours (N=100) | 17.4139 | 88.45 $\pm$ 1.46 | 91.61 $\pm$ 1.71 | 66.42 $\pm$ 3.27 | 66.56 $\pm$ 3.28 | 36.56 $\pm$ 6.70 | 36.37 $\pm$ 6.97
The code is licensed under the MIT licence.

6
reproduce.sh
View File

@ -3,9 +3,9 @@
#python search.py --dataset cifar100 --data_loc '../datasets/cifar100' --n_runs 3 --n_samples 10
#python search.py --dataset ImageNet16-120 --data_loc '../datasets/ImageNet16' --n_runs 3 --n_samples 10
python search.py --dataset cifar10 --data_loc '../datasets/cifar10' --n_runs 3 --n_samples 100
python search.py --dataset cifar10 --data_loc '../datasets/cifar10' --n_runs 3 --n_samples 100
python search.py --dataset cifar10 --trainval --data_loc '../datasets/cifar10' --n_runs 3 --n_samples 100
python search.py --dataset cifar100 --data_loc '../datasets/cifar100' --n_runs 3 --n_samples 100
python search.py --dataset ImageNet16-120 --data_loc '../datasets/ImageNet16' --n_runs 3 --n_samples 100
python search.py --dataset cifar100 --data_loc '../datasets/cifar100' --n_runs 3 --n_samples 100
python search.py --dataset ImageNet16-120 --data_loc '../datasets/ImageNet16' --n_runs 3 --n_samples 100
python process_results.py --n_runs 3