Quick produce table and full run
This commit is contained in:
parent
204a72f95e
commit
656e75cba8
12
README.md
12
README.md
@ -4,7 +4,7 @@
|
||||
|
||||
The datasets can also be downloaded as instructed from the NASBench-201 README: [https://github.com/D-X-Y/NAS-Bench-201](https://github.com/D-X-Y/NAS-Bench-201).
|
||||
|
||||
To exactly reproduce our results:
|
||||
To reproduce our results:
|
||||
|
||||
```
|
||||
conda env create -f environment.yml
|
||||
@ -13,11 +13,19 @@ conda activate nas-wot
|
||||
./reproduce.sh
|
||||
```
|
||||
|
||||
Will produce the following table:
|
||||
For a quick run you can set `--n_runs 3` to get results after 3 runs:
|
||||
|
||||
| Method | Search time (s) | CIFAR-10 (val) | CIFAR-10 (test) | CIFAR-100 (val) | CIFAR-100 (test) | ImageNet16-120 (val) | ImageNet16-120 (test) |
|
||||
|:-------------|------------------:|:-----------------|:------------------|:------------------|:-------------------|:-----------------------|:------------------------|
|
||||
| Ours (N=10) | 1.73435 | 88.99 $\pm$ 0.24 | 92.42 $\pm$ 0.33 | 67.86 $\pm$ 0.49 | 67.54 $\pm$ 0.75 | 41.16 $\pm$ 2.31 | 40.98 $\pm$ 2.72 |
|
||||
| Ours (N=100) | 17.4139 | 89.18 $\pm$ 0.29 | 91.76 $\pm$ 1.28 | 67.17 $\pm$ 2.79 | 67.27 $\pm$ 2.68 | 40.84 $\pm$ 5.36 | 41.33 $\pm$ 5.74
|
||||
|
||||
The size of `N` is set with `--n_samples 10`. To produce the results in the paper, set `--n_runs 500`:
|
||||
|
||||
| Method | Search time (s) | CIFAR-10 (val) | CIFAR-10 (test) | CIFAR-100 (val) | CIFAR-100 (test) | ImageNet16-120 (val) | ImageNet16-120 (test) |
|
||||
|:-------------|------------------:|:-----------------|:------------------|:------------------|:-------------------|:-----------------------|:------------------------|
|
||||
| Ours (N=10) | 1.73435 | 89.25 $\pm$ 0.08 | 92.21 $\pm$ 0.11 | 68.53 $\pm$ 0.17 | 68.40 $\pm$ 0.14 | 40.42 $\pm$ 1.15 | 40.66 $\pm$ 0.97 |
|
||||
| Ours (N=100) | 17.4139 | 88.45 $\pm$ 1.46 | 91.61 $\pm$ 1.71 | 66.42 $\pm$ 3.27 | 66.56 $\pm$ 3.28 | 36.56 $\pm$ 6.70 | 36.37 $\pm$ 6.97
|
||||
|
||||
|
||||
The code is licensed under the MIT licence.
|
||||
|
@ -3,9 +3,9 @@
|
||||
#python search.py --dataset cifar100 --data_loc '../datasets/cifar100' --n_runs 3 --n_samples 10
|
||||
#python search.py --dataset ImageNet16-120 --data_loc '../datasets/ImageNet16' --n_runs 3 --n_samples 10
|
||||
|
||||
python search.py --dataset cifar10 --data_loc '../datasets/cifar10' --n_runs 3 --n_samples 100
|
||||
python search.py --dataset cifar10 --data_loc '../datasets/cifar10' --n_runs 3 --n_samples 100
|
||||
python search.py --dataset cifar10 --trainval --data_loc '../datasets/cifar10' --n_runs 3 --n_samples 100
|
||||
python search.py --dataset cifar100 --data_loc '../datasets/cifar100' --n_runs 3 --n_samples 100
|
||||
python search.py --dataset ImageNet16-120 --data_loc '../datasets/ImageNet16' --n_runs 3 --n_samples 100
|
||||
python search.py --dataset cifar100 --data_loc '../datasets/cifar100' --n_runs 3 --n_samples 100
|
||||
python search.py --dataset ImageNet16-120 --data_loc '../datasets/ImageNet16' --n_runs 3 --n_samples 100
|
||||
|
||||
python process_results.py --n_runs 3
|
||||
|
Loading…
Reference in New Issue
Block a user