iicd/xautodl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

re-organize NATS-Bench

Browse Source
This commit is contained in:
D-X-Y 2020-08-18 06:29:30 +00:00
parent ffd23a6cbd
commit 3529b993ff
10 changed files with 102 additions and 153 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
README.md
View File

@ -67,7 +67,8 @@ At the moment, this project provides the following algorithms and scripts to run
<tr> <!-- (7-th row) --> <tr> <!-- (7-th row) -->
<td align="center" valign="middle"> ... </td> <td align="center" valign="middle"> ... </td>
<td align="center" valign="middle"> ENAS / REA / REINFORCE / BOHB </td> <td align="center" valign="middle"> ENAS / REA / REINFORCE / BOHB </td>
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NAS-Bench-201.md">NAS-Bench-201.md</a> </td> <td align="center" valign="middle"> Please check the original papers </td>
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NAS-Bench-201.md">NAS-Bench-201.md</a> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NATS-Bench.md">NATS-Bench.md</a> </td>
</tr> </tr>
<tr> <!-- (start second block) --> <tr> <!-- (start second block) -->
<td rowspan="1" align="center" valign="middle" halign="middle"> HPO </td> <td rowspan="1" align="center" valign="middle" halign="middle"> HPO </td>

9
README_CN.md
View File

@ -59,10 +59,17 @@
<td align="center" valign="middle"> <a href="https://openreview.net/forum?id=HJxyZkBKDr"> NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search</a> </td> <td align="center" valign="middle"> <a href="https://openreview.net/forum?id=HJxyZkBKDr"> NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search</a> </td>
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NAS-Bench-201.md">NAS-Bench-201.md</a> </td> <td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NAS-Bench-201.md">NAS-Bench-201.md</a> </td>
</tr> </tr>
<tr> <!-- (6-th row) -->
<td align="center" valign="middle"> NATS-Bench </td>
<td align="center" valign="middle"> <a href="https://xuanyidong.com/assets/projects/NATS-Bench"> NATS-Bench: Benchmarking NAS algorithms for Architecture Topology and Size</a> </td>
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NATS-Bench.md">NATS-Bench.md</a> </td>
</tr>
<tr> <!-- (6-th row) --> <tr> <!-- (6-th row) -->
<td align="center" valign="middle"> ... </td> <td align="center" valign="middle"> ... </td>
<td align="center" valign="middle"> ENAS / REA / REINFORCE / BOHB </td> <td align="center" valign="middle"> ENAS / REA / REINFORCE / BOHB </td>
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NAS-Bench-201.md">NAS-Bench-201.md</a> </td> <td align="center" valign="middle"> Please check the original papers. </td>
<td align="center" valign="middle"> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NAS-Bench-201.md">NAS-Bench-201.md</a> <a href="https://github.com/D-X-Y/AutoDL-Projects/tree/master/docs/NATS-Bench.md">NATS-Bench.md</a> </td>
</tr> </tr>
<tr> <!-- (start second block) --> <tr> <!-- (start second block) -->
<td rowspan="1" align="center" valign="middle" halign="middle"> HPO </td> <td rowspan="1" align="center" valign="middle" halign="middle"> HPO </td>

13
docs/NATS-Bench.md
View File

@ -7,11 +7,23 @@ We analyze the validity of our benchmark in terms of various criteria and perfor
We also show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms on it. All logs and diagnostic information trained using the same setup for each candidate are provided. We also show the versatility of NATS-Bench by benchmarking 13 recent state-of-the-art NAS algorithms on it. All logs and diagnostic information trained using the same setup for each candidate are provided.
This facilitates a much larger community of researchers to focus on developing better NAS algorithms in a more comparable and computationally effective environment. This facilitates a much larger community of researchers to focus on developing better NAS algorithms in a more comparable and computationally effective environment.
**coming soon!** **coming soon!**
## How to Use NATS-Bench ## How to Use NATS-Bench
## The Procedure of Creating NATS-Bench
1, train all architecture candidate in the size search space with 90 epochs and use the random seed of `777`.
```
bash ./scripts/NATS-Bench/train-shapes.sh 00000-32767 90 777
```
The checkpoint of all candidates are located at `output/NATS-Bench-size` by default
## To Reproduce 13 Baseline NAS Algorithms in NAS-Bench-201 ## To Reproduce 13 Baseline NAS Algorithms in NAS-Bench-201
### Reproduce NAS methods on the topology search space ### Reproduce NAS methods on the topology search space
@ -50,6 +62,7 @@ python ./exps/algos-v2/search-cell.py --dataset ImageNet16-120 --data_path $TORC
### Reproduce NAS methods on the size search space ### Reproduce NAS methods on the size search space
### Final Discovered Architectures for Each Algorithm ### Final Discovered Architectures for Each Algorithm
The architecture index can be found by use `api.query_index_by_arch(architecture_string)`. The architecture index can be found by use `api.query_index_by_arch(architecture_string)`.

30
exps/NAS-Bench-201/xshapes.py → exps/NATS-Bench/main-sss.py
View File

@ -1,8 +1,8 @@
############################################################### ##############################################################################
# NAS-Bench-201, ICLR 2020 (https://arxiv.org/abs/2001.00326) # # NATS-Bench: Benchmarking NAS algorithms for Architecture Topology and Size #
############################################################### ##############################################################################
# Copyright (c) Xuanyi Dong [GitHub D-X-Y], 2019.08 # # Copyright (c) Xuanyi Dong [GitHub D-X-Y], 2020.07 #
############################################################### ##############################################################################
import os, sys, time, torch, argparse import os, sys, time, torch, argparse
from typing import List, Text, Dict, Any from typing import List, Text, Dict, Any
from PIL import ImageFile from PIL import ImageFile
@ -189,9 +189,9 @@ def filter_indexes(xlist, mode, save_dir, seeds):
if __name__ == '__main__': if __name__ == '__main__':
parser = argparse.ArgumentParser(description='NAS-Bench-X', formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser = argparse.ArgumentParser(description='NATS-Bench (size search space)', formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--mode', type=str, required=True, choices=['new', 'cover'], help='The script mode.') parser.add_argument('--mode', type=str, required=True, choices=['new', 'cover'], help='The script mode.')
parser.add_argument('--save_dir', type=str, default='output/NAS-BENCH-202', help='Folder to save checkpoints and log.') parser.add_argument('--save_dir', type=str, default='output/NATS-Bench-size', help='Folder to save checkpoints and log.')
parser.add_argument('--candidateC', type=int, nargs='+', default=[8, 16, 24, 32, 40, 48, 56, 64], help='.') parser.add_argument('--candidateC', type=int, nargs='+', default=[8, 16, 24, 32, 40, 48, 56, 64], help='.')
parser.add_argument('--num_layers', type=int, default=5, help='The number of layers in a network.') parser.add_argument('--num_layers', type=int, default=5, help='The number of layers in a network.')
parser.add_argument('--check_N', type=int, default=32768, help='For safety.') parser.add_argument('--check_N', type=int, default=32768, help='For safety.')
@ -206,10 +206,12 @@ if __name__ == '__main__':
args = parser.parse_args() args = parser.parse_args()
nets = traverse_net(args.candidateC, args.num_layers) nets = traverse_net(args.candidateC, args.num_layers)
if len(nets) != args.check_N: raise ValueError('Pre-num-check failed : {:} vs {:}'.format(len(nets), args.check_N)) if len(nets) != args.check_N:
raise ValueError('Pre-num-check failed : {:} vs {:}'.format(len(nets), args.check_N))
opt_config = './configs/nas-benchmark/hyper-opts/{:}E.config'.format(args.hyper) opt_config = './configs/nas-benchmark/hyper-opts/{:}E.config'.format(args.hyper)
if not os.path.isfile(opt_config): raise ValueError('{:} is not a file.'.format(opt_config)) if not os.path.isfile(opt_config):
raise ValueError('{:} is not a file.'.format(opt_config))
save_dir = Path(args.save_dir) / 'raw-data-{:}'.format(args.hyper) save_dir = Path(args.save_dir) / 'raw-data-{:}'.format(args.hyper)
save_dir.mkdir(parents=True, exist_ok=True) save_dir.mkdir(parents=True, exist_ok=True)
if not isinstance(args.srange, str): if not isinstance(args.srange, str):
@ -218,7 +220,8 @@ if __name__ == '__main__':
to_evaluate_indexes = set() to_evaluate_indexes = set()
for srange in srangestr.split(','): for srange in srangestr.split(','):
srange = srange.split('-') srange = srange.split('-')
if len(srange) != 2: raise ValueError('invalid srange : {:}'.format(srange)) if len(srange) != 2:
raise ValueError('invalid srange : {:}'.format(srange))
assert len(srange[0]) == len(srange[1]) == 5, 'invalid srange : {:}'.format(srange) assert len(srange[0]) == len(srange[1]) == 5, 'invalid srange : {:}'.format(srange)
srange = (int(srange[0]), int(srange[1])) srange = (int(srange[0]), int(srange[1]))
if not (0 <= srange[0] <= srange[1] < args.check_N): if not (0 <= srange[0] <= srange[1] < args.check_N):
@ -226,10 +229,12 @@ if __name__ == '__main__':
for i in range(srange[0], srange[1]+1): for i in range(srange[0], srange[1]+1):
to_evaluate_indexes.add(i) to_evaluate_indexes.add(i)
assert len(args.seeds) > 0, 'invalid length of seeds args: {:}'.format(args.seeds) if not len(args.seeds):
raise ValueError('invalid length of seeds args: {:}'.format(args.seeds))
if not (len(args.datasets) == len(args.xpaths) == len(args.splits)): if not (len(args.datasets) == len(args.xpaths) == len(args.splits)):
raise ValueError('invalid infos : {:} vs {:} vs {:}'.format(len(args.datasets), len(args.xpaths), len(args.splits))) raise ValueError('invalid infos : {:} vs {:} vs {:}'.format(len(args.datasets), len(args.xpaths), len(args.splits)))
assert args.workers > 0, 'invalid number of workers : {:}'.format(args.workers) if args.workers <= 0:
raise ValueError('invalid number of workers : {:}'.format(args.workers))
target_indexes = filter_indexes(to_evaluate_indexes, args.mode, save_dir, args.seeds) target_indexes = filter_indexes(to_evaluate_indexes, args.mode, save_dir, args.seeds)
@ -239,4 +244,3 @@ if __name__ == '__main__':
torch.set_num_threads(args.workers) torch.set_num_threads(args.workers)
main(save_dir, args.workers, args.datasets, args.xpaths, args.splits, tuple(args.seeds), nets, opt_config, target_indexes, args.mode == 'cover') main(save_dir, args.workers, args.datasets, args.xpaths, args.splits, tuple(args.seeds), nets, opt_config, target_indexes, args.mode == 'cover')

18
exps/NAS-Bench-201/xshape-file.py → exps/NATS-Bench/sss-file-manager.py
View File

@ -1,10 +1,10 @@
############################################################### ##############################################################################
# NAS-Bench-201, ICLR 2020 (https://arxiv.org/abs/2001.00326) # # NATS-Bench: Benchmarking NAS algorithms for Architecture Topology and Size #
############################################################### ##############################################################################
# Copyright (c) Xuanyi Dong [GitHub D-X-Y], 2020.01 # # Copyright (c) Xuanyi Dong [GitHub D-X-Y], 2020.01 #
############################################################### ##############################################################################
# Usage: python exps/NAS-Bench-201/xshape-file.py --mode check # Usage: python exps/NATS-Bench/sss-file-manager.py --mode check #
############################################################### ##############################################################################
import os, sys, time, torch, argparse import os, sys, time, torch, argparse
from typing import List, Text, Dict, Any from typing import List, Text, Dict, Any
from shutil import copyfile from shutil import copyfile
@ -55,9 +55,9 @@ def copy_data(source_dir, target_dir, meta_path):
if __name__ == '__main__': if __name__ == '__main__':
parser = argparse.ArgumentParser(description='NAS-Bench-X', formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser = argparse.ArgumentParser(description='NATS-Bench (size search space) file manager.', formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--mode', type=str, required=True, choices=['check', 'copy'], help='The script mode.') parser.add_argument('--mode', type=str, required=True, choices=['check', 'copy'], help='The script mode.')
parser.add_argument('--save_dir', type=str, default='output/NAS-BENCH-202', help='Folder to save checkpoints and log.') parser.add_argument('--save_dir', type=str, default='output/NATS-Bench-size', help='Folder to save checkpoints and log.')
parser.add_argument('--check_N', type=int, default=32768, help='For safety.') parser.add_argument('--check_N', type=int, default=32768, help='For safety.')
# use for train the model # use for train the model
args = parser.parse_args() args = parser.parse_args()

2
scripts-search/NASNet-space-search-by-GDAS-FRC.sh
View File

@ -25,7 +25,7 @@ else
data_path="$TORCH_HOME/cifar.python/ImageNet16" data_path="$TORCH_HOME/cifar.python/ImageNet16"
fi fi
save_dir=./output/search-cell-${space}/GDAS-${dataset}-BN${track_running_stats} save_dir=./output/search-cell-${space}/GDAS-FRC-${dataset}-BN${track_running_stats}
OMP_NUM_THREADS=4 python ./exps/algos/GDAS.py \ OMP_NUM_THREADS=4 python ./exps/algos/GDAS.py \
--save_dir ${save_dir} \ --save_dir ${save_dir} \

36
scripts-search/X-X/train-shapes-01.sh
View File

@ -1,36 +0,0 @@
#!/bin/bash
#####################################################
# Copyright (c) Xuanyi Dong [GitHub D-X-Y], 2020.01 #
#####################################################
# bash ./scripts-search/X-X/train-shapes-01.sh 0 4
echo script name: $0
echo $# arguments
if [ "$#" -ne 2 ] ;then
echo "Input illegal number of parameters " $#
echo "Need 2 parameters for hyper-parameters-opt-file, and seeds"
exit 1
fi
if [ "$TORCH_HOME" = "" ]; then
echo "Must set TORCH_HOME envoriment variable for data dir saving"
exit 1
else
echo "TORCH_HOME : $TORCH_HOME"
fi
srange=00000-32767
opt=01
all_seeds=777
cpus=4
save_dir=./output/NAS-BENCH-202/
SLURM_PROCID=$1 SLURM_NTASKS=$2 OMP_NUM_THREADS=${cpus} python exps/NAS-Bench-201/xshapes.py \
--mode new --srange ${srange} --hyper ${opt} --save_dir ${save_dir} \
--datasets cifar10 cifar10 cifar100 ImageNet16-120 \
--splits 1 0 0 0 \
--xpaths $TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python/ImageNet16 \
--workers ${cpus} \
--seeds ${all_seeds}

43
scripts-search/X-X/train-shapes-v2.sh
View File

@ -1,43 +0,0 @@
#!/bin/bash
#####################################################
# Copyright (c) Xuanyi Dong [GitHub D-X-Y], 2020.01 #
#####################################################
# SLURM_PROCID=0 SLURM_NTASKS=6 bash ./scripts-search/X-X/train-shapes-v2.sh 12 777
#
# SLURM_PROCID=0 SLURM_NTASKS=4 bash ./scripts-search/X-X/train-shapes.sh 30000-32767 90 777
# SLURM_PROCID=0 SLURM_NTASKS=4 bash ./scripts-search/X-X/train-shapes.sh 00000-09999 90 777
#
echo script name: $0
echo $# arguments
if [ "$#" -ne 2 ] ;then
echo "Input illegal number of parameters " $#
echo "Need 2 parameters for hyper-parameters-opt-file, and seeds"
exit 1
fi
if [ "$TORCH_HOME" = "" ]; then
echo "Must set TORCH_HOME envoriment variable for data dir saving"
exit 1
else
echo "TORCH_HOME : $TORCH_HOME"
fi
#srange=01000-03999,04050-05000,06000-09000,11000-14500,15000-18500,20000-23500,25000-27500,29000-30000
#srange=00000-00999,04000-04049,05001-05999,09001-10999,14501-14999,18501-19999,23501-24999,27501-28999,30001-32767
#srange=00000-09999
srange=10000-29999
opt=$1
all_seeds=$2
cpus=4
save_dir=./output/NAS-BENCH-202/
OMP_NUM_THREADS=${cpus} python exps/NAS-Bench-201/xshapes.py \
--mode new --srange ${srange} --hyper ${opt} --save_dir ${save_dir} \
--datasets cifar10 cifar10 cifar100 ImageNet16-120 \
--splits 1 0 0 0 \
--xpaths $TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python/ImageNet16 \
--workers ${cpus} \
--seeds ${all_seeds}

49
scripts-search/X-X/train-shapes.sh
View File

@ -1,49 +0,0 @@
#!/bin/bash
#####################################################
# Copyright (c) Xuanyi Dong [GitHub D-X-Y], 2020.01 #
#####################################################
# [mars6] CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/X-X/train-shapes.sh 00000-05000 12 777
# [mars6] bash ./scripts-search/X-X/train-shapes.sh 05001-10000 12 777
# [mars20] bash ./scripts-search/X-X/train-shapes.sh 10001-14500 12 777
# [mars20] bash ./scripts-search/X-X/train-shapes.sh 14501-18000 12 777
# [saturn4] bash ./scripts-search/X-X/train-shapes.sh 18001-19500 12 777
# [saturn4] bash ./scripts-search/X-X/train-shapes.sh 19501-23500 12 777
# [saturn4] bash ./scripts-search/X-X/train-shapes.sh 23501-27500 12 777
# [saturn4] bash ./scripts-search/X-X/train-shapes.sh 27501-30000 12 777
# [x] bash ./scripts-search/X-X/train-shapes.sh 30001-32767 12 777
#
# CUDA_VISIBLE_DEVICES=2 bash ./scripts-search/X-X/train-shapes.sh 01000-03999,04050-05000,06000-09000,11000-14500,15000-18500,20000-23500,25000-27500,29000-30000 12 777
# SLURM_PROCID=1 SLURM_NTASKS=5 bash ./scripts-search/X-X/train-shapes.sh 01000-03999,04050-05000,06000-09000,11000-14500,15000-18500,20000-23500,25000-27500,29000-30000 90 777
# [GCP] bash ./scripts-search/X-X/train-shapes.sh 00000-09999 90 777
# [UTS] bash ./scripts-search/X-X/train-shapes.sh 30000-32767 90 777
echo script name: $0
echo $# arguments
if [ "$#" -ne 3 ] ;then
echo "Input illegal number of parameters " $#
echo "Need 3 parameters for start-and-end, hyper-parameters-opt-file, and seeds"
exit 1
fi
if [ "$TORCH_HOME" = "" ]; then
echo "Must set TORCH_HOME envoriment variable for data dir saving"
exit 1
else
echo "TORCH_HOME : $TORCH_HOME"
fi
srange=$1
opt=$2
all_seeds=$3
cpus=4
save_dir=./output/NAS-BENCH-202/
OMP_NUM_THREADS=${cpus} python exps/NAS-Bench-201/xshapes.py \
--mode new --srange ${srange} --hyper ${opt} --save_dir ${save_dir} \
--datasets cifar10 cifar10 cifar100 ImageNet16-120 \
--splits 1 0 0 0 \
--xpaths $TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python/ImageNet16 \
--workers ${cpus} \
--seeds ${all_seeds}

52
scripts/NATS-Bench/train-shapes.sh Normal file
View File

@ -0,0 +1,52 @@
#!/bin/bash
##############################################################################
# NATS-Bench: Benchmarking NAS algorithms for Architecture Topology and Size #
##############################################################################
# Copyright (c) Xuanyi Dong [GitHub D-X-Y], 2020.01 #
##############################################################################
# CUDA_VISIBLE_DEVICES=0 bash scripts/NATS-Bench/train-shapes.sh 00000-05000 12 777
# bash ./scripts/NATS-Bench/train-shapes.sh 05001-10000 12 777
# bash ./scripts/NATS-Bench/train-shapes.sh 10001-14500 12 777
# bash ./scripts/NATS-Bench/train-shapes.sh 14501-18000 12 777
# bash ./scripts/NATS-Bench/train-shapes.sh 18001-19500 12 777
# bash ./scripts/NATS-Bench/train-shapes.sh 19501-23500 12 777
# bash ./scripts/NATS-Bench/train-shapes.sh 23501-27500 12 777
# bash ./scripts/NATS-Bench/train-shapes.sh 27501-30000 12 777
# bash ./scripts/NATS-Bench/train-shapes.sh 30001-32767 12 777
#
# CUDA_VISIBLE_DEVICES=2 bash ./scripts/NATS-Bench/train-shapes.sh 01000-03999,04050-05000,06000-09000,11000-14500,15000-18500,20000-23500,25000-27500,29000-30000 12 777
# SLURM_PROCID=1 SLURM_NTASKS=5 bash ./scripts/NATS-Bench/train-shapes.sh 01000-03999,04050-05000,06000-09000,11000-14500,15000-18500,20000-23500,25000-27500,29000-30000 90 777
# [GCP] bash ./scripts/NATS-Bench/train-shapes.sh 00000-09999 90 777
# [UTS] bash ./scripts/NATS-Bench/train-shapes.sh 30000-32767 90 777
##############################################################################
echo script name: $0
echo $# arguments
if [ "$#" -ne 3 ] ;then
echo "Input illegal number of parameters " $#
echo "Need 3 parameters for start-and-end, hyper-parameters-opt-file, and seeds"
exit 1
fi
if [ "$TORCH_HOME" = "" ]; then
echo "Must set TORCH_HOME envoriment variable for data dir saving"
exit 1
else
echo "TORCH_HOME : $TORCH_HOME"
fi
srange=$1
opt=$2
all_seeds=$3
cpus=4
save_dir=./output/NATS-Bench-size/
OMP_NUM_THREADS=${cpus} python exps/NATS-Bench/main-sss.py \
--mode new --srange ${srange} --hyper ${opt} --save_dir ${save_dir} \
--datasets cifar10 cifar10 cifar100 ImageNet16-120 \
--splits 1 0 0 0 \
--xpaths $TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python \
$TORCH_HOME/cifar.python/ImageNet16 \
--workers ${cpus} \
--seeds ${all_seeds}