+
+
+
| Type | +ABBRV | +Algorithms | +Description | +
|---|---|---|---|
| NAS | +TAS | +Network Pruning via Transformable Architecture Search | +NeurIPS-2019-TAS.md | +
| DARTS | +DARTS: Differentiable Architecture Search | +ICLR-2019-DARTS.md | +|
| GDAS | +Searching for A Robust Neural Architecture in Four GPU Hours | +CVPR-2019-GDAS.md | +|
| SETN | +One-Shot Neural Architecture Search via Self-Evaluated Template Network | +ICCV-2019-SETN.md | +|
| NAS-Bench-201 | +NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search | +NAS-Bench-201.md | +|
| NATS-Bench | +NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size | +NATS-Bench.md | +|
| ... | +ENAS / REA / REINFORCE / BOHB | +Please check the original papers | +NAS-Bench-201.md NATS-Bench.md | +
| HPO | +HPO-CG | +Hyperparameter optimization with approximate gradient | +coming soon | +
| Basic | +ResNet | +Deep Learning-based Image Classification | +BASELINE.md | +
+
+Searching for A Robust Neural Architecture in Four GPU Hours is accepted at CVPR 2019.
+In this paper, we proposed a Gradient-based searching algorithm using Differentiable Architecture Sampling (GDAS).
+GDAS is baseed on DARTS and improves it with Gumbel-softmax sampling.
+Concurrently at the submission period, several NAS papers (SNAS and FBNet) also utilized Gumbel-softmax sampling. We are different at how to forward and backward, see more details in our paper and codes.
+Experiments on CIFAR-10, CIFAR-100, ImageNet, PTB, and WT2 are reported.
+
+
+## Requirements and Preparation
+
+Please install `Python>=3.6` and `PyTorch>=1.5.0`.
+
+CIFAR and ImageNet should be downloaded and extracted into `$TORCH_HOME`.
+
+### Usefull tools
+1. Compute the number of parameters and FLOPs of a model:
+```
+from utils import get_model_infos
+flop, param = get_model_infos(net, (1,3,32,32))
+```
+
+2. Different NAS-searched architectures are defined [here](https://github.com/D-X-Y/AutoDL-Projects/blob/main/xautodl/nas_infer_model/DXYs/genotypes.py).
+
+
+## Usage
+
+### Reproducing the results of our searched architecture in GDAS
+Please use the following scripts to train the searched GDAS-searched CNN on CIFAR-10, CIFAR-100, and ImageNet.
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts/nas-infer-train.sh cifar10 GDAS_V1 96 -1
+CUDA_VISIBLE_DEVICES=0 bash ./scripts/nas-infer-train.sh cifar100 GDAS_V1 96 -1
+CUDA_VISIBLE_DEVICES=0,1,2,3 bash ./scripts/nas-infer-train.sh imagenet-1k GDAS_V1 256 -1
+```
+If you are interested in the configs of each NAS-searched architecture, they are defined at [genotypes.py](https://github.com/D-X-Y/AutoDL-Projects/blob/main/xautodl/nas_infer_model/DXYs/genotypes.py).
+
+### Searching on the NASNet search space
+
+Please use the following scripts to use GDAS to search as in the original paper:
+```
+# search for both normal and reduction cells
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NASNet-space-search-by-GDAS.sh cifar10 1 -1
+
+# search for the normal cell while use a fixed reduction cell
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NASNet-space-search-by-GDAS-FRC.sh cifar10 1 -1
+```
+
+**After searching**, if you want to re-train the searched architecture found by the above script, you can use the following script:
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts/retrain-searched-net.sh cifar10 gdas-searched \
+ output/search-cell-darts/GDAS-cifar10-BN1/checkpoint/seed-945-basic.pth 96 -1
+```
+Note that `gdas-searched` is a string to indicate the name of the saved dir and `output/search-cell-darts/GDAS-cifar10-BN1/checkpoint/seed-945-basic.pth` is the file path that the searching algorithm generated.
+
+The above script does not apply heavy augmentation to train the model, so the accuracy will be lower than the original paper.
+If you want to change the default hyper-parameter for re-training, please have a look at `./scripts/retrain-searched-net.sh` and `configs/archs/NAS-*-none.config`.
+
+
+### Searching on a small search space (NAS-Bench-201)
+
+The GDAS searching codes on a small search space:
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/GDAS.sh cifar10 1 -1
+```
+
+The baseline searching codes are DARTS:
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/DARTS-V1.sh cifar10 1 -1
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/DARTS-V2.sh cifar10 1 -1
+```
+
+**After searching**, if you want to train the searched architecture found by the above scripts, please use the following codes:
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201/train-a-net.sh '|nor_conv_3x3~0|+|nor_conv_3x3~0|nor_conv_3x3~1|+|skip_connect~0|skip_connect~1|skip_connect~2|' 16 5
+```
+`|nor_conv_3x3~0|+|nor_conv_3x3~0|nor_conv_3x3~1|+|skip_connect~0|skip_connect~1|skip_connect~2|` represents the structure of a searched architecture. My codes will automatically print it during the searching procedure.
+
+
+**Tensorflow codes for GDAS are in experimental state**, which locates at `exps-tf`.
+
+# Citation
+
+If you find that this project helps your research, please consider citing the following paper:
+```
+@inproceedings{dong2019search,
+ title = {Searching for A Robust Neural Architecture in Four GPU Hours},
+ author = {Dong, Xuanyi and Yang, Yi},
+ booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
+ pages = {1761--1770},
+ year = {2019}
+}
+```
diff --git a/AutoDL-Projects/docs/ICCV-2019-SETN.md b/AutoDL-Projects/docs/ICCV-2019-SETN.md
new file mode 100644
index 0000000..13c5fa1
--- /dev/null
+++ b/AutoDL-Projects/docs/ICCV-2019-SETN.md
@@ -0,0 +1,53 @@
+# [One-Shot Neural Architecture Search via Self-Evaluated Template Network](https://arxiv.org/abs/1910.05733)
+
+
+
+Highlight: we equip one-shot NAS with an architecture sampler and train network weights using uniformly sampling.
+
+One-Shot Neural Architecture Search via Self-Evaluated Template Network is accepted by ICCV 2019.
+
+
+## Requirements and Preparation
+
+Please install `Python>=3.6` and `PyTorch>=1.2.0`.
+
+### Usefull tools
+1. Compute the number of parameters and FLOPs of a model:
+```
+from utils import get_model_infos
+flop, param = get_model_infos(net, (1,3,32,32))
+```
+
+2. Different NAS-searched architectures are defined [here](https://github.com/D-X-Y/AutoDL-Projects/blob/main/lib/nas_infer_model/DXYs/genotypes.py).
+
+
+## Usage
+
+Please use the following scripts to train the searched SETN-searched CNN on CIFAR-10, CIFAR-100, and ImageNet.
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts/nas-infer-train.sh cifar10 SETN 96 -1
+CUDA_VISIBLE_DEVICES=0 bash ./scripts/nas-infer-train.sh cifar100 SETN 96 -1
+CUDA_VISIBLE_DEVICES=0,1,2,3 bash ./scripts/nas-infer-train.sh imagenet-1k SETN 256 -1
+```
+
+### Searching on the NAS-Bench-201 search space
+The searching codes of SETN on a small search space (NAS-Bench-201).
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/SETN.sh cifar10 1 -1
+```
+
+**Searching on the NASNet search space** is not ready yet.
+
+
+# Citation
+
+If you find that this project helps your research, please consider citing the following paper:
+```
+@inproceedings{dong2019one,
+ title = {One-Shot Neural Architecture Search via Self-Evaluated Template Network},
+ author = {Dong, Xuanyi and Yang, Yi},
+ booktitle = {Proceedings of the IEEE International Conference on Computer Vision (ICCV)},
+ pages = {3681--3690},
+ year = {2019}
+}
+```
diff --git a/AutoDL-Projects/docs/ICLR-2019-DARTS.md b/AutoDL-Projects/docs/ICLR-2019-DARTS.md
new file mode 100644
index 0000000..eaf3b8b
--- /dev/null
+++ b/AutoDL-Projects/docs/ICLR-2019-DARTS.md
@@ -0,0 +1,32 @@
+# DARTS: Differentiable Architecture Search
+
+DARTS: Differentiable Architecture Search is accepted by ICLR 2019.
+In this paper, Hanxiao proposed a differentiable neural architecture search method, named as DARTS.
+Recently, DARTS becomes very popular due to its simplicity and performance.
+
+## Run DARTS on the NAS-Bench-201 search space
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/DARTS-V2.sh cifar10 1 -1
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/algos/GDAS.sh cifar10 1 -1
+```
+
+## Run the first-order DARTS on the NASNet/DARTS search space
+This command will start to use the first-order DARTS to search architectures on the DARTS search space.
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/DARTS1V-search-NASNet-space.sh cifar10 -1
+```
+
+After searching, if you want to train the searched architecture found by the above scripts, you need to add the config of that architecture (will be printed in log) in [genotypes.py](https://github.com/D-X-Y/AutoDL-Projects/blob/main/lib/nas_infer_model/DXYs/genotypes.py).
+In future, I will add a more eligent way to train the searched architecture from the DARTS search space.
+
+
+# Citation
+
+```
+@inproceedings{liu2019darts,
+ title = {{DARTS}: Differentiable architecture search},
+ author = {Liu, Hanxiao and Simonyan, Karen and Yang, Yiming},
+ booktitle = {International Conference on Learning Representations (ICLR)},
+ year = {2019}
+}
+```
diff --git a/AutoDL-Projects/docs/NAS-Bench-201-PURE.md b/AutoDL-Projects/docs/NAS-Bench-201-PURE.md
new file mode 100644
index 0000000..56a4afb
--- /dev/null
+++ b/AutoDL-Projects/docs/NAS-Bench-201-PURE.md
@@ -0,0 +1,183 @@
+# [NAS-BENCH-201: Extending the Scope of Reproducible Neural Architecture Search](https://openreview.net/forum?id=HJxyZkBKDr)
+
+**Since our NAS-BENCH-201 has been extended to NATS-Bench, this `README` is deprecated and not maintained. Please use [NATS-Bench](https://github.com/D-X-Y/AutoDL-Projects/blob/main/docs/NATS-Bench.md), which has 5x more architecture information and faster API than NAS-BENCH-201.**
+
+We propose an algorithm-agnostic NAS benchmark (NAS-Bench-201) with a fixed search space, which provides a unified benchmark for almost any up-to-date NAS algorithms.
+The design of our search space is inspired by that used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph.
+Each edge here is associated with an operation selected from a predefined operation set.
+For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes 4 nodes and 5 associated operation options, which generates 15,625 neural cell candidates in total.
+
+In this Markdown file, we provide:
+- [How to Use NAS-Bench-201](#how-to-use-nas-bench-201)
+
+For the following two things, please use [AutoDL-Projects](https://github.com/D-X-Y/AutoDL-Projects):
+- [Instruction to re-generate NAS-Bench-201](#instruction-to-re-generate-nas-bench-201)
+- [10 NAS algorithms evaluated in our paper](#to-reproduce-10-baseline-nas-algorithms-in-nas-bench-201)
+
+Note: please use `PyTorch >= 1.2.0` and `Python >= 3.6.0`.
+
+You can simply type `pip install nas-bench-201` to install our api. Please see source codes of `nas-bench-201` module in [this repo](https://github.com/D-X-Y/NAS-Bench-201).
+
+**If you have any questions or issues, please post it at [here](https://github.com/D-X-Y/AutoDL-Projects/issues) or email me.**
+
+### Preparation and Download
+
+[deprecated] The **old** benchmark file of NAS-Bench-201 can be downloaded from [Google Drive](https://drive.google.com/file/d/1SKW0Cu0u8-gb18zDpaAGi0f74UdXeGKs/view?usp=sharing) or [Baidu-Wangpan (code:6u5d)](https://pan.baidu.com/s/1CiaNH6C12zuZf7q-Ilm09w).
+
+[recommended] The **latest** benchmark file of NAS-Bench-201 (`NAS-Bench-201-v1_1-096897.pth`) can be downloaded from [Google Drive](https://drive.google.com/file/d/16Y0UwGisiouVRxW-W5hEtbxmcHw_0hF_/view?usp=sharing). The files for model weight are too large (431G) and I need some time to upload it. Please be patient, thanks for your understanding.
+
+You can move it to anywhere you want and send its path to our API for initialization.
+- [2020.02.25] APIv1.0/FILEv1.0: [`NAS-Bench-201-v1_0-e61699.pth`](https://drive.google.com/open?id=1SKW0Cu0u8-gb18zDpaAGi0f74UdXeGKs) (2.2G), where `e61699` is the last six digits for this file. It contains all information except for the trained weights of each trial.
+- [2020.02.25] APIv1.0/FILEv1.0: The full data of each architecture can be download from [
+NAS-BENCH-201-4-v1.0-archive.tar](https://drive.google.com/open?id=1X2i-JXaElsnVLuGgM4tP-yNwtsspXgdQ) (about 226GB). This compressed folder has 15625 files containing the trained weights.
+- [2020.02.25] APIv1.0/FILEv1.0: Checkpoints for 3 runs of each baseline NAS algorithm are provided in [Google Drive](https://drive.google.com/open?id=1eAgLZQAViP3r6dA0_ZOOGG9zPLXhGwXi).
+- [2020.03.09] APIv1.2/FILEv1.0: More robust API with more functions and descriptions
+- [2020.03.16] APIv1.3/FILEv1.1: [`NAS-Bench-201-v1_1-096897.pth`](https://drive.google.com/open?id=16Y0UwGisiouVRxW-W5hEtbxmcHw_0hF_) (4.7G), where `096897` is the last six digits for this file. It contains information of more trials compared to `NAS-Bench-201-v1_0-e61699.pth`, especially all models trained by 12 epochs on all datasets are avaliable.
+- [2020.06.30] APIv2.0: Use abstract class (NASBenchMetaAPI) for APIs of NAS-Bench-x0y.
+- [2020.06.30] FILEv2.0: coming soon!
+
+**We recommend to use `NAS-Bench-201-v1_1-096897.pth`**
+
+
+The training and evaluation data used in NAS-Bench-201 can be downloaded from [Google Drive](https://drive.google.com/open?id=1L0Lzq8rWpZLPfiQGd6QR8q5xLV88emU7) or [Baidu-Wangpan (code:4fg7)](https://pan.baidu.com/s/1XAzavPKq3zcat1yBA1L2tQ).
+It is recommended to put these data into `$TORCH_HOME` (`~/.torch/` by default). If you want to generate NAS-Bench-201 or similar NAS datasets or training models by yourself, you need these data.
+
+## How to Use NAS-Bench-201
+
+**More usage can be found in [our test codes](https://github.com/D-X-Y/AutoDL-Projects/blob/main/exps/NAS-Bench-201/test-nas-api.py)**.
+
+1. Creating an API instance from a file:
+```
+from nas_201_api import NASBench201API as API
+api = API('$path_to_meta_nas_bench_file')
+# Create an API without the verbose log
+api = API('NAS-Bench-201-v1_1-096897.pth', verbose=False)
+# The default path for benchmark file is '{:}/{:}'.format(os.environ['TORCH_HOME'], 'NAS-Bench-201-v1_1-096897.pth')
+api = API(None)
+```
+
+2. Show the number of architectures `len(api)` and each architecture `api[i]`:
+```
+num = len(api)
+for i, arch_str in enumerate(api):
+ print ('{:5d}/{:5d} : {:}'.format(i, len(api), arch_str))
+```
+
+3. Show the results of all trials for a single architecture:
+```
+# show all information for a specific architecture
+api.show(1)
+api.show(2)
+
+# show the mean loss and accuracy of an architecture
+info = api.query_meta_info_by_index(1) # This is an instance of `ArchResults`
+res_metrics = info.get_metrics('cifar10', 'train') # This is a dict with metric names as keys
+cost_metrics = info.get_compute_costs('cifar100') # This is a dict with metric names as keys, e.g., flops, params, latency
+
+# get the detailed information
+results = api.query_by_index(1, 'cifar100') # a dict of all trials for 1st net on cifar100, where the key is the seed
+print ('There are {:} trials for this architecture [{:}] on cifar100'.format(len(results), api[1]))
+for seed, result in results.items():
+ print ('Latency : {:}'.format(result.get_latency()))
+ print ('Train Info : {:}'.format(result.get_train()))
+ print ('Valid Info : {:}'.format(result.get_eval('x-valid')))
+ print ('Test Info : {:}'.format(result.get_eval('x-test')))
+ # for the metric after a specific epoch
+ print ('Train Info [10-th epoch] : {:}'.format(result.get_train(10)))
+```
+
+4. Query the index of an architecture by string
+```
+index = api.query_index_by_arch('|nor_conv_3x3~0|+|nor_conv_3x3~0|avg_pool_3x3~1|+|skip_connect~0|nor_conv_3x3~1|skip_connect~2|')
+api.show(index)
+```
+This string `|nor_conv_3x3~0|+|nor_conv_3x3~0|avg_pool_3x3~1|+|skip_connect~0|nor_conv_3x3~1|skip_connect~2|` means:
+```
+node-0: the input tensor
+node-1: conv-3x3( node-0 )
+node-2: conv-3x3( node-0 ) + avg-pool-3x3( node-1 )
+node-3: skip-connect( node-0 ) + conv-3x3( node-1 ) + skip-connect( node-2 )
+```
+
+5. Create the network from api:
+```
+config = api.get_net_config(123, 'cifar10') # obtain the network configuration for the 123-th architecture on the CIFAR-10 dataset
+from models import get_cell_based_tiny_net # this module is in AutoDL-Projects/lib/models
+network = get_cell_based_tiny_net(config) # create the network from configurration
+print(network) # show the structure of this architecture
+```
+If you want to load the trained weights of this created network, you need to use `api.get_net_param(123, ...)` to obtain the weights and then load it to the network.
+
+6. `api.get_more_info(...)` can return the loss / accuracy / time on training / validation / test sets, which is very helpful. For more details, please look at the comments in the get_more_info function.
+
+7. For other usages, please see `lib/nas_201_api/api.py`. We provide some usage information in the comments for the corresponding functions. If what you want is not provided, please feel free to open an issue for discussion, and I am happy to answer any questions regarding NAS-Bench-201.
+
+
+### Detailed Instruction
+
+In `nas_201_api`, we define three classes: `NASBench201API`, `ArchResults`, `ResultsCount`.
+
+`ResultsCount` maintains all information of a specific trial. One can instantiate ResultsCount and get the info via the following codes (`000157-FULL.pth` saves all information of all trials of 157-th architecture):
+```
+from nas_201_api import ResultsCount
+xdata = torch.load('000157-FULL.pth')
+odata = xdata['full']['all_results'][('cifar10-valid', 777)]
+result = ResultsCount.create_from_state_dict( odata )
+print(result) # print it
+print(result.get_train()) # print the final training loss/accuracy/[optional:time-cost-of-a-training-epoch]
+print(result.get_train(11)) # print the training info of the 11-th epoch
+print(result.get_eval('x-valid')) # print the final evaluation info on the validation set
+print(result.get_eval('x-valid', 11)) # print the info on the validation set of the 11-th epoch
+print(result.get_latency()) # print the evaluation latency [in batch]
+result.get_net_param() # the trained parameters of this trial
+arch_config = result.get_config(CellStructure.str2structure) # create the network with params
+net_config = dict2config(arch_config, None)
+network = get_cell_based_tiny_net(net_config)
+network.load_state_dict(result.get_net_param())
+```
+
+`ArchResults` maintains all information of all trials of an architecture. Please see the following usages:
+```
+from nas_201_api import ArchResults
+xdata = torch.load('000157-FULL.pth')
+archRes = ArchResults.create_from_state_dict(xdata['less']) # load trials trained with 12 epochs
+archRes = ArchResults.create_from_state_dict(xdata['full']) # load trials trained with 200 epochs
+
+print(archRes.arch_idx_str()) # print the index of this architecture
+print(archRes.get_dataset_names()) # print the supported training data
+print(archRes.get_comput_costs('cifar10-valid')) # print all computational info when training on cifar10-valid
+print(archRes.get_metrics('cifar10-valid', 'x-valid', None, False)) # print the average loss/accuracy/time on all trials
+print(archRes.get_metrics('cifar10-valid', 'x-valid', None, True)) # print loss/accuracy/time of a randomly selected trial
+```
+
+`NASBench201API` is the topest level api. Please see the following usages:
+```
+from nas_201_api import NASBench201API as API
+api = API('NAS-Bench-201-v1_1-096897.pth') # This will load all the information of NAS-Bench-201 except the trained weights
+api = API('{:}/{:}'.format(os.environ['TORCH_HOME'], 'NAS-Bench-201-v1_1-096897.pth')) # The same as the above line while I usually save NAS-Bench-201-v1_1-096897.pth in ~/.torch/.
+api.show(-1) # show info of all architectures
+api.reload('{:}/{:}'.format(os.environ['TORCH_HOME'], 'NAS-BENCH-201-4-v1.0-archive'), 3) # This code will reload the information 3-th architecture with the trained weights
+
+weights = api.get_net_param(3, 'cifar10', None) # Obtaining the weights of all trials for the 3-th architecture on cifar10. It will returns a dict, where the key is the seed and the value is the trained weights.
+```
+
+To obtain the training and evaluation information (please see the comments [here](https://github.com/D-X-Y/AutoDL-Projects/blob/main/lib/nas_201_api/api_201.py#L142)):
+```
+api.get_more_info(112, 'cifar10', None, hp='200', is_random=True)
+# Query info of last training epoch for 112-th architecture
+# using 200-epoch-hyper-parameter and randomly select a trial.
+api.get_more_info(112, 'ImageNet16-120', None, hp='200', is_random=True)
+```
+
+# Citation
+
+If you find that NAS-Bench-201 helps your research, please consider citing it:
+```
+@inproceedings{dong2020nasbench201,
+ title = {{NAS-Bench-201}: Extending the Scope of Reproducible Neural Architecture Search},
+ author = {Dong, Xuanyi and Yang, Yi},
+ booktitle = {International Conference on Learning Representations (ICLR)},
+ url = {https://openreview.net/forum?id=HJxyZkBKDr},
+ year = {2020}
+}
+```
diff --git a/AutoDL-Projects/docs/NAS-Bench-201.md b/AutoDL-Projects/docs/NAS-Bench-201.md
new file mode 100644
index 0000000..4fcb7f5
--- /dev/null
+++ b/AutoDL-Projects/docs/NAS-Bench-201.md
@@ -0,0 +1,254 @@
+# [NAS-BENCH-201: Extending the Scope of Reproducible Neural Architecture Search](https://openreview.net/forum?id=HJxyZkBKDr)
+
+**Since our NAS-BENCH-201 has been extended to NATS-Bench, this README is deprecated and not maintained. Please use [NATS-Bench](https://github.com/D-X-Y/AutoDL-Projects/blob/main/docs/NATS-Bench.md), which has 5x more architecture information and faster API than NAS-BENCH-201.**
+
+We propose an algorithm-agnostic NAS benchmark (NAS-Bench-201) with a fixed search space, which provides a unified benchmark for almost any up-to-date NAS algorithms.
+The design of our search space is inspired by that used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph.
+Each edge here is associated with an operation selected from a predefined operation set.
+For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes 4 nodes and 5 associated operation options, which generates 15,625 neural cell candidates in total.
+
+In this Markdown file, we provide:
+- [How to Use NAS-Bench-201](#how-to-use-nas-bench-201)
+- [Instruction to re-generate NAS-Bench-201](#instruction-to-re-generate-nas-bench-201)
+- [10 NAS algorithms evaluated in our paper](#to-reproduce-10-baseline-nas-algorithms-in-nas-bench-201)
+
+Note: please use `PyTorch >= 1.2.0` and `Python >= 3.6.0`.
+
+You can simply type `pip install nas-bench-201` to install our api. Please see source codes of `nas-bench-201` module in [this repo](https://github.com/D-X-Y/NAS-Bench-201).
+
+**If you have any questions or issues, please post it at [here](https://github.com/D-X-Y/AutoDL-Projects/issues) or email me.**
+
+### Preparation and Download
+
+[deprecated] The **old** benchmark file of NAS-Bench-201 can be downloaded from [Google Drive](https://drive.google.com/file/d/1SKW0Cu0u8-gb18zDpaAGi0f74UdXeGKs/view?usp=sharing) or [Baidu-Wangpan (code:6u5d)](https://pan.baidu.com/s/1CiaNH6C12zuZf7q-Ilm09w).
+
+[recommended] The **latest** benchmark file of NAS-Bench-201 (`NAS-Bench-201-v1_1-096897.pth`) can be downloaded from [Google Drive](https://drive.google.com/file/d/16Y0UwGisiouVRxW-W5hEtbxmcHw_0hF_/view?usp=sharing). The files for model weight are too large (431G) and I need some time to upload it. Please be patient, thanks for your understanding.
+
+You can move it to anywhere you want and send its path to our API for initialization.
+- [2020.02.25] APIv1.0/FILEv1.0: [`NAS-Bench-201-v1_0-e61699.pth`](https://drive.google.com/open?id=1SKW0Cu0u8-gb18zDpaAGi0f74UdXeGKs) (2.2G), where `e61699` is the last six digits for this file. It contains all information except for the trained weights of each trial.
+- [2020.02.25] APIv1.0/FILEv1.0: The full data of each architecture can be download from [
+NAS-BENCH-201-4-v1.0-archive.tar](https://drive.google.com/open?id=1X2i-JXaElsnVLuGgM4tP-yNwtsspXgdQ) (about 226GB). This compressed folder has 15625 files containing the trained weights.
+- [2020.02.25] APIv1.0/FILEv1.0: Checkpoints for 3 runs of each baseline NAS algorithm are provided in [Google Drive](https://drive.google.com/open?id=1eAgLZQAViP3r6dA0_ZOOGG9zPLXhGwXi).
+- [2020.03.09] APIv1.2/FILEv1.0: More robust API with more functions and descriptions
+- [2020.03.16] APIv1.3/FILEv1.1: [`NAS-Bench-201-v1_1-096897.pth`](https://drive.google.com/open?id=16Y0UwGisiouVRxW-W5hEtbxmcHw_0hF_) (4.7G), where `096897` is the last six digits for this file. It contains information of more trials compared to `NAS-Bench-201-v1_0-e61699.pth`, especially all models trained by 12 epochs on all datasets are avaliable.
+- [2020.06.30] APIv2.0: Use abstract class (NASBenchMetaAPI) for APIs of NAS-Bench-x0y.
+- [2020.06.30] FILEv2.0: coming soon!
+
+**We recommend to use `NAS-Bench-201-v1_1-096897.pth`**
+
+
+The training and evaluation data used in NAS-Bench-201 can be downloaded from [Google Drive](https://drive.google.com/open?id=1L0Lzq8rWpZLPfiQGd6QR8q5xLV88emU7) or [Baidu-Wangpan (code:4fg7)](https://pan.baidu.com/s/1XAzavPKq3zcat1yBA1L2tQ).
+It is recommended to put these data into `$TORCH_HOME` (`~/.torch/` by default). If you want to generate NAS-Bench-201 or similar NAS datasets or training models by yourself, you need these data.
+
+## How to Use NAS-Bench-201
+
+**More usage can be found in [our test codes](https://github.com/D-X-Y/AutoDL-Projects/blob/main/exps/NAS-Bench-201/test-nas-api.py)**.
+
+1. Creating an API instance from a file:
+```
+from nas_201_api import NASBench201API as API
+api = API('$path_to_meta_nas_bench_file')
+# Create an API without the verbose log
+api = API('NAS-Bench-201-v1_1-096897.pth', verbose=False)
+# The default path for benchmark file is '{:}/{:}'.format(os.environ['TORCH_HOME'], 'NAS-Bench-201-v1_1-096897.pth')
+api = API(None)
+```
+
+2. Show the number of architectures `len(api)` and each architecture `api[i]`:
+```
+num = len(api)
+for i, arch_str in enumerate(api):
+ print ('{:5d}/{:5d} : {:}'.format(i, len(api), arch_str))
+```
+
+3. Show the results of all trials for a single architecture:
+```
+# show all information for a specific architecture
+api.show(1)
+api.show(2)
+
+# show the mean loss and accuracy of an architecture
+info = api.query_meta_info_by_index(1) # This is an instance of `ArchResults`
+res_metrics = info.get_metrics('cifar10', 'train') # This is a dict with metric names as keys
+cost_metrics = info.get_compute_costs('cifar100') # This is a dict with metric names as keys, e.g., flops, params, latency
+
+# get the detailed information
+results = api.query_by_index(1, 'cifar100') # a dict of all trials for 1st net on cifar100, where the key is the seed
+print ('There are {:} trials for this architecture [{:}] on cifar100'.format(len(results), api[1]))
+for seed, result in results.items():
+ print ('Latency : {:}'.format(result.get_latency()))
+ print ('Train Info : {:}'.format(result.get_train()))
+ print ('Valid Info : {:}'.format(result.get_eval('x-valid')))
+ print ('Test Info : {:}'.format(result.get_eval('x-test')))
+ # for the metric after a specific epoch
+ print ('Train Info [10-th epoch] : {:}'.format(result.get_train(10)))
+```
+
+4. Query the index of an architecture by string
+```
+index = api.query_index_by_arch('|nor_conv_3x3~0|+|nor_conv_3x3~0|avg_pool_3x3~1|+|skip_connect~0|nor_conv_3x3~1|skip_connect~2|')
+api.show(index)
+```
+This string `|nor_conv_3x3~0|+|nor_conv_3x3~0|avg_pool_3x3~1|+|skip_connect~0|nor_conv_3x3~1|skip_connect~2|` means:
+```
+node-0: the input tensor
+node-1: conv-3x3( node-0 )
+node-2: conv-3x3( node-0 ) + avg-pool-3x3( node-1 )
+node-3: skip-connect( node-0 ) + conv-3x3( node-1 ) + skip-connect( node-2 )
+```
+
+5. Create the network from api:
+```
+config = api.get_net_config(123, 'cifar10') # obtain the network configuration for the 123-th architecture on the CIFAR-10 dataset
+from models import get_cell_based_tiny_net # this module is in AutoDL-Projects/lib/models
+network = get_cell_based_tiny_net(config) # create the network from configurration
+print(network) # show the structure of this architecture
+```
+If you want to load the trained weights of this created network, you need to use `api.get_net_param(123, ...)` to obtain the weights and then load it to the network.
+
+6. `api.get_more_info(...)` can return the loss / accuracy / time on training / validation / test sets, which is very helpful. For more details, please look at the comments in the get_more_info function.
+
+7. For other usages, please see `lib/nas_201_api/api.py`. We provide some usage information in the comments for the corresponding functions. If what you want is not provided, please feel free to open an issue for discussion, and I am happy to answer any questions regarding NAS-Bench-201.
+
+
+### Detailed Instruction
+
+In `nas_201_api`, we define three classes: `NASBench201API`, `ArchResults`, `ResultsCount`.
+
+`ResultsCount` maintains all information of a specific trial. One can instantiate ResultsCount and get the info via the following codes (`000157-FULL.pth` saves all information of all trials of 157-th architecture):
+```
+from nas_201_api import ResultsCount
+xdata = torch.load('000157-FULL.pth')
+odata = xdata['full']['all_results'][('cifar10-valid', 777)]
+result = ResultsCount.create_from_state_dict( odata )
+print(result) # print it
+print(result.get_train()) # print the final training loss/accuracy/[optional:time-cost-of-a-training-epoch]
+print(result.get_train(11)) # print the training info of the 11-th epoch
+print(result.get_eval('x-valid')) # print the final evaluation info on the validation set
+print(result.get_eval('x-valid', 11)) # print the info on the validation set of the 11-th epoch
+print(result.get_latency()) # print the evaluation latency [in batch]
+result.get_net_param() # the trained parameters of this trial
+arch_config = result.get_config(CellStructure.str2structure) # create the network with params
+net_config = dict2config(arch_config, None)
+network = get_cell_based_tiny_net(net_config)
+network.load_state_dict(result.get_net_param())
+```
+
+`ArchResults` maintains all information of all trials of an architecture. Please see the following usages:
+```
+from nas_201_api import ArchResults
+xdata = torch.load('000157-FULL.pth')
+archRes = ArchResults.create_from_state_dict(xdata['less']) # load trials trained with 12 epochs
+archRes = ArchResults.create_from_state_dict(xdata['full']) # load trials trained with 200 epochs
+
+print(archRes.arch_idx_str()) # print the index of this architecture
+print(archRes.get_dataset_names()) # print the supported training data
+print(archRes.get_comput_costs('cifar10-valid')) # print all computational info when training on cifar10-valid
+print(archRes.get_metrics('cifar10-valid', 'x-valid', None, False)) # print the average loss/accuracy/time on all trials
+print(archRes.get_metrics('cifar10-valid', 'x-valid', None, True)) # print loss/accuracy/time of a randomly selected trial
+```
+
+`NASBench201API` is the topest level api. Please see the following usages:
+```
+from nas_201_api import NASBench201API as API
+api = API('NAS-Bench-201-v1_1-096897.pth') # This will load all the information of NAS-Bench-201 except the trained weights
+api = API('{:}/{:}'.format(os.environ['TORCH_HOME'], 'NAS-Bench-201-v1_1-096897.pth')) # The same as the above line while I usually save NAS-Bench-201-v1_1-096897.pth in ~/.torch/.
+api.show(-1) # show info of all architectures
+api.reload('{:}/{:}'.format(os.environ['TORCH_HOME'], 'NAS-BENCH-201-4-v1.0-archive'), 3) # This code will reload the information 3-th architecture with the trained weights
+
+weights = api.get_net_param(3, 'cifar10', None) # Obtaining the weights of all trials for the 3-th architecture on cifar10. It will returns a dict, where the key is the seed and the value is the trained weights.
+```
+
+To obtain the training and evaluation information (please see the comments [here](https://github.com/D-X-Y/AutoDL-Projects/blob/main/lib/nas_201_api/api_201.py#L142)):
+```
+api.get_more_info(112, 'cifar10', None, hp='200', is_random=True)
+# Query info of last training epoch for 112-th architecture
+# using 200-epoch-hyper-parameter and randomly select a trial.
+api.get_more_info(112, 'ImageNet16-120', None, hp='200', is_random=True)
+```
+
+Please use the following script to show the best architectures on each dataset:
+```show the best architecture
+python exps/NAS-Bench-201/show-best.py
+```
+
+
+## Instruction to Re-Generate NAS-Bench-201
+
+There are four steps to build NAS-Bench-201.
+
+1. generate the meta file for NAS-Bench-201 using the following script, where `NAS-BENCH-201` indicates the name and `4` indicates the maximum number of nodes in a cell.
+```
+bash scripts-search/NAS-Bench-201/meta-gen.sh NAS-BENCH-201 4
+```
+
+2. train earch architecture on a single GPU (see commands in `output/NAS-BENCH-201-4/BENCH-201-N4.opt-full.script`, which is automatically generated by step-1).
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201/train-models.sh 0 0 389 -1 '777 888 999'
+```
+This command will train 390 architectures (id from 0 to 389) using the following four kinds of splits with three random seeds (777, 888, 999).
+
+| Dataset | Train | Eval |
+|:---------------:|:-------------:|:------------:|
+| CIFAR-10 | train | valid / test |
+| CIFAR-10 | train + valid | test |
+| CIFAR-100 | train | valid / test |
+| ImageNet-16-120 | train | valid / test |
+
+Note that the above `train`, `valid`, and `test` indicate the proposed splits in our NAS-Bench-201, and they might be different with the original splits.
+
+3. calculate the latency, merge the results of all architectures, and simplify the results.
+(see commands in `output/NAS-BENCH-201-4/meta-node-4.cal-script.txt` which is automatically generated by step-1).
+```
+OMP_NUM_THREADS=6 CUDA_VISIBLE_DEVICES=0 python exps/NAS-Bench-201/statistics.py --mode cal --target_dir 000000-000389-C16-N5
+```
+
+4. merge all results into a single file for NAS-Bench-201-API.
+```
+OMP_NUM_THREADS=4 python exps/NAS-Bench-201/statistics.py --mode merge
+```
+This command will generate a single file `output/NAS-BENCH-201-4/simplifies/C16-N5-final-infos.pth` contains all the data for NAS-Bench-201.
+This generated file will serve as the input for our NAS-Bench-201 API.
+
+[option] train a single architecture on a single GPU.
+```
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201/train-a-net.sh resnet 16 5
+CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201/train-a-net.sh '|nor_conv_3x3~0|+|nor_conv_3x3~0|nor_conv_3x3~1|+|skip_connect~0|skip_connect~1|skip_connect~2|' 16 5
+```
+
+
+## To Reproduce 10 Baseline NAS Algorithms in NAS-Bench-201
+
+We have tried our best to implement each method. However, still, some algorithms might obtain non-optimal results since their hyper-parameters might not fit our NAS-Bench-201.
+If researchers can provide better results with different hyper-parameters, we are happy to update results according to the new experimental results. We also welcome more NAS algorithms to test on our dataset and would include them accordingly.
+
+**Note that** you need to prepare the training and test data as described in [Preparation and Download](#preparation-and-download)
+
+- [1] `CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201-algos/DARTS-V1.sh cifar10 1 -1`, where `cifar10` can be replaced with `cifar100` or `ImageNet16-120`.
+- [2] `CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201-algos/DARTS-V2.sh cifar10 1 -1`
+- [3] `CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201-algos/GDAS.sh cifar10 1 -1`
+- [4] `CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201-algos/SETN.sh cifar10 1 -1`
+- [5] `CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201-algos/ENAS.sh cifar10 1 -1`
+- [6] `CUDA_VISIBLE_DEVICES=0 bash ./scripts-search/NAS-Bench-201-algos/RANDOM-NAS.sh cifar10 1 -1`
+- [7] `bash ./scripts-search/NAS-Bench-201-algos/R-EA.sh cifar10 3 -1`
+- [8] `bash ./scripts-search/NAS-Bench-201-algos/Random.sh cifar10 -1`
+- [9] `bash ./scripts-search/NAS-Bench-201-algos/REINFORCE.sh cifar10 0.5 -1`
+- [10] `bash ./scripts-search/NAS-Bench-201-algos/BOHB.sh cifar10 -1`
+
+In commands [1-6], the first args `cifar10` indicates the dataset name, the second args `1` indicates the behavior of BN, and the first args `-1` indicates the random seed.
+
+**Note that** since 2020 March 16, in these scripts, the default NAS-Bench-201 benchmark file has changed from `NAS-Bench-201-v1_0-e61699.pth` to `NAS-Bench-201-v1_1-096897.pth`, and thus the results could be slightly different from the original paper.
+
+
+# Citation
+
+If you find that NAS-Bench-201 helps your research, please consider citing it:
+```
+@inproceedings{dong2020nasbench201,
+ title = {{NAS-Bench-201}: Extending the Scope of Reproducible Neural Architecture Search},
+ author = {Dong, Xuanyi and Yang, Yi},
+ booktitle = {International Conference on Learning Representations (ICLR)},
+ url = {https://openreview.net/forum?id=HJxyZkBKDr},
+ year = {2020}
+}
+```
diff --git a/AutoDL-Projects/docs/NeurIPS-2019-TAS.md b/AutoDL-Projects/docs/NeurIPS-2019-TAS.md
new file mode 100644
index 0000000..fb87976
--- /dev/null
+++ b/AutoDL-Projects/docs/NeurIPS-2019-TAS.md
@@ -0,0 +1,71 @@
+# [Network Pruning via Transformable Architecture Search](https://arxiv.org/abs/1905.09717)
+
+[](https://paperswithcode.com/sota/network-pruning-on-cifar-100?p=network-pruning-via-transformable)
+
+Network Pruning via Transformable Architecture Search is accepted by NeurIPS 2019.
+In this paper, we proposed a differentiable searching strategy for transformable architectures, i.e., searching for the depth and width of a deep neural network.
+You could see the highlight of our Transformable Architecture Search (TAS) at our [project page](https://xuanyidong.com/assets/projects/NeurIPS-2019-TAS.html).
+
+
+
+
+
+
+
| Type | +ABBRV | +Algorithms | +Description | +
|---|---|---|---|
| NAS | +TAS | +Network Pruning via Transformable Architecture Search | +NeurIPS-2019-TAS.md | +
| DARTS | +DARTS: Differentiable Architecture Search | +ICLR-2019-DARTS.md | +|
| GDAS | +Searching for A Robust Neural Architecture in Four GPU Hours | +CVPR-2019-GDAS.md | +|
| SETN | +One-Shot Neural Architecture Search via Self-Evaluated Template Network | +ICCV-2019-SETN.md | +|
| NAS-Bench-201 | +NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search | +NAS-Bench-201.md | +|
| NATS-Bench | +NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size | +NATS-Bench.md | +|
| ... | +ENAS / REA / REINFORCE / BOHB | +Please check the original papers. | +NAS-Bench-201.md NATS-Bench.md | +
| HPO | +HPO-CG | +Hyperparameter optimization with approximate gradient | +coming soon | +
| Basic | +ResNet | +Deep Learning-based Image Classification | +BASELINE.md | +