Since the development of the first real deep neural network AlexNet in 2012, deep learning has made great progress in computer vision and natural language processing. Lots of these breakthroughs often come alone with the new architecture design of deep neural networks. We are interested in pushing the boundary of deep learning performance by advancing the design of neural architectures, while taking the trade-off between the accuracy of the network and its computation cost into consideration.