JuliaCon 2018

Abstract
Deep neural networks can be highly useful. Nonetheless, some problems require imperative programming elements such as variables, conditional execution, and loop clauses. We present Nili.jl, a Julia package which combines the logic of imperative programming with the adaptivity of machine learning, together producing a differential programming framework adapted to the age of AI.
Description
Deep neural networks have been tremendously successful in approaching a variety of learning problems. Nonetheless, the basic building blocks that make up such models are mostly designed to facilitate gradient based parameter learning, rather than to enable productive programming, or to provide a fully fledged programming language. Imperative programming languages on the other hand provide programming elements such as variables, conditional execution clauses, and definitions of subroutines that facilitate succinct descriptions of programs, but lack the flexibility that gradient based learning allows. We set on combining the best of both worlds in a new Julia framework that on the one hand supports imperative programming elements, and on the other, allows for gradient based parameter learning. The main challenge in combining the two is to provide an efficient algorithm for calculating the gradient when variables, and loops are included. In such cases, the complexity of calculating the gradient using vanilla backpropagation grows exponentially with the number of iterations and the number of variables. To circumvent this problem, we provide new auto-differentiation mechanisms that can handle such cases in linear time, and a new framework that supports them. We supplement our framework with operations that are typical of neural networks, paving the way to new kinds of differentiable programs. In particular, we show how combining the logic of imperative programming with the adaptivity of machine learning enables us to solve image classification tasks that standard neural networks fail to solve.