Course Description

This course covers the principles and techniques of differential programming, which are becoming increasingly important in fields such as machine learning, computer vision, graphics, physics simulation, and scientific computing. Dif- ferentiation and optimization are key ingredients in the success of deep learn- ing models, which are used in many applications such as image and speech recognition, natural language processing, and autonomous driving. Backprop- agation and automatic differentiation are essential techniques for computing gradients in deep learning models, allowing us to train them efficiently on large datasets. Differentiable physics and rendering are emerging areas that enable us to simulate physical and visual phenomena using deep learning tech- niques. The ability to solve partial differential equations (PDEs) using neural networks is a promising area of research that has the potential to transform scientific computing by enabling us to model complex physical systems more efficiently. In addition, differentiable convex optimization provides a power- ful tool for solving optimization problems that arise in many areas of science and engineering. By mastering the principles and techniques of differential programming, students will be well-equipped to tackle a variety of real-world problems and make significant contributions to the fields of machine learning, computer graphics, physics simulation, and scientific computing. Students will learn how to apply these techniques to a range of problems, including computer vision, graphics, physics simulation, and scientific comput- ing. They will develop an understanding of the theory behind these techniques as well as practical skills in implementing and using them in software.

Assignments

experience. By the end of the course, students will be able to design and implement differentiable programs and use them to solve a variety of problems in different domains.

Learning Goals

Upon successful completion of this course, the students should be able to:

  • Define differential programming and its relevance to machine learning, computer vision, graphics, physics simulation, and scientific computing.
  • Explain differentiation and backpropagation and their applications in training deep learning models.
  • Apply differential programming to solve problems in differentiable physics and rendering.
  • Define neural rendering and describe the neural rendering pipeline, in-
  • Apply neural networks to solve partial differential equations (PDEs).
  • Define convex optimization and explain how it benefits from differentia- bility.
  • Read and review scientific papers. final grade. Homework Exercises Four homework exercises within the course contribute to the final grade. They are intended to prepare you for the project work. The first two exercises introduce you to the frameworks and play with different datasets and deep architectures. These frameworks are used in the following two homework exercises and for the final project. The exercises are designed as high-level tasks. They are intended to train independent thinking, which is needed for solving research and industry problems. The main source of information can be obtained through online deep learning tutorials. For additional support, use the office hours for individual assistance. The exercises must be solved in pairs using course material and online sources. Plagiarism will be sanctioned following the university policy. Deadlines and grading:
  • Homework #1 is due on Week 3
  • Homework #2 is due on Week 4
  • Homework #3 is due on Week 6 Late submissions are accepted but penalized -25% per day. Detailed grading Research Papers The second half of the course will be organized as a reading group, focus- ing on related latest research papers. During each class, two papers will be presented and discussed. Every student is expected to read one paper and write a review. Everybody needs to at least skim through both papers. After each paper presentation, we will have an in-depth discussion on paper details should be considered for the discussion. The review must include: (1) a short contribution summary, (2) one question for discussion, (3) one strength, and (4) one weakness. The review should be limited to 400 words and should be self-contained. Every student will present one research paper. The presentation will be limited to a predefined time (e.g., 15 min). After the presentation, there will be a predefined time (e.g., 15 min) for a discussion led by one other student. Every student will moderate the discussion of one other paper. Deadlines and grading:
  • Review must be submitted on the evening before the class. Reviews will be graded according to the above-mentioned 4 points.
  • Presentation must be submitted 3 days before the presentation and dis- cussed with the instructor before the presentation day.
  • The presentation will be graded based on content quality, structure, and how well it is put into context in accordance with the related work. Late submissions are not accepted. Project The main task of the course is the research project. The project goal is to identify, extend and evaluate an existing approach, combine several existing approaches for solving a novel problem, or develop a completely novel approach for solving a selected problem. We will present some topics and open problems in class, but students should try to think about their own ideas using their or three. This will help you develop your teamwork skills, make use of your complementary skills and allow you to work on more complex topics. Each individual must have explicit responsibilities stated in the report. All team members must join forces for the following:
  • Project proposal in the form of a 3– to 5–minute pitch presentation.
  • Literature overview, which can start with papers discussed in the class similar to the project topic. You must search for additional papers that cite and are cited by them using Google Scholar or equivalent tools. You will use this review in preparation for the related work section of the final report.
  • Development should make use of the frameworks used in homework as- signments and other online sources. Do not recreate already existing tools if they are available. The code must be submitted alongside the report in a git repository with appropriate documentation for their use. The use of existing work must be clearly indicated and stated in the final report.
  • Evaluation must include an experiment design that validates the cor- rectness of the contribution. It should also try to show the qualitative and quantitative improvements in existing solutions.
  • The report must follow the structure of a research paper and should be 8–10 pages long (including the references) in ToG (Transactions on Graphics) format. Start writing early on and try to complete individual sections when you are done with the corresponding work (e.g., write Related work during the literature review, write the Methodology section once you have finished developing your system, write the evaluation after you have evaluated it, etc.).
  • Supplemental material should be submitted in the form of a short video (max. 3 min) demonstrating the project idea and the results. Deadlines and grading:
  • Project proposal is due on Week 5
  • Related work overview is due on Week 6
  • Code, presentation and final report are due on Week 14 Late submissions are accepted but penalized -25% per day. Detailed grading is explained in the final report template. assessments
  • Homework exercises (30%)
  • Homework #1 (5%)
  • Homework #2 (10%)
  • Homework #3 (10%)
  • Research presentations (30%)
  • Reviewing papers (10%)
  • Presenting a selected paper (20%)
  • Project (45%)
  • Proposal
  • Code and evaluation
  • Report
  • Presentation

Course Policies

homework will get a penalty as defined for each assignment.

Learning Material

Automatic Differentiation

  • Wengert, R. E. (1964). A simple automatic derivative evaluation program. Communications of the ACM, 7(8), 463–464.
  • A. Radul, Introduction to Automatic Differentiation , 2013. [Online]. A vailable:https://alexey.radul.name/ideas/2013/introduction- to-automatic-differentiation/.
  • Hogan, R. J. (2014). Fast reverse-mode automatic differentiation using expres- sion templates in c++. ACM Transactions on Mathematical Software (TOMS), 40(4), 1–16.
  • C. H. Bischof, M. R. Haghighat, et al., “Hierarchical approaches to auto- matic differentiation,” Computational Differentiation: Techniques, Ap- plications, and Tools , pp. 83–94, 1996.
  • S. P. Jones, Automatic differentiation for dummies . [Online]. A vailable: https : / / 2019 . ecoop . org / details / ecoop - 2019 - papers / 11 / Automatic-Differentiation-for-Dummies .
  • D. Oktay, N. McGreivy, J. Aduol, A. Beatson, and R. P. Adams, Ran- domized automatic differentiation , 2021. arXiv: 2007.10412 [cs.LG] .
  • J. M. Siskind, “Automatic differentiation: Inverse accumulation mode,” in Program Transformations for ML Workshop at NeurIPS 2019 .

Backpropagation

  • T. Vieira, Backprop is not just the chain rule . [Online]. A vailable:https: //timvieira.github.io/blog/post/2017/08/18/backprop-is-not- just-the-chain-rule/ .
  • W. Wang, Z. Dang, Y. Hu, P. Fua, and M. Salzmann, Backpropagation- friendly eigendecomposition, 2019. arXiv: 1906.09023 [cs.LG] .

Differentiable Optimization

  • A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and Z. Kolter, Differentiable convex optimization layers, 2019. arXiv: 1910.12430 [cs.LG].
  • B. Amos and J. Z. Kolter, Optnet: Differentiable optimization as a layer in neural networks , 2021. arXiv: 1703.00443 [cs.LG] .
  • D. Maclaurin, D. Duvenaud, and R. Adams, “Gradient-based hyper- parameter optimization through reversible learning,” in Proceedings of the 32nd International Conference on Machine Learning , F. Bach and D. Blei, Eds., ser. Proceedings of Machine Learning Research, vol. 37,
  • Lille, France: PMLR, 2015, pp. 2113–2122. [Online]. A vailable: https: //proceedings.mlr.press/v37/maclaurin15.html.
  • Y. Li, A. Božič, T. Zhang, Y. Ji, T. Harada, and M. Nießner, “Learning to optimize non-rigid tracking,” in Proc. Computer Vision and Pattern Recognition (CVPR), IEEE , 2020.
  • A. Mensch and M. Blondel, Differentiable dynamic programming for structured prediction and attention, 2018. arXiv: 1802.03676 [stat.ML].

Differentiable Programming

  • K. Farrahi and J. Hare, COMP6248 Differentiable Programming (and Deep Learning) , 2022. [Online]. A vailable: http : / / comp6248 . ecs . soton.ac.uk.
  • R. O’Connor, Differentiable programming - a simple introduction , 2022. [Online]. A vailable:https://www.assemblyai.com/blog/differentiable- programming-a-simple-introduction/ .
  • M. C. Lin, CMSC 838B / 498Z: Differentiable Programming , 2021. [Online]. A vailable: https : / / www . cs . umd . edu / class / fall2021 / cmsc838b/.
  • M. Slater, Differentiable programming from scratch, 2022. [Online]. A vail- able: https://thenumb.at/Autodiff/.

Differentiable Physics

  • Y. Hu, L. Anderson, T.-M. Li, et al., “Difftaichi: Differentiable program- ming for physical simulation,” ICLR, 2020.
  • Z. Huang, Y. Hu, T. Du, et al., “Plasticinelab: A soft-body manipulation benchmark with differentiable physics,” arXiv preprint arXiv:2104.03311, 2021.
  • A. McNamara, A. Treuille, Z. Popović, and J. Stam, “Fluid control using the adjoint method,” ACM Transactions On Graphics (TOG) , vol. 23, no. 3, pp. 449–456, 2004.
  • F. de A vila Belbute-Peres, K. Smith, K. Allen, J. Tenenbaum, and J. Z.

Kolter, “End-to-end differentiable physics for learning and control,” Ad-

  • vances in neural information processing systems , vol. 31, 2018.
  • J. Liang, M. Lin, and V. Koltun, “Differentiable cloth simulation for inverse problems,” in Advances in Neural Information Processing Sys- tems, vol. 32, Curran Associates, Inc., 2019. [Online]. A vailable: https: / / proceedings . neurips . cc / paper _ files / paper / 2019 / file / 28f0b864598a1291557bed248a998d4e-Paper.pdf.
  • Y.-L. Qiao, J. Liang, V. Koltun, and M. C. Lin, Scalable differentiable physics for learning and control , 2020. arXiv: 2007.02168 [cs.LG] .
  • X. Li, T.-K. L. Wong, R. T. Q. Chen, and D. Duvenaud, Scalable gra- dients for stochastic differential equations , 2020. arXiv: 2001 . 01328 [cs.LG].
  • K. Um, R. Brand, Y. R. Fei, P. Holl, and N. Thuerey, “Solver-in-the- loop: Learning from differentiable physics to interact with iterative pde- solvers,” Advances in Neural Information Processing Systems , vol. 33, pp. 6111–6122, 2020.
  • P. Holl, V. Koltun, and N. Thuerey, “Learning to control pdes with differentiable physics,” arXiv preprint arXiv:2001.07457 , 2020.
  • B. Ummenhofer, L. Prantl, N. Thuerey, and V. Koltun, “Lagrangian fluid simulation with continuous convolutions,” in International Con- ference on Learning Representations , 2020.
  • M. Geilinger, D. Hahn, J. Zehnder, M. Bächer, B. Thomaszewski, and S.
  • Coros, Add: Analytically differentiable dynamics for multi-body systems with frictional contact , 2020. arXiv: 2007.00987 [cs.GR] .
  • S. L. Brunton, B. R. Noack, and P. Koumoutsakos, “Machine learning for fluid mechanics,” Annual review of fluid mechanics , vol. 52, pp. 477– 508, 2020.
  • T. Dorigo, A. Giammanco, P. Vischia, et al. , Toward the end-to-end op- timization of particle physics instruments with differentiable program- ming: A white paper , 2022. [Online]. A vailable: https://arxiv.org/ abs/2203.13818.

Differentiable Rendering

  • A. Tewari, J. Thies, B. Mildenhall, et al. , Advances in neural rendering , 2022. [Online]. A vailable: https://www.neuralrender.com/.
  • S. Zhao, W. Jakob, and T.-M. Li, “Physics-based differentiable ren- dering: From theory to implementation,” in ACM SIGGRAPH 2020

Courses, ser. SIGGRAPH ’20, Virtual Event, USA: Association for Com-

  • puting Machinery, 2020, isbn: 9781450379724. doi: 10.1145/3388769. 3407454. [Online]. A vailable: https://doi.org/10.1145/3388769. 3407454.
  • Papers with Code - Neural Rendering . [Online]. A vailable: https : / / paperswithcode.com/task/neural-rendering.
  • Awesome Neural Rendering . [Online]. A vailable: https://github.com/ weihaox/awesome-neural-rendering.
  • T.-M. Li, M. Aittala, F. Durand, and J. Lehtinen, “Differentiable monte carlo ray tracing through edge sampling,” ACM Transactions on Graph- ics (TOG) , vol. 37, no. 6, pp. 1–11, 2018.
  • S. Bangaru, T.-M. Li, and F. Durand, “Unbiased warped-area sampling for differentiable rendering,” ACM Trans. Graph., vol. 39, no. 6, 245:1– 245:18, 2020.
  • T. Zeltner, S. Speierer, I. Georgiev, and W. Jakob, “Monte carlo estima- tors for differential light transport,” Transactions on Graphics (Proceed- ings of SIGGRAPH) , vol. 40, no. 4, Aug. 2021. doi: 10.1145/3450626. 3459807.
  • D. Vicini, S. Speierer, and W. Jakob, “Path replay backpropagation: Dif- ferentiating light paths using constant memory and linear time,” Trans- actions on Graphics (Proceedings of SIGGRAPH) , vol. 40, no. 4, 108:1– 108:14, Aug. 2021. doi: 10.1145/3450626.3459804.
  • C. Zhang, Z. Dong, M. Doggett, and S. Zhao, “Antithetic sampling for monte carlo differentiable rendering,” ACM Trans. Graph., vol. 40, no. 4, 77:1–77:12, 2021.
  • S. Laine, J. Hellsten, T. Karras, Y. Seol, J. Lehtinen, and T. Aila, Modu- lar primitives for high-performance differentiable rendering , 2020. arXiv: 2011.03277 [cs.GR] .

Differential Geometry processing

  • D. Smirnov and J. Solomon, “HodgeNet: Learning spectral geometry on triangle meshes,” ACM Transactions on Graphics (TOG) , vol. 40, no. 4, 166:1–166:11, 2021.
  • Y. Wang, V. Kim, M. Bronstein, and J. Solomon, “Learning geomet- ric operators on meshes,” in Representation Learning on Graphs and Manifolds 2019 (ICLR workshop) , 2019.
  • R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman, and D. Cohen-

Or, “Meshcnn: A network with an edge,” ACM Transactions on Graphics

  • (TOG), vol. 38, no. 4, pp. 1–12, 2019.
  • J. Huang, H. Zhang, L. Yi, T. Funkhouser, M. Nießner, and L. Guibas,

Texturenet: Consistent local parametrizations for learning from high-

  • resolution signals on meshes , 2019. arXiv: 1812.00020 [cs.CV] .
  • J. Schult, F. Engelmann, T. Kontogianni, and B. Leibe, Dualconvmesh- net: Joint geodesic and euclidean convolutions on 3d meshes , 2020. arXiv: 2004.01002 [cs.CV] .
  • A. Deva Prasad, A. Balu, H. Shah, S. Sarkar, C. Hegde, and A. Krishna- murthy, “Nurbs-diff: A differentiable programming module for nurbs,”

Computer-Aided Design, vol. 146, p. 103 199, 2022, issn: 0010-4485. doi:

  • https : / / doi . org / 10 . 1016 / j . cad . 2022 . 103199. [Online]. A vail- able: https : / / www . sciencedirect . com / science / article / pii / S0010448522000045.

Systems and Libraries

  • A. Paszke, S. Gross, F. Massa, et al. , Pytorch: An imperative style, high- performance deep learning library , 2019. arXiv: 1912.01703 [cs.LG] .
  • Nvidia kaolin . [Online]. A vailable: https://developer.nvidia.com/ kaolin.
  • W. Jakob, S. Speierer, N. Roussel, et al., Mitsuba 3 renderer, version 3.1.1, https://mitsuba-renderer.org, 2022.
  • Y. Hu, T.-M. Li, L. Anderson, J. Ragan-Kelley, and F. Durand, “Taichi:

A language for high-performance computation on spatially sparse data

  • structures,” ACM Transactions on Graphics (TOG), vol. 38, no. 6, pp. 1– 16, 2019.
  • Y. Hu, J. Liu, X. Yang, et al. , “Quantaichi: A compiler for quantized simulations,” ACM Transactions on Graphics (TOG) , vol. 40, no. 4, 2021.

Course Calendar

W eek 1: Lectures:

  • Overview and Introduction
  • Differentiation
  • Optimization W eek 2: Lectures:
  • Code Differentiation
  • Automatic Differentiation Labs:
  • Consultations on Homework #1 W eek 3: Lectures:
  • Solving Differential Equations Using Neural Networks
  • Ordinary Differential Equations
  • Partial Differential Equations
  • Deadlines: Homework #1 Labs:
  • Homework #1 defense
  • Consultations on Homework #2 W eek 4: Lectures:
  • Physics-Informed Deep Learning
  • NN-Based Models for Simulating Dynamical Systems
  • Deadlines: Homework #2 Labs:
  • Homework #2 defense W eek 5: Lectures
  • Differentiable Rendering
  • Deadlines: Project proposal Labs:
  • Introduction to Nvidia Kaolin W eek 6: Lectures:
  • Project Pitches
  • Deadlines: Homework #3 Labs:
  • Homework #3 defense W eek 7: Lectures:
  • Neural Rendering
  • Deadlines: Project-related work overview Labs:
  • Project-related consultations W eek 8: Lectures:
  • Papers’ Club #1 Labs:
  • Project-related consultations W eek 9: Lectures:
  • Papers’ Club #2 Labs:
  • Project-related consultations W eek 10: Lectures:
  • Papers’ Club #3 Labs:
  • Project-related consultations W eek 11: Lectures:
  • Papers’ Club #4 Labs:
  • Project-related consultations W eek 12: Lectures:
  • Lectures: Research Paper review - Part 1 Labs:
  • Project-related consultations W eek 13: Lectures:
  • Lectures: Research Paper review - Part 2 Labs:
  • Project-related consultations W eek 14: Lectures:
  • Lectures: Research Paper review - Part 3 Labs:
  • Project-related consultations W eek 15: Lectures:
  • Lectures: Project Presentations
  • Deadlines: Project code, report, presentation