Hardware-Software Co-Design Real-time AI

Course Overview

The aim of this course is to expose the students to the recent advances in AI, and machine, and FPGA-based SoCs for machine learning. This course focuses on the FPGA-based acceleration of machine learning and deep learning algorithms for real-time edge computing. This course offers an interactive practical introduction to hardware/software co-design, machine learning and computer vision, deep learning based on Xilinx Pynq (Python productivity for Zynq ) solution. Throughout the course, students will work on top of Xilinx’s Python ecosystem and Jupyter notebooks. This course introduces the basics of deep learning and enables students to build basic computer vision, Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) inference algorithms on Pynq platform using pre-trained models. Examples are the Quantized Neural Network (QNN), Binarized Neural Network (BNN) and Quantized LSTMs.

Textbook

This course doesn’t use a specific textbook. Considering the rapid growth in the field, it relies on the publicly available information. The major references are: Pynq Community Website and Pynq Jupyter Notebooks:

• http://www.pynq.io/home.html
• https://pynq-testing.readthedocs.io/en/latest/jupyter_notebooks.html

The lab assignments are designed based on the following GitHub repositories:

• https://github.com/Xilinx/PYNQ
• https://github.com/Xilinx/PYNQ_Workshop
• https://github.com/Xilinx/PYNQ-ComputerVision
• https://github.com/Xilinx/QNN-MO-PYNQ
• https://github.com/Xilinx/BNN-PYNQ
• https://github.com/Xilinx/LSTM-PYNQ
• https://github.com/Xilinx/pytorch-quantization
• https://github.com/Xilinx/PYNQ-DL

Topics

1. Introduction to AI big bang, and basics of machine learning and deep learning
2. Real-time edge intelligence and FPGA-based SoCs
3. Overview of Pynq platform
4. Basics of hardware/software co-design and FPGA-based SoCs.
5. Getting started with Jupyter notebooks (notebook)
6. Overlay design methodology and Pynq overlay design
7. Basics of computer vision and OpenCV
8. Basics of deep learning inference (Convolutional and Recurrent Neural Networks)
9. Quantized Neural Network (QNN) on PYNQ
10. Binarized Neural Network (BNN) on PYNQ
11. Quantized LSTM on PYNQ

Outcomes

By end of this semester, the students should demonstrate the following competencies:

1. Basics understanding of machine learning, deep learning and computer vision
2. Ability to understand the principles hardware/software co-design
3. The Jupyter Notebook framework deployed in the Xilinx Pynq context
4. Hardware/software codesign based on Xilinx Pynq solution
5. Basic knowledge of computer vision algorithms and OpenCV
6. Basic Knowledge of Deep learning paradigm including CNNs and RNNs models (with a focus on the real-time inference)
7. Understanding of FPGA overlay design and Xilinx deep learning and computer vision APIs