Environment-aware Dynamic Partitioning of Neural Networks - Research Data and Code
Description
Supporting data for "Environment-aware Dynamic Partitioning of Neural Networks". We present a method to dynamically partition neural networks on resource-constrained IoT devices to increase FPS and decrease total inference latency. Currently, in real-time neural network systems, input data collected from sensors is transmitted to local servers for inference. Network transmission often introduces significant latency. In feedforward neural networks, the output of each layer is passed to the next layer, with data size progressively decreasing through successive layers. Instead of transmitting the original data, we exploit this downward size trend by using an edge device (either co-located with the sensor or embedded within it) to transmit smaller intermediate results to the local server, which processes the remaining layers. Furthermore, we quantize and compress these intermediate results using various techniques to reduce their size before transmission by up to 97.11%. Even with lossy compression, we maintain a mean average precision (mAP) of 0.52. Includes a partition point calculator to calculate the best partition point for the YOLOv8 and YOLOv3-tiny neural networks. Included data is for Nvidia Jetson Orin Nano 8GB (YOLOv8) and Nvidia Jetson Nano 2GB (YOLOv3-tiny). Code for measuring the data is also included.
Files
Steps to reproduce
The code included is for YOLOv8 or YOLOv3-tiny. We ran YOLOv8 or the Nvidia Jetson Nano 8GB and YOLOv3-tiny on the Nvidia Jetson Nano 2GB. PyTorch was used for YOLOv8 and TensorFlow was used for YOLOv3-tiny. You can use the same code for other models but you will have to change the model and/or the name of the layers in the code.
Institutions
- Mississippi State University