Generative AI like the Large Language Models (LLMs) has become more available for the general consumer in recent years. Publicly available services, e.g., ChatGPT, perform token generation on networked cloud server hardware, effectively removing the hardware entry cost for end users. However, the reliance on network access for these services, privacy and security risks involved, and sometimes the needs of the application make it necessary to run LLMs locally on edge devices. A significant amount of research has been done on optimization of LLMs and other transformer-based models on non-networked, resource-constrained devices, but they typically target older hardware. Our research intends to provide a 'baseline' characterization of more recent commercially available embedded hardware for LLMs, and to provide a simple utility to facilitate batch testing LLMs on recent Jetson hardware. We focus on the latest line of NVIDIA Jetson devices (Jetson Orin), and a set of publicly available LLMs (Pythia) ranging between 70 million and 1.4 billion parameters. Through detailed experimental evaluation with varying software and hardware parameters, we showcase trade-off spaces and optimization choices. Additionally, we design our testing structure to facilitate further research that involves performing batch LLM testing on Jetson hardware.

A million developers across the globe are now using the NVIDIA Jetson platform for edge AI and robotics to build innovative technologies. Plus, more than 6,000 companies -- a third of which are startups -- have integrated the platform with their products. These milestones and more will be celebrated during the NVIDIA Jetson Edge AI Developer Days at GTC, a global conference for the era of AI and the metaverse, taking place online March 20-23. Register free to learn more about the Jetson platform and begin developing the next generation of edge AI and robotics. Atlanta-based Kris Kersey, the mind behind the popular YouTube channel Kersey Fabrications, is one developer using the NVIDIA Jetson platform for his one-in-a-million technological innovations.

Alif Semiconductor, supplier of low-power, secure, AI/ML-enhanced fusion processors and microcontrollers (MCUs), announced collaboration with Telit, a global leader in the Internet of Things (IoT), to deliver developer kits that provide cloud-connected hardware and software reference designs for a wide variety of distributed and IoT edge applications. The kits focus on connected AI/ML-enhanced vision, voice, vibration, and sensor applications such as AI cameras, smart home, city infrastructure, biometric access control, and wearables. Ensemble devices utilize innovative aiPM power management technology that feature a High Efficiency, always-on region that senses the environment using initial AI/ML processing, while a separate High Performance region wakes as needed to rapidly execute additional heavy AI/ML workloads and returns to sleep. In addition to smart power management, the Ensemble family provides multiple layers of security based on a secure identity and strong root-of-trust for complete lifecycle management handling keys, certificates, secure boot, remote updates and more. The developer kits include Telit's wireless connectivity modules from Wi-Fi and Bluetooth to LTE and 5G cellular, including low-power wide-area (LPWA) offerings in Cat-M and NB-IoT for IoT applications that require lower power consumption and longer battery life.

NVIDIA JetPack SDK is the most comprehensive solution for building end-to-end accelerated AI applications. JetPack SDK provides a full development environment for hardware-accelerated AI-at-the-edge development. All Jetson modules and developer kits are supported by JetPack SDK. JetPack SDK includes Jetson Linux Driver Package with bootloader, Linux kernel, Ubuntu desktop environment, and a complete set of libraries for acceleration of GPU computing, multimedia, graphics, and computer vision. It also includes samples, documentation, and developer tools for both host computer and developer kit, and supports higher level SDKs such as DeepStream for streaming video analytics, Isaac for robotics, and Riva for conversational AI.

The Nano 2GB is connected and the Nano 4GB is ... [ ] in the background. Nvidia has asserted its leadership in Artificial Intelligence (AI) with a GPU architecture that continues to evolve with the growing demands of both training and inferencing AI workloads. The latest Ampere architecture provided a huge jump in performance with a new architecture that also allows the GPU to be partitioned to act as seven individual inference engines. As a result of Ampere, Nvidia's own supercomputer Selene based on the DGX A100 server ranks fifth in the TOP500 supercomputers and number one in the Green500 supercomputers. However, Nvidia is focused on more than just extreme computing as demonstrated by its proposed acquisition of Arm. Even without acquiring Arm, Nvidia has been pushing the boundaries of the AI down to lower-power and small form factor applications.

NVIDIA expanded the NVIDIA Jetson AI at the Edge platform with an entry-level developer kit priced at just $59, opening the potential of AI and robotics to a new generation of students, educators and hobbyists. The Jetson Nano 2GB Developer Kit is designed for teaching and learning AI by creating hands-on projects in such areas as robotics and intelligent IoT. To support the effort, NVIDIA also announced the availability of free online training and AI-certification programs, which will supplement the many open-source projects, how-tos and videos contributed by thousands of developers in the vibrant Jetson community. "While today's students and engineers are programming computers, in the near future they'll be interacting with, and imparting AI to, robots," said Deepu Talla, vice president and general manager of Edge Computing at NVIDIA. "The new Jetson Nano is the ultimate starter AI computer that allows hands-on learning and experimentation at an incredibly affordable price."

The SparkFun JetBot AI Kit is a robot platform that can be powered by your own Jetson Nano Developer Kit by NVIDIA. This SparkFun kit is based on the open-source NVIDIA JetBot! We understand that not everyone has access to multiple 3D printers on each floor, and a whole warehouse of electronics so we wanted to build a kit from ready to assemble parts to get you up and running as quickly as possible. This version of the SparkFun JetBot AI Kit is a great launchpad for creating entirely new AI projects for makers, students and enthusiasts who already own a Jetson Nano and are looking to take the next step with their machine learning platform. It's straightforward to set up and use and is compatible with many popular accessories.

According to Cisco's forecast, there will be 850 ZB of data generated by mobile users and IoT devices by 2021. With a surge in data, challenges like latency will emerge. And, if one has to derive intelligence from algorithms in real-time, the traditional systems cannot be trusted for long. Thanks to the efforts of top companies to place supercomputers in the pockets, edge computing systems have garnered significant attention. Over the past couple of years chipmakers have been bullish on developing integrated solutions for edge cases.

OKdo recently announced the latest addition to their range of products, the NVIDIA Jetson Nano Developer Kit. The new developer kit offers out-of-the-box support for full desktop Linux and is compatible with many peripherals that support 4K, including Ethernet, USB, and HDMI. It also features a 128-core NVIDIA Maxwell GPU that allows users to develop AI applications in video analytics and robotics. While the unit is primarily geared toward makers and learners, this kit can be just as valuable for engineers and embedded developers. The Jetson Nano Developer Kit allows developers to run multiple neural networks in parallel for applications such as image classification, object detection, segmentation, and speech processing.

Nvidia unveiled a new federated learning edge computing reference application for radiology to help hospitals crunch medical data for better disease detection while protecting patient privacy. Called Clara Federal Learning, the system relies on Nvidia EGX, a computing platform which was announced earlier in 2019. It uses the Jetson Nano low wattage computer which can provide up to one-half trillion operations per second of processing for tasks like image recognition. EGX allows low-latency artificial intelligence at the edge to act on data, in this case images from MRIs, CT scans and more. Nvidia made its announcement of Clara on Sunday at the Radiological Society of North America conference in Chicago.