Ollama

Fei Liu, Zejun Kang, Xing Han

With the growing demand for offline PDF chatbots in automotive industrial production environments, optimizing the deployment of large language models (LLMs) in local, low-performance settings has become increasingly important. This study focuses on enhancing Retrieval-Augmented Generation (RAG) techniques for processing complex automotive industry documents using locally deployed Ollama models. Based on the Langchain framework, we propose a multi-dimensional optimization approach for Ollama's local RAG implementation. Our method addresses key challenges in automotive document processing, including multi-column layouts and technical specifications. We introduce improvements in PDF processing, retrieval mechanisms, and context compression, tailored to the unique characteristics of automotive industry documents. Additionally, we design custom classes supporting embedding pipelines and an agent supporting self-RAG based on LangGraph best practices. To evaluate our approach, we constructed a proprietary dataset comprising typical automotive industry documents, including technical reports and corporate regulations. We compared our optimized RAG model and self-RAG agent against a naive RAG baseline across three datasets: our automotive industry dataset, QReCC, and CoQA. Results demonstrate significant improvements in context precision, context recall, answer relevancy, and faithfulness, with particularly notable performance on the automotive industry dataset. Our optimization scheme provides an effective solution for deploying local RAG systems in the automotive sector, addressing the specific needs of PDF chatbots in industrial production environments. This research has important implications for advancing information processing and intelligent production in the automotive industry.

Han-Woo Choi, J. Jeong

To support data-driven decision-making in a Manufacturing Execution System (MES) environment, a system that can quickly and accurately analyze a wide range of production, quality, asset, and material information must be deployed. However, existing MES data management approaches rely on predefined queries or report templates that lack flexibility and limit real-time decision support. In this paper, we proposes a domain-specific Retrieval-Augmented Generation (RAG) architecture that extends LangChain’s capabilities with Manufacturing Execution System (MES)-specific components and the Ollama-based Local Large Language Model (LLM). The proposed architecture addresses unique MES requirements including real-time sensor data processing, complex manufacturing workflows, and domain-specific knowledge integration. It implements a three-layer structure: an application layer using FastAPI for high-performance asynchronous processing, an LLM layer for natural language understanding, and a data storage layer combining MariaDB, Redis, and Weaviate for efficient data management. The system effectively handles MES-specific challenges such as schema relationships, temporal data processing, and security concerns without exposing sensitive factory data. This is an industry-specific, customized approach focusing on problem-solving in manufacturing sites, going beyond simple text-based RAG. The proposed architecture considers the specificity of data sources, real-time and high-availability requirements, the reflection of domain knowledge and workflows, compliance with security and quality control regulations, and direct interoperability with MES systems. The architecture can be further enhanced through integration with various manufacturing systems, an advanced LLM, and distributed processing frameworks while maintaining its core focus on MES domain specialization.

Donghao Huang, Zhaoxia Wang

Energy efficiency is a critical consideration in deploying large language models (LLMs) due to their significant environmental footprint. This study addresses this challenge by leveraging Ollama, an open-source framework that facilitates efficient local deployment of LLMs, thereby reducing energy consumption alongside enhancing privacy and accessibility. We evaluate the performance and efficiency of open-source LLMs from the Qwen2.5 and Llama3 families (0.5B–90B parameters) across diverse hardware platforms, including consumer-grade devices. Our evaluation spans high-end systems (RTX A6000, RTX 4090), consumer laptops like Apple M3/M4, and accessible platforms like NVIDIA Jetson AGX Orin, showcasing pathways to widespread AI adoption. Key findings include: (1) Ollama’s quantization enables efficient LLM deployment on consumer hardware while preserving performance; (2) Windows Subsystem for Linux enhances speeds across all models, delivering up to 21× improvements for the smallest model while also improving energy efficiency, thereby broadening adoption on common computing platforms; (3) Medium-sized models (7B–32B) achieve an ideal balance of performance and efficiency; (4) Larger models achieve the highest performance and exhibit better cross-platform consistency but are significantly less energy efficient. These results highlight strategies for inclusive, privacy-preserving, and sustainable AI deployment, advancing the democratization of LLM technology while reducing energy consumption and infrastructure demands.

F. S. Marcondes, Adelino Gala, Renata Magalhães, Fernando Perez de Britto, Dalila Durães, Paulo Novais

Varun Rajesh, Om Jodhpurkar, Pooja Anbuselvan, M. Singh, Ashok Jallepali, S. Godbole, Pradeep Kumar Sharma, Hritvik Shrivastava

We present a systematic, empirical evaluation of five local large language model (LLM) runtimes on Apple Silicon: MLX, MLC-LLM, llama.cpp, Ollama, and PyTorch MPS. Experiments were conducted on a Mac Studio equipped with an M2 Ultra processor and 192 GB of unified memory. Using the Qwen-2.5 model family across prompts ranging from a few hundred to 100,000 tokens, we measure time-to-first-token (TTFT), steady-state throughput, latency percentiles, long-context behavior (key-value and prompt caching), quantization support, streaming performance, batching and concurrency behavior, and deployment complexity. Under our settings, MLX achieves the highest sustained generation throughput, while MLC-LLM delivers consistently lower TTFT for moderate prompt sizes and offers stronger out-of-the-box inference features. llama.cpp is highly efficient for lightweight single-stream use, Ollama emphasizes developer ergonomics but lags in throughput and TTFT, and PyTorch MPS remains limited by memory constraints on large models and long contexts. All frameworks execute fully on-device with no telemetry, ensuring strong privacy guarantees. We release scripts, logs, and plots to reproduce all results. Our analysis clarifies the design trade-offs in Apple-centric LLM deployments and provides evidence-based recommendations for interactive and long-context processing. Although Apple Silicon inference frameworks still trail NVIDIA GPU-based systems such as vLLM in absolute performance, they are rapidly maturing into viable, production-grade solutions for private, on-device LLM inference.