05/June/2024 | Written By: YoungHoon Jeon, NamJun Jo, Jaeho Lee, Junyeop Lee

AI at your fingertips: On-Device LLM

In a world teeming with web-based AI services, AI has become a natural companion in our daily tasks. However, the convenience of AI has always depended on internet connectivity. But what about environments where internet access is limited?

Imagine a future where you can maximize your work efficiency with AI assistance even while traveling, commuting on a high-speed train, or retreating in a remote mountain. With Upstage's on-device LLM - WriteUp, this is not a distant future, but a reality you can experience right now.

Deep dive into On-Device LLM

What is On-Device LLM?

On-device LLM refers to the technology where AI models run directly on user devices instead of in the cloud. This approach offers several advantages, such as convenience of use without requiring an internet connection and enhanced data privacy, as data is not transmitted to external servers. Due to its secure and stable nature, on-device LLM has become highly desirable.

How to implement On-Device LLM?

Implementing on-device LLM requires advanced technology due to several challenges, including model optimization and efficient computation.

• Quantization: LLM quantization involves reducing the size of AI models to optimize memory usage and improve processing speed. This technique allows the model to maintain performance while minimizing its size. However, since the quality of the output can degrade with quantization, finding the optimal balance within the user’s context is crucial.

• Optimization: Advanced optimization technologies are essential to ensure smooth performance and seamless integration with user devices. Without the use of cloud resources, it is necessary to efficiently leverage the computing power of each individual device. This involves differentiating between CPU-only environments and those with dedicated GPUs or NPUs, and developing optimized code tailored to each specific chip. We have embraced this challenge by implementing a computing platform optimized for parallel computation.

Enjoy Our latest technology Now

With our proprietary LLM, Solar, we successfully applied quantization, reducing the parameter size by over 50%. This allowed us to create an LLM that fits within the RAM of a laptop, enabling multitasking without compromising performance.

In addition to model size quantization, we performed prompt engineering and engine optimization to deliver the best possible service. Through advanced prompt engineering, we achieved significant improvements in task performance. Collaboration with Intel engineers enabled us to optimize the engine for WriteUp Windows, significantly enhancing the user experience by increasing processing speed.

On-Device LLM Service - WriteUp

Our product, Write Up, is an LLM specifically tailored for writing tasks. With just a laptop, you can utilize Write Up anywhere, even in remote areas without phone or internet access. Now, you can receive AI assistance for writing, even in the most secluded locations.

Success Story: Enhancing risk management system with WriteUp

Imagine successfully deploying a critical update to your company's risk management system, a task requiring precision, coordination, and intense focus. After weeks of meticulous planning and execution, the update is live, bringing significant enhancements to back-office operations, managerial oversight, and front-office trading. The individual responsible for this success deserves a well-earned break and decides to vacation in a remote location, away from the office bustle and without access to the company’s internet.

In such a scenario, WriteUp becomes an invaluable tool. Using WriteUp's advanced capabilities, including tone adjustment and real-time editing, the individual crafts a concise and clear email update to inform back-office coworkers, managers, and front-office traders about the improvements. The email is tailored to be informative yet reassuring, highlighting the positive impact of the update on daily operations and trading activities.

By simply providing the key points to be communicated, WriteUp can swiftly transform them into appropriately tailored messages for each respective audience.

original text: ”Portfolios A-D process credit products and bonds now. Fixed an issue with inflated results. Added new risk pipelines. Cache issue fixed.”

WriteUp for Windows, Soon available

We are excited to announce that our on-device LLM service, WriteUp, will soon be available for Windows users. For the latest official announcements, follow us on LinkedIn. Stay tuned for updates and visit our page today!