Our journey to expanding the LLM ecosystem: #OpenSource #Hackathon #SchoolTour

Company
Written By Hailey(박성민) .
2024/06/12 | Written by: Sungmin Park
 

Intro

The world of AI is changing the way we work, evolving with every technological breakthrough. The transformative impact of Large Language Models (LLMs) on AI product development has become a game-changer, maximizing business impact. That's why we drive innovation by expanding our boundaries in the LLM ecosystem.

We have experienced the broad impact of LLMs being utilized across various industries at several global conferences and community events. In March, we participated in GTC 2024, where we operated our own booth and engaged in startup pitches to showcase Upstage's full-stack LLM ecosystem and Solar LLM to prospective customers and users worldwide. Also, we have attended community events related to LLM to explore potential business opportunities and gain a deeper understanding of LLM research and applications.

What we found was that many people have a significant interest in establishing a tailored full-stack LLM ecosystem to make AI adoption more efficient. Our goal is to develop a full-stack LLM ecosystem that encompasses the entirety of the AI adoption journey, from A to Z, fostering the advancement of LLM application development within the global LLM community.

The photos show Upstage participating in LLM community events around the world, including in the US and Vietnam.



The Rise of the LLM Ecosystem

Ever wondered why the LLM ecosystem is experiencing such rapid growth and gaining attention in industries? The reason is that LLM has totally transformed how we develop and use advanced AI applications, making them super versatile for a wide range of business needs.

At the heart of this transformation stands the LLM ecosystem, which plays a pivotal role in promoting the efficient handling of vast amounts of data, fostering the development of intricate machine learning models, and deploying systems capable of executing complex tasks with ease. Given the rapid pace at which AI is evolving, maintaining a vibrant LLM ecosystem is essential to ensure that language learning processes are adequately supported and that the full potential of LLMs is fully harnessed.

Before delving into understanding the development of a LLM, it is essential to know its core components. These include a hyper-scale cloud infrastructure and supercomputing capabilities, along with hyper-scale data centers, which are vital for establishing a robust operational environment for the model. Furthermore, a backbone model and tuning techniques are required for its cost-effective application. Lastly, a large amount of high-quality training data is essential for building and fine-tuning the model's performance.

Upstage Full-stack LLM Ecosystem

With these components, the process of creating an LLM involves three phases: pre-training, supervised fine-tuning, and alignment. To carry out this process effectively, we have pinpointed four essential components for a comprehensive LLM framework: an open-source ETL solution for LLM data processing (Dataverse), scaling LLMs through depthwise scaling and continued pretraining (Depth Up Scaling), stepwise-direct preference optimization (sDPO), and an open-source LLM evaluations solution (Evalverse). These fundamental elements are vital to the successful development of LLMs, ensuring that the models perform exceptionally well across a range of applications. We've also launched the "Up 1 Trillion Token Club" to boost LLM growth and the "Open Ko-LLM Leaderboard" to evaluate the performance of Korean Large Language Models as part of supporting a vibrant LLM ecosystem.

Our core LLM ecosystem is designed to enhance LLM, pushing them to reach optimal performance and versatility across a diverse range of language-based tasks.




Open Source LLMs for everyone

The open-source LLM that we mentioned has gained more attention in the AI market due to its features of accessibility, transparency, and cost-effectiveness. We endeavor to create open-source LLMs that are beneficial to everyone, fostering a vibrant and innovative ecosystem. Below we present four of our recently launched open-source projects.


Solar Mini is a pre-trained large language model (LLM) developed by Upstage, designed to be easily customizable, in particular through fine-tuning, for various enterprise use-cases. It has gained recognition by reaching the top of the Open LLM Leaderboard on Hugging Face. Its exceptional performance has been proved in several tasks, including translation, math solving, and categorization, which resulted in exceeding the performance of GPT4. This model is available under the Apache 2.0 license and can be easily integrated into powerful, purpose-trained LLMs through our console or platforms like Amazon SageMaker JumpStart, AWS Marketplace, Langchain and BentoML.

Dataverse is a freely accessible open-source project designed to streamline the extract, transform, and load (ETL) process using Python. Within the LLM sphere, the importance of robust data pre-processing techniques cannot be underestimated. To foster a vibrant open-source ecosystem, Upstage has launched Dataverse, which aims to not only bridge this gap in our community by sharing evolving data engineering techniques but also to make it easily accessible in one cue.


Evalverse is a freely accessible, open-source project designed to support your LLM (Large Language Model) evaluation needs. It is designed to streamline the evaluation of LLMs by integrating a variety of evaluation methodologies, catering to the needs of AI research engineers and scientists. It also supports no-code evaluation processes for people who may have less experience working with LLMs and users can receive a well-organized report with figures summarizing the evaluation results.

The Open Ko-LLM Leaderboard objectively evaluates the performance of Korean Large Language Models (LLMs), adopting five types of evaluation methods: ARC (AI2 Reasoning Challenge), HellaSwag, MMLU (Massive Multitask Language Understanding), TruthfulQA, and KoCommonGEN V2. Researchers can share their results on the leaderboard, fostering transparency of a vibrant Korean LLM evaluation ecosystem. Its biggest advantage is the ability to evaluate the performance of Korean LLM models through Korean benchmarks. Additionally, participants can build their credentials by winning 'This Month's LLM' at the Open Ko-LLM Leaderboard.

Propelling Global LLM Application Development to New Heights

We are truly making AI beneficial for everyone by working to democratize access to the technology—both in terms of the tools people can use and in opening up new opportunities for developing LLM applications.

Hosting hackathons in various countries such as South Korea, the United States, Vietnam, and beyond is part of our efforts to expand our LLM ecosystem globally. This would be a chance for students, researchers, and developers to dive into our full-stack LLM tech and play around with their techy ideas. We believe that running these events will contribute to the growth of the LLM ecosystem.

Hackathon projects at AGI House SF utilizing the full stack Solar LLM from Upstage.

Photo from the Gen AI Hackathon for building RAG applications and fine-tuning, held on June 1st in San Francisco.Photo from the Gen AI Hackathon for building RAG applications and fine-tuning, held on June 1st in San Francisco.

Photo from the Gen AI Hackathon for building RAG applications and fine-tuning, held on June 1st in San Francisco.

Over the past six months, we have successfully hosted and organized 8 global tour programs, including hackathons and school tours. These events have led to numerous valuable ideas that can significantly improve people's daily lives. For instance, one idea involves creating educational videos using solar power, another suggests an API that can convert technical concepts into diagrams with a Layout Analyzer, and there's an app project to help leukemia patients enjoy tasty, immune-boosting meals throughout their recovery. These initiatives are part of the efforts to foster innovation within the global LLM ecosystem. The following paragraph outlines our past efforts and our plans for hosting more global tour events. Moreover, the upcoming hackathon tour will be held in Vietnam, Japan, and Thailand.

<Upstage's Global Tour for Expanding the LLM Ecosystem>

Shaping the future of Large Language Models

We are grateful for every opportunity to witness cutting-edge technology that paves the way for the future. We believe this is just the beginning of our journey, and we are excited to foster more collaborations and contributions with global industries to enhance work efficiency. Stay tuned for the limitless possibilities with our full-stack LLM and vibrant LLM ecosystem, which we are improving.

If you are interested in our work, solutions, or want to leverage Solar LLM for your business, we encourage you to reach out to us using the link below. We would love to connect, share our insights, and explore how we can work together to shape the future of AI.

Hailey(박성민) .
Previous

LLM Trend Report (2024 Q2)

Next

Upstage Optimizes Large Language Model for Intel Core Ultra Processors