[Case Study] Leveraging LLM for the Translation and Summarization of Professional Articles and Documents

Introduction

As we venture further into the digital age, the significance of Generative AI, specifically the LLM, in a multitude of sectors cannot be overstated. This cutting-edge technology offers revolutionary solutions for businesses, challenging conventional methods and heralding a new era of efficiency and innovation. From deploying AI-driven customer service chatbots to facilitating advanced data analytics, the introduction of Generative AI underscores its expansive potential and adaptability across various industries.

The rising momentum in integrating AI solutions into business frameworks has spurred investigations into the optimal utilization of LLM's capabilities. By closely aligning with client needs, such explorations have illuminated promising avenues for the effective implementation of Generative AI in the business realm. Whether for content generation, data interpretation, or as this case study demonstrates, professional translation and summarization, the multifaceted nature of LLM is set to profoundly influence and reshape business operations and strategies.

Challenge

The client has been translating and summarizing the latest US industry news from Korea daily for years to share with their colleagues at a consistent time. Finding US news in Korea and translating it is arduous, not just because of the summarization but also because of the time difference.

Besides, for accurate and insightful delivery to executives, a professional translation and summary with knowledge about the industry and market is crucial. Moreover, given the need to share multiple news pieces daily, they wanted the summaries to be provided in a consistent format.

At TecAce, we possess numerous bilingual engineers familiar with the relevant industries and located in the US. This enables us to address many of the client's challenges. However, even experts proficient in both Korean and English found it time-consuming to find, translate, and summarize 5 to 6 news pieces daily.

Solution 1

We attempted translation using famous Generative AIs available in the market. We tried translating and summarizing the news into Korean using ChatGPT 4.0, Bard, Clova X, and DeepL. ChatGPT 4.0 yielded the most satisfactory results, using relatively accurate Korean words and maintaining context during translation and summarization.

However, the client's requirement wasn't just simple translation and summarization. They needed concise expression using industry-specific terms and a consistent format for each news summary. We decided to tackle this problem with prompt engineering on GPT 4.0.

We tested prompts that can translate, summarize, and shape the news on three services using GPT 4.0: ChatGPT 4.0, WRTN (strong in Korean), and Bing with web search capability.

Advantages and Disadvantages of GPT4.0 Generative AI tools

	Advantages	Disadvantages
ChatGPT 4.0	- Easily applicable prompts that function well - High contextual understanding	- Tends to be overly verbose - Occurrences of hallucination
Wrtn (GPT 4.0)	- Uses advanced Korean-specific words - Offers multiple word choices in Korean	- Doesn’t fully grasp the overall context and produces sentences that don’t always fit
Bing Chat (GPT 4.0)	- Convenient to use directly in a browser - Summarizes simply and clearly	- Occasional service malfunctions - Sometimes the summary is too simplistic to understand.

Prompt engineering improved the results significantly, but they still didn't match the quality of translations done by experts. The appropriate use of professional terms was significantly lacking.

Solution 2

The ultimate solution combined prompt engineering with fine-tuning the GPT model to use professional terms accurately. For prompt engineering, we first wrote prompts in English and later translated them into Korean. We manually curated summaries and underwent several fine-tuning processes for better understanding.

Another challenge was instilling confidence in the AI's translations and summaries. A feature was implemented to evaluate how well the translated summaries met criteria like 'Accuracy, Comprehensiveness, Conciseness, and Contextualization' based on algorithms defined in various studies.

Lastly, considering it’s a service, UX/UI was not neglected. Since AI is not perfect and might require editing, a UX was devised to create the final summary while viewing both the original and extended summary simultaneously.

Outcomes and Benefits

The service integrates capabilities of Generative AI (LLM) that it performs relatively well in the business. While currently applied only to news content in English/Korean, considering the variety of content and languages, the potential applications are endless.

With the system, the client saved over 60% of the time previously spent on this task and could allocate more time to finding and analyzing meaningful news. They can now produce more reliable information by getting evaluations based on the four criteria.

With fine-tuning, hallucination decreased, and the use of professional terms improved. Another advantage of fine-tuning is reducing prompt lengths, lowering token usage costs, and receiving faster results.

Conclusion

The "News Copilot" case study underscores the powerful capabilities of Generative AI, like LLM, in improving the translation and summarization of professional articles. Specifically designed to translate and summarize U.S. industry news for Korean professionals, the AI tool demonstrated remarkable efficiency gains. However, striking the right balance between automation and human input was crucial.

While AI tools like ChatGPT 4.0 offered commendable translations, challenges persisted, particularly in context understanding and industry-specific terminology. The solution was found in blending prompt engineering and model fine-tuning to achieve accurate translations enriched with professional terms.

Innovatively, the DeepKU feature was introduced, going beyond basic translations by identifying and elaborating on critical industry issues and keywords. By emphasizing an intuitive user experience, the system enabled users to interact and edit AI-generated content with ease, even amidst the tool's imperfections. The result was a synergy of AI and human expertise, cutting down time spent on such tasks by over 60% and allowing businesses to shift focus to in-depth news analysis.

In essence, the potential of AI tools such as LLM is vast in professional translation. Their true value emerges when tailored to specific needs and when paired seamlessly with human oversight and expertise.

🇰🇷 Here is the Korean version of the article

[Case Study] LLM을 사용하여 전문 문서를 번역하고 요약 하는 방법.pdf

Download PDF • 1.18MB