Project

My project consists of two components, training an NMT model that can act as a first-pass translator for Ottoman Turkish, and testing the performance of OpenAI’s Generative Pre-Trained Transformer model, GPT-4. Released in February 2023, GPT-4 showed what some AI and natural language processing (NLP) researchers call emergent capabilities. In my research, emergent capabilities refers to Large Language Models (LLMs) like GPT-4 demonstrating surprisingly high performance in natural language tasks for which they were not explicitly trained. One such task is translation of Ottoman Turkish into English.

Machine Translation and GPT-4

In the domain of NLP, machine translation (MT) is a task that requires parallel data. Practically, this means that MT datasets consist of sentences in source and target languages that are translations of one another. NMT model training differs from that of GPT-4, because translation is a sequence-to-sequence task. In NMT, the source sentence is encoded and the target sentence is decoded, which means that the model generates the source sentence in the target language. GPT-4, on the other hand, is an autoregressive model. In simplified terms, GPT-4 generates a sentence by decoding it word by word.

To the best of our knowledge, GPT-4 has never been trained specifically for translation, let alone for the translation of Ottoman Turkish. While the details about the training data and implementation are proprietary information that OpenAI has not publicly disclosed, it is safe to assume that GPT-4 has not been explicitly trained on Ottoman Turkish texts. However, GPT-4 probably encountered Ottoman texts as part of its training since it has seen all of the Internet in some fashion.

Despite the gaps of knowledge surrounding the ‘how’ and the ‘why,’ I established empirically through Semantics of Empire that GPT-4 outperforms other LLMs and NMT models trained for Modern Turkish to English translation. Below is a chart comparing the performance of several models–including some that I fine-tuned–on three test texts: an Ottoman novel, Ahmet Mithat Efendi’s Felâtun Bey ve Râkim Efendi, a late 18th century manuscript Hulasatü’l-itibar by Ahmed Resmi, and an early 18th century manuscript, Osman Ağa’s account of his imprisonment.

Experiment

I was initially surprised by GPT-4’s performance; the model-translated texts looked similar to scholarly translations. These initial findings encouraged me to test GPT-4’s performance more methodologically. I designed the following experiment, which will be a part of my M.S. thesis in Symbolic Systems, supervised by Prof Mark Algee-Hewitt, and more broadly a component of my History dissertation, advised by Prof Ali Yaycıoğlu. This experiment is conducted fully online through a website that I built: Ottoman Translator

My website uses the OpenAI GPT-4 API to query the model to translate Ottoman Turkish sentences submitted by you, the experiment participants. Upon typing or pasting a sentence in the text box, you click the ‘Translate’ button. The English translation of the sentence will appear under the text box. After reading the translation, click the ‘Edit This Translation’ button to offer corrections. A pop up will appear with the sentence translated by the model. You are asked to edit or correct the translated sentence based on your domain knowledge and expertise.

Guidelines for Using the Translation Tool

• Do not input sentences that are too long, i.e. over 70-80 words, depending on the character length of your words. The model performance decreases drastically with longer texts. This limitation stems from the model’s architecture.
• Do not click the Translate button repeatedly. It might take up to 30 seconds for the translation to appear. If there is an issue, you will see an error message pop up immediately.
• Crucially, don’t forget to offer edits and corrections to the translated sentences. Each translation costs me some amount of money for which I have very limited funds. As a graduate student, I need to collect data for research purposes and do not have the means to offer this as a free service. I appreciate your understanding and collaboration in this regard.
• Note that the translations may not be fully accurate. They are not scholarly translations. Always read and carefully evaluate them.
• You may use the translations that you generate on my website for research and teaching purposes. If you do so, cite my work with the instructions provided at the very end of this blog post.

Data Usage

I use the Google Sheets API to collect your input, the model’s translation, and your correction. I am not collecting any identifying information including, but not limited to, IP addresses. My experiment does not fall under the domain of Stanford University’s Institutional Review Board (IRB) because I am testing model performance with human annotators, which does not constitute a human experiment. Throughout Summer 2024, I will use the data generated in my experiment to test GPT-4’s performance on a set of Ottoman Turkish examples from a broader time period than my PhD research focus. I will also incorporate this data into my model training.

Future Directions

Ultimately, my goal is to produce a model that can compete with GPT-4 as a first pass translator and host it freely for all scholars and researchers in the Ottoman Empire and Middle Eastern Studies fields. This is why at this point in time I am inviting scholars of Ottoman Turkish to participate in this experiment and help me build the digital future for our field. All comments and questions can be directed to me via email.

Citations

If you use translations from this model in your scholarship, or engage with this project in any other way, please cite this post as you would with any publication. You may use this format for citations: Merve Tekgürler. Experimenting with GPT-4 for Machine Translation from Ottoman Turkish into English. First Published: 05/01/2024. URL: https://github.com/mervetekgurler/mervetekgurler.github.io/blob/7ad5adcc1271396ce2fa08803c156ce4d10ebd9f/_posts/2024-05-01-GPT4-translator.md

The American Historical Association is the largest membership association of professional historians in the world. The annual meeting, which in 2024 took place in San Francisco, was my first time at the AHA. I presented a poster about my research at the intersection of Ottoman history and computational linguistics. I created this page to share supplemantary information, including translations and links to data.

Briefly put, the poster focused on topic modelling on 4 court histories from the 18th and 19th centuries, covering primarily the period from the 1750s through early 1800s. These histories were acquired from scholarly editions and theses, which can be found in the bibliography below. I segmented them following the internal divisions of the texts, first dividing them by year and second by the subheadings. I preprocessed these text segments to clear stopwords and normalize words, removing inflections through a customized stemming algorithm and further rule-based text cleaning. I used Gensim’s LDA (Latent Dirichlet Allocation) to extract 50 topics. Text segments, both in original and in preprocessed version and the topics that are matched to those segments with their probabilities can be found here.

Topic Models

In this section below you can find more details about the 10 topics directly referenced in the poster. I provided below the original word cloud, its English translation, and the probabilities associated with the words in each topic. Words and probabilities for all 50 topics can be found here.

You scroll down this page to see the topics in sequential order or can use these links to navigate on the page.

1. Topic 2
2. Topic 12
3. Topic 14
4. Topic 15
5. Topic 18
6. Topic 21
7. Topic 22
8. Topic 35
9. Topic 36
10. Topic 49

Topic 2

Word Clouds

Words, Translations, Probabilities

Topic 12

Word Clouds

Words, Translations, Probabilities

Topic 14

Word Clouds

Words, Translations, Probabilities

Topic 15

Word Clouds

Words, Translations, Probabilities

Topic 18

Word Clouds

Words, Translations, Probabilities

Topic 21

Word Clouds

Words, Translations, Probabilities

Topic 22

Word Clouds

Words, Translations, Probabilities

Topic 35

Word Clouds

Words, Translations, Probabilities

Topic 36

Word Clouds

Words, Translations, Probabilities

Topic 49

Word Clouds

Words, Translations, Probabilities

Bibliography

Primary Sources

Ahmed Cevdet Paşa. Târîh-i Cevdet, vols 1, 2, 3. 2nd ed. İstanbul: Matbaa-I Osmâniyye, 1892. In M. İpşirli. “Ahmed Cevdet Paşa. Târîh-i Cevdet.” Türk Tarih Kurumu Yayınları, 2018. Ahmed Resmi Efendi. Hulâsatüʼl-iʻtibâr. [manuscript, 1783]. In E. Menchinger. “Hulâsatüʼl-iʻtibâr = A Summary of Admonitions: A Chronicle of the 1768-1774 Russian-Ottoman War.” İstanbul: Isis Press, 2011. Ahmed Vâsıf Efendi. Mehâsinü’l-Âsâr ve Hakāʼiku’l-Ahbâr, 1752-1774. [manuscript, 1787]. In N. Sağlam. “Ahmed Vâsıf Efendi ve Mehâsinü’l-Âsâr ve Hakāʼiku’l-Ahbâr’ı.” [dissertation, 2014]. Ahmed Vâsıf Efendi. Mehâsinü’l-Âsâr ve Hakāʼiku’l-Ahbâr, 1774-1779. [manuscript, 1787]. In Mücteba İlgürel. “Ahmed Vasıf Efendi Mehâsînü’l – Âsâr ve Hakâikü’l – Ahbâr 1774-1779 (H. 1188- 1193).” Ankara: Türk Tarih Kurumu Yayınları, 2014. Ahmed Vâsıf Efendi. Mehâsinü’l-Âsâr ve Hakāʼiku’l-Ahbâr, 1782-1787. [manuscript, 1787]. In Mücteba İlgürel. “Ahmed Vasıf Efendi Mehâsînü’l – Âsâr ve Hakâikü’l – Ahbâr.” Ankara: Türk Tarih Kurumu Basımevi, 1994. Enveri Sadullah Efendi. Tarih-i Enveri I, 1768-1774. [manuscript, 1775]. In M. S. Çalışkan. “(Vekayi’nüvis) Enveri Sadullah Efendi ve Tarihinin I . Cildi’nin Metin ve Tahlili (1182 - 1188 /1768 - 1774).” [dissertation, 2000]. Enveri Sadullah Efendi. Tarih-i Enveri II, 1774-1783. [manuscript, 1783]. In H. Çiçek. “(Vekāyiʻnüvis) Sadullah Enverî Efendi ve Tarihi’nin II. Cildinin Metin Ve Tahlili (1187–1197/1774–1783).” [dissertation, 2018]. Enveri Sadullah Efendi. Tarih-i Enveri, 1786-1792. [manuscript, 1794]. In Ü.F.Bayram. “Enverî Târîhi: Üçüncü Cild (Metin Ve Değerlendirme).” [dissertation, 2014]. Nuri For comparison: Ahmed Câvid Bey. Müntehabât. [manuscript, 1802]. Transcribed and analyzed by Adnan Baycar. “Ahmed Câvid Bey’in Müntehabâtı (Tahlil ve Tenkitli Metin).” [dissertation, 1999].