Assessing accuracy, clarity, and naturalness with AQuA

Augmented Quality Assessment (or AQuA) is a tool developed by SIL International that seeks to increase the pace and thoroughness of translation quality assurance. It harnesses artificial intelligence (AI) techniques, assisting human reviewers in objectively assessing multiple facets of translation quality by enhancing and complementing preexisting translation work. AQuA is able to produce an increasingly detailed suite of quality-related diagnostics, with several augmented quality assessment methods in the areas of accuracy, clarity, and naturalness.

Current Functionality

The AQuA team has been working in a multidisciplinary and cross-organizational manner to validate the following methods for efficiently probing the “big three” qualities of Bible translations:

Accuracy – Semantic similarity, word correspondence, and missing words
Clarity – Machine question answering
Naturalness – Sentence and word length (Lix readability)

AQuA is currently in-beta, available as a standalone app to select trial users. It has undergone several field tests across the globe in the past year, where it has helped translation teams identify missing verse portions, unnaturally-translated verses, and incorrectly or inconsistently translated words.

Using AQuA Metrics to Explore the Translation Landscape

The following graph plots the mean and range of word correspondence scores of several well-known English Bible translations to the New Testament Greek, as measured by AQuA. We can see that the more formally equivalent translations (Young’s Literal Translation, the King James Version) have the highest average word correspondence scores to the Greek, while the paraphrase translations (The Message, The Living Bible) have the lowest scores.

Plotting known translations in this way allows someone to get a rough idea of where a new, unknown translation may fit into the landscape, even if this translation is in another language. The following graph adds in several non-English translations (in red). Notice how the Spanish NVI shares a similar average word correspondence score to the English NIV; we see a similar pattern also with the Spanish RVR and the KJV, as well as with the Spanish NTV and the NLT.

AQuA in Action

In the AQuA app, the user can view multiple quality metrics of their translation side-by-side, giving them a new perspective on their draft. AQuA’s unique visualizations map the consistency of certain metrics across books, chapters, and verses, and allow for easy spotting of outliers; see the example below.

The figure below maps the semantic similarity between the English Good News Translation and the English New American Standard Bible. AQuA shows us that the biggest semantic differences between the two translations are in the wisdom literature books (Job, Proverbs, and Ecclesiastes).

Zooming into the book of Ruth, we can plot the semantic similarity and word correspondence metrics against each other. Mousing over the low-scoring outlier in the bottom left shows us the verse reference, project, and comparison texts.

If we want to take a closer look at any given verse, we can view AQuA’s guesses of which words present in the comparison text may be missing from the draft.

Of course, AQuA works for more than just English. Consultants, for example, may be interested in comparing their project text directly to the Greek, as in the example below. With the help of Augmented Quality Assessment, several translation teams have been able to locate areas in need of revision in their translations that otherwise would not have been found.

Plans for scaling up

The AQuA team is hard at work on the back-end to accompany a potentially rapidly-growing user base. Integration into Clear Dashboard and Scripture Forge are planned for the coming months, and a possible AQuA platform.bible extension is on the horizon as well. As the number of AQuA users grows, we will continue to develop onboarding and training materials, such as articles, tutorials, and explainer videos to ensure proper use and understanding of the tool.

To contact the AQuA team, please fill out this contact form on our website.