Assessing accuracy, clarity, and naturalness with AQuA

The Augmented Quality Assessment (or AQuA) tool harnesses artificial intelligence (AI) techniques to “augment” the translation quality assessment process. AQuA is able to produce an increasingly detailed suite of quality-related diagnostics, with at least five augmented quality assessment methods in the areas of accuracy, clarity, and naturalness.

Current Functionality

The AQuA team has been working in a multidisciplinary and cross-organizational manner to validate the following methods for efficiently probing the “big three” qualities of Bible translations:

  • Accuracy – Semantic similarity, agreement similarity, and word alignment
  • Clarity – Machine question answering
  • Naturalness – Consistency, reading level 

The figure below plots the output of these 5 methods to create a quality “fingerprint” of two different translations. In this case, the plot indicates that the translation represented in blue is similar to the translation represented in green with respect to most qualities. The notable exception is clarity, where the blue translation shows noticeably more clarity.

Use in the real world!

Over the past year, the AQuA team has created a series of private beta prototypes and reports integrating its AI-driven quality metrics. These prototypes have been applied to analyze real world translations in 5 translations from Middle East/North Africa (MENA), Mexico and Southeast Asia.

A figure from one of these reports is included below. This report confirms (with AQuA) the state of a draft prior to and after review, where one can see AQuA-generated quality warnings addressed during the review process. This also confirms that the review process is improving quality (with respect to accuracy in this case) and that AQuA is generating warnings aligned with the human quality assessment process.

Plans for scaling up

Although there are still more experiments and fine-tuning needed, the private beta usage of AQuA tools and real world trials have confirmed viability. In the near future, the AQuA team will:

  • Further develop and release:
    • A standalone AQuA application that allows users to upload Bible revisions/drafts for analysis via AQuA generated quality information; 
    • An open and accessible AQuA “component,” where this component enables developers to integrate AQuA into existing applications like Scripture Forge and Paratext ; and
    • A prototype Oral Bible Translation (OBT) quality assessment interface that demonstrates the applicability of AQuA methods to audio recordings. 
  • Leverage the determined best practices around the use of this technology to develop/deploy AQuA training materials on and other relevant community sites.
  • Continue to develop new and different methodologies to assess text and oral Bible translations, such that the AQuA suite of “diagnostics” advances with the rapidly changing technology landscape and better represents the variety of quality information for the translator or consultant