Week 10: Evaluation pt. 1 (The Prequel)

Hello friends! I’ve found a couple of potential testers, so it’s time to plan out the platform evaluation process step by step.

1 – Introductions and Start Translation

The whole process will begin with meeting only the family member (FM) or caretaker of the child, who’ll act as the “translator”/platform user. After a brief conversation to learn some basic information (e.g. languages spoken in their household, by their child, experiences with translation or similar platforms), I’ll walk FM through an overview of the evaluation steps and a brief demo of the platform. If possible, I’ll obtain consent for recording or, alternatively, note-taking to detail the child’s response to the books. Then, FM will begin using the platform—uploading a provided book, selecting the languages and cultures of conversion, and starting the translation process.

2 – Narrative Interview (20 minutes)

The translation takes quite some time: around a minute per page. So, in the meantime, I’m hoping to conduct a brief narrative interview (conversational) with FM to understand their personal experiences with a multicultural household and/or raising a child with two languages. Some information I’m aiming from each person include: 

1) What “generation” of immigrant they are, and their level of connection with their native language.

2) How much they value the preservation of said native language in their household. 

3) What obstacles, if any, that they or their child have faced in language development. 

4) Whether they find a platform like this useful for children’s language education. 

3 – Transcreation 

At this point, the translation should be completed, and FM can move on to the transcreation step. Image-editing is around as slow as translation (~40 seconds to a minute per page), but more interactive, so we’ll likely intermittently continue our conversations from the narrative interview. Before entering prompts for editing, FM will be made aware that the goal of image-editing will be to localize each page to the target culture; however, if all edits for a page are not satisfactory or contain alarming content, they have the option to select the original version. For the moment, I’m not including any guidelines to FM’s prompt-writing. As we’ll see later, I’ve included something to mitigate user-prompt-caused bad editing. Thus, I won’t provide any suggestions or other influence on the FM while they enter their prompt. I will, however, ask the FM their intention behind each prompt (on specific cultural details and what they envision for localization) after it is submitted. 

4 – Feedback and Reading 

Once transcreation has finished, and FM has filled out the feedback form, their child will read both versions of the book (or FM will read aloud to the child). I’ll leave the reading process mostly up to FM to keep it as comfortable for the child as possible. If permitted to observe or record the reading, I’ll note down the child’s reactions—what version of the book better retains their interest, which pages they gravitate to or turn away from. At the end of the reading, FM will decide, based on the reading and/or through a selection-based interaction with the child, which version of the book the child seems to prefer. 

5 – ??? 

The steps I’ve laid out are all subject to change depending on in-evaluation interactions. I may also add additional analyses outside of translator-reader pairs by simply conversing with primary care faculty or directors and demonstrating the platform to them. As I continue to test the image-editing and translation, more flaws pop up (including faulty character recognition, outright strange editing outputs), so I do not expect the transcreated or even translated versions of the book to be favored by my subjects. Instead, I’ll focus on identifying the benefits and pitfalls of this platform and its corresponding translation/image-editing services, the intricacies of language education in households and schools, and the future support a platform like mine can offer for dual-language development in young children. 

New Features – Intent and Title-Skipping

From the start, it seemed questionable to allow users full control over image-editing prompts. A bit of playing around with the models showed that certain prompts did not…exactly produce the intended outputs. Pasumarthi et al. (2024) developed a solution to bridge the gap between a user’s editing “intention” and prompts suitable for each model. The pipeline, which we’ll call “intent,” passes the user’s prompt, alongside information on the specializations of each editing model, to a Vision-Language Model (I’m using Gemini 2.0 Flash). The VLM identifies the user’s intent and reformats the prompt to better fit the capabilities of each image-editing model (e.g. “Make it Canadian” might be reformatted into a prompt that includes “insert Canadian flag” for a model specialized on object insertion). I’ve integrated this intent-detection pipeline into the page-by-page transcreation step of my platform, which should ideally boost the overall transcreation’s alignment with a user’s editing goals.

Oh, and I’ve decided that title pages will be skipped in translation. OCR is awful with the creatively drawn titles, and the translation turns out to be deeply awkward on the page. Example shown below:

That’s supposed to be a French translation, by the way.

References

Pasumarthi, A., Sharma, A., Patel, J. H., Bheemaiah, A., Vadlamannati, S., Chang, S., … & Khanuja, S. (2024, December). HILITE: Human-in-the-loop Interactive Tool for Image Editing. In 2024 IEEE International Conference on Big Data (BigData) (pp. 7380-7385). IEEE.

​​Trujillo, Y. A., & Castañeda-Trujillo, J. E. (2024). Fostering Mother Language and Cultural Preservation: Perspectives of Three Latina Immigrant Mothers in the United States. Journal of Latinos and Education, 1–15. https://doi.org/10.1080/15348431.2024.2416058