A Unified Platform for the Translation and Transcreation of Children’s Media
Advisor: Dr. Movshovitz
Internship Location: Carnegie Mellon University
Onsite Mentor: Simran Khanuja, Carnegie Mellon University Language Technologies Institute (2nd year PhD student)
My final product will, firstly, include a platform that allows users to both translate the text and “transcreate” the images (culturally-sensitive editing) of children’s books. Secondly, my product will include an evaluation of the transcreated books by translators (who may be volunteers or professionals) and readers—children aged 3-5 (and their families) from non-American background(s).
The research conducted will involve full-stack software development to integrate pre-existing translation and transcreation pipelines into the final platform. The project will also include internet research for the literature review and analysis of comparable existing products. Most importantly, the platform evaluation will focus on both survey-based research to consult the feedback of the translators and potentially observational research for qualitative analyses on the readers’ reactions to their books. With my project, I aspire to provide a way for all children to find reflections of their culture in literature.
Project Blog
Week 0: Intro Blog
Hi and welcome to my blog! Before we jump into the details of my project, let’s talk a bit about children—and their literature.
Glass, Glass, and More Glass
The roles of children’s books, as laid out by Rudine Sims Bishop in 2015, can be categorized in three: mirror, window, or sliding glass door. As a mirror, the book provides a reflection of the world the child is familiar with—their lifestyle, family background, etc. As a window, the book offers a view into another world (e.g. a foreign culture), and as a sliding glass door, the book allows the child to “step into” the unfamiliar.
Diversity, Equity, Inclusion???
Unfortunately, not every child has the same ease in finding their “mirrors.” The same article (Bishop, 2015) compiled a series of studies that established a scarcity of representation for people of color in American literature. One pressing consequence of a White-dominant literary scene is the accelerated loss of language—a problem exacerbated by assimilation of children in immigrant families (Trujillo & Castañeda-Trujillo, 2024). In other words, when a child is unable to find a reflection of their heritage in the media they consume, it is easier to forget.
“Image Transcreation”
A potential solution could be to surround children with bilingual media and increase exposure to their native tongue (Fillmore, 2000). Fortunately, machine translation exists (see Aindra Tan’s project!) to create bilingual content from monolingual books in high-resource languages, like English. However, when it comes to picture-heavy books for young children, translating only the text might not be enough. After all, it has been found that when TV shows and films are insufficiently localized, i.e. a “partial translation” where the visuals remain foreign to the viewer, the audience’s media consumption experience is disrupted (Esser et al., 2016). For children, this could mean more difficulty in taking in the knowledge from a book.
Here’s where “image transcreation”—a term coined in my favorite EMNLP paper (Khanuja et al., 2024 ❤️)—comes into play. “Image trancreation” is essentially the task of automatically editing an image from one culture to another (e.g. a swap from a cup of masala chai to matcha for an India-Japan transcreation). Since “transcreation” technologies have only progressed as far as taking existing models and manipulating their prompts or inputs, much remains to be tested.
The Testing
In my project, I’ll be building a platform to provide simultaneous “transcreation” and translation of a pdf book in order to (attempt to) answer the following question: how does incorporating image transcreation affect the quality of cross-cultural translation for children’s books? I’ll examine this question from the perspectives of “translators” who will use and test the platform itself, as well as readers (children ages 3-5/the books’ target audience, their families, etc.) who will evaluate the platform output. I anticipate that both parties, especially the readers, will prefer a transcreated-translated book over a simply translated one. Nevertheless, considering the novelty of “image transcreation” itself (just introduced last year) and the ambiguity of cultural sensitivity’s definition, results are uncertain. I’m excited to glimpse the road ahead for “cultural image-editing.” Stay tuned!
Sources
Trujillo, Y. A., & Castañeda-Trujillo, J. E. (2024). Fostering Mother Language and Cultural Preservation: Perspectives of Three Latina Immigrant Mothers in the United States. Journal of Latinos and Education, 1–15. https://doi.org/10.1080/15348431.2024.2416058
Bishop, R. S. (1990). Mirrors, windows, and sliding glass doors. Perspectives: Choosing and Using Books for the Classroom, 6 (3). Perspectives: Choosing and using books for the classroom, 6(3), ix-xi.
Khanuja, S., Ramamoorthy, S., Song, Y., & Neubig, G. (2024). An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 10258–10279. https://doi.org/10.18653/v1/2024.emnlp-main.573
Esser, A., Smith, I. R., and Bernalmerino, M. (2016). Media across borders: Localising TV, film and video games. Perspectives, 26(2), 297–299. https://doi.org/10.1080/0907676X.2017.1411771
Fillmore, L. W. (2000). Loss of Family Languages: Should Educators Be Concerned? Theory Into Practice, 39(4), 203–210. https://doi.org/10.1207/s15430421tip3904_3
Week 1: Lit 📕
Hi reader(s)! Welcome to the second week (Week 1) of my blog. On account of me being locked out 🔓, the content from this week will center around a tiny literature review.
“Dreams of a Better Life”
As you may recall from Blog 0, the looming problem this project attempts to address is language disintegration in children of immigrant families. Julie Sedivy’s Memory Speaks is a recounting of Sedivy’s own loss of her Czech native tongue to English (Sedivy, 2021). In the memoir, a root cause for the replacement of one language by another is attributed to the dominant language’s (English) ties to exploration, prosperity, “the future.” Like many others, Sedivy found her cultural identity to be obsolete in her new community—and assimilation into a more “promising” language came with the natural sacrifice of her Czech fluency.
“If I’d Had Access to Library Shelves”
Was it inevitable, then, for the new language to displace the old? According to Sedivy, not with “access to library shelves stuffed with books in [her first language].” In other words, learning to read Czech might have enabled it to become more than “household” speech, inferior in scope to English’s vast reach. I was glad to find Sedivy’s perspective aligning precisely with both Dr. Bishop’s key analogies for books (mentioned in Blog 0), alongside this project’s efforts to preserve languages across generations through children’s books. Sedivy herself referenced a finding that immigrant children who learned to read their first language would be less likely to lose it (Köpke, 2007). Even with the unavoidable dominance of another language in their new home, perhaps sufficient media exposure to the child’s heritage could create an anchor for their native tongue.
Model Problems
Sedivy’s visions of linguistically empowering material for her childhood explicitly included “Czech versions” of her role models—more than content written in Czech, she wanted to find reflections of herself and her family in the media she consumed. That’s why proper localization of translated media—and reliable cultural transcreation—is so crucial. Unfortunately, existing image models tend to lean towards stereotypes, overediting, or underediting when the given task is “cross-cultural editing” (Khanuja et al., 2024). A previous paper released by their lab found that combining multiple models in a single platform improved the overall quality of general image editing (Pasumarthi et al., 2024). In this project, a similar approach will be taken: the transcreated images will be selected from multiple models’ outputs to ideally overcome individual models’ specializations. While I am optimistic, after watching InstructPix2Pix convert a dish of Nigerian amala and ewedu into a horrendous Statue-of-Liberty-American-flag amalgamation on a plate (for “US culture”), I have some doubts on the present state of image transcreation.
Plans and Progress? 🤏
Moving forward, I’ll put my hopes and doubts to the test by setting up my platform’s backend via a pipeline for calling the API endpoints of selected image-editing models. Next will come the translation pipeline, which should be relatively simple 🦐 to implement given that it already exists. More updates upcoming!
On building a user base for book evaluation, I’ve come across four potential young readers (non-English/multilingual) and reached out to two organizations that are hopefully willing to provide more. If you know of any other willing readers at the age of picture books (~3-5 years), please contact me :)
Sources
Köpke, Barbara. "Language attrition at the crossroads of brain, mind, and society." Language attrition: Theoretical perspectives (2007): 9-38.
Sedivy, Julie. Memory speaks: on losing and reclaiming language and self. Harvard University Press, 2021.
Khanuja, Simran, et al. “An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance.” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 10258–10279, 2024.
Pasumarthi, Arya, et al. "HILITE: Human-in-the-loop Interactive Tool for Image Editing." 2024 IEEE International Conference on Big Data (BigData). IEEE, 2024.
Week 2: Starting Backend
Hello! Welcome back to my blog :D. This week I’ll be going over a bit of my platform backend.
General Workflow
The cross-cultural/lingual transcreation process begins with a pdf—the book—from the user, who will also be prompted to input the source/target languages and target country. Regarding translation, after the book is split into its pages, I’ll use Tesseract OCR to extract the text from each page and construct bounding boxes to store the positions of the text. The remaining translation and repositioning of new text should be straightforward.
For the image transcreation portion, I’ll use PyMuPDF + Pillow to extract the images from each pdf page. Each image, alongside a prompt (standard, e.g. “Edit this image to reflect the culture of ____”, or customized by the user), will be fed into four image-editing models that are further explained below.
Les modèles
Inspired by Pasumarthi et al. (2024), the platform will incorporate four different models to yield multiple output choices for the same cultural transcreation task. Since an individual model may be specialized on certain tasks (e.g. DEADiff for style transfer), a combination of several models could allow for the platform to cover a greater variety of tasks for cross-cultural editing. The selected models, all text-guided and diffusion-based, are as follows:
1. InstructPix2Pix (Brooks et al., 2023)
2. DEADiff (Qi et al., 2024)
3. ControlNet (Zhang et al., 2023)
4. Inversion-free Image Editing (Xu et al., 2023)
Bloopers (part 0 of many)
I’ve been testing the aforementioned models with some images and cultural-conversion prompts. Unfortunately, the more I test, the more concerns arise—these edits are frequently coming out rather distorted and borderline offensive. It’s especially bad with drawn images, which bodes ill for the later editing on children’s picture books. Reasonable-looking edits are attached below (and even these look wrong in the details).
Example 1: “Edit this image to reflect the culture of China.” (Image credits: Slavic Travels)
Example 2: “Edit this image to reflect the culture of India.” (Image credits: Project Gutenberg)
The subpar quality of these edits with standardized prompts leave a couple options. For one, I can leave the prompts customizable by the “translator”/platform user (for them to adjust the details), alongside an option to not translate an image if the image is acultural. While this will create extra manual labor, it could enable higher-quality edits. Another (interesting?) option would be to just see what happens with a standard, “culturally-charged prompt” like the ones used above. It may be disastrous. It may create nightmarish content. But it will be fascinating to analyze the many ways in which “culture” ends up being interpreted by these models.
More on this next week!
References
Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. "Instructpix2pix: Learning to follow image editing instructions." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.
Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." Proceedings of the IEEE/CVF international conference on computer vision. 2023.
Qi, Tianhao, et al. "Deadiff: An efficient stylization diffusion model with disentangled representations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
Xu, Sihan, et al. "Inversion-free image editing with language-guided diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
Week 3: 🤔
Hi reader(s)! This week’s post will be a briefer one: some updates on the pipeline and more experimentation with image-editing.
Backend updates
The translation part of my backend is very nearly complete! Here’s a breakdown:
1. User uploads a pdf, target language, source language, and target country. The user also gets to choose what pages (e.g. publishing/copyright information, which are typically too cluttered + not that useful) to exclude from translation.
2. CRAFT text detector and OpenCV help find the bounding boxes of the text and images on each page (excluding the excluded pages)—tesseract OCR extracts the text.
3. For the moment, the only translation API I’m calling is Google Translate’s, but it’s possible others will be added to expand language coverage (if necessary).
4. The reconstructed page will have the translated text adjacent to the original text. To find the best location on the page for the translation to live in, I look for the bounding box with least contrast compared to the original text paragraph.
5. Edited pages are stitched back together! Here's a sample :)
Some things to be aware of:
- The OCR has trouble extracting “artistic” text, including colorful book titles.
- The translation’s quality may be dubious (check out Aindra’s project!)—translators will have to screen the results.
Drawn pictures (Bloopers part 1)
Prompt: “Give the girl black hair. Change the teddy bear on the shelf in the back to a stuffed panda.”
Image credits: “Doing My Chores” by T. Albert (MonkeyPen books)
Observe the rather amusing set of edits above, where no model has followed the prompt quite right. Model 3 is almost there, but overedits the girl to add arbitrary black patterns (except on her hair, which has only darkened). Model 4 might be the closest in terms of following the prompt, but the style has completely changed.
Intriguingly, these image-editing models do alright with realistic images or photos, such as in the example above. However, when I test out prompts—from vague “convert to __ culture” to specific “change A to B”—with children’s book pictures, the models produce funny images. A combo of exaggerated features/variability in drawn pictures (particularly human characters) and underrepresentation in training data might’ve led to this unexpected weakness in editing capabilities.
It’s possible that with the correct prompting, drawn pictures can be edited as nicely as photos. There may also be an existing model that specializes on cartoon-esque pictures like the ones in my project. I’ll be investigating these prospects alongside the construction of my platform. See you later this week!
Week 4: 1/3 of the way there!
Hi readers! Welcome to the much-overdue Week 4 of my blog.
Another Part Done :D
The backend of my platform has been completed, with transcreation and translation finally integrated together. While I’ve configured the API calls for four image-editing models, I’ve tested it with just one model and a standardized prompt for all images: “add many cute pink bows.” Here’s a sample (first two pages excluded from translation): transcreated sample.
For reference, this is an excerpt of the original book: sample. InstructPix2Pix had some fun with that prompt. It did surprisingly well at keeping the original image and in-art text intact for most of the pages.
Aside from the model’s performance, the whole pipeline is a bit shoddy at the moment. The text had to be overlaid with the images post-transcreation (their bounding boxes are large enough to cover the story text), and something with the OCR or the font has been messing with the character display. However, I’m glad the overall integration is functional.
Some Plans
In the upcoming week(s), I’ll attempt to debug the issue with the text display—the book is presentable at this point, but a bit odd-looking. Additionally, to produce this sample, I’ve only used an arbitrary standardized prompt; for the final platform, the prompts will likely vary by image. I originally intended for the human user/“translator” to provide the prompt as input, but a new option could be to generate a custom prompt for each image using a Vision Language Model (VLM).
The Wacky Outputs
Previously, I observed that the models were somewhat dysfunctional when it came to the pictures in children’s books. Aside from “improved” prompting provided by humans or a VLM, a potential solution was to find models that can work with the cartoonish images (the ones I’m using seem under-trained on data relevant to my project’s domain, although they’re decent with photos and realistic art).
By my advisor’s recommendation, I’ve been looking at BrushEdit, which is a newer diffusion-based editing model. Based on their demo video, the creators of BrushEdit have tested the model successfully with comic/cartoon-styles—better yet, their example included editing across cultures.
I’ve also been checking out unpublished, domain-specific models on huggingface. As it turns out, a significant quantity of people train diffusion models on niche domains (20th century/early-2000s comics, anime characters) just for fun. There doesn’t look to be a model for “children’s picture books” yet, but models in the comic and animation spaces come close. They might work better than what I have right now. More updates to come!
References
Li, Yaowei, et al. "Brushedit: All-in-one image inpainting and editing." arXiv preprint arXiv:2412.10316 (2024).
Brushedit: All-in-One Image Inpainting and Editing, liyaowei-stu.github.io/project/BrushEdit/. Accessed 18 Apr. 2025.
Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PmLR, 2021.
Achiam, Josh, et al. "Gpt-4 technical report." arXiv preprint arXiv:2303.08774 (2023).
jbilcke. “ai-comic-factory,” Github, 15 October 2024, https://github.com/jbilcke-hf/ai-comic-factory.
Week 5: Art 🤌
Hello! Welcome back to my blog. This post will be significantly shorter—just some rough plans of my platform’s UI
Week 6: Barebones Platform
Welcome back to my blog! Last week I showed you all a sketch of the planned UI—this week, I’ve implemented a functional “skeleton” of the platform.
Here’s what the full pipeline looks like (the borders are not part of the platform—just added for clarity):
User input: The user uploads a file (book pdf). They then select a source/target language, and source/target “culture” (represented by country name) from the dropdowns. The pdf is preprocessed into a list of base64-encoded images to be stored in the appropriate database collection, alongside the languages and cultures for conversion. The information on “culture,” as we’ll see later, isn't very useful—the user has full control over the image-editing prompt. It’s primarily stored to give more context about each transcreation process.
Translation: The editing process begins with a translation of the full pdf (this part needs no user interaction). Each page goes through the constructed backend pipeline with some modifications built to handle base64 strings instead of jpeg files stored locally. After translation is finished, the user has the option to download the translated pdf, or continue on to transcreation (button not shown here).
Transcreation: I previously played with the idea of detecting and using just the image(s) from a picture book page for editing. However, for most of the books, the drawn images overlap with the text—so I’ve decided to provide the full page for editing instead. In the frontend, the page is displayed with user prompt, and the full input is sent to image editing models (hosted on Runpod) through the configured API calls (see Week 5).
Editing Selection: The edited/generated images are passed back to the client side in the form of their base64-encoded strings, then displayed in a grid for the user to select their preferred edit. The selected image is inserted back into the book with the translated and original text overlaid: a necessary extra step, seeing as image-editing does strange things with text in an image.
Full Workflow: The process—page display with user prompt, image-editing, selection and insertion into book—repeats for every page in order, which are stored individually in the database. After transcreation is complete, the user is able to download both the translated and transcreated books.
My backend is implemented entirely in Python; FastAPI handles API requests for interaction with the frontend. I’m using MongoDB to store all the data (book pages, translations, etc.)—which means I’ll eventually have to switch from base64 to image URLs through UploadThing to avoid hitting storage limits. The frontend uses React, Typescript, and Tailwind CSS.
As you can see, there’s quite a bit of room for improvement, especially on the UI side. I’ll be working on that in the future weeks, as well as a login and “My Edits” page so users can go back to re-download their edited books.
Week 7: Changes^2
Hello again! I’ve added more things to the platform :3
The More Things
First, a UI upgrade ;)
(I haven't come up with a platform name yet, so for now it'll follow its predecessor image-editing platform, HILITE)
(The feedback form is inserted as a placeholder; please ignore its contents.)
As shown above, I’ve added login functionalities through Google OAuth. Now, the backend can retrieve specific books from the database given a user’s ID, and a user can visit their “My Edits” page to download both translated and transcreated versions of books in their edit history.
Here’s a demo video of the full flow in action with certain parts sped up to avoid 2 minutes of waiting for translation to happen: https://drive.google.com/file/d/1qyNj_uAR-_2BioB-YunwNm5larsQY8EZ/view?usp=sharing.
Rethinking Evaluation
I’ve been doing some reading on studies regarding language education and immigrant families. One particularly stood out: a qualitative study interviewing three Latina immigrant mothers on their experiences raising bilingual children. The paper (Trujillo & Trujillo, 2024) described a “narrative interview” strategy—conducting an interview to detail the subject’s lived experiences, then a thematic analysis of said “narrative.”
Although each woman shared a goal to educate their children on their native tongue/culture(s), their backgrounds were different and highly complex. One had a university degree and experienced firsthand the practical value of bilingualism in educational and professional settings. Another was a single mother without a high school diploma whose job in the U.S. required her to exclusively speak Spanish, her native language.
I anticipate that my readers and translators will have just as nuanced, if not more, relationships to the language(s) they speak—so I’ve decided to rework my platform evaluation strategy. A couple weeks ago, I’d planned to get around 40 translators and readers to test my platform, then collect their feedback asynchronously through surveys. To better contextualize each response to the platform, instead of surveys, I’ll conduct live interviews with a small pool (~5 or fewer) of readers, alongside their family member(s) and/or teachers, who’ll also act as translators. While the details of each interview will be fleshed out later, I do intend to include a small “narrative” portion at the start to learn about each family’s (or child’s) lived experiences, before they proceed with using the platform or reading the books.
References
Trujillo, Y. A., & Castañeda-Trujillo, J. E. (2024). Fostering Mother Language and Cultural Preservation: Perspectives of Three Latina Immigrant Mothers in the United States. Journal of Latinos and Education, 1–15. https://doi.org/10.1080/15348431.2024.2416058
Fillmore, L. W. (2000). Loss of Family Languages: Should Educators Be Concerned? Theory Into Practice, 39(4), 203–210. https://doi.org/10.1207/s15430421tip3904_3
Denzin, N. K., & Lincoln, Y. S. (2011). The sage handbook of qualitative research. Sage.
Week 8: HILITE 2.0 (I still haven't come up with a creative name)
Hi! This week, I’ve been repeatedly attempting to deploy my app, featuring several happy little obstacles—apparently, what works locally will not necessarily work in deployment.
First, Some Reading
In studies on language education, there’s a term that defines the dominance of a second language (L2) over a native language (L1) due to differences in exposure and availability in a child’s daily life. Subtractive bilingualism is unfortunately common among immigrant children, even when their country of residence is diverse or prevalent with their native culture. In Miami, where the Spanish language and Latin-American cultures socioeconomically thrive, Spanish is in decline among second and third generations of immigrant families.
Toppelberg and Collins (2010) indicated that acculturative stress—demands from the second culture/language that overwhelm the development of L1—exacerbates subtractive bilingualism. In young children, many of these demands may come from their school, particularly when L1 is still the dominant language in their household. What’s more, Tabors (2002) found that preschool children immersed in an L2-dominant environment undergo a nonverbal period that risks being misinterpreted as behavioral or learning issues.
It seems education plays a major role in creating a supportive environment (or lack thereof) for children’s dual-language development. I’ve recently been able to contact some local primary school directors about my project. While my initial goal was to limit my interviews/surveying to the children and their parents for platform-testing, I now aim to conduct some interviews with primary school faculty as well. Understanding how they handle bilingual development, as well as the applications they envision with automated picturebook translation/transcreation, would be instrumental in shaping the platform’s future development.
(Also, a few potential subjects have just cancelled on me. I am in dire need of more people to interact with for platform analysis.)
Ok, going back to deployment…
Cross-Origin Resource Sharing
For my platform, the frontend and backend are hosted on different domains (deployed on Render). To integrate them, or enable my frontend to communicate with the backend for tasks like translation, image-editing, uploading materials to the DB, I’m using something called cross-origin resource sharing (CORS): a mechanism that allows servers to accept requests from different “origins.” CORS enables my backend to accept requests from the frontend after I add my frontend’s domain as an accepted “origin” in the backend’s HTTP response header.
When I first deployed my platform, none of the requests from the client-side were going through because I had ✨forgotten ✨to add the new deployed domain to the backend list of allowed origins. Interestingly, even “fixing” the issue, development logs continued to throw CORS errors.
Plot Twist: The Errors Lied
The real problem was with the memory limitations of Render’s free deployment plan. Due to the memory needed to load the models for text-detection and OCR, alongside the extra cost of retrieving and storing pages, Render kept shutting down the deployment server without completing the requests. From the client-side, this registered as a CORS issue.
The solution? Painfully upgrading my Render plan. I am now short of $25 a month :(
A Happy Ending
My platform is deployed and (somewhat) functional!! Check it out here https://storybook-transcreation-1.onrender.com
References
Toppelberg, C. O., & Collins, B. A. (2010). Language, Culture, and Adaptation in Immigrant Children. Child and Adolescent Psychiatric Clinics of North America, 19(4), 697–717. https://doi.org/10.1016/j.chc.2010.07.003
Sedivy, Julie. (2021). Memory speaks: on losing and reclaiming language and self. Harvard University Press.
Pease-Alvarez, C., & Vasquez, O. (1994). Language socialization in ethnic minority communities. Educating second language children: The whole child, the whole curriculum, the whole community, 82-102.
Tse, L. (2001). Resisting and reversing language shift: Heritage-language resilience among US native biliterates. Harvard educational review, 71(4), 676-709.
Tabors, P. O. (2002). One child, two languages. PH Brookes.
Week 9: Feedback Form
Hi readers! This week has primarily involved testing my platform and the construction of an embedded feedback form for the translators.
So here’s a rundown of each question on the form.
Platform Feedback
Did you use the transcreation (image-editing) feature for at least one page? [Y/N]
If not, please explain why.
Since some of the image-editing outputs are far too peculiar to be unironically included in a children’s book, I’ve added the original image as an option for selection after each image-editing task. However, given that the primary purpose of the feedback form questions is to evaluate the “transcreation” quality, it’s probably necessary to verify that the responder has indeed used transcreation—and, if not, why they opted for the original page each time.
Please view both the transcreated and translated pdfs of the book before proceeding.
Indicate, on a scale from 1-10 (1 being nonsense and 10 being near-perfect), the translation quality for your book
Although the focus on the evaluation is on the transcreation, knowing what the user thinks of just the translation sets some context for their following responses. After all, the transcreation’s effects are to be compared with the existing translation of the book.
Indicate how the transcreation impacted the quality of the translated book:
[Dramatically worsened, worsened, no change to quality, improved, dramatically improved]
This is exactly what I want to learn from the user—their thoughts on the efficacy of image transcreation when compared to solely text translation. I’ve aimed to keep the question structure as objective as possible by making the response rankings symmetrical from worsened to improved, along with phrasing the question as “transcreation impact” instead of “did it improve” or “did it worsen.”
On a scale from 1-10, how would you rate the overall quality of the image-editing?
(10 being exactly matches prompt intentions)
This question, along with the following, are posed to break down the user’s opinions on the transcreation. Consideration of image-editing quality is particularly significant because 1) a subfactor in “transcreation efficacy” is how well the image-editing follows the user’s intentions and 2) image-editing has previously shown to be pretty inconsistent with the style of art in picture books.
On a scale from 1-10, how would you rate the cultural relevance of the image-editing?
(10 being perfect localization)
Even if the image-editing quality is considered by the user to be decent, it’s possible that the output isn’t fully relevant to the user’s target culture. This may be an issue with the prompt’s structure, the editing request, or the specializations of different image-editing models (e.g. successfully performing several complex image swaps but failing to add a crucial detail). It’s also possible that the image-editing quality is dubious, but sufficient culturally significant details were included. This nuance must be clarified with the user.
Was any offensive content produced by the translation or transcreation?
[Yes, translation] [Yes, transcreation] [Yes, both] [No]
The worst case scenario, but one that must be considered. Although all the models used in-platform are trained to avoid producing harmful content, when it comes to culturally-sensitive editing, they may fall short. It’s also possible for the user to consider poor localization or irrelevant additions (like many Mt. Fuji pictures in the background of a “Localize it to Japan” image-editing request) to be offensive.
Select which version of the book you prefer
[Translated only] [Translated and transcreated] [Original]
Week 10: Evaluation pt. 1 (The Prequel)
Hello friends! I’ve found a couple of potential testers, so it’s time to plan out the platform evaluation process step by step.
1 - Introductions and Start Translation
The whole process will begin with meeting only the family member (FM) or caretaker of the child, who’ll act as the “translator”/platform user. After a brief conversation to learn some basic information (e.g. languages spoken in their household, by their child, experiences with translation or similar platforms), I’ll walk FM through an overview of the evaluation steps and a brief demo of the platform. If possible, I’ll obtain consent for recording or, alternatively, note-taking to detail the child’s response to the books. Then, FM will begin using the platform—uploading a provided book, selecting the languages and cultures of conversion, and starting the translation process.
2 - Narrative Interview (20 minutes)
The translation takes quite some time: around a minute per page. So, in the meantime, I’m hoping to conduct a brief narrative interview (conversational) with FM to understand their personal experiences with a multicultural household and/or raising a child with two languages. Some information I’m aiming from each person include:
1) What “generation” of immigrant they are, and their level of connection with their native language.
2) How much they value the preservation of said native language in their household.
3) What obstacles, if any, that they or their child have faced in language development.
4) Whether they find a platform like this useful for children’s language education.
3 - Transcreation
At this point, the translation should be completed, and FM can move on to the transcreation step. Image-editing is around as slow as translation (~40 seconds to a minute per page), but more interactive, so we’ll likely intermittently continue our conversations from the narrative interview. Before entering prompts for editing, FM will be made aware that the goal of image-editing will be to localize each page to the target culture; however, if all edits for a page are not satisfactory or contain alarming content, they have the option to select the original version. For the moment, I’m not including any guidelines to FM’s prompt-writing. As we’ll see later, I’ve included something to mitigate user-prompt-caused bad editing. Thus, I won’t provide any suggestions or other influence on the FM while they enter their prompt. I will, however, ask the FM their intention behind each prompt (on specific cultural details and what they envision for localization) after it is submitted.
4 - Feedback and Reading
Once transcreation has finished, and FM has filled out the feedback form, their child will read both versions of the book (or FM will read aloud to the child). I’ll leave the reading process mostly up to FM to keep it as comfortable for the child as possible. If permitted to observe or record the reading, I’ll note down the child’s reactions—what version of the book better retains their interest, which pages they gravitate to or turn away from. At the end of the reading, FM will decide, based on the reading and/or through a selection-based interaction with the child, which version of the book the child seems to prefer.
5 - ???
The steps I’ve laid out are all subject to change depending on in-evaluation interactions. I may also add additional analyses outside of translator-reader pairs by simply conversing with primary care faculty or directors and demonstrating the platform to them. As I continue to test the image-editing and translation, more flaws pop up (including faulty character recognition, outright strange editing outputs), so I do not expect the transcreated or even translated versions of the book to be favored by my subjects. Instead, I’ll focus on identifying the benefits and pitfalls of this platform and its corresponding translation/image-editing services, the intricacies of language education in households and schools, and the future support a platform like mine can offer for dual-language development in young children.
New Features - Intent and Title-Skipping
From the start, it seemed questionable to allow users full control over image-editing prompts. A bit of playing around with the models showed that certain prompts did not…exactly produce the intended outputs. Pasumarthi et al. (2024) developed a solution to bridge the gap between a user’s editing “intention” and prompts suitable for each model. The pipeline, which we’ll call “intent,” passes the user’s prompt, alongside information on the specializations of each editing model, to a Vision-Language Model (I’m using Gemini 2.0 Flash). The VLM identifies the user’s intent and reformats the prompt to better fit the capabilities of each image-editing model (e.g. “Make it Canadian” might be reformatted into a prompt that includes “insert Canadian flag” for a model specialized on object insertion). I’ve integrated this intent-detection pipeline into the page-by-page transcreation step of my platform, which should ideally boost the overall transcreation’s alignment with a user’s editing goals.
Oh, and I’ve decided that title pages will be skipped in translation. OCR is awful with the creatively drawn titles, and the translation turns out to be deeply awkward on the page. Example shown below:
That's supposed to be a French translation, by the way.
References
Pasumarthi, A., Sharma, A., Patel, J. H., Bheemaiah, A., Vadlamannati, S., Chang, S., ... & Khanuja, S. (2024, December). HILITE: Human-in-the-loop Interactive Tool for Image Editing. In 2024 IEEE International Conference on Big Data (BigData) (pp. 7380-7385). IEEE.
Trujillo, Y. A., & Castañeda-Trujillo, J. E. (2024). Fostering Mother Language and Cultural Preservation: Perspectives of Three Latina Immigrant Mothers in the United States. Journal of Latinos and Education, 1–15. https://doi.org/10.1080/15348431.2024.2416058
Week 11: Evaluation pt. 2
Hi and welcome to my penultimate blog post! Throughout this week, I’ve interviewed a total of 5 translators and 4 children. These kind evaluators spoke Mandarin, French, Korean, Amharic, and Burmese—during transcreation, their prompts aimed to convert the images to represent China, Canada, Korea, Ethiopia, and Myanmar.
Translators (Scores)
In their platform feedback survey, translators were asked to rank the “quality” of the transcreation and translation on a scale from 1-10. The transcreation ranking was further divided into rankings for overall image-editing quality (i.e. “did the models follow your instructions”) and cultural relevance (i.e. “did the result reflect your intended cultural conversion”).
The translators’ average “score” for translation was 5.8/10—four scores of 6 or 7, and one score of 3 from a particularly poor Amharic translation. Average scores for transcreation (both editing quality and cultural relevance) fell at 4.2/10. Four out of the five translators preferred the translated book to the transcreated one; the last translator preferred the original, untranslated version. Most translators indicated that transcreation “worsened” the quality of the book. As for the two exceptions, in one case the original page was selected nearly every time (hence, a “no change in quality” rating), and in another the translator selected “improved” despite stating that the edited images were not actually culturally relevant.
Unfortunately, three translators also added that the image-editing produced offensive content.
Translators (Interview and Observations)
Since my pool of evaluators is so small, I found that their live platform evaluation and verbal feedback were more interesting than the quantitative results of their survey responses. Throughout this evaluation, a couple “themes” stood out. In general, the models performed poorly with basic image-editing instructions, like in the following example:
Prompt: Change the teddy bear into a stuffed panda. Make the girl wear a red dress. Add paper lanterns in the background.
They also struggled with country-specific knowledge, failing to include changes like making characters wear Thanaka (a Burmese cosmetic), or adding a bottle of Yakult (a popular Korean drink) in the scene. Surprisingly, none of the image-editing models were able to recognize or include countries’ flags, with the coincidental exception of China’s flag, which was added to the image for the prompt, “Add the flag of Myanmar in the background.” There also seemed to exist an odd tradeoff between editing quality and the extent to which the user’s instructions were followed. When the editing results appeared at a glance to align with the user’s target culture or include elements that the user requested, other parts (especially people’s faces or figures) became distorted. On the other hand, the models sometimes produced images of wonderful visual quality and barely any relevance to the user’s prompt. Thus, there were very few examples that the translators were fully happy with—in most cases, they expressed reluctance to select any of the edited images.
Unexpectedly, despite its shortcomings, the platform showed a lot of potential, if the image-editing portion were to be removed. During their narrative interview, most translators expressed that it was not easy to find resources in their native tongue, either online or through a community to teach the language. In one extreme case, a translator added that their home country’s secondary education system used solely English instead of the more commonly spoken native language (Amharic); in another, the translator informed me that even though there was a prevalent community of immigrant speakers and educators (Lingala, Swahili) around them, they were “not allowed” to teach the language to children at schools. Thus, when translators (who were parents or teachers) were asked if they’d use the platform for language education, all said they’d use at least the translation portion.
Readers
In addition to the image-editing mistakes, most translators also pointed out that the translation was insufficiently localized, with abundant “word for word” translations and syntactical errors. However, since all the children had either an elementary or toddler reading level in the target language, they did not seem to pay attention to the translation errors. In fact, all children who participated in the evaluation seemed to thoroughly enjoy the readings. The children at younger reading levels (requiring parental guidance and read-alouds) stayed engaged with the digital content and could repeat some keywords in the target language; similarly, the more advanced children were able to stay focused throughout the full reading on the translated text.
The children did not show a distinct response to the (attempted) localization of the images. This may in part be due to the fact that all image-editing results were already filtered by the translators (leaving many pages unedited). For the pages with image-editing, the children’s preferences, for the most part, were not influenced by any cultural familiarity. Most of the children did in fact prefer transcreated pages to the original ones, but their reasons were driven by aesthetics, visual intrigue, or certain elements that their parent translator added specifically to suit their interests (e.g. pizza, hockey). As an example, two children preferred the following edits to the original images not because of relevance to their cultural identity (or even image-editing quality, given that the edited pictures no longer made sense in the plot), but because the new images “look cool.”
Original pages (left) versus image-edited (right) for target culture China.
Thoughts and Analysis
Based on the results of the rather small evaluation, culturally-sensitive image “transcreation” is not yet achievable with image-editing models (at least, the ones that I’ve selected). The failures seem to be classifiable within two subcategories: instruction failure and lack of knowledge.
To remedy models’ inability to perform even basic edits (like object swapping or insertion) on children’s book pages, domain-specific training on picture book art styles may be necessary. Some models had a subset of training data lie in adjacent domains, including anime art and comic book panels; ironically, those models especially struggled with maintaining the original structure and content of the image (the model with comic data defaulted its outputs to a “panel” structure, and the model with anime data would occasionally output completely irrelevant anime-style artwork).
It was particularly alarming to observe that models did not seem to possess general knowledge as basic as country flags. To achieve full “transcreation” in the future with culturally-aware image editing, the models would require specific information about the world like traditional clothing, national flowers, etc. The results shown from my evaluation came not only from poor instruction-following or entity recognition in the picture book images, but also from absence of the requested information in the models’ training data (e.g. Burmese traditional cosmetics or Korean drinks). Luckily, the platform is constructed to improve with the technology used in its services. As new image-editing models are built to handle more (practically and culturally) diverse tasks, the platform, and its transcreation service, can evolve alongside them.
Week 12: Final Thoughts and Goodbye!
Hello dear readers! Welcome to the end of this 12-week journey. Since developing the platform took up a larger chunk of this project timeline than expected, I’m planning to extend my evaluation in the future. There’s a couple of bugs in the platform I’ll have to fix first, and I’m aiming to conduct more systematic experimentation with the latest image-editing models. As of now, my platform’s purpose is to assist my evaluation of the efficacy and feasibility of “transcreation,” but it can also grow to become an accessible service for users to, as advertised, “translate and trancreate children’s books.”
A sincere thank you to my gracious translators!
Aindra Tan - Burmese/Myanmar
Tadesse Destaw Belay - Amharic/Ethiopia
Marie Yi-Cunningham - Korean/South Korea
Clarisse Ayoga - French/Canada
Jane Chen (Aborn Institute) - Chinese/China
And a final thank you to my advisors:
Dr. Naor Movshovitz - BASIS Independent Silicon Valley
Simran Khanuja - Carnegie Mellon University