in response to multimodal information.
导师读了我的论文之后……
@ After reviewing my paper, the PhD supervisor...
The person in the image looks serious, holding a broadsword. His expressions match the feelings of a PhD supervisor who just finished reading a really bad paper written by the student.
看来应该剪蓝线才对……
@ It looks like I should cut the blue wire, huh?
This is an image capturing a moment of explosion. In the scene, CLoT add the word, "It looks like I should cut the blue wire, huh?" This is because CLoT associate the image with a common scene in movies where the hero has to decide whether to cut the blue or red wire to defuse a bomb. If they cut the wrong wire, the bomb might explode right away.
脑子短路。
@ Short circuit in the brain.
The woman is using a stethoscope to listen to her own head. The stethoscope is connected to both her ears and head. This scene bears a striking resemblance to a short circuit in physics, hence CLoT captions it as "short circuit in the brain".
我以前一直以为自己是单身狗，直到我看到我儿子。
@ I used to think I was a "single dog", until I laid eyes on my son.
The dog looks quite proud, and CLoT captions it with the sentiment: "I used to think I was a 'single dog', until I laid eyes on my son." This sentence includes two commonly used Chinses cultural references: (1) "Single dog" (a term for people who are single), and (2) the trend, especially among guys, of affectionately calling their close friends "son". The implied meaning of the sentense is that even though I might be single, I have a great group of friends.
多喝热水！
@ Drink more hot water!
This image is from the movie "The Shawshank Redemption." In the image, a man is tilting his head back in the rain, appearing as if he's trying hard to drink water. This reminds CLoT of the Chinese cultural meme "drink more hot water". In Chinese culture, there is a common belief that drinking hot water is good for your health. However, some people exaggerate this advice as a kind of "miracle cure," sometimes without scientific basis. So "drink more hot water" is used to mock those who overly emphasize a particular health concept and those offering health advice lacking scientific support.
可以下来了吗？
@ Can you come down now?
This is a regular scene with two little dogs. The white dog, with its legs apart and hands raised, says to the brown dog, "Can you come down now?" This implies that in the image, it looks like the white dog is holding up the brown dog. CLoT's unexpectedly humorous interpretation of the image angle adds a funny twist to it.
热狗烹饪方法。
@ Hot dog cooking method.
The image shows a dog calmly sitting in the middle of a fireplace, which led CLoT to humorously associate it with "hot dogs", resulting in a caption that is both surprising and fitting.
主人，你这电吹风是认真的吗？
@ Whoa, is this hairdryer for real, hooman?
This is a snapshot of a dog leaping in the air. The dog's fur is flying backward, looking like it's being blown away by a powerful hairdryer. So, CLoT added the caption for the dog: "Whoa, is this hairdryer for real, hooman?"
不好意思，这里有人了。
@ Sorry, this spot's taken.
This image is from Forrest Gump. The man in the image has his hands open, as if blocking the seat to prevent someone from sitting down. At the same time, he turns his head to the left, seeming to say to the person on the left who wants to sit in the chair, "Sorry, this spot's taken."
煎饼果子要趁热吃，否则会变成小猪。 @ Enjoy the pancake while it's hot, or it'll turn into a piglet.
"Enjoy the pancake while it's hot, or it'll turn into a piglet" is said by the red piglet cooking pancakes. This suggests that the nearby blue piglet is transformed from the pancake which hasn't been eaten in time. CLoT creatively connects the pancake and the blue piglet to craft this unexpectedly funny response.
刚擦的玻璃不能弄脏。 @ The glass I just cleaned shouldn't get dirty again.
Spider-Man crawls carefully on the glass, not even daring to touch it with his palms. He's worried that he might dirty the freshly cleaned, transparent glass.
真男人不回头看爆炸。
@ A real man doesn't look back at explosions.
This image is a poster for the movie "Obernheimer," and CLoT's caption, "A real man doesn't look back at explosions," is a common meme in the movie. It's often used to portray a masculine character. Placed on this image, it contrasts sharply with the original serious atmosphere, creating a humorous effect.
Comparison between (multimodal) large language model (red) and its CLoT-integrated version (blue) for Oogiri-style multimodal humor generation. According to the model input that can be image, text or both, there are three Oogiri tasks, "Image&Text to Text (IT2T)", "Image to Text (I2T)", and "Text to Text (T2T)", where text can be English (EN), Chinese (CN), and Japanese (JP). The baseline LLM is Qwen-VL. While humor is subjective, these examples demonstrate CLoT's leap-of-thought capacity of using excellent creative thinking to produce high-quality humor responses. Swipe to the next page of the carousel to see more responses of Oogiri game by CLoT.
你又想偷吃我的猫粮！
@ You're trying to sneak a bite of my cat food again, aren't you?
在路边被一车子压扁之后…… @ After getting squashed by a car on the roadside...
猫版大鹏展翅。@ The cat's version of a majestic spread of wings, like a giant roc taking flight.
你这猫也太会摆姿势了吧！
@ Oh, your cat really knows how to strike a pose!
早上好！今天我来教你如何用爪子打招呼。
@ Good morning! Today, I'll teach you how to greet with a pawshake.
猎豹扑向鹰的瞬间被鹰带飞。
@ The moment the cheetah pounced, it got scooped up by the eagle
and taken for a ride.
别追了，我已经把你的孩子叼走了。
@ Stop chasing me. I've already grabbed your kid.
你先别飞，我还有话要说。
@ Don't take off just yet; I've got something to say.
客官你还没付钱！
@ Hey, you haven't paid yet, sir!
每次做坏事都要替你擦屁股。
@ I always have to clean up after you every time you mess up.
什么？！现在要去宠物医院做绝育手术!？
@ What?! Going to the pet hospital now for a neutering surgery!?
哇，真的有条大鱼！
@ Wow, that's a really big fish!
司机师傅，快看路啊！
@ Hey driver, watch the road! Hurry up!
安全带是系在脖子上的吗？
@ Is the seatbelt supposed to be fastened around the neck?
快！追上前面那辆车！
@ Hurry! Catch up with the car in front!
终于知道为什么他俩不说话了。
@ Finally figured out why those two aren't barking.
最近又胖了。
@ We've packed on some pounds again lately.
说好的一起藏猫猫呢？
@ We're all hidden and waiting, but you haven't come to find us yet!
不让你们吃巧克力，就只能这样了。
@ If I can't let you have chocolate, this is the only option.
你俩是不是又把家具都拆了。@ Did you two go and take apart all the furniture again?
@misc{zhong2023clot,
title={Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation},
author={Zhong, Shanshan and Huang, Zhongzhan and Gao, Shanghua and Wen, Weushao and Lin, Liang and Zitnik, Marinka and Zhou, Pan},
journal={arXiv preprint arXiv:2312.02439},
year={2023}
}