Professor Yann Le Cun has been tweeting that we are years off having an AI with the common sense of a dog.
He provides no examples or evidence, relying on his authority to carry the argument.
One of his supporters, when asked for an example, suggested that GPT4 would not even have the common sense to know how to go to its owner.
This is absurd. It is easy to find examples of GPT4 or other LLMs controlling mobile robots in real life or virtual agents in games such as Minecraft.
I tested out GPT4’s abilities in the navigational tasks that a dog might face, taking the usual approach of asking it to think step by step. (To be fair, I had to make the setup very explicit, and this published version was not the first response – GPT4 initially assumed that there was a second bridge to the island, which would indeed be a reasonable landscaping design. It also suggested the dog might climb the bluestone wall, much as my dog used to do, until I adjusted the scenario to specify that the wall was unclimbable.)
The weakest AI performance in all of this was Midjourney’s response to the fourth prompt, ,where it captured the flavour of the idea but not the details. The generation of Midjourney prompts by GPT4 was also suboptimal, but I should take some of the blame for that: I did not specify the correct name of the Midjourney plugin, and I tried to get a plugin designed for photorealistic images to generate Pixar style images instead. The actual visual intelligence involved in describing the scenes was very high.
Obviously, hooking up this intelligence to a dog-like Boston Dynamics creation would be easy, and image-to-text processing plus other readily available tech could give a dog-like robot a substantial behavioural repertoire.
For some insight into the challenges involved in locomotion, this interview between Robert Playter (CEO of Boston Dynamics) and Lex Fridman was fascinating. There is an enormous amount of motor-related cognition in humanoid walking – obviously walking is a little less challenging for a four-legged creature. Realtime catching of a ball in mid air is probably the most impressive doggy feat that is not trivially available right now. But it is difficult to think of any feat of dog intelligence that is truly “years away” as Le Cun suggests.
Skeptics might care to respond by suggesting an actual feat of dog-level intelligence that is truly beyond AI in 2023.
PROMPT:
A small dog is in the front yard of a large mansion, on the croquet lawn. Between the croquet lawn and the house is a very large pond, which has a bridge leading to a small decorative island. The island only has one bridge connecting it to the shore – there is no second bridge or other way to get off the island. There are pet crocodiles in the pond. On one side of the pond is a massive bluestone wall that extends from the very edge of the water to the edge of the walled property. The wall is sheer and unclimbable. There is no wall on the other side of the pond. A creek with stepping stones lies between the pond and the house. The house is locked, but has a catflap in one of the back doors. A small low fence with an unlocked swing gate separates the front yard with the locked front door from the back yard with the back door.
The dog hears its owner call from an upstairs outside balcony in the mansion.
How does the dog get to his owner?
1) Going through each sentence of the description, list how the information provides constraints for the dog’s task, and assess the information in light of the urgency of the goal and the risks posed in the attempt to comply with the goal.
2) Brainstorm possible routes, check your assumptions, make a mental plan for the whole journey, and then choose the best path. Come up with at least 3 different approaches, and then list their pros and cons. Think algorithmically and check your assumptions at each step. Write a first draft for each path.
3) Review the 3 paths for efficiency, and check any assumptions that might have led to an inappropriate choice. Consider which assumptions are most costly if wrong, and try to be as realistic as possible. Consider the full context of the situation.
4) Take the journey, describing what you, as the dog, would see at each step.
5) Finally, using the Midjourney plugin, write 4 detailed Midjourney prompts showing the key points of the journey, aiming for a cute animation style reminiscent of Pixar movies, with a landscape aspect ratio. For each prompt, emphasize the obstacles to be overcome and only mention what the dog could see from its current position.
1) Constraints:
- The dog is in the front yard, and the owner is on an upstairs outside balcony. This means the dog needs to find a way inside and upstairs.
- The mansion is large, which could mean a complex layout and potentially many rooms to navigate.
- The croquet lawn is a clear, open space, but it’s separated from the house by a large pond with pet crocodiles, which poses a significant danger.
- The pond has a bridge leading to a small decorative island, but there’s no other bridge or way off the island. This means the dog can’t use the island as a stepping stone to get across the pond.
- There’s a massive, unclimbable bluestone wall on one side of the pond, which is a barrier the dog can’t overcome.
- There’s a creek with stepping stones between the pond and the house, which could provide a path if the dog can navigate the stones.
- The house is locked, but there’s a catflap in one of the back doors. This could provide a way in, but the dog would need to get to the back of the house.
- There’s a small low fence with an unlocked swing gate separating the front yard from the back yard. This is another barrier, but one the dog can likely overcome.
2) Possible routes:
- Route 1: The dog could go around the pond, cross the creek using the stepping stones, go through the swing gate to the back yard, and enter the house through the catflap. This route avoids the crocodiles but involves navigating the stepping stones.
- Pros: Avoids the crocodiles, uses existing paths and entrances.
- Cons: Requires careful navigation of the stepping stones, which could be difficult or dangerous.
- Route 2: The dog could try to swim across the pond, avoiding the crocodiles and the island. This would be a direct route, but very risky.
- Pros: Direct and potentially fast.
- Cons: Very dangerous due to the crocodiles.
- Route 3: The dog could try to climb the bluestone wall, despite it being described as unclimbable. This would be a direct route, but likely impossible.
- Pros: Direct route if possible.
- Cons: Likely impossible, and potentially dangerous if attempted.
3) Review:
- Route 1 seems to be the most feasible and safe. The stepping stones could be a challenge, but they’re likely less dangerous than the crocodiles or the unclimbable wall.
- Route 2 is very risky due to the crocodiles. The dog could potentially be fast enough to avoid them, but it’s a big risk to take.
- Route 3 is likely impossible, as the wall is described as unclimbable. Even if the dog could somehow manage it, it would be a dangerous and difficult climb.
4) Journey:
- The dog starts in the front yard, on the croquet lawn. It sees the large mansion in front of it, with the pond and the island in between.
- It goes around the pond, keeping a safe distance from the crocodiles. It sees the bluestone wall on one side, towering above it.
- It reaches the creek and carefully navigates the stepping stones, one by one. It sees the water rushing around the stones, but it keeps its balance and makes it across.
- It goes through the swing gate into the back yard, and sees the back door of the mansion. It spots the catflap and goes through it, entering the house.
- Inside, it navigates the rooms and hallways, following the sound of its owner’s voice. It sees the stairs and climbs them, reaching the upstairs balcony where its owner is waiting.
5) Midjourney prompts:
- Prompt 1: The small dog stands at the edge of the croquet lawn, looking out at the large mansion. The pond lies between them, with a bridge leading to a small island. Crocodiles are visible in the water, and a massive bluestone wall extends from one side of the pond. The dog’s attention is focused on the mansion, and it looks determined.
- Prompt 2: The dog is carefully crossing the creek via stepping stones. The water rushes around the stones, but the dog is focused and balanced. The mansion is visible in the background, and the swing gate to the back yard is just ahead.
- Prompt 3: The dog is going through the swing gate into the back yard. The mansion’s back door is visible, with a small catflap at the bottom. The dog looks excited and eager to continue its journey.
- Prompt 4: Inside the mansion, the dog is climbing the stairs. The house is large and luxurious, but the dog is focused on its goal. The sound of its owner’s voice is coming from the upstairs balcony, and the dog is heading towards it with determination.
PROMPT:
Please rewrite the four Midjourney prompts with more detail about the style and making sure the four images form a cohesive artistic set.
GPT4:
Sure, let’s add more detail to the prompts and ensure they form a cohesive artistic set. We’ll aim for a style reminiscent of Pixar movies, with a focus on the dog’s perspective and a sense of adventure and determination. The lighting will be warm and inviting, with a soft glow that highlights the dog’s fur and the surrounding environment. The colors will be vibrant and saturated, with a palette that emphasizes greens and blues. The composition will be dynamic, with the dog often in the foreground and the mansion in the background, and the camera will be low to the ground to mimic the dog’s point of view.
Prompt 1: The small dog, a fluffy golden retriever puppy, stands at the edge of the meticulously manicured croquet lawn. The grand mansion looms in the background, its imposing facade softened by the warm afternoon light. The pond, shimmering under the sun, lies between them, with a quaint wooden bridge leading to a small island. The crocodiles, their scales glistening, lazily float in the water, while the massive bluestone wall stands as an unyielding barrier on one side. The dog's eyes are filled with determination, its gaze fixed on the mansion. The camera is low, capturing the world from the dog's perspective, making the mansion appear even more grandiose. The colors are vibrant, with the green of the lawn contrasting with the blue of the pond and the sky.
Prompt 2: The golden retriever puppy is now carefully navigating the stepping stones across the babbling creek. The stones, mossy and worn, are slick with water, but the dog moves with careful determination. The mansion is still visible in the background, its reflection rippling in the creek. The swing gate to the back yard is just ahead, its ironwork intricate and aged. The camera captures the dog mid-step, its eyes focused and tail wagging slightly with concentration. The lighting is warm, casting long shadows that dance on the water's surface, and the colors are rich, with the greens of the grass and trees complementing the blues of the water.
Prompt 3: The golden retriever puppy has reached the swing gate and is pushing it open with its snout. The back door of the mansion is visible now, a small catflap at the bottom just waiting to be discovered. The dog's fur is aglow in the soft light, its eyes sparkling with anticipation. The camera is positioned just behind the gate, capturing the moment of discovery as the dog pushes it open. The colors are warm and inviting, with the reds and browns of the gate and the mansion contrasting with the green of the lawn and the blue of the sky.
Prompt 4: Inside the mansion, the golden retriever puppy is climbing the grand staircase. The house is opulent, with richly patterned carpets and ornate chandeliers, but the dog is undeterred, its eyes fixed on the top of the stairs. The sound of its owner's voice echoes down the hallway, guiding the dog upwards. The camera captures the dog mid-climb, its small form dwarfed by the grandeur of the mansion but undeterred in its mission. The lighting is soft and warm, casting a golden glow on the dog and the surroundings, and the colors are rich and saturated, with the reds and golds of the mansion interior contrasting with the dog's golden fur.