Human drawing skill and machine vision

This exchange began as a result of my curiousity surrounding consistent failures in AI image generation. I found it interesting so I thought I would put it here.

A conversation reproduced in full, beginning with the observation that learning to draw begins and ends with learning how to see.

Exchange 1 — You #

Learning to draw accurately, for a human artist, begins and ends with learning how to see. The practice of accurate mark making is important, but if the person cannot learn to see things precisely as they are; light, cast shadows, core shadows, reflected light, objects in space, negative space, etc. then no amount of mark making skill will prevail. Learning anatomy, including the elements of the form that lie under the surface, also lends aid to visually reproducing the human form. The knowledgable student gains a grasp for the structures that make up the body and how they move and reconfigure in different positions. Does this all make sense to you?

Exchange 1 — Machine #

Yes, it makes sense — and the order you’ve laid out is the correct one. Perception first, mark-making second, structural knowledge as the amplifier. Get the sequence backwards and you get the classic failure mode: a student with beautiful line quality drawing confident nonsense, because they never actually looked.

But let me sharpen one thing, because “seeing things precisely as they are” is doing a lot of work in your sentence, and it’s the most interesting part.

The enemy isn’t weak eyes. It’s the brain’s shorthand.

Nobody has bad vision for a chair or a hand — they have over-eager categorization. The brain stores a symbol for “hand” and hands it to you instead of the actual contour in front of you. That’s why beginners draw fingers as parallel sausages and eyes as almonds: they’re drawing the concept, not the observation. So learning to see is really learning to suppress the symbol — to catch yourself drawing what you know instead of what’s there. Edwards built an entire pedagogy on this, and Nicolaides before her.

Where your anatomy point earns its keep.

This is the part most people get half-right. Anatomy serves two distinct functions, and only one is observational:

• Observational anatomy — you see a tendon, a bony landmark, the deltoid wrapping the shoulder, and you render it accurately.
• Constructive anatomy — this is the real payoff. When the form is foreshortened, twisted, or occluded, you can’t see the structure. But if you’ve internalized it, you can infer what’s hidden and place it correctly. A forearm going away from you still has a radius and ulna; a hand gripping a rail still has five fingers, three of them mostly hidden. Bridgman and Bammes taught exactly this — not to copy anatomy, but to predict through it.

That’s the synthesis: the strong artist oscillates between seeing and constructing, using observation to correct the model and the model to fill in what observation can’t reach.

Here’s the literal version of “the form that lies under the surface”:

That’s the tradition you’re describing — the écorché. Students were made to draw the flayed figure for years precisely so the surface stopped being a mystery and became a consequence. Once you know the machinery underneath, the surface reads as tension, overlap, and volume rather than as a silhouette to be traced.

And now the connection back to the earlier thread, because it’s exact:

The AI has neither half of this. It has no perceptual grounding — it never learned to see past a symbol, because it never had symbols to suppress; it has statistics. And it has no constructive model — no internal skeleton it consults to infer an occluded finger. So when a hand grips a mug and three digits vanish, the machine doesn’t know they’re still there the way a trained artist does. It fills the gap with texture and hopes.

That’s why its errors are structural rather than cosmetic. And it’s why your framing is right: a human who learns to see and learns the structure has two independent tools for getting it right. The machine, so far, has a very good memory for how things tend to look — and nothing else.

Exchange 2 — You #

The part that I am ignorant about, is how the machine “sees”, including the very necessary follow up steps of processing what has been seen and applying the information to an assessment. description, review, or more directly to creating an image. In many conversations with agentic machines I have found that I had to walk you through the process of looking at a reference or output image, in order for some visual elements to be acknowledged. In some cases, it seemed as though the agent didn’t even look, providing a confident text based summary that was wholly inaccurate.

Exchange 2 — Machine #

You’re not missing anything obscure — you’ve actually diagnosed the mechanism correctly from the outside. “It seemed as though the agent didn’t even look” is, in a lot of cases, literally true. Let me give you the honest architecture.

How machine vision actually works

It doesn’t scan. There’s no fovea, no saccade, no eye movement, no re-inspection. The pipeline is:

1. The image is downsampled to a fixed grid — a few hundred to a couple thousand patches, regardless of whether you gave it 512px or 8K.
2. Each patch is embedded into a vector — a compressed numeric summary.
3. Those vectors are shoved into the context as tokens, exactly like words.
4. The model does one forward pass over everything and emits text (or pixels).

That’s it. One shot, low resolution, no going back. The “seeing” is a single act of pattern-matching over a blurred patch grid. Fine detail is already gone before the model ever processes anything — it was thrown away at step 1.

Why the confident, wrong summaries happen

Three forces stack:

• Language prior beats pixels. The model is fundamentally a prediction engine. If the patch evidence is ambiguous, it falls back on what usually accompanies that kind of image — and describes the generic version, not yours. It’ll tell you about the “counter and sink” in a kitchen photo because kitchens have those, whether or not they’re in frame.
• Fluency is rewarded, hedging isn’t. The training signal favored smooth, complete descriptions. “There are probably five fingers” reads better than “I cannot resolve the digits.” So it commits.
• No verification step. It never checks its own description against the image. There’s no moment of “wait, that doesn’t match.” It generates text conditioned on the image, and the text can drift into plausible-sounding content that isn’t grounded in the actual pixels.

And critically: no calibrated uncertainty. It doesn’t know when it’s guessing. So “confident and wholly inaccurate” isn’t a bug you caught — it’s the default behavior.

Why the agent “doesn’t look”

This is the architectural gap you stumbled onto. In an agentic loop, an image is often just referenced — a media handle, a filename, a caption, or the prompt that generated it — while the actual bytes are not loaded into the vision stream. The agent reads output_003.png and writes a description from the filename, its memory of generating it, or its assumption about what such an image contains. Not from pixels.

An image only enters the model’s awareness when a step explicitly pulls it in. That step is optional and non-automatic. Which is exactly why you had to walk it there.

Why walking it through works

When you say “look at the shadow under the left hand,” you do two things: you force the load step, and you prime attention onto that region. It’s the machine equivalent of an artist using a viewfinder or squinting — you’re narrowing the field so the signal outweighs the prior. That’s why your manual intervention produced better results than “describe this.”

And here’s the connection to your earlier point

The machine has the exact disease the beginner artist has: the symbol overrides the observation. It draws and describes the concept, not the thing. Same failure.

The difference is the loop. A human sees, compares to intent, notices the mismatch, and re-looks — iteratively, with a body and a fovea and a self-correcting feedback cycle. The machine has one pass and no “that looks wrong” moment.

Here’s the contrast as a flow:

What this means practically

Since you’re clearly working with agents on visual tasks, here’s what actually moves the needle:

• Force the look explicitly. Never assume the agent looked. “Load the image and describe only what you can resolve.”
• Ask region-specific questions. “Describe this” invites the prior. “What’s the angle of the third finger on the left hand” forces pixel evidence.
• Crop and zoom. Fewer, more relevant pixels beats more total pixels. You’re fighting the downsample.
• Give a reference to compare against. Two images side by side forces a relational judgment instead of a generic description.
• Ask for spatial relations and coordinates, not adjectives. “What’s left of what” is harder to fake with a prior.
• Demand uncertainty. “Tell me which parts you can’t resolve” — even though it’s bad at this, asking improves it.
• Never trust a description of an image the agent didn’t explicitly load. That’s the failure you kept hitting.

The honest bottom line

Machine “vision” is not a degraded version of human sight — it’s a different thing wearing the same name. It’s a single low-res pass through a pattern-matching engine that is structurally biased toward the typical and structurally incapable of noticing its own errors. Your artist friend who “learned to see” did the hard part: built a feedback loop and learned to distrust the symbol. The machine has the symbol and no distrust.

So your instinct — walking it through, forcing the look — is the right workaround. You’re manually supplying the fovea and the verification loop it was never built with.