The Valuable Dev

The Expert Blind Spot In Software Development

learning involves the working memory

Once upon a time, in the Fabulous Land of Web Shops Where Rivers Were Made of Honey, there was me, fresh and young PHP developer with little experience, and my team leader who was in the industry for 10 years.

It was late in the evening, and I was still at the office. We had to finish a backend for an important client. Quickly, as always. I was exhausted.

My team leader was full of passion, agitating a weird diagram full of boxes and arrows: “Here’s my idea. We should first create two interfaces, put two decorators here and maybe three factories there; we need the system to be flexible as much as we can, the client expect to keep his backend for 20 years. How, and what about abstracting this? It might not be DRY enough though.”

My knowledge in design pattern was pretty poor at the time. Even the difference between inheritance and composition wasn’t clear to me. I was looking at his schema like Champollion might have looked at the Rosetta Stone for the first time, asking myself many questions.

They were first flowing from my mouth: what’s a decorator? Why do we create interfaces? Why this factory, there? The explanations brought even more complex concepts I couldn’t really grasp. The stress of the impostor syndrome was beginning to kick in. I was a fraud, a newbie who wanted to play in the Gods’ playground.

My brain was boiling. Sweat was pearling on my forehead. My confusion was growing at each explanation given. It’s when the cleaning lady began to turn on her vacuum cleaner that I gave up. I said yes to everything and went home.

As you might expect, the project didn’t go well. My team leader blamed me because I didn’t ask enough questions for me to understand. The solution they found: organizing monthly meetings to point every single mistake I was making, often threatening to fire me if I couldn’t improve. It was a time when CI/CD, code review, and pair programming didn’t exist. I was alone. Ultimately, I crashed into the burnout wall.

Was I too stupid not to understand the confusing explanations of my team leader? I could have done better for sure, but I would never blame a beginner for not understanding my explanations.

I didn’t understand what was happening at the time, but I swear to the Twelve Gods that I would remember this experience not to reproduce it. I wanted to be a better mentor when it would be my turn to be the Senior Ninja Developer of the Wizardry Crown explaining magic tricks to my apprentices.

Years forward, a miracle happened. I stumbled upon the theory of the expert blind spot, finally explaining partly what happened on this cursed project.

Teaching is one of the best way for testing ourselves to see if we really understood what we’ve learned. That’s why teaching and learning are tightly linked. Teaching well will increase our understanding of the subject we want to master and, therefore, our level of expertise.

While we teach, we learn.

Seneca

But we need to understand and avoid the expert blind spot to teach well. This might begin to explain why, in my experience, good developers are not necessarily good teacher.

This article will specifically speak about teaching from a person to another. Things are a bit different when you try to learn by yourself with a book, for example. We’ll talk about:

  • What the expert blind spot is.
  • The theory of cognitive load.
  • How work understanding and learning.
  • Why the expert blind spot is a consequence of our learning process.
  • Actionable solutions to avoid this expert blind spot.

Let’s shed some light on the expert blind spot.

What’s the Expert Blind Spot?

Definition

Let’s first define what an expert is, to be sure we agree on the meaning of the key terms. From the Oxford Dictionary:

A person with special knowledge, skill, or training in something.

An expert is somebody who knows what he doesn’t know, as mentioned in the Dunning-Kruger effect. According to this theory, beginners have often a lot of confidence about their knowledge and skills because they are not aware about what they don’t know. As a result, they conclude that they have a lot of knowledge.

This is called illusion of competence.

The more we become skilled and knowledgeable, the more we discover how much you don’t know. It’s really useful to know what to focus on to increase our expertise.

We live on an island surrounded by a sea of ignorance. As our island of knowledge grows, so does the shore of our ignorance.

John Archibald Wheeler

Even if experts are more aware about what they don’t know, their awareness of the details of what they know decrease. It sounds pretty weird but it’s exactly what the expert blind spot is. They can still apply their knowledge to perform tasks, their only use a higher level of abstractions.

What’s the Problem of the Expert Blind Spot?

What’s the problem here? If experts can still use the knowledge they’re not entirely aware of in details, that’s good enough.

When expert developers explain their knowledge to beginners, they will brush away every detail they’ve abstracted. The learners, even if they need this hidden knowledge to understand and learn the whole concept, won’t be able to access it. They won’t understand.

The experts will have difficulties to understand why the beginners don’t understand. For them, the concept feels obvious. The learners, on the other side, won’t be able to ask the good questions either, since they’re not aware about what they don’t know. How to ask the good questions if you have no idea what kind of answer you want?

This brings us back to the lengthy story I was narrating in the introduction.

Even worse: beginners might misunderstand what the experts try to teach them, without even knowing it. The absence of knowledge is better than having misconceptions, because it’s harder to unlearn than to learn.

Speaking about mentoring...
If you want to step up your soft skills, you might be interested by the book I’m writing.

Why On Earth There Is Such a Thing as the Expert Blind Spots?

Studies on cognition show us why there’s such a thing as the expert blind spot. First, we need to see how cognitive learning works.

The Types of Memories

According to the cognitive load theory, we have two types of memory: the working memory and the long term memory.

The Working Memory

The working memory is very limited in space (we can only process 4 elements of information at the same time!) and time (around 20s). These numbers can change slightly from a person to another.

We can use this memory to:

  • Learn a new resource.
  • Perform a task.

As you can imagine, problems pop up when we try to learn too much at the same time. Our working memory can’t process everything simultaneously because of its limited space. Similarly, when we perform a task, we need to recall what we have in our long term memory (what we’ve learned) in the working memory to do that task, with the spatial constraints we saw above.

That’s why simplicity is so important in software development. It explains why people try to decompose their code into easy-to-understand modules, not to put too much in our working memory at once.

Learning in the working memory
Attempt to learn the Linux pipe

The Long Term Memory

When we learn a new resource using effective learning methods, what’s in our working memory will go in our long term memory. To do so, we need to understand the knowledge we’re trying to process.

To understand, we need to have every piece of information required in our working memory. From there, we’ll create schemata in our long term memory to organize the new knowledge. Later, we can use our new schemata to perform a task.

Understand in the working memory and long term memory
Understanding the Linux pipe thanks to new pieces of information and prior knowledge of the command-line.

Pieces of information and Chunking

We didn’t define what these pieces of information are. Each of them will occupy one spot in our working memory. The content itself will depend on what is learned and who’s learning it.

What’s in our long term memory are layers of schemata abstracting one another. When performing a task, we need to reduce the number of schemata we bring back to our working memory not to overload it.

Let’s take an example. Carl, your colleague developer, is running the shiny Ubuntu on his powerful laptop. He wants to know how many files beginning with a z are in a folder.

In my example, you are the famous command-line expert of the whole company. Naturally, because of your extreme empathy and your out-of-this-world generosity, you try to solve his problem.

“Hey Carl!” you begin in the most friendly tone ever heard in the company. “It’s super easy! All you have to do is a good old ls | grep "^z" | wc -l. You see, you first run ls, you pipe the output with grep, then with wc and voilà! Your dreams come true!”

But Carl doesn’t have your knowledge. Raised in the comfortable and boring GUI world, he doesn’t understand anything about your magical skills. He knows ls, but he doesn’t understand what a pipe is, and he has no clue about regular expressions. What about grep and wc? They’re mysteries to him.

Do you think he would understand your explanation? I’m doubtful too. When you type your command-line, you don’t have to remind yourself what a pipe is and how it works in details. You bring from your long term memory to your limited working memory the concept of the pipe and the commands.

These concepts are abstracted under the word pipe, wc, and grep for you. You don’t need to even think about it. The concept is automatically applied to the task at hand. Because of your expertise in the command line, you save space in your working memory. It reduces the mental energy when you need to perform that task too.

For Carl, things are different. To understand this command, he has to bring the concept of pipe into his working memory, what’s grep, what’s wc, how regular expressions work, and link all of that.

It’s a lot. Too much, even for a command which seems so trivial to you. Unfortunately, you’ve fallen into the expert blind spot trap. You assumed that everybody has the knowledge (or schemata) you have in your long term memory, without even realizing it.

I know what you might think. “Carl has to ask questions! What’s a pipe? What’s wc? What’s grep? This is what Dave should say!”.

You’re right, it can help. But nobody likes to look stupid, especially when you’re a beginner. He might ask one or two questions, but for the pipes he might as well try to make some assumptions on how it works; from there, misconceptions can quickly spread in his poor brain.

Another detail: you said to Carl that the task at hand was “super easy”. Super easy for you, but not for Dave. He might not want to look stupid by asking questions about something super easy.

I’m coming back to my little story in the article’s introduction: even if I was asking questions, the answers brought so many more foreign concepts that my working memory was totally overloaded. It was impossible to understand in that case.

Finally, as a beginner, Carl might not know that he doesn’t know. In that case, all the useful questions won’t even pop up in his brain.

What should we do to help the poor Carl?

Possible expert blind spot while expert explain the command line
In red: possible expert blind spots while an expert explains the command line.

How to Avoid the Expert Blind Spot

What’s the first step we should all take to solve a problem? Clearly stating what the problem is. This is what we just did.

A problem well stated is a problem half solved.

Charles Kettering

Now, we need to keep the concept of the expert blind spot in mind when we mentor somebody.

Simplifying the Learning Material

The cognitive load theory advise to lower the number of pieces of information in the teaching material. This approach is good when we teach beginners. After all, they don’t have a lot of knowledge on the topic, so they need first to create some foundations to build upon them and keep them in their long term memory.

This can be tricky to do. Simplification can lead us on the path of erroneous information, where begins the Terrible World of Misconceptions™.

To bring back my command-line example from above, I would have:

  1. Asked questions, to be sure that Carl has at least a basic understanding of the shell, during informal discussions if possible. More on that below.
  2. Explained the concept of pipe, and showed to our learner some simple examples.
  3. Given the solution.
  4. Asked to do other tasks with pipes, for our learner to put the concept in the long term memory.

The goal here to teach the concept of the Linux pipe instead of explaining every single details of the command. Knowing some CLIs like grep or wc is important too but it can wait. Carl doesn’t have to learn everything at once.

I would encourage him as well to use the command-line as much as he can, for him to discover its power and to practice what he learned, if he wants to. Being too pushy is rarely a good idea, we need to be patient and let people take action by themselves.

Let’s take another example. Some years ago, I was managing a remote team with a colleague. We understood very quickly that they didn’t have any experience with writing unit tests. We decided to teach them the basics:

  1. We picked a feature we had to implement in our codebase.
  2. We began to write functions and their unit tests, explaining at the same time our approach to the remote team.
  3. As the demonstration was progressing, we were asking questions: how would you test this method? What do we want to test, here? Is it a good idea to test this controller?
  4. Then, we reviewed thoroughly their unit tests during the next weeks. At the beginning, we didn’t try to correct everything, keeping in mind that multiple approaches were possible. Being too picky, at least at the beginning, would have discouraged them, confused them, and overloaded their working memory.

It’s important to practice in order to learn. It shouldn’t be too easy or to hard:

  • If it’s too easy, the learners won’t learn anything.
  • If it’s too hard, the learners will be quickly discouraged.

Accordingly, the learners need to be rewarded with small successes and breakthrough as often as possible.

This approach is good for beginners. It might be better for experts to practice before looking at any explanation.

Humility and Respect as a Teacher

Let’s take 5 minutes to speak about Dave, your colleague developer. Dave is a nice guy, but he’s very sure of himself. Even if his knowledge in programming could be improved significantly, he thinks he knows everything.

Again the Dunning Kruger effect kicking in.

In my experience, this is coupled with the beginners’ insecurity, feeling obliged to compare themselves to experts. Add a pinch (or a tonne) of impostor syndrome and you’ll see Dave trying to be right in every circumstance, throwing his “knowledge” at your face to heal his ego.

Don’t get me wrong: experts can have these problems too. But I think it’s important to show to beginners that “experts” don’t know everything. We are mere mortals, we do mistakes, and we should not be ashamed to recognize it. This includes the expert blind spot.

In fact, learning is accepting as well what we don’t know in order to fix our weaknesses. Teaching can reveal them if we keep in mind that some of our knowledge, as “experts”, is way too abstracted to be understood by beginners.

This will show to beginners that:

  1. Nobody knows everything, so they should not be ashamed of their blind spots, misconceptions, and mistakes.
  2. It’s necessary to accept what we don’t know to know what to learn next.

Consequently, we should respect the colleagues we work with, even if they don’t know things we judge they should. How many times I heard developers claiming, strongly and loudly to somebody asking a simple question: “but… you don’t know that??!”. This “but-it’s-simple” attitude doesn’t bring anything good, for anybody.

Curiosity demands that we ask questions, that we try to put things together and try to understand this multitude of aspects as perhaps resulting from the action of a relatively small number of elemental things and forces acting in an infinite variety of combinations.

Richard Feynman

We have to accept our weaknesses as much as others’.

Being Interested in the Others

What’s the best way to be aware of the level of knowledge beginners have in some precise area? In my experience, an informal, relaxed conversations around a cup of tea is perfect for that. Yes, you need to drink tea, it’s the best beverage in the world. I proved it in very unscientific ways.

Let’s imagine the following scene: you have no clue about GraphQL. An expert in GraphQL tries to teach you the concept. She goes through a demonstration first. Then, she tries to assess your level of knowledge. We can imagine the teacher saying:

  • “So, what do we do now? Yes! Indeed, very good!”
  • “And that, where do you put it?”
  • “And this? No, not at all. Think about it again.”
  • “This is not good either. You should do that and this. Let’s try again.”

In practice, learners feel these questions quite arrogant. I had a colleague teaching exactly this way. Unfortunately, he was giving the impression to the others (including the learners) that he was patronizing. This wasn’t the intent of the teacher, but it’s how it felt to the learners.

As a beginner, if you have some experience with GraphQL, our expert might go into some details: “I love you can do X or Y, it’s pretty powerful, isn’t it?”. She should then pay attention to your answers and your body language. If you’re looking to the ground, not really knowing what to answer, or simply acquiescing, your teacher might have found the limits of your knowledge.

This method has multiple advantages:

  1. The teachers will develop a friendly relationship with the learner.
  2. The learner will feel considered, and not simply as somebody who doesn’t know.
  3. Others won’t judge the teacher as the expert-who-think-she-knows-everything. Difficult to teach when this image begins to stick to you.

It’s important to assess the knowledge of your colleagues without judging or blaming. Every developer has different learning backgrounds; consequently, they have all different level of knowledge for different subjects. That’s great because a team made of complementary individuals will have a higher level of collective intelligence.

In short, as a teacher, you need to adapt to your audience.

Retaining the Steps of your Knowledge Acquisition

Now that we guess what our learners don’t know, how can we know what piece of information we might omit for the good understanding of our beloved colleagues? After all, it’s difficult to explain something you abstracted away in your long term memory. You need to retrieve it yourself first.

There are different ways to capture each step of your learning process on an external medium, in a layered fashion. Then, when you fall into the trap of the expert blind spot while teaching, you can come back to these resources to see what you might have forgotten to explain.

What are these resources? How to create them?

Mapping your Schemata

I love creating mindmaps of books or articles I read. You can capture the layers of your knowledge in there. You can hide some of these layers to see the root of the knowledge, and expend the branches to see the different level of abstractions.

In short, mind maps allow you to apply, in a very visual way, the principle of chunking we saw at the beginning of this article. It can, more or less, represent the schemata in your long term memory.

It means that you need to mindmap the sources of knowledge which are explaining well and thoroughly the concepts they teach. You need to know as well what prior knowledge you need in order to understand the material.

Then, when your learners look at you as you were an alien speaking Goa’uld, you can come back to your mind maps to see what piece of information they don’t know and you’ve forgotten to explain. Even if the answer is not in the mindmap but in your prior knowledge, only going through it can trigger what you forgot to explain.

Additionally, mindmaps are a fantastic way to store knowledge in your long term memory by actively going through them on a regular basis, using spaced repetitions. They will help you to go through the Expert Path™.

Writing a Development Journal

I keep a journal where I write the bugs I encounter and how I solved them, the traps and inconsistencies programming languages throw to our wonderful faces, and other joyful details I want to log.

When a beginner encounter difficulties to understand my explanations how to solve a difficult bug or tasks, I can search in my journal using tags and search queries to understand what was my thinking process at the time.

Then, I can try to remember the prior knowledge required to explain it to my learners, if they don’t know it already. Similarly, using mindmaps can help remember how I was thinking at the time, what difficulties I had, and how I solved them.

Writing a Blog

A big benefit of writing a blog is the appropriation (or reinforcement) of knowledge. Again, going through my blog can help me to remember what was the state of my knowledge at the time I wrote a precise article, and what I read to acquire this knowledge.

Then, I can have a mental model about what to explain to the learner, and have all the details I need in order to avoid the expert blind spot.

Thriving to be a Good Mentor

We can learn a great deal by being a good mentor

It’s time to summarize. What did we see in this article?

  • The working memory, used to understand, learn, and perform, is limited.
  • Experts need to abstract their knowledge (“chunking”) and brush away the details. The goal: save some space in the working memory to learn new concepts, or to take action.
  • The expert blind spot is the set of all the details experts can’t recall easily.
  • It’s not because experts abstracted the details of their knowledge that they can’t retrieve it to perform a given task. It’s a more automatic (or intuitive) way of using knowledge.
  • When experts teach to beginners, they might not be able to explain these abstracted details.
  • Informal communications and friendly relationships can give precious information about what a learner doesn’t know. It will help the teacher not falling into the expert blind spot.
  • Writing or representing the steps and layers of our knowledge (via mindmaps, blogs, or journals for example) can remind us the details we might have chunked.

There is no magical recipes: avoiding the expert blind spot is ultimately a personal quest. In short: we should try, experiment, learn, and practice!

Share Your Knowledge