Ask the wrong people and you build the wrong thing
Not long ago my wife and I received an email from our kid’s school. (They attend a CPS school.) The email was a survey of some sort that would be used to make decisions about curriculum in next year’s school program. It always excites me when parents, teachers and administrators get to collaborate on school programs.
You can imagine my frustration as I clicked on the link to the survey and was greeted with some cryptic error message from Google Forms. I wish I could remember what the error said but even as someone with a technical background, the error didn’t point to any specific action that could be taken to resolve it. Thanks to the pandemic, I’m well equipped with handling the idiosyncrasies of CPS’ implemenation of Google Apps. I logged out of all my Google accounts and then logged back in with my daughter’s CPS email address and I was granted access to the survey.
The question being asked was if we would prefer more STEM classes or more Arts related classes for extra curriculur items next year. I quickly suspected that this was probably going to lead to a case of Selection Bias as the number of people who figured out how to participate in the survey are probably more technical leaning than those that just gave up. Out of curiosity I asked a few people in my circle. The people who I’d consider technical, poked around and figured things out, while other people just gave up, assuming that there was something broken on the site, which was a fair assumption given the generic nature of the error message.
This experience got me thinking about how often we make “educated” decisions based on poor information. CPS could think that they were implementing the wishes of their student community only to find out they were addressing a subset. I’ve fallen victim to this mistake myself when I wasn’t
When my team and I were designing our infrastructure platform at Basis we had a tendency to talk to the loudest developers in the room. Those developers had very specific needs and requirements. But we failed a lesson that I’m sure every product manager in the world knows. The loudest people aren’t always representative of the larger user body. This is exactly what we encountered as we built out our chat bot. Using feedback from the noisy developers pushed us towards a model where there were many different options for building environments and packages. Instead of creating a tight streamlined process, we created different avenues for people to build and manage their environments. We supported custom datasets that were seldom used. We created different methods of creating environments, so maybe you only needed the database server or maybe you only wanted the database server and the jobs server. This created headaches for the people that didn’t want that functionality, which forced us to create omnibus commands that strung together multiple commands.
I’d really like to be angry at the developers for this but the truth is the mistake was all my own. Developers, like anyone else, have different things that they’re attracted to. Some developers love to understand the stack from top to bottom and want configurability at every level. Others are wholly disinterested in infrastructure and want to just point to a repository and say “make an environment out of this”. Despite what my personal feelings are on how much or how little interest they have, the reality is you’re probably not moving them from whatever their stated position is. And even if you do, without using a very heavy hand, you’re likely to just alienate them.
The lesson learned is to make sure that you’re talking to the audience that you actually want to talk to. Who are you asking? How are you asking them? Think about how they might self-select out of your surveys or questions and see if you can mitigate that. To engage with the entire audiece means you might need multiple methods of interviewing people. Developers who respond to surveys might not be the same developers that will respond to 1-on–1 interviews. Don’t make the mistake of optimizing for a subset of your audience or user base. Put care and thought into reaching your target audience.
