Principles for web developer research

[foolip]

When conducting research and running surveys, what are the most important things to get right? What are common pitfalls we should avoid?

A few things that come to mind for me:

• Transparency and doing work in public.
• Web developer time is precious, and we should not waste it. How can we minimize the amount of web developer attention we "use" to get (just) the information we need?
• Who is a web developer, and which web developers do we want to learn about/from?

[robnyman]

Outlined this in the doc about joint research, adding it here for context:

To ensure research efforts done together across multiple browser vendors and organizations, this document aims for agreement on general principles and approaches regarding research:

Principles

• Research results should always be published publicly, unless there is a strong reason for not to
• Research needs to be focused enough to get useful and actionable data
• Balanced number of stakeholders to avoid too much Design by committee
• Create short and clear surveys, to respect developers’ survey fatigue and lack of time
• Do research often enough, so it’s not just an annual check-in
• Ensure it’s not to time consuming for stakeholders to create and analyze

Benefits

• Much quicker to take for developers, therefore causing less survey fatigue
• Good to get a quick indication on certain topics
• Quick turnaround time for stakeholders on getting results
• Can be run with a good cadence, e.g. monthly
• Present the result publicly so everyone has the same view and information

[foolip]

Filing #2, a variation of "Web developer time is precious" springs out. For surveys that web developer volunteer to do, it makes a huge difference if they survey is fun and/or educational. IMHO, State of JS/CSS are very strong on the fun/educational aspects.

[robnyman]

I've created Principles for joint developer research
to outline our values and approach.

One area that still needs to be discussed is

Who is a web developer, and which web developers do we want to learn about/from?

[captainbrosset]

Who is a web developer, and which web developers do we want to learn about/from?

The web platform supports a lot of different use cases. Different people use it in different ways. I'd probably want to avoid focusing on a definition of who a web developer is that is too narrow. In fact, even the name "web developer" might not necessarily include all users of the web platform.

My feeling is that we should learn from as many folks as possible in this research, but open to hearing if there's a need to focus on a given type of web platform users rather than others.

[dontcallmedom]

I think the question of who we want to learn from is intimately tied to the question of what we will do with the results.

In general, I think we will want to use it to help prioritize improvements to the platform and its developer experience; but there are many ways to prioritize among for instance:

• experts developers
• junior developers
• people learning to be developers
• people doing Web development outside of a professional context
• people not doing Web developers because they've given up on it

I'm also guessing that depending on the research, we may target different parts of these, and different subsets (e.g. developers using CSS vs JS).

Some research benefits from breadth (as many respondents as possible), others may benefit a lot more from depth (why are so few web sites adopting technology X).

So maybe we shouldn't answer the question in the abstract (who is a web developer), but it's probably worth starting to explore what kind of research would need what kind of input, and then dive into the principles that drive audience identification across these different kinds.