This is an evolving document that attempts to distill some parts of my thinking about how I (should) do research. It’s a work in progress.

Words that Describe My Taste in Research

These are words that describe the kind of research that I grativitate towards and strive to do myself. I certainly appreciate other work that doesn’t necessarily fit these descriptors.

  • Empirical
  • Minimally opinionated
  • Grounded
  • Useful
  • Synthetic
    • I find myself gravitating towards research that combines existing concepts in interesting ways.
    • A risk for this tendency is jumping into a synthesis that is already obvious prior to its implementation; I try to be cautious about this.
  • Not purely critical
    • I lean towards optimism. As soon as I’ve raised a critique, I often find myself considering how to address it.

The Value of Research

Having a concept of what makes research valuable is helpful to me when I evaluate ideas. I think that there are two main ways that research can be valuable:

  • Reducing uncertainty on important questions
  • Revealing new important uncertainties

Under this lens, the overall value of science is kind of an integral of “important uncertainty reduced” over time. The emphasis on “important” is intentional - there are a lot of questions, most of them are answerable given enough investment, but only some of those answers are important. Importance depends on timing.

This definition also clarifies that there are some activities that are research-adjacent, but do not directly produce research value. These include:

  • Programming
  • Making figures
  • Reading papers
  • Going to meetings

I do all of these activities, and some amount of them is necessary to do research. When I ask myself how things are going, time spent on these activities counts for something, but it doesn’t count nearly as much as having findings that reduce uncertainty on important questions.

On Selecting Research Topics

My selection criteria for question-answering research projects emphasizes return-on-investment based on the above definition of value. As someone working with limited resources, the “holy grail” of research ROI is a project where the experiments are cheap and fast, and the results reduce uncertainty dramatically on a very important question.

In general I prefer to be able to work iteratively. I place a lot of emphasis on projects where the iterative cycle can be made short.

On Empirical Research Operations

These are principles that I use to orient myself. Sometimes I ignore them and go with my gut. Sometimes that works, sometimes it doesn’t.

  • Prioritize for important uncertainty reduction (see Research as a Stochastic Decision Process)
  • Do the most important thing now; postpone everything else indefinitely
  • Look for a big effect size at the beginning of the project
  • Cut your losses; beware the sunk cost fallacy
  • Shorten the iterative cycle
  • Trade other expenses for compute expense
  • Keep the computer working
  • Maintain momentum