entr

There's no shortage of ways to automatically run your tests when files change (e.g. jest has jest --watch). I find these mechanisms useful for maintaining flow. Having Vim open on one screen and a test-runner open on another screen is fantastic.

There's not a custom runner for everything, but you might not need one. I've found that entr often gets the job done. entr lets you "run arbitrary commands when files change."

When I was writing the cli caching script, I kept git ls-files | entr bats test running. Every time I saved my script or the test, the tests would run.

The entr homepage specifies several good use-cases and how to watch for new files, etc. Give it a try.