I was inheriting the startup method and then running Test::Class->runtests, which ran the tests from the base class and the subclass separately.
ie. Foo inherits from Bar which inherits from Test::Class