As Yitzchak Scott-Thoennes correctly points out this is nothing to do with XS, but with Test::Exception not being used at compile time. You can get around this by either wrapping the conditional loading up in a BEGIN block:
CPAN::Forum
Test-Exception - Re: Test::Exception and XS code