At what point should I stop trying to handle this myself and hire a litigator instead?

Several specific factors, individually or especially in combination, are strong signals that hiring an attorney rather than continuing to handle the matter yourself is the financially sound choice. These include the claimed amount being substantial relative to what your business can realistically afford to spend litigating, the presence of an attorney fees clause in the underlying contract that could significantly increase your total exposure, a personal guarantee being genuinely in play that puts your own personal assets at risk, or the underlying facts being genuinely and seriously disputed rather than representing a straightforward, largely undisputed unpaid balance.

As a general rule of thumb, if two or more of these specific factors apply simultaneously to your situation, the cost of an attorney's retainer is very likely to be less, in expected financial terms, than the risk of continuing to handle a genuinely complex or high-stakes matter without professional legal representation.