When do I actually have to pay departure tax if I leave Canada?

Departure tax in Canada for non-residents is generally triggered when CRA considers you to have become a non-resident for tax purposes, not simply because you travel or work abroad temporarily.

At that point, CRA may apply a deemed disposition to certain assets you own, meaning they are treated as though they were sold at fair market value on the date you left Canada. Any resulting tax is typically reported on your departure return for that tax year.

Coming back to Canada later does not automatically reverse the departure tax rules. The key factor is whether CRA considered you a non-resident during the period you were away. Temporary absences where residential ties to Canada are maintained often do not trigger the same treatment.

CRA explains the deemed disposition and residency rules here:
CRA Leaving Canada (Emigrants) Guide