ike_sa: Defer task manager destruction after child destruction