Handle collisions between rekey and the following delete properly