Properly remove listener when listen() times out