Spiking neural networks (SNNs) have attracted great attention due to their distinctive properties of low power consumption and high computing efficiency on neuromorphic hardware. An effective way to obtain deep SNNs with competitive accuracy on large-scale datasets is ANN-SNN conversion. However, it requires a long time window to get an optimal mapping between the firing rates of SNNs and the activation of ANNs due to conversion error. Compared with the source ANN, the converted SNN usually suffers a huge loss of accuracy at ultra-low latency. In this paper, we first analyze the residual membrane potential error caused by the asynchronous transmission property of spikes at ultra-low latency, and deduce an explicit expression for the residual membrane potential error (RMPE) and the SNN parameters. Then we propose a layer-by-layer calibration algorithm for these SNN parameters to eliminate RMPE. Finally, a twostage ANN-SNN conversion scheme is proposed to eliminate the quantization error,the truncation error, and the RMPE separately. We evaluate our method on CIFAR and ImageNet, and the experimental results show that the proposed ANNSNN conversion method has a significant reduction in accuracy loss at Ultra-lowlatency. For ImageNet, when T is ≤ 64, the delay required by our method is about 1/2 of the other methods.