The measurement of the inflationary stochastic gravitational-wave background (SGWB) is one of the main goals of future GW experiments. In direct GW experiments, an obstacle to achieving it is the isolation of the inflationary SGWB from the other types of SGWB. In this paper, as a distinguishable signature of the inflationary SGWB, we argue the detectability of its universal property: antipodal correlations, i.e., correlations of GWs from the opposite directions, as a consequence of the horizon re-entry. A phase-coherent method has been known to be of no use to detect the angular correlations in SGWB due to a problematic phase factor. We thus investigate whether we can construct a phase-incoherent estimator of the antipodal correlations in the intensity map. We find that, unfortunately, the answer is no: there is no estimator that is sensitive to the antipodal correlations but does not suffer from the problematic phase factor. Our analysis clarifies the importance of averaging in defining the estimator and will give an insight on what kind of angular correlations in SGWB is detectable or not.