In this paper, a semi-supervised graph-based method for estimating 3D body pose from a sequence of silhouettes, is presented. The performance of graph-based methods is highly dependent on the quality of the constructed graph. In the case of the human pose estimation problem, the missing depth information from silhouettes intensifies the occurrence of shortcut edges within the graph. To identify and remove these shortcut edges, we measure the similarity of each pair of connected vertices through the use of sliding temporal windows. Furthermore, by exploiting the relationships between labeled and unlabeled data, the proposed method can estimate the 3D body poses, with a small set of labeled data. We evaluated the proposed method on several activities and compared the results with other recent methods. Our method significantly reduced the mean squared error, showing the positive effect of removing shortcut edges.