There is a Hadīth which says:

Two men sneezed before the Prophet. The Prophet said to one of them: May Allah bestow His Mercy on you; but he did not say that to the other. On being asked, the Prophet said: That one praised Allah (at the time of sneezing), while the other did not praise Allah. (Abū Da’ūd, Kitābu’l-Adab)

From this, I gather that one should only reply to a sneeze by saying yarhamukullāh if the person who sneezed said al-hamdulillāh. But how would I know if the other person said it or not. He may have said it in his heart. Also, what is the significance of these two utterances?

I’ll first answer the last part of your question:

It is desired from a Muslim to remember Allah as much as he can everyday. This remembrance gives him the strength and courage to brave the storms of life, and places him in the right perspective of being a servant of the Almighty. As such this remembrance is also an expression of servitude from a person for his Creator. There are various forms of this; one very important one is an expression of gratitude for the various gifts and blessings the Almighty has bestowed us with. These gifts and blessings are obviously countless. As a thanksgiving for these blessings, the Prophet (sws) has suggested various utterances at various instances. Sneezing is one of them, and a very meaningful one too – since the relief and comfort a person gets when he is through with it requires that we thank the Lord of the worlds for this favour.

A person is required to reply with yarhamukullāh only if he has heard the person who has sneezed say al-hamdulillāh; if he has said this in his heart or in a tone which is not audible to the listener then of course it is not necessary to reply in the prescribed manner.