Although social consensus is a common way to establish the veracity of a memory, I don't think it is the only way.

If I "remember" setting my wallet by the door, I can verifiy that later, when I look for it before grocery shopping.

My "memory" of where I left my wallet serves as a "prediction" of where I will find it later.

And if those types of predictions usually end up being true, I might think of my memory as "good". But if these types of predictions start to be “wrong “ (i.e. not useful) more often, I might start to wonder if my memory is failing.

Some "memories" don't have any way of being "tested" or "proven" other than by social consensus - i.e. as a "prediction" that others will agree with my testimony of events. But other memories can be tested in non-social ways, all by ourself.