Below is 1870 Census, Wylliesburg PO Charlotte Co., VA page 44, near other black or "M" Elams.
For Josephus Gregory to list three members of this family in his farm diary, I would say there was some connection either before the war ended or possibly by 1873/74 this Elam
family was living on the Gregory place and farming with Josephus.  I remember from childhood there being 3 tenant houses on the place, like 80 some years ago.
The diary was a neighborhood diary as well as a farm Journal.  He mentioned deaths of neighbors, strange weather conditions, etc.

Previous dwelling was a Snead Elam 23, appeared to be "M" for mullato, with an Edna (hard to read) 65, also "M" apparently and a couple of others.