The Zinn Education Project published an article about Memorial Day, or “Decoration Day”, written by David W. Blight. According to him, "The First Decoration Day" was led by people who had recently been freed from slavery in Charleston, SC on May 1, 1865.
At the end of the Civil War, "Thousands of black Charlestonians, most former slaves, remained in the city and conducted a series of commemorations to declare their sense of the meaning of the war. The largest of these events, and unknown until some extraordinary luck in my recent research, took place on May 1, 1865. During the final year of the war, the Confederates had converted the planters’ horse track, the Washington Race Course and Jockey Club, into an outdoor prison. Union soldiers were kept in horrible conditions in the interior of the track; at least 257 died of exposure and disease and were hastily buried in a mass grave behind the grandstand. Some twenty-eight black workmen went to the site, re-buried the Union dead properly, and built a high fence around the cemetery. They whitewashed the fence and built an archway over an entrance on which they inscribed the words, “Martyrs of the Race Course.” "
Full article available here.
* The image used in the banner is from World War I, not from the 1865 events in Charleston.