Brown was born in Kennington, the son of church minister James Baldwin Brown and his wife, a sister of the sculptor Henry Leifchild who lived at Kirkstall Road
He studied at Uppingham and Oriel College, Oxford and became a Fellow at Brasenose College in 1874. He became the first holder of the Watson Gordon Chair of Fine Art at the University of Edinburgh in 1880 and held the chair until his retirement in 1930. In Edinburgh he lived initially at 3 Grosvenor Street in the west of the city before moving to 50 George Square
Cremated at West Norwood Cemetery