Willumsen grew up as an only child in his parents’ home in Copenhagen. His father ran a pub and later a gentlemen’s outfitter’s. Willumsen believed that the sculptor Jens Adolph Jerichau may have been his real father. His relationship to his mother was always good, while his relationship to his father was more problematic. When his mother died in 1899, he executed a large sepulchral monument to his parents, which was erected in Vestre Kirkegård in 1901. His father did not die until 1910.
Willumsen was twice married. His first marriage was to Juliette Meyer, with whom he had two sons, Jan and Bode. In the 1890s Juliette produced a number of works in white faience in the style of the period. Juliette modelled for him on occasion, among other things for the etching Lady Out Walking and The Family Vase (The Danish Museum of Decorative Art).
In 1903 he married the sculptress Edith Wessel, with whom he had two daughters, Gersemi and Anse. Edith, who created figures in wood, bronze, marble and multicloured wax, appeared frequently in Willumsen’ works. She is A Mountain Climber, Hagemanns Kollegium and The Royal Museum of Fine Arts), the mother in The Evening Soup and in The Painter and His Family and the woman in After the Storm. The two girls also appear in a number of works.
Willumsen spent his old age together with the French dancer and painter Michelle Bourret, who can be found in several of his paintings from the 1930s. All Willumsen’s wives were artists and they both supported and helped him in his art.