The garden dates back to the early 1870s when it was designed by the vicar Johan Carl Wisløff and his wife Amalie Wisløff. The botanist professor Schübeler at the University Botanical Garden in Christiania (now Oslo) had encouraged them to create the garden, and they did for many years test out plants for him. The garden soon became well known because of its large collection of plants, many of which had previously not been cultivated this far north.

When the reverend Wisløff died in 1893, he was buried in the rowan grove at the lower end of the vicarage garden. His simple memorial stone is still there.