Hadsel Rectory is situated on a hill overlooking a rich cultural landscape and the fjord. Large stately trees, the old stone-enclosed cemetery, and the octagonal church are the nearest neighbors to the rectory courtyard. In addition to the main house, the courtyard features a large barn with a loft, as well as a carriage house and an outhouse. The church and all the buildings on the rectory are built of wood materials and are painted red.

The courtyard is dominated by the large and stately two-story main house, built at an angle, with a slate-covered roof, four tall chimneys, and small-paned windows.