Construction of the building started in 1843 under then Chaplain of Hong Kong Vincent Stanton. It was completed in 1848 and was used as a school building by the St Paul's College, one of the oldest secondary school in the territory, in 1851. When the new Diocese of Victoria was created, Stanton handed over the college property to the newly appointed Bishop, Dr. George Smith. The school have since relocated to its present site in Bonham Road next to the University of Hong Kong.
The building sits on a knoll and known for its imposing architecture and proportions. Inside are spacious rooms with high ceilings. It has been listed as a Grade I historic building but is not open to the public.