The Fitzwilliam Hotel was built in 1807 by Ward and Robeson. When first opened, the Hotel served as an inn, a store, and a post office. By the late 1840's the building housed a harness shop, union store, drug store and post office. Throughout the years the Hotel changed hands frequently and served as everything from a tavern to a tearoom. Unfortunately, the building saw less and less use through the 1930s, and was eventually torn down in 1946. Granite posts from the porch can still be seen and mark where the hotel stood.
The most famous guest ever to grace the Fitzwilliam Hotel was President Ulysses S Grant, who stayed in the Hotel on August 12, 1883.