Soon after President James Madison's death, Dolley Madison had to sell Montpelier, primarily to settle the debts of her profligate son (from her former marriage). She inherited a row house (pictured above with its iron balcony) on Lafayette Square in Washington, DC, as part of Madison's estate and moved there in 1837.
Madison Place borders the east side of Washington’s famed Lafayette Square, across Pennsylvania Avenue from the White House. The Dolley Madison House, an unpretentious Federal-style structure at the northeast corner of the square (Madison Place and H Street), was built in 1818-20 by Richard Cutts, a Massachusetts congressman who was Dolley's brother-in-law (he was married to Dolley’s sister, Anna). Cutts, Comptroller of the U.S. Treasury from 1817 to 1829, had borrowed money for the construction of this house from James Madison. In 1829, after Cutts had lost most of his fortune in unsuccessful business ventures, ownership of the house reverted to Madison, although he never lived in it. Upon James Madison’s death in 1836, Dolley inherited the house and moved here to spend virtually her entire widowhood. During her residence, from 1837 until her death in 1849, she advised various First Ladies and played a prominent role in Washington society.
The house served as home of the Cosmos Club from 1886 to 1952. The historic character of the house was subsequently preserved in the 1960s through the efforts of another first lady, Jacqueline Kennedy, architect John Carl Warnecke, and the National Trust for Historic Preservation. The restored building is painted yellow and features a distinctive 19th-century iron balcony. Now federally owned and used by the U.S. Court of Claims, it is not open to public visitation.
One of several enslaved persons who accompanied Dolley Madison from Montpelier upon her return to Washington was Paul Jennings. Like Dolley Madison, Paul Jennings had previously lived on Lafayette Square (in the White House) while James Madison served as President. In dire financial straits, Dolley Madison had to sell Paul Jennings in 1847; later that same year he was purchased by her Lafayette Square neighbor Daniel Webster.
In 1865, Paul Jennings wrote the first memoir ever written about life in the White House, titled A Colored Man's Reminiscences of James Madison. In this work he also described what it was like to see the former First Lady living in poverty on Lafayette Square after her husband's death. He wrote: "Mrs. Madison was a remarkably fine woman. She was beloved by every body in Washington, white and colored. In the last days of her life she was in a state of absolute poverty, and I think sometimes suffered for the necessities of life. While I was a servant for Mr. Webster, he often sent me to her with a market basket full of provisions, and told me whenever I saw anything in his house I thought she was in need of, to take it to her. I often did this, and occasionally gave her small sums from my own pocket."