J. P. Morgan Home, Madison Avenue and 36th Street. J.P. Morgan's home; Built: 1853; Razed: 1928. John Jay Phelps built the Italianate mansion as one of three identical brownstone homes on the block of Madison Avenue between 36th and 37th Streets. The other two homes were built by Isaac N. Phelps and William E. Dodge. J.P. Morgan, Sr. bought the Phelps home in 1882. He made few changes to the 3 story exterior but modernized the inside of the house considerably. In 1928 the house was demolished to make way for the Morgan Library Annex.