I love the National Portrait Gallery and American Art Museum. They are housed in the same building at the other end of the 42 bus line. Ride the Metro Center bound 42 all the way to the end of the line and you'll be right across the street from the entrance (G St NW between 7th and 9th Sts NW). The American Art exhibits change pretty frequently, but the Portrait Gallery's collections seem to rotate in and out at a slower pace. More exciting though, is the building itself. Originally constructed to house the Patent Office (it was supposed to be fireproof), the south wing was completed in the 1830s and the Patent Office moved into the space in 1840.
Over the years the building has established a rather sordid history, growing by several wings, housing numerous agencies and government offices, surviving the Civil War (as a morgue for a time), hosting Presidential inaugural balls and, of course, being partially destroyed by a large fire (1877).
It was designated as home to the Smithsonian's American Art Museum and National Portrait Gallery in 1968. In 2000 it was closed for a two year renovation and according to government schedule reopened in July 2006. It was worth the wait though. Here are a few photos from the building. These are from the third floor, which is hardly ever crowded.