If you're trying to correlate putting a basketball hoop on a tennis court with demolishing a large part of a building and replacing it with a building twice the size of the White house, I guess you won't get it  Of course each president has changed the interior.  There have been swimming pools, bowling alleys that don't alter the exterior facade.  While I thing the patio is in poor taste, from a historical perspective, not an issue.  Putting big flagpoles outside, tacky and out of perspective, but go for it.  Planting roses, vegetables, your favorite pot variety, go for it. Yes, the major gutting and addition of a balcony were major but necessary for preservation of the building.  The balcony addition while controversial was within the architectural aesthetic.  This addition fits in as well with the architecture as those stupid ferris wheels in Paris, London, Seattle and wherever else.
 Well, architecture is outside the scope of my inquiry.  But now I'm curious.  What is so radically different about the architecture of the ballroom?  James McCrery is a recognized expert in neo-classical architecture, and these elevations look to me like they match the White House architecture.  But you disagree?  Why?
Scale.  you want a garage twice as big as the main house?
If you want something that big for function, you put it behind the main house (don't know if there is room).
 There we run into the question of what is the front of the White House, and what is the back.  The address is on Pennsylvania Avenue, but there is no defined "front door" to the White House.  Employees enter either the West Wing directly or through the East Wing, and guests at state dinners enter through the East Wing, which is why Teddy Roosevelt first built the East Wing in 1902, because 200 dignitaries arriving, mostly in carriages then, at the door facing Pennsylvania Ave jammed up the avenue and pissed off the locals.  So he constructed the East Wing, which has its own long private drive off of Pennsylvania Ave and a turnaround at the south end for carriages and automobiles.  
Effectively, most of the square footage occupied by the ballroom will be "behind" the White House, which is to say, to the south of Pennsylvania Ave. To build a ballroom anywhere but as the East Wing would entail far more excavation, and would obscure the view of the White House from the President's Park and the Ellipse. And to my mind, that would be a greater violation of "history" than to put the ballroom where state guests have been arriving since Teddy Roosevelt.
It's not clear to me whether the structure will be taller than the White House.
But, okay, it's the size of the thing that violates your sense of aesthetics.  I get it.