browse-url-new-window-flag is a variable defined in `browse-url.el'.
Its value is

Non-nil means always open a new browser window with appropriate browsers.
Passing an interactive argument to M-x browse-url, or specific browser
commands reverses the effect of this variable. Requires Netscape version
1.1N or later or XMosaic version 2.5 or later if using those browsers.

You can customize this variable.