The only reason not to upgrade is if you’re using a 32-bit version of Windows that can’t run 64-bit programs, but most computers built in the last eight years or so should be 64-bit compatible. Starting with Java 8 Update 20, the Update tab in the Java Control Panel enables users to automatically update 64-bit JREs (in addition to 32-bit versions) that are installed on their system. From a 64-bit browser, download 64-bit Java from the manual page. It’s more securenot to mention faster and more stable. Users should download 64-bit Java software, if they are running 64-bit browsers. If it were my machine (and I have been in this situation), I’d switch to 64-bit just because I want all of that RAM to be used. You should upgrade to the 64-bit version. Copy back your data from the backup or elsewhere.
Ultimately, you have to ask yourself some tough questions. “Concerned” is hard for me to judge, as is whether or not it’s worth your time to go to 64-bit.