Need to support legacy browsers, but don't want the full weight of es5-shim loaded for everyone? Sure, just load it for browsers that need it! It should be easy enough to add more polyfills as you need them. You can also include some legacy IE polyfills to complete the experience.
Alternatively, you can use the polyfill.io service which takes care of this for you and more.