I think adding a new module would be easier. The JSON5 format is very permissive and would add many new rules (e.g. identifiers as object keys). I think a new parser (maybe copied from the existing JSON parser) would make it easier to adjust the grammar. I am not sure if you like ragel, but I found it to be very pleasant to work with.
Having taken a quick cursory look at the new JSON5 module, I think it would make sense to have a distinct decoder for JSON5, but have common code for the encoder. Having decode() and validate() call the appropriate implementation depending on flags. This would reduce the amount of code duplication, and ensure that the flags used by the modules stay compatible.
/grubba