For video streams (at least H.264 and H.265), the config packet containing SPS/PPS must be prepended to the next packet (the following keyframe). For audio streams (at least OPUS), they must not be merged.