Just a very quick update. According to Thomas, who designed gPodder, the problem may lie with the underlying infrastructure (Qt MultimediaKit using GStreamer).

Is there a way we could upgrade the GStreamer components (gst-av etc.) to a newer version?

Also he suggested we should try the Android version. If the problem does not occur there, it must be the multimediakit. Unfortunately, I do not have an Android device.