Now some hours later, i see that it actually still happens, even with forced IPv4.
EDIT: i had just tried re-enabling IPv6, but that made it even worse.
So my conclusion currently is:
This happens randomly at times and i dont know why (is it high traffic times?)