My friend told me: "I think it has something to do with your VPN not being compatible with Window’s way of assessing connectivity, maye it tries a DNS request but that is blocked since you use the VPN’s DNS ? I don’t really know ; but the fact that the exclamation mark only appears when using the VPN suggests a similar cause. Anyway, if you can still use the Internet normally, just ignore the exclamation mark."
What I did was that I changed my VPN.
I had this issue while using NordVPN and then uninstalled NordVPN and installed ExpressVPN and the problem solved.