A few years later and this is still a problem. The article Unbloating the buffers describes a way of configuring your network to fix this:

“I traded about 10% of bandwidth (263Mbit down/41Mbit up per iperf3) for:”

• constant average bandwidth on both upload and download
• no impact of download on upload
• network load has no visible impact on latency
• effective traffic prioritisation

The solution isn’t so straightforward, though. You have to have control over your routing endpoint at home in order to set up AQM with a tool like CAKE.