VOIP update: gigabit fiber cures most of what ails you
Ok, so I had a couple of weeks of continually degrading call quality and then realized that ATT had just started offering fiber service in my neighborhood, so I jumped on that like a gymnast on a trampoline and dropped my cable connection like a ton of bricks. After a lot of fooling around with getting the install done and turning off the cable service and then re-configuring my internal network to handle the now ridiculously fast network connection... I can assure you that if you have crappy audio quality over a cable modem it is most likely your ISP's jittery lossy junk connection and no amount of QOS will make their part of it any better.
My impression is that VOIP technology works really hard to mask the shittiness of shitty internet. I use opus codec for example using CSipSimple and it's a super high quality codec with forward error correction and blablabla. So if your internet jitters and packets arrive late or not at all, it will cover up that loss up to several percent packet loss in such a way that it just sounds like you're muffled (it can reconstruct the lower-frequency components even with packet losses). Well, it turns out that by the end of my cable modem experience, sure enough I was experiencing something like 2% packet loss and jitter that would have packet arrival times be between something like 20 ms normally, and 100ms every second or two. It seemed to coincide with time of day, and I suspect a poor/corroded/temperature sensitive splitter connection somewhere along the coax together with the shared nature of the cable in my neighborhood. Monitoring incoming traffic during a call using "fireqos status wan-in" would show steady 100 kbps incoming for several seconds (when using ulaw) and then 15-30 kbps for a second where a bunch of packets would drop out... and it would do this randomly on average around 5 second intervals. No amount of QOS on my end could solve this issue.
Also, testing this issue was hard. Things like dlsreports speed test try hard but if the ISP is prioritizing icmp/ping packets it can give a very misleading view of packet loss and jitter for the UDP/RTP packets that make up your voice.
Now, the FIBER connection is a whole different story. It includes an IPv6 prefix I can delegate, and I now have a fully routable public IPv6 network with no extra lag/delay. In fact, it showed up a bunch of issues in my network that weren't there at the lower speeds. I moved the routing load from a consumer Buffalo wifi router to my home server which has a 4 core Celeron J1900 processor, 16 Gigs of RAM and 3 bonded NICs. It provides an NFS server, an SMB server, a web server, dnsmasq, and FireHOL/FireQOS traffic control. Pushing packets through the QOS and firewall doesn't even hit 1% CPU usage on this machine, whereas the Buffalo router was maxing out its CPU at 150-200Mbps. So, now I consistently get over 800 Mbps symmetric internet connection. The best part is that packet loss is nonexistent, the fiber transceiver has a battery backup, the jitter is around 1 or 2 ms under load, and I can dedicate 10Mbps to voice calls without even noticing it as roundoff-error.
Still, the WiFi part can be tricky especially with neighbors interfering, but I can assure you gigabit fiber is the way to go 😉
Yes, I still run QOS on the gigabit, because regardless of how big your pipe is, someone will come along and open up a big video buffering task and suck up the bandwidth if you let them. Prioritizing your voice and your ssh keystrokes ahead of all else still makes sense.
Onward to internet the way it was meant to be.