March Tech Recap

Hey everyone. We shipped a bunch of new stuff in March. I want to highlight a few under-the-hood improvements you may have missed. Our big focus was playback issues on lbry.tv and we also made some protocol improvements. Let's start at the top of the stack and work our way down.

lbry.tv Kubernetes cluster

lbry.tv has been growing like wild over the last few months. To keep up with the increasing load, we moved the app to a Kubernetes cluster that is easy to scale up as demand spikes. The cluster runs a bunch of LBRY SDK instances, a proxy in front of them to handle authentication and wallet loading, a database for the proxy, plus monitoring and alerting (Prometheus + Grafana).

So far k8s has really delivered on its promise. Our apdex scores (the most holistic metric we track for lbrytv) have been consistently great and we had no trouble handling almost 900 simultaneous streams, whereas a month ago 150 streams would have crushed the app.

That's not to say that lbry.tv has been perfect. We did have some downtime and slowness in March. But we've eliminated a whole class of problems and can focus our efforts on the next lowest-hanging fruit.

Distributed lbry.tv Players

In addition to moving to the k8s cluster, we split out the code that downloads and serves the LBRY streams into its own codebase and set up instances in a few locations around the world (two in the US and two in Europe for now). The player caches acts as a video CDN, caching streams and serving them from a closer location. This makes videos start faster for more users.

Video Transcoding

As we wrote in the post on buffering, high-quality videos are often unstreamable because the video bitrate is higher than your bandwidth. The solution is transcoding the video down to a reasonable size before uploading it, and now that's as easy as passing a flag to the publish command (and automatic in our apps). One caveat is that using the flag requires that ffmpeg is installed on your system. In the future we may remove this dependency.

If you want to know what's going on under the hood, here's the source.

On-Demand Header Download

Our biggest protocol improvement in March was a huge boost to the header download process. Any client that doesn't have a full copy of the blockchain (and that's most of our clients) needs at least the headers to verify transactions and URL resolution. Without headers, you are trusting that the wallet server is not lying to you or withholding information (and if you're doing that, what's the point of using a blockchain?). The problem is that our headers are at 80MB right now, and will only get bigger. Downloading them the first time the app runs is a terrible experience for the users (its the cause of the infamous "catching up..." green screen), and may be prohibitive on low-bandwidth connections.

The solution we came up with is something we call "on-demand header download". It means the SDK will request and download the headers it needs right when it needs them. With this feature, the SDK starts pretty much instantly since startup is no longer blocked by header download.

In addition, we finally got rid of several hacks. On Android, we used to bundle the headers right into the APK. No need to do that anymore, and very timely - we were about to hit the 100MB file size limit on Play Store apps. We also stopped keeping an always-up-to-date headers file in S3, since there are no longer any speed benefits to downloading the headers from there.

BitTorrent MVP

This one has the smallest impact on users today, but the biggest potential for LBRY down the road. The latest SDKs are now part of the BitTorrent network, and can download torrents published to LBRY

The integration is seamless. Here's Jeremy's demo. The stream he downloads has a BitTorrent infohash in its metadata instead of a lbry sd_hash. If you wanna see it in action on the command line (or at the playground), resolve lbry://bigbuckbunny-torrent1. You'll see this in the source:

source: {
    bt_infohash: "dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c",
    hash: "1f4dc7dfd0407801552d46a400e4d610b4fc14df896f7fb7769e02182d7ac2635938172055e97fca577b15eb6e78f459",
    media_type: "video/mp4",
    name: "Big Buck Bunny.mp4",
    size: "276134947"
},

When you call get on this URL (as you would on any video you want to watch), the SDK will connect to BitTorrent using libtorrent, look up the infohash, and download the video. Simple!

And More

I talked about only a few of the things we did last month. In total, we merged 196 PRs from 49 users, across 34 LBRY repos. Thank you contriutors! I wanna shout out all of you here:

AlessandroSpallina, Bharat123rox, BrannonKing, Coolguy3289, EricBrianAnil, GlenCooper, MarcoFalke, Meantub, PeterMortensen, PierreRochard, SwapzoneIO, achow101, addy1510, adityakaria, akinwale, andybeletsky, apetresc, bvbfan, ceoger, eggplantbren, eukreign, finer9, gahag, gustavonalle, gwillen, gy741, harding, hebasto, instagibbs, jackrobison, jessopb, jring-o, jsigwart, kauffj, kcseb, ken2812221, laanwj, lbrynaut, lyoshenka, nikooo777, osilkin98, promag, ryanofsky, sdaftuar, seanyesmunt, shyba, tiger5226, tzarebczan, ykris45.