quick update

yeah I haven't updated my site in a year. i've gone thru quite a lot. i doubt anyone really is keeping track tho, so w/e. i'll make a life update post at some point but right now i just need to rant.

(also shoutout to adam neely for the idea of labelling the subpoints as chords :3)

music production software sucks

part I - the blissful, ignorant world of the FOSS-adjacent ecosystem

If you're reading this, chances are you come from the FOSS-adjacent ecosystem where DRM for software is more or less a non-issue - even with proprietary software (firmware, drivers, licensed software) it's just a matter of grabbing the package and installing the binary first and performing the registration later. At worst, an AppImage, or a flatpak (but that's a different issue compared to the hellworld to """professional""" audio software).

In the music production world, the only analog to this paradism is REAPER. It's a rather powerful, proprietary DAW that costs 225 dollars for commercial use (which they define as generating over 20k dollars in revenue yearly) or 60 dollars for noncommercial use. It's even available on Linux (bless those guys for dealing with Linux audio bullshit). They also provide a 60-day trial without needing you to even create an account - you can just install the binary!

This is how software should be distributed. Unfortunately, this is a rarity in the audio world.

part IIm - the ilok

Everyone - meet the iLok.

Made by PACE, iLok is a DRM soulution that the audio companies all fawn for.

Initially, the iLok was created as a USB hardware DRM storage pool where your licenses are stored on the USB key itself - when the software in question (whether it is your DAW or it's one of the many plugins you use) needs to authenticate, it will do some shenanigans with the iLok USB key to verify that the computer is authorized to use the plugin. (IDEA: R.E the iLok?)

We all know how flaky USB keys are. Countless horror iLok stories can be found with a quick Google search. The music world does seem to have settled on the iLok pretty early on though, so there really wasn't a need to have 15 different proprietary hardware DRM schemes.

But you know what? For an era where software was mainly distributed with firm releases (and a price tag to match), that was fair game.

part IIIm7b5 - SaaS strikes

...and then the SaaS business model hits.

It's not a secret that the "as a service" model kinda doesn't work - see the profitability figures of the barrage of streaming services, Xbox Game Pass, and other consumer-facing services for proof of that. Generally speaking, the "as a service" model only really works if your product is on the cusp of taking over the entire market - the early days of Netflix comes to mind, with the less-thought-of example of Ma Bell a close second (yes, all you SaaS companies are just wannabe monopolies, come at me).

So it's not a surprise that in a world where name matters more than technical superiority and subjective quality, the SaaS model works awfully well for pro audio companies - brand loyalty is one hell of a drug and many-a-mixing engineer are guilty of this - even me!

It's also not a secret that SaaS demands a different scheme of DRM as opposed to the big-box model of software - WideVine/Fairplay is our analog to the eldrich horror that I will explain in the next part. WideVine and Fairplay both demand a certain set of features from your device (namely - the OS you run, the level of hardware acceleration, and most importantly the ability for zero-copy) as well as a constant online-only connection - the cardinal sin of SaaS is to go off the internet.

Of course the pro audio world has it's own analog to this.

part IVmaj7 - iLok Cloud , or "Mom they SaaSed my hardware DRM!"

To be honest - I think that iLok Cloud is an improvement over the design of the original iLok USB - it stores your licences on PACE's servers and all you need to do is log in to gain access to all the software you're licensed to! Sounds great, but remember the cardinal sin of SaaS - you must never be offline.

iLok Cloud is completely inaccessible without the a connection to PACE servers. Not even for a second - you must have a constant connection or the iLok software on your own computer will tell all your plugins and even your DAW to stop working.

This. Is. A. Problem.

Especially since PACE servers aren't the most reliable. Just look for "ilok server down" on Google for proof.

Always-online applications are unacceptable in the professional world - work must always be done at work time, and the iLok Cloud scheme is completely at odds with this scheme.

The silver lining is that the traditional iLok USB scheme is still available as an option, but most pro audio companies are trending toward iLok Cloud as the main option.

part V - A symptom, not the cause.

Really, there isn't a pushback against SaaS even in the pro audio world - the effect of provenance and the status of brands far outweight the technical deficiency of the DRM scheme that cripples the industry and makes it less accessible to all.

And this is not unique to the pro audio world - this is merely a foil to get you, the reader, to think about how DRM affects you in personal life - maybe you're a FOSShead and can't watch some sporting event that's behind a restrictive FairPlay policy after already paying the monthly fee. Maybe you're an artist who needs to make ends meet by cutting out your Adobe subscription only to realize you can't pay the cancellation fee without selling your soul to the devil.

SaaS is a cancer among the world of both consumer and professional software alike. The best part? This is only one example!

part VIm

So what do we do from here?

I don't know the full answer to counter this monopolistic "as a service" model. But I know one thing for sure - we cannot let this terrible model stop us from consuming and creating - doing so will only mean that the monopolists win.

Note that I'm not advocating for piracy or breaking DRM here - I'm merely stating that isolating yourself from creating and enjoying works of art will not help the cause. Some perfectly fine solutions are buying physical media, purchasing perpetual licenses whenever available, and buying music off Bandcamp - spending as much of your money outside the SaaS ecosystem and making it as unappetizing to shareholders as possible is probbaly a good start.

Keep doing you - watching, listening, enjoying, creating, by however you see fit!

End of rant,

r/l/s

Overview

This project took place over Summer 2023 as part of the Google Summer of Code (GSoC). This project's objective was to implement the in-loop filter portion of the new Versatile Video Codec (VVC, H.266) decoder currently under development in FFmpeg. The In-Loop filters that I chose to implement are the Sample Adaptive Offset (SAO) and Deblocking filters. These filters are already implemented in C, and my objective was to implement x86_64 assembly versions that were more performant than the C "template" version.

The need for SIMD-accelereated code for this new decoder is clear - even with SIMD, software decoding of the VVC requires lots of computational power, and compilers are currently unable to optimize code to fully utilize the SIMD aspects of modern processors.

SAO Filter

SAO is an in-loop filter used in both HEVC and VVC to improve the objective quality of the reconstructed picture by supressing banding artifacts and ringing artifacts caused by quantization errors of high frequency components. The SAO filter can be split into two parts - a band filter and an edge filter.

As part of a "qualification task" to join FFmpeg's GSoC 2023 program, I had to bring over the SAO filter from HEVC to VVC. This is possible as the filter is largely unchanged between the two. During this period before GSoC, I was largely able to bring forward the band filter. However, the edge filter alluded me, and I was not able to complete that.

Thus, my first GSoC task was to complete the edge filter implementation. This was largely done by the middle of June, and my code was merged in this PR.

Deblocking Filter

As with the SAO filter, the deblocking filter can been split into two parts: the luma deblocking filter and the chroma deblocking filter. I chose to start with the chroma deblocking filter. The deblocking filters also shared commonality with the HEVC version to some degree - which I intended to port. I also tried writing the "stronger" version of these filters, but I was ultimately confined to only working on the parts that were already implemented in HEVC.

I chose to start with the chroma deblocking filter, as that had a checkasm test that would be easily modified to work with the new filter. This would guide me in working on the chroma deblocking filter. While the checkasm test did prove helpful, it also gave me a lot of issues!

The chroma deblocking filter took until early August to complete - mainly because I did not catch some pixel boundaries and I was getting lots of segfaults that I was not able to pin down. I then tried writing a version that respected theseboundries, but found myself stuck. However, I decided to benchmark the version I had that worked, and the results came back much more performant than I had expected, so my work will be submitted as such.

This left me with not a lot of time to work on the luma deblocking filter. I had honestly wished that I can say that the luma deblocker was complete to some level and in my final GSoC submission, but this is not to be. Complicating matters was the lease on the place I was renting ended on the 24th, leaving me very little time between moving to work on the filter. I had identified where the HEVC and VVC filters differed (mainly in accessing and calculating tc, and using tc to clip pixels to a limit), but YUView still shows lots of pixels where off-by-one errors on luma pixels occured.

Current State

• AVX2 SAO Code: Merged

• AVX2 Chroma Deblocker: Currently blocked by checkasm throwing an error on the GitHub Actions check. I am not sure what that is about, as on my system, checkasm reports all functions passses. Furthermore, all conformance checks pass, so I believe the issue is with checkasm and the Github Actions tester.

• AVX2 Luma Deblocker: Still needs more work. I am not sure where the error is, as I have accounted for all tc changes. Further time will be needed for the Luma deblocker to be ready.

Out-of-scope(?) work

This was entirely out of scope of my original GSoC project, but I got a MacBook Pro with an ARM-based M1 Pro chip after I had submitted my GSoC proposal. One night, after bashing my head on the chroma filter to no end, I decided to work on a distraction, and did the exact same port of the SAO filter from HEVC to VVC on ARM NEON. I spent a few hours on this and it has been merged in this PR.

Future work

• Luma Deblocker: the luma deblocker currently found in this PR and the codepack below does not work. Obviously, this needs to be fixed.

• Chroma Deblocker: shift != 0 (which I believe is where the pixels are close to an end) should be handled in assembly. The strong case should also not be too hard

• ARM NEON: Once the issues with the luma deblocker are complete and I have a sense of all the changes needed to port the luma deblocking filter, an ARM NEON port of the filter might be a good idea.

Before I go into the next part, I would like to say that I am not abandoning the FFvvc project after the end of GSoC. My availability might not be as good as I will have to juggle between classes, club activities, work, and this project. However, I will continue working on this to the best of my ability.

Where to find my code

• All code: I have made a codepack file that contains all files that I have created and/or modified over the course of GSoC 2023. Copying this into the FFvvc fork should be a good place to start if you intend to finish the work I started.

• AVX2 SAO Filter: (merged)

• AVX2 Chroma Deblocking Filter: (not merged yet)

• AVX2 Luma Deblocking Filter: (broken, not merged)

• ARM NEON SAO Filter: (merged)

Acknowledgement

I would like to extend my utmost appreciation to Nuo Mi, my mentor, who has been an amazing mentor to work with. Despite the timezone difference, Nuo Mi has been able to explain exactly what I needed in order to get to the point where I am right now. I would like to also thank the FFmpeg project as a whole for taking me up on this GSoC project.

I would like to thank my parents for being extremely supportive of me working through GSoC even if they can't quite grasp what I'm doing.

I would like to thank my roommates Tom, August, Hemant, and Kelton for keeping me sane over the summer while I was couped up in my room staring at assembly - thanks for all the beers!

And finally - thank you, the reader, for reading my submission/blog post! I'm glad I was able to keep your attention, and I hope you got something out of my overall GSoC experience!

Till Next Time,

r/l/s