These fixes are mainly to address performance issues related to MetaSounds on the first frame when told to play. For more information on the problem and profiling done to identify it, you can scroll down to after the fix. Note: MetaSounds is in Early Access and there is no doubt in my mind that the power and performance of MetaSounds will be exponentially better by the time UE5 is released officially. These fixes are for folks like me who are impatient and want to try out ideas that were just not possible to do easily in UE4.
1. Re-Enable Asynchronous Sound Building.
According to Aaron Mcleran, this used to be on but they ran into technical issues working on Valley of Ancients and had to turn it off temporarily to fix other issues. Now those issues are fixed and we can re-enable it.
//ENGINE OVERRIDE - Enabling Async Task - Updating Graph seems to cause issues when doing the builder async
//BuilderTask = nullptr;
// ENGINE OVERRIDE END
2. Set BlockRate to 50 to ease CPU cost of playing a MetaSound
3. A bit hacky – Cache the Metadata into a Map to reduce the overhead of searching through for the same type of Metadata.
//ENGINE OVERRIDE - Caching pins to reduce cost by half for processing.
TMap<FName, FMetasoundFrontendClassMetadata> CachedMap;
// Inject receive nodes for each transmittable input
for (const FSendInfoAndVertexName& InfoAndVertexName : SendInfoAndVertexes)
bool bSuccess = false;
FMetasoundFrontendClassMetadata ReceiveNodeMetadata = FMetasoundFrontendClassMetadata();
//UE_LOG(LogTemp, Warning, TEXT("TypeName %s"), *InfoAndVertexName.SendInfo.TypeName.ToString());
bSuccess = true;
ReceiveNodeMetadata = *CachedMap.Find(InfoAndVertexName.SendInfo.TypeName);
ReceiveNodeMetadata = FMetasoundFrontendClassMetadata();
bSuccess = GetReceiveNodeMetadataForDataType(InfoAndVertexName.SendInfo.TypeName, ReceiveNodeMetadata);
//ENGINE OVERRIDE END
4. Make sure to enable stream caching and force streaming on your audio files.
The Problems (More Details) – Synchronous sound building and Slow Metadata Searching On Input
I have recently jumped into UE5 to test out MetaSounds for some cool ideas for Project MIX. First thing I noticed however, is that runtime performance is severely impacted by large hitches during play (Remember that UE5’s MetaSounds are super super hot off the press, nothing is set in stone and everything is likely to improve and change).
Here is the MetaSound I am using for example. Nothing too complex, but I am playing many of these at a time so its understandable that there would be some performance concerns.
When playing multiple sounds on the same frame (around 5-10), we see these massive spikes. Processing Sources – 37.044 ms FAsyncMetaSoundBuilder – 10.418 ms.
The first thing was getting some great insider knowledge from Aaron Mcleran saying that FAsyncSoundBuilder was actually built to run asynchronously. (Fix is above for that)
When going through and switching it back to an Asynchronous task we now get this
What we see now is the FAsyncMetaSoundBuilder issue is now gone without any real delay issues even with using the metasounds with Quartz.
This now leaves Processing sources as our next big hitter. After a few hours of digging I was able to narrow it down to this function in MetasoundSource.cpp
Every time FindClassesWithClassName is run, it can hit the cpu anywhere from 0.5ms to 2ms depending on complexity of your input graph pins. Essentially if your Metasound has 100’s of inputs, you will see a cost of about 1ms per pin when the sound plays, which is something that is likely to be improved. FOR NOW up above, I have a bit of a hacky fix where we make sure to only run this function once per pin type. So if you have 10 float inputs, it will cost 1ms to gather the metadata instead of 10 ms.
The reason is it seems to have to re-execute a query in order to get the proper meta-data. This to me feels like it is something that will be improved upon since it doesn’t quite make sense why it needs to do that on-play.
After the hack fix in MetasoundSource.cpp listed above, we see this as a result.
You can see the largest spike now happens much more rarily and is MAX about 12.8ms. Still reallllly high but comparatively now at least the game is playable. Hopefully this helps other folks like myself who want to dive into some awesome new tech!
In short, Project Mix is a visual node-based music creation game that is currently in early alpha stages. The goal is to allow people who want to make music the ability to do so real-time while those who just want to listen or have very minimal interaction can play with mixes that others create. Think of it like Little Big Planet’s Edit mode vs Play Mode.
Where it started.
Really it was just a simple question that sparked this entire idea. Can nodes connect to other nodes to create sound in a fun way? Walking down this path has lead me on an incredible journey so far.
Very first proof of concept
First Pre-Alpha Composition
Where it is as of July 2021
Timing, Features, and refining of the base visuals.
I already went over a ton of this stuff in this blog post. Basically I had to nail down critical timing issues and the overall concept of how I’d like the nodes to actually connect to each other. Plus, it was time to create the bulk of the features. But here are some cliff notes.
Ok! So if it looks good then why not release it now?
Well… 2 things, Licensing and User Experience. I have to make sure I cover my bases when it comes to having people share “Mixes” with each other using the game’s format. So I am currently working to secure licenses for sounds I would like to use in the game.
For user experience, I really want to have something that feels fun to edit. Right now its still quite a bit more complex than I’d like so I am honing in on some really cool ways to achieve this! Also, I’d like to integrate it to a workshop of some sort to help make things easier for players to share their creations with each other.
Basic Menus (Yanno know like loading and saving etc)
User Experience Upgrade allowing for easier editing and usability
Modulators (Will allow ways to adjust compositions in fun ways)
Oh yea… SOUNDS! Refining the actual sound library mix is going to use as I get permissions!
Something I have been dabbling with more and more is a randomized version of project mix that can generate its own melodies and beats to keep things interesting enough for anyone to hop in 24/7. This could also, in the future, cycle between other player’s creations endlessly which I think would be fantastic!
Potentially this idea could even be a “twitch plays music” type thing as well in which players can input commands via chat to tell project mix what to do! This is all just shooting in the air at the moment and really will change its priorities based on feedback and overall attention.
Thats it! Twitter is where I am the most active, so please lemme know what you think on there!
It has been quite awhile since I have done a formal write-up going over the details of project mix and what to do next. Right off the bat, we have some sort of an art style starting to evolve! However, I think there might be a few other options I’d like to explore.
Going From TimeSynth to Quartz in UE4.26
As much as I loved the Time Synth, this is something that was absolutely crucial to making progress. I was spending so much time trying to integrate features that already exist for every other audio thing in the engine and fix audio crackles / decoding issues with the Time Synth Component. The good thing is, I now have way more of an idea of how audio actually works in unreal! I scrapped about 1000 lines of engine override code and now directly integrated everything using quartz.. it was a ton of work but ended up being really powerful!
What is Quartz? This is the new subsystem that allows you to Play any audio component quantized to a beat clock. This means that you no longer have to activate 5 plugins to get something like this working AND .. the biggest win is it is truly available on ANY AUDIO COMPONENT. Which means it also works with audio cues!
Here is a very very basic setup of Quartz.
On Begin Play I get the quartz subsystem and make a new clock! This node also allows you to override an existing clock which means its safe to call multiple times. Clocks are sorted by name and can be called upon by name later too so make sure you have an actual name in there. From there I save the quartz clock as a variable and then subscribe to a quantization event. You can think of this as hooking into the system so that every time a beat plays, it fires an event.
NOTE: This event is always going to be at the start of a beat, which means that it will give plenty of time (If your frame-rate isn’t terrible) to setup the audio that is needed to play next beat. This part has always been tricky to work around even with the Time Synth. Basically you have to treat everything like a queue and make sure that whatever system you are writing is ahead a certain amount of time so that when you call this “Play Quantized” node on an audio component, it is exactly 1 beat behind where you want it to actually play.
If you’d like to see more, you can head over to “Dan Reynolds Audio” on Youtube. He goes over a pretty nice introduction of exactly this type of thing and a little more details.
The point here is that there is so much potential with how to use this system I could talk for hours about it!
How does Project Mix work with Quartz?
Some of the key things that quartz did was fix timing issues. After awhile TimeSynth would desync and become rather unreliable. It also had quite a few crashes that I was accounting for in a really bad way!
Everything syncs to a beat!
The beat clock is the most important thing to a system like this. Quartz Subscription to quantization event means that we can essentially treat one of these as our main driver for activation! So starting off, I subscribe to a quartz event on 1/16 notes. 1/32 is a bit too close to tick for comfort currently and complicates the process of setting up timing a bit too much in my opinion. I might switch it later, but I’ve found 1/16 to be the most reliable.
Once subscribed to a 1/16th note quantization event, I then send this event to every node and link in the system. From here they will decide what to do based on which beat they think they should be activated on next. Remember what I said before where you have to queue up a beat before-hand in order to properly quantize? Yea…. that makes thing quite a pain is the ass! However, essentially what I do is just make sure the entire system is a beat ahead so that to the player, they won’t really know the difference!
To understand how everything works with project MIX a bit more in-depth lets talk about nodes and links. In the MIX system I call them Devices and all of the Devices operate from a brain that I call “The Machine”
This is where the heartbeat is established and everything related to events for the system is handled in here as well. The machine’s job is to essentially distribute the heartbeat to every device in the system! This also means we can control order of ticking, what actually is ticking, etc all from one place without having to do custom tick groups or any junk like that.
Machine Device Nodes
These operate simply be being the sound makers and general activators. They have access to links and also have all of the information on what sample they are supposed to play. We can go over samples later! Simply put though, nodes are the intersections in a highway type system that tell other links to activate and whether or not to play sound.
Nodes can have an infinite amount of links that activate it or that need to be activated after the node is activated.
Machine Device Links
Links are connected to a START and and END node. When a START node is activated, we then tell the END node that it should be activated in however many beats the link is active for. Links can only be activated once per duration of activation. This means that if a Node is to fire off twice while the link is still active, it will ignore the second activation request. Simple enough right? Well… this is where things get hairy.
Solving some pretty critical sync issues! ( a bit of a ramble but could be useful to someone in a similar tricky scenario)
One of the trickiest hurdles to solve was the order of beats. Imagine this scenario, Nodes and links can activate when they think they should be active. However timing and everything is very important to be setup during these activated states. For instance.. when a Node gets activated, it sets up how long it needs to be active for and tells links that they should do the same. Unfortunately, there is really only one event that gets sent out to do this. Which means in some cases when dealing with 1000’s of nodes all trying to figure out when they should activate, you run into situations where the activation was sent but the order of activation ends up making it so that it needs to wait until the next heartbeat in order to properly activate it! Which is not ideal at all since 1 more heartbeat would lead to the result of something being off sync. The fix was making sure to treat the state machine on the devices as a desired stated instead of instant changes. This allows for a loop to constantly run that handles the state changes which gets populated anytime the state needs to change. This allows for the order to work as requested and for everything to be finished in the same heart beat event!
For those that are C++ savy, here is what I mean.
When a device state change happens, we add it to an array on the machine that is constantly running until it finishes during the quantization beat event.
void AMIXDevice::SetDeviceState_Implementation(EMIXDeviceState NewDeviceState)
if (NewDeviceState != NextDeviceState)
NextDeviceState = NewDeviceState;
The event that handles sending out the quantization event to the devices.
void AMIXMachine::SendQuantizationToDevices_Implementation(FName ClockName, EQuartzCommandQuantization QuantizationType, int32 NumBars, int32 Beat, float BeatFraction)
// Send quantization events out to connected devices.
for (int32 DeviceIndex = 0; DeviceIndex < Devices.Num(); DeviceIndex++)
AMIXDevice* Device = Devices[DeviceIndex];
if (IsValid(Device) && Device->QuantizationType == QuantizationType)
Device->MachineQuantizationEvent(NumBars, Beat, BeatFraction);
// Now handle state changes properly. This is a dynamic array that will get populated any time a state change happens!
for (int32 DeviceIndex = 0; DeviceIndex < DevicesNeedingStateChanges.Num(); DeviceIndex++)
AMIXDevice* Device = DevicesNeedingStateChanges[DeviceIndex];
if (IsValid(Device) && Device->QuantizationType == QuantizationType)
DeviceIndex -= 1;
The result is this! The ability to accurately handle about 200 devices in a scene all at 90fps at 300bpm with 1/16 notes!
This is very very much still subject to change, but one of the defining characteristics of project mix is the ability to compose something out of nothing. This requires a sample library that is setup in a way to handle multiple velocities and per note samples. The DAW that I primarily use is FL-studio. Ultimately I would love to find a way to export directly from FL-studio into the library format I use so that I could compose an entire song and get it imported into the link and node format (Maybe some day! Baby steps first!)
It Starts with a DAW
In FL-studio I setup a project that has markers for each note on an 88 key note scale. All Samples are recorded in a way that is faithful to the copyright agreements for each of the different companies I obtain them from. This being said, I still needed to contact each company individually to make 100% sure that it wasn’t in violation of their terms as some of them require further licensing/permissions before allowing something like this to not be considered “Reselling a Sample Pack”. Lastly, nothing is used without modification and nothing is saved off 1-1. Furthermore when the product is cooked, it is compressed by 40% so they will never be the same as buying the plugins, kits, or whatever and using them for production.
Importing Samples Into Unreal
Now that we have the files setup, we can go ahead and get them into unreal. This part is mostly tedius, but I have setup a tool in unreal using an editor utility widget that can search and parse the files by name to get them into a data format I expect.
The folder structure I use is Audio/Samples/Type/Velocity. (Soft ,Medium, and Hard). Velocity refers to how hard a key was pressed on the piano. This allows us to blend between sounds rather nicely.
From here, I need to generate the Audio Synesthesia data which will be used by the system to generate visualization data for the nodes to use when the sound activates. I was able to do this by making an editor utility widget to handle creation.
Once I have that all setup, I create a data sample asset which is a custom one that has a TMAP in it for storing all of this per note. ANOTHER NOTE: (Pun intended) whenever setting up data structures like this in unreal its extremely important to use soft references or primary data assets. This allows you to handle the Loading of your data manually instead of unreal thinking you need to load everything whenever you reference the data structure directly. Its really good practice to do so and will prevent turmoil in the long run!
Node Settings for Samples.
On each node I made a structure for playing samples.
Channel refers to what track on the final mixer to use. There are 16 available so that I can mix each instrument individually and then on top of that each sample has volume controls too. This is also something that is pretty subject to upgrades and changes since its widely going to be how complex I am wanting to go in the future.
Utilizing the basics of Audio Synesthesia for loudness visualization.
Why audio synesthesia and not just the build in FFT data?
I have found that the built in FFT system doesn’t account for perceived loudness nearly as well and also has points where it seems to not be as accurate as I would like it to be. Audio Synesthesia is pretty spectacular and decently easy to use.
In the future I would love to do spectral information too and not just loudness, but for now I am starting off simple. All of those NRT assets I created now can be utilized when making the sounds. This is actually a ton more complex than anticipated. In blueprint they require you to bind to an active sound in order to get the current playback time of a sound component. I CANNOT use blueprint that much anymore due to the CPU Overhead of the blueprint virtual machine. In code I really didn’t want to set this up with delegates, it would just massively complicate the process of checking playback time on potentially 100s of active sounds. So instead, I added some engine overrides to the audio component.
This gives me the playback time on every sound that is playing from that component. One issue you’ll notice out of the gate is “FindActiveSound” has an audio thread check in it to ensure that its only called on the audio thread… I just commented that out that for now. I could potentially route it much nicer later using the proper “Run On Audio Thread” commands, but I have found that the audio team in particular is a bit overzealous with their check()’s in code to make sure the audio system is as stable as possible (Makes a ton of sense, but also means that most of the time it seems to be ok breaking some of these rules).
Using that, I can then gather the needed information for the perceived loudness of a sound on tick for any of the active sounds in my audio pool.
The Custom Audio Pool on The Machine
When a node plays a sound, I end up making an audio component on the fly and saving a reference to it in an audio pool structure. This is where I can then grab all of the sounds on tick and update the visual information for each sound in the pool. When they are finished I can then make sure they are properly removed. With this method, I am only processing what I need to and then I can also handle the proper routing of visual information as well. This information is 100% stored on the machine so that it only gets calculated once! Any information the node needs is sent over to it after its calculated.
Node Visuals Using the NRT Data
Finally we then can use this to drive a material that handles the visuals! This part is still in blueprint, but likely will be moved later.
For the nodes, we have 2 fresnels. On for the sharp outer line and another for the sharp inner circle. This is placed on a spherical object and based on the settings we can control how much it animates. I called the parameter “Size” initially but it really should be called ActiveLoudness.
Material for nodes
The links and any other visualizer on the project has access to the NRT data and can use it knowing the only cost is the animated visual itself rather than calculating the data again.
If anyone has any questions about
What’s Next for Project Mix?
There are a ton of roads that I could take for a project like this. Some people have suggested having the interactivity being driven from a DAW like ableton real-time for live concerts or experimentation. I think that would be pretty sweet but also super complex and I am also 99% sure epic is already doing something like that on a much larger scale for artists. So my initial focus is going to be as more of a toy beat maker thing.
Getting Interactivity with the nodes in a playable state.
Making new nodes on the fly in-game
Controlling links in-game
Potentially making links distance-based for their timing which would help gamify it a bit.
Sequence Nodes (Allowing nodes to play on a sequence)
Next Major Milestone is to be able to start with a blank level and make an entire composition from scratch. This will help prove the system out on a scale of usability that will allow it to grow wings in other areas. I am excited! Woot!
Thanks for making it this far and if you’d like updates or to stay in touch be sure to follow me on Twitter and/or subscribe to me on youtube.
So, I had an idea. I wanted to make fireworks as a way to celebrate the 4th of july. And I figured lets try to do it with project mix.
What you see in the video is 150 Connected nodes. Each node has their own note and sound settings.
All of these are defined by presets in which I manually create by importing a wav file from FL-studio with each octave I want to have available to use. Like I’ve talked about in the past with the post about audio visualization using the baked analysis that unreal just added, there are only a handful of actual sound files which makes my life so much easier when it comes to quick importing and management. However, I don’t currently have support for just.. One-Shot type sounds.
I tried to take an approach to create this that is a bit additive. I started with the basic kick to get the loop that I wanted to use, layered in the snares and high hats, and then went to town emulating the “firework” feel I was going for. This meant that I needed to create a few buttons to help make things faster like Randomize Notes, Randomly place in a spherical pattern, and selection helpers.
Right now there is NO MIXING setup at all. This is about as raw as audio playback can get in unreal. Just play the damn sound… This is also why certain parts might sound a bit too loud or soft. So the next phase of development for this system is pretty clear. Project Mix.. needs.. well.. a MIX of some sort. But that’s not the most worrying thing to me. I think the biggest issue with development right now is that I have to hit play.. wait for the loop to start… and hope that it sounds the way I want. This makes things extremely slow and aggravating to work with. So, I think I am gonna take an interesting step forward into editor-based nodes that actually play .. while I am tweaking them (Obviously with some control on when to play). Imagine kinda like planet coaster, where the roller-coaster is running along the track you are currently building. This is something that I think is needed in order to not rip my hair out creating the next scene and ultimately.. the final vision for this project ( Which I am currently keeping to myself ).
Then… well there is the time synth.. which aims to solve all of my synchronization problems… so I gotta get going on that too. I did watch dan’s amazing overview of it and I think it has a ton of potential with one fatal flaw… time synth components are made to not be used spatially… 150 times in a scene. Therefore.. I’d have to tell the nodes to call the master time synth in order to play things on time efficiently. Maybe if I do that I can convince the awesome audio folks to help me out with getting 3D sound to work properly per node.
So to finish it off.. here is a few bullet points for the future.
Better Editor Development workflow
An actual soundclass and submix setup.
Update the core audio system to use the time synth component instead of my jenky synchronization master blueprint.
Those three are enough to keep me busy for awhile. On that note, have a great day <3
Hi, welcome to the first post on my website. This is pretty rad woop!
First off, what is Project Mix?
I’ve been exploring with an interactive node-based visual audio simulation in UE4 purely as a way to dive deep into the audio engine’s potential and to try and get something that I think will be really interesting fun to mess with in my free time. The video below shows where it has been sitting for the last month since I messed with it last. Its got a ton of work left with it, but I think it is starting to show some interesting promise.
More Refined Testing
4.22 Added a pretty amazing feature to unreal’s audio engine.
Baked Spectral Analysis Curves and Envelopes on Sound Waves
Sound waves can now be pre-analyzed for envelope and spectral energy to drive Blueprints during playback. This allows sound designers to create compelling audio-driven systems, while offloading the spectral analysis work to improve runtime performance. In addition, analysis data from a proxy sound wave can be substituted for a sound wave’s analysis data, allowing designers to spoof isolated sound events when trying to drive gameplay.
Every .wav file now has settings on it for being able to analyze the data and output it as something that could potentially be used for gameplay / animation techniques. Well.. turns out I might have a use for both of those things. For now I wanted to see how it works and how easy it was to implement.
Here are some settings I am experimenting with inside of the wav file uasset.
Boring Explaination of why I chose these numbers. Everything is pretty much maxed out and from what I could tell the uasset does not appear to gain size by much when making these values very large. Honestly, they are all undocumented so I have no idea if cranking up these values or adding more and more frequencies to analyze is a horrible thing. I imagine it would be, but right now ¯_(ツ)_/¯
The biggest ones to pay attention to are “Frequencies to Analyze” and “Frame Size”. I am guessing that essentially if I have a 2048 FFTSize and a 4096 frame size then I essentially am allowing 2 frames per section of data? Maybe? WHo knows? WHAT I DO KNOW is that the frequencies to analyze actually do matter and it will do its best later on to try and match or lerp between the correct frequencies if you are trying to say grab something in the 300hz range but there is only data for 100hz and 500hz. (That was actually some nicely commented code under the hood. Thanks Epic!)
Blueprint Time. (Its actually not that bad)
Ok.. this part actually isn’t as hard as it might seem. Really this was super easy overall, just had to find the right nodes.
GetCookedFFTData is the magic node we are looking for and you get it directly from ANY AUDIO COMPONENT. It’s just built right in now which is fantastic. You hook this up to Event tick and you’ll get the Magnitude of the frequency you want whenever that sound is told to play. Since its directly inside of an audio component that also means that you don’t have to sync data up… I am .. really impressed it has this much integration to be honest. I was expecting like some back alley bridge of connectivity between 3 different systems.
With a little bit of a lerp on the magnitude to help smooth out some jitters, the result is not bad at all and I am rolling forward with this as a part of my sound player base class to be able to easily animate things with that data. Essentially now I get animation curves … for free? Hopefully the max settings thing doesn’t bite me later on. To test it I just replaced what used to be 3 animation curves ( One for attack, sustain, and release ) and now have just the straight up data from from the analysis. Which I think is pretty awesome. I am gonna further explore the capabilities of this as I harden a few more of the core systems for this thing. You should be able to notice the animations are much more accurate to the sound you actually hear.
I am gonna keep going. If anyone has any questions / ideas / feedback I am all ears! But that’s all I got for now. I am gonna try to post regularly but … who knows? Just trying it out.