# Blend Modes in Unity

You’ve probably heard about blend modes available in image and video editing tools like Photoshop or After Effects. It is an important technique for content creation and has long since become an integral part of them.

But what about video games?

Say you need to use Color Dodge blending for a particle system. Or your UI-artist made a beautiful assets in Photoshop, but some of them are using Soft Light blend mode? Or maybe you’ve wanted to create some weird Lynch-esque effect with Divide blending and apply it to a 3D mesh?

In this article I will describe the mechanics behind popular blend modes and try to simulate their effect inside Unity game engine.

In case you are not interested in the details and just looking for a complete solution to use the blend mode effect in Unity, try this package on the Asset Store: http://u3d.as/b9w

# Blending algorithms

For starters, let’s define what exactly we need to do. Consider two graphic elements, where one overlaps the other.

This is the example of Normal blend mode: each pixel of the lower layer (a) is completely overridden by the pixels of the layer that covers it (b). The most trivial case possible — most of the objects “blends” like this.

While in Screen blend mode, colors of both layers are inverted, multiplied, and then inverted again. Let’s implement this algorithm using Cg programming language:

fixed4 Screen (fixed4 a, fixed4 b)
{
fixed4 r = 1.0 - (1.0 - a) * (1.0 - b);
r.a = b.a;
return r;
}

Note we are passing the alpha value of the upper layer (b.a) to the alpha of the resulting layer (r.a), to be able to control the opacity of the material.

Overlay algorithm is conditional: for the “dark” areas we are multiplying the colors, for “light” — using the expression similar to Screen.

fixed4 Overlay (fixed4 a, fixed4 b)
{
fixed4 r = a < .5 ? 2.0 * a * b : 1.0 - 2.0 * (1.0 - a) * (1.0 - b);
r.a = b.a;
return r;
}

In Darken mode RGB components of the layers are compared and only the “darkest” left in the resulting color.

fixed4 Darken (fixed4 a, fixed4 b)
{
fixed4 r = min(a, b);
r.a = b.a;
return r;
}

Other modes are implemented in a similar ways and writing about all of them will be boring, so (in case you are interested) I put the implementation of the 18 remaining blend modes in Cg here: gist.github.com/Elringus/d21c8b0f87616ede9014

Thus, in general terms, the problem can be summarized as follows: for each pixel of the object (b) find the pixel that is drawn “behind” it (a) and blend their colors using selected algorithm.

# Implementing with GrabPass

Having all the blending algorithms implemented it may seem that only the trivial part of the work is left: to get the “a” — color of the pixels located “behind” our object. However, this step proved to be the most problematic.

In order to get that “a” layer we need to access the frame buffer, and we need to do this inside the fragment shader to be able to perform the blending algorithm per pixel. At the same time, the logic of the rendering pipeline won’t allow us to do this directly.

The final image (which contains the data we need) is formed after the fragment shader execution, so we can’t access it directly from the Cg program. Therefore, we need to find a workaround.

In fact, the need for the final image data within the fragment shader occurs quite often. Most of the post processing effects implementation, for example, is unthinkable without accessing the final image inside fragment function. For such cases, there is the so-called render to texture technic: the data from the frame buffer is copied to a special texture, which is exposed for reading on consecutive fragment shader run.

There are several ways to work with render texture in Unity. In our case, the most appropriate would be to make use of GrabPass.

GrabPass is a special passtype — it grabs the contents of the screen where the object is about to be drawn into a texture. This texture can be used in subsequent passes to do advanced image based effects.

Just what we need!

Let’s create a simple shader for UI-graphics, add a GrabPass to it and return the result of colors blending using the Darken algorithm: GrabDarken.shader

To evaluate the results, we will use the same textures we used in Photoshop during blend algorithms demonstration.

As you can see, the results in Unity and in Photoshop are visually identical. Win? Not really…

Render to texture is a somewhat “heavy” operation. Even on average PC, using more than 100 of such effects leads to a noticeable frame rate drop. The situation is aggravated by the fact that GrabPass performance is inversely related to the display resolution. Imagine what performance would be if we run such procedure on some iPad with an ultra HD display? In my case, even a couple of UI-objects using GrabPass inside an empty scene dropped framerate below 20 FPS.

# Implementing with Unified Grab

One optimization suggests itself: why not use a single, unified GrabPass? The original image remains unchanged per frame, so it should be possible to “grab” once and then use the data for all the subsequent blend operations.

Unity allows to implement the plan in a convenient way. Just pass a string inside GrabPass expression and the unified grab texture with this name will be created and used in all the passes per frame.

GrabPass { "_SharedGrabTexture" }

Now every material instance using this shader will first check if GrabPass was already performed at the current frame and will try to reuse the “shared” grab texture. Like this, we are able to perform many blending operations at once without any serious performance penalty.

Unfortunately, this solution has one significant drawback: as different objects use the same texture to get information about the “back” layer, it becomes identical to them. That is, such objects can’t “see” each other and do not consider this information when blended.

The problem becomes obvious if we stack two objects with the blend effect onto each other:

In addition, even one single GrabPass per frame may be too “expensive” for mobile devices, which means we have to look for alternative approaches.

# Implementing with BlendOp

If using GrabPass in any way is a no-go, let’s try to achieve the goal without it. One option: try to change blending mode performed after the fragment shader execution (in context of Unity rendering pipeline):

This step is mostly used for processing semi-transparent objects and there is not much we can modify here. The blending logic is controlled by the special expression, which uses two keywords to specify operations for source and destination colors.

Blend SrcFactor DstFactor

The source color (returned from the fragment shader) is multiplied by the value returned from the first operand (SrcFactor), the destination color (the color of the “back” layer) is multiplied by the second operand (DstFactor) and the resulting values are added. List of operands, in turn, is fairly limited: it is possible to use 1, 0, the source and target colors, as well as the results of their inversion.

The optional BlendOp command somewhat expands our “power”, allowing to replace the addition of two operands with subtraction, taking their minimum or maximum.

Using these instruments, I was able to reproduce the following blend modes:

• Darken:

BlendOp Min
Blend One One

• Lighten:

BlendOp Max
Blend One One

• Linear Burn:

BlendOp RevSub
Blend One One

• Linear Dodge:

Blend One One

• Multiply:

Blend DstColor OneMinusSrcAlpha

Let’s modify our UI Darken shader to use BlendOp instead of GrabPass: BlendOpDarken.shader

Using the same textures to evaluate the result.

The problem is obvious: we are using alpha blending stage for our needs, so the transparent areas of the texture are not handled correctly. On the other hand, opaque objects are rendered correctly and with great performance. Thereby, if we need to blend an opaque object and the blend mode could be reproduced with the Blend and BlendOp expressions — it could be the best way to go.

Edit 21.06.15: With Unity 5.1 release the engine now supports so called “advanced blend modes” which allows using 11 blend modes with the BlendOp function.  Device has to support GL_KHR_blend_equation_advanced or GL_NV_blend_equation_advanced extensions in order for this to work. For more info: http://docs.unity3d.com/ScriptReference/Rendering.BlendOp.html

# Implementing with Framebuffer Fetch

Previously I mentioned that it is impossible to access the frame buffer from the fragment shader. Actually, there is an exception.

In 2013, the EXT_shader_framebuffer_fetch function has been added to the OpenGL ES 2.0 specification, which allows retrieving frame buffer data from inside the fragment shader. Moreover, a few months ago, in Unity 4.6.3 release this feature was exposed to the Cg.

Let’s modify our UI shader to use Framebuffer Fetch: FrameBufferFetchDarken.shader

Perfect. What else could we actually wish? No extra operations, maximum performance, any blend logic can be implemented with ease… Except that illustration above is a piece of a screenshot captured from iPad Air. And in Unity editor, for example, our shader would simply refuse to work.

The thing is that only the iOS devices fully supports OpenGL ES specification. And on other platforms (even if their graphic subsystem uses the OpenGL ES API), the framebuffer fetch function may not work at all.

# Summary

We looked through four different blend mode implementations in Unity game engine:

• GrabPass is costly, but also gives the most correct results in any situations;
• Unified Grab provides a significant performance boost when using blend effect with multiple objects, but also prevents the objects from blending with each other;
• BlendOp is the fastest solution but also is the most limited one — only a few modes could be achieved and we can’t use semi-transparent objects with it;
• Frame Buffer Fetch is also extremely fast, allows simulating any blend modes, but could only be used on iOS devices.

I wasn’t able to find one universal solution, but using all four allows to make use of blend mode effect in the most cases.

To see all the work in action, here is a video demonstrating blend modes applied to the GUI, particle systems, 3D meshes and sprites in Unity:

And a WebGL demo: https://elringus.me/static/blend-modes-webgl/

In case someone will want to experiment with the solutions described in the article, here is a simple Unity project with the materials I used for the demonstration: https://github.com/Elringus/BlendModeExample

• Dusko Pejacki

Great job! Can’t wait to try the examples.

• Thank you!
Should any questions arise, feel free to ask them here :)

• Rebecca Fernandez

Hello – great article! I’m giving this a try, but I’m a bit stuck. The objects I want to try this with are in a specific layer in Unity that is looked after by a specific camera. Within this layer there is nothing else behind the objects, however there are layers behind this one that I would like GrabPass to grab (background, clouds, etc). Do you know of a way I can do this? At the moment I’m just getting the default background grey colour from grabpass, which is frustrating!

• Hello Rebecca,
GrabPass should “grab” anything that is rendered prior to its execution. Therefore, if those other layers are rendered before the objects you want to blend — it should work fine. I’ve made a project with the setup you’ve described and it seems to work ok: https://github.com/Elringus/BlendModeLayered

• Rebecca Fernandez

Ah – so the only difference between our setups was that I was using the unified GrabPass method. When I tried the regular GrabPass it worked perfectly.
So perhaps the background cameras haven’t finished drawing by the time the first GrabPass is called?

• Yep, it could be so. Or maybe grab with the same name was used somewhere before and captured that empty space.

• Hello, I’ve just come across your article and I think you’ve shown the difficulty in proper blending with Unity quite well. The hoops you have to jump through are quite crazy. However, I think you’ve missed a more obvious solution (or perhaps less obvious). I posted this on the Unity forums ages ago and never received a reply: http://forum.unity3d.com/threads/default-blending-incorrect.207152/. I believe the default Unity shaders are all wrong and are using an incorrect alpha blending mode. If you use your custom shaders with the proper blending mode, you will be able to blend properly using the alpha that is already written to the frame buffer.

I’ve been working on a product that does this color blending for TV. We’ve been in development for a couple of years now using all modified shaders that take this ‘proper’ alpha blending into effect and haven’t yet had an issue.

• Well, the default Unity shaders were made to support a bunch of things and work on many different platforms, so maybe the cost for this was some not-very-optimal design decisions. Besides, the source Shader Lab and CG code translates to either GLSL or HLSL, so it’s possible, that it gets optimized to some degree.

As for the color blending, I’m not sure, how modifying the blend step could help in getting the framebuffer info at the fragment shader step. As far as I understand, it’s not Unity limitation, but a more low-level API/architecture thing. Because, for example, in OpenGL ES 2.0 there is a function that allows fetching the framebuffer in the fragment shader, while in other API it’s not possible and we have to use render to texture.

• When you specify the the Blend mode you are able to tell it to use the “Destination” as opposed to the “Source”. The issue is, the destination value for the alpha is never correct. My guess is this slips by because none of the default shaders ever use the destination alpha. If instead you use a modified blending function, the destination alpha is written correctly and you’re able to use that (which bypasses fetching the framebuffer altogether — it’s already stored!). On Android you would have to enable 32-bit frame buffer to get it working correctly.

• Romain Macre

If you want to write a function of this type in a fragment shader:
fixed4 Overlay (fixed4 a, fixed4 b),
changing the blend mode is no use to store both the source and destination, which you need to use in your function.
Unless I’m missing something.

• Steph

I’m very interested in your work :D
I admit I’m still a noob at unity and programming… I looked into your example project, but couldn’t find the blend mode effect script or anything that would tell me, how i can change the blend mode :/
Sorry, perhaps I’m just a little bit stupid…. but I would love it, if I could use this :)

• It’s really simple, but you’ll have to modify the shader itself in order to change the blend mode. Just replace the body of this function: http://git.io/vEdxW with any other from the list: http://git.io/vEdx0 and it should do the work.

• Oh, strange, it seems I’ve only seen a half of your message. About the background: you are correct, the effect will use the background (or anything that is drawn before the object) to blend it’s texture with, but it will affect only the object which uses the effect. You may check this demo for example: http://elringus.me/static/blend-modes-webgl/

About the Asset Store: I’m actually selling the asset and it’s currently goes with 30% discount for Unity Pro users :) http://u3d.as/b9w

• Pingback: 混合模式的数学原理 – 破土踩坑()