How to Blend Fiction with Reality in After Effects

By Vernon Manlapaz

In late 2015, while working as a 3D-animation and visual effects manager, I found myself with a bit of free time in the evenings. Always looking for a chance to improve my skills and develop a faster workflow, I began shooting videos around Los Angeles with my iPhone and then trying to integrate believable computer-generated objects into the scenes. I imagined paper cranes being released into the wild, furniture coming to life, and full-sized Lego planes taking off at a local air show.

I'm still doing it today. I enjoy creating things that invoke wonder in people who see it. My hope is that the unexpected scenes invite you to imagine and create again, and to view the world in a different way. But first, I have to get the effect right.

One of the biggest challenges is blending the 3D object I create into the shot I’ve captured on my phone. In this tutorial, I’ll walk you through some of the steps I take in Adobe After Effects to merge fantasy with reality.

Click to play the video. Making this 3D render of a fish match with handheld footage of boaters at Echo Park required some planning while on location and fine-tuning in After Effects.

STEP 1: GET THE FOOTAGE

Once I have an idea for a new video, I capture the footage with my iPhone X using the Filmic Pro app. (It gives me manual control over focus, white balance, film rate, and more.) I tend to look for scenes with good lighting and objects on the ground with strong contrast, such as rocks or floor patterns. These details make for excellent markers when it comes time to track the footage later.

Once I have my footage, I use a Ricoh Theta V camera to capture 360-degree images of the environment at different exposures. Later, I’ll merge these images into a single high-dynamic range (HDR) image, which I’ll use as a source for reflections and indirect lighting when creating my 3D object. For this project, I captured 13 exposures to ensure I had as much lighting information as possible, but five would probably be sufficient for most purposes.

While on location, I shoot a series of 360-degree images at different exposures to use as a lighting reference later.

STEP 2: CREATE AN HDR IMAGE

After getting the footage and necessary location shots, I bring the series of 360-degree images into Adobe Photoshop to create an HDR image. In Photoshop, choose File > Automate > Merge to HDR Pro and select all of the images.

Photoshop can merge the bracketed shots into a single high dynamic range image.

Once Photoshop has done its work, you can check that the resulting image has the range you need by adding an Exposure adjustment layer and scrubbing the Exposure slider left and right. The image should preserve the details and not get washed out and grainy. Save the image as a .HDR file.

STEP 3: TRACK THE FOOTAGE

Next, I use Adobe After Effects to track the iPhone camera footage. This data, when imported into my 3D modeling software, will define where the ground plane is and how the camera is moving, so the 3D object can move in sync. With the footage open in After Effects, I choose Animation > Track Camera. So long as the footage contains high-contrast details, After Effects should do most of the work for you.

After Effects can track camera movement to export into your preferred 3D software.

STEP 4: RENDER THE 3D OBJECT

Now it's time to model the fish. In Maya, I import the camera tracking data and the HDR image I created earlier, making sure to orient the image so the scene matches the video footage—in this example, the camera should be facing the fountain.

While I don’t have the space here to go into detail on using Maya, you can easily find 3D models for use in your project online. Once I have my fish moving and lit the way I want, I export four different render passes: one of the fish itself, one of the environment reflecting off the surface of the fish, one of the fish’s reflection, and one of the fish’s shadow. By rendering each of these passes separately, I’ll have more control when blending the different elements in After Effects.

STEP 5: FINE-TUNE LIGHTING

After rendering everything, I import the fish layer into After Effects by selecting File > Import > File and choosing the first frame in the rendered image sequence.

Right away, I can see everything come together. Still, the fish doesn’t quite fit with the rest of the scene. The surface of the fish should reflect some of the blue from the water around it. To fix this, I add a Levels layer and tweak the R, G, and B values individually until there’s a bit of blue shade reflecting on the fish’s left side that's close to the blue value of the water in the footage.

I adjust the R, G, and B channels individually to pump up the blue color on the fish.

After applying color correction (see right image), the left side of the fish better matches the blue of the water around it.

Because the fish is shiny, its surface should look slightly reflective. To get this effect, I now import the render pass of the reflection on the surface of the fish. I set the new layer’s blend mode to Screen and set Opacity to 30% for a more subtle effect.

First, I import the reflection of the surface of the fish separately for better control. After adjusting the reflection layer’s blend mode, the fish has an added sheen.

STEP 6: ADD THE REFLECTION

To make the fish appear to be floating over the lake, I need to add a reflection to the water’s surface. 

Moving forward in the timeline to where the fish crosses over the water, I import the rendered reflection pass. At its default setting, the reflection is too stark: It should ripple like the water’s surface and appear mostly translucent.

To fix this, I first set the reflection layer’s Opacity to 30%.

Then, to make the reflection better match the water’s surface, I use three different effects, each located in the Effects menu: a Directional Blur set to match the direction the waves are moving; a Fast Box Blur for an overall softness to the edges; and a Ripple effect, which distorts the reflection to match the waves. The goal is to match the fish’s reflection to those around it, such as the swans in the lake.

After an opacity change as well as blur and ripple effects, the fish’s reflection looks far more natural.

STEP 7: ADD A SHADOW

In addition to creating a reflection, the floating fish will also create a shadow, both on land and water. Returning to the beginning of the timeline, I import the rendered shadow pass.

The imported Shadow render is far too crisp.

As usual, it needs a lot of work to look natural. The good news is that I can use the same settings as the reflection pass to match the flow of the shadow over the water. With the shadow layer selected, once again apply the Directional Blur, Fast Box Blur, and Ripple effects.

Now with the Fish, Reflection, and Shadow layers turned on, I have what you see below.

To finish, I go to Composition > Composition Settings and crop the video to 1080x1080 for Instagram, and then export the finished file.

After more than a decade of working professionally in visual effects, and more than four years of making my own personal projects, I’ve learned that patience is the key to improving your skills. While modern computers can render objects and animations more quickly than ever, learning the tricks of the trade still takes time. So keep at it and have fun.