Jeff Komarow

Our phones have powerful Gyroscopes built in, and Unity lets us harness that power for Virtual Production. In this episode we are going to write a simple script that connects the real world with the virtual world. 

The first step is to setup a mobile device. Plug your mobile device into your computer, whether it be mac or PC, android or iPhone. Note- you will need a Mac to control a iPhone. 

On your device, download an app called Unity Remote. Open the app and Unity, and play the Unity Scene.

Next follow along to the rest of the video.

Now, all that we are controlling here is a square, this is not creative enough, we should be more creative, so lets have this representation of a camera look more like a camera In the next episode we are going to create a 3D object right here without leaving Unity. Check it out! 

Why I switched to iPhone after 12 years of android:

1.​ The camera hole. Apple design is better.
2.​ The lidar camera, scanning and autofocus use.
3. The three camera design. Same as canon pioneered back in the day, hmm canon, should you look into this “revolutionary design” in front of your eyes?!
4.​ The ability to record multiple inputs at the same time.
5.​ I wanted to switch it up.
6. ​The screen is really really bright, and really really big, epic.
7.​ 10 bit on a phone is a first I think, beats everyone else.
8.​ AR tech is something apple cares about a lot right now.
9.​ Filmic pro is a good app.
10. Screen mirroring is easier and sometimes better with apple.
11. Sensor is optically stabilized and digitally stabilized- can turn on and off.
12. The best camera is the one on you.

In the future, there will be a perfect camera. But Jeff, how could you say such a thing? In 10 months there is going to be another camera with another new feature. I’m here to say that no, one day there will be a camera that will give you total creative control over the image.

Let me explain, the camera will be the size of a phone. Big cameras and big lenses will not be needed because better digital tech will replace it. There will be light field cameras that unlike cameras today, they record rays of light coming in as data, today cameras only record light intensity. So these cameras exist today, but making it happen 24 times a second is very, very hard. Once this is figured out, the idea of focus will become a decision in the editing room, not on set. No more buzzed focus takes, no more virtual focus. Also the light field camera gives you 3D with no additional lens needed.

Second, the zoom will be a decision in editing as well. Today it’s already possible to shoot 8k and punch in to 4k.

Cameras  in the future will have 1000 megapixels and no need to have a zoom. The unique characteristics of a zoom lens, flares, and breathing, that will be added into your edit software when you zoom, it will look the same with computer aided lens distortion.

Third, the iris will not be needed.

Cameras will have a ISO so sensitive, the brightest and darkest parts of the scene will not be lost. The file sizes will be- big. 

This is the same principle as the new 32 bit recorders. They record such a big array that you don’t need to adjust the gain. 

This perfect camera is a long time away, until then I am sticking with my iPhone 12 Pro Max.

This episode we are going to explore our first tutorial, setting up video feed into unity. As of now, We implement digital video into unity through code using the WebCamtexture. 

I believe in future versions of Unity this process may become easier, but we may still do it this way by code because of how much control we have. 

If you are new to coding, Don’t run away, I will help you here I will hold your hand through this. We do not need to do any abstract coding here, just follow and repeat the code.

If you are coming from the filmmaking world like myself and you think coding is too much, think of it as just like making a video project: ten thousand little steps, we are going to go one by one and before you know it we will have done it. 

Coding lets us explore options and ideas in the virtual world that is not possible from a program with a graphical user interface. We have to look at and manipulate raw information. 

I learned how to do this from a tutorial by sloan kelly and he is extremely kind to give me his permission to revise this tutorial for you today. Here is a link to his channel, follow his socials and subscribe to his channel.

The first step is we are going to have unity and our code editor of choice setup. If you are new to the series consider watching my unity starter tutorial here.

To setup a camera feed into Unity we are going to need a capture card piece of hardware. Today thanks to overseas manufacturing and Amazon’s competitive dominance, you can get a capture card for as low as 15 bucks. This does really good 720p. Perfect for simple capture and using a camera you might already have as a webcam. Here is a step up to 1080p. We are going to use this. This device also solves a problem with a hdmi in and out. 

This product costs about $35 dollars, it’s incredible because until now, this hardware was costing five times more. 

To capture HD-SDI, there are two easy options: First, get a good capture card like the time tested blackmagic hdmi and SDI capture card. Problem solved. Secondly, you can try a SDI to HDMI converter.

As of right now, the way to get video into Unity is webcam texture. We setup a webcam button, and an input switcher, and this tutorial is great because the button is built with context in mind. 

The first step in this process is creating the object that unity will show the camera video on. Everything in Unity is an object. We make a 2 dimensional plane called a rawimage. Raw means that it will be a plane continually manipulated during runtime. We are now taking the camera of Unity and looking at the 2D image.

In addition I have made my own little ideations. I have added a simple stopwatch in the corner. This is just for fun, it’s just an idea of what could be added as useful tools for a project. Where a new tool can be made from a few lines of code. 

When you plug a USB camera option into the PC, and you run the program, Unity will automatically find the camera. This is the same way that we are going to find the camera on a mobile device, whether Android Smartphone or iPhone.

This small line of code here is what Unity reads out as the active camera, 

So, important things I want to cover here, I think this system will be replaced entirely in the near future with updates. There are many other workflows to consider for your project, one that we will implement in different situations is sending the video and the unity background and compositing them in a third party computer and system. If you separate operations by computers you can gain more flexibility and versioning, which could alleviate problems.

So give this a shot, Play around with video inside of Unity. With this as the basis of our virtual operations, anything is possible after this. 

This tutorial is designed for anyone to quickly learn the main buttons and workspaces in Unity.

Let’s take a look inside of Unity 3D Software and talk about how we interact with this interface.

First off, similar to other cloud software like adobe, we have a manager for the versions of the software. So step one is to download Unity Hub. Once installed go to Add and add the latest build of Unity. The choices you see here are for the device you want the project to play on, you have android, IOS, WebGL, Xbox, Ect. 

The main screen of your project is Project. The Hierarchy is where you stage your game objects. The Inspector is where you edit your game objects and other settings. The Scene is where your objects are viewable in 3D space, and lastly Game is where you can view through the virtual camera!

Inside of unity is an animator, recorder, sprite editor, render pipelines, and more. 

It is intuitive to figure out how to move the camera around. 

There are other windows that are important to bring up. Go to window > and add a window like the Console, when you make an error, this will inform you and tell you what you did wrong in your code, smart troubleshooting!

Instead of giving everyone a copy of Unity with EVERY single feature, they break the program up into downloadable packages. Its found in the Package Manager You just Install what you need without wasting space.

When you are building your project for exporting out of Unity, you use this screen Build Settings, here is the scene that you choose to build, there can he a hierarchy structure for large projects here. There are many additional settings for your projects you can adjust.

You can change the layout of Unity easily by pressing the top right corner, the three dots.

Now let me explain how Unity files work. A unity Project inside a folder, is a collection of files and folders. The main one you look for is assets, inside is where we store folders for fonts, animations, backgrounds, objects, scripts.

Looking at an untouched scene has these objects in it. A light and a camera. 

Now- what do we do with Unity?!

There is a lot we can do with Unity!

It is based around objects, everything is an object. 

We create a basic empty object by right clicking ,create, empty object.

In the inspector we are able to control the objects. 

We can control the objects by adding components here. There are different built in components. You can easily save components where you tweaked the settings to your liking as a preset

We control objects using scripts. Simple as that.

You can drag a script onto an object or go to add component and create a new one.

Open up the scrip in Visual Studio Community or Visual Studio Code these code editors have compatibility with Unity and things that make it much more enjoyable to code with. 

A blank c# script can be easily generated in Unity. At the top is what groups of code are needed to reference for the script to work.

If you are using the System in unity in a script, you need to add

using UnityEngine.System to the top of the script.

Next section is start, this is what is called when we start the script. Then is void update, this is what happens or updates every frame. So if we are at 60 frames per second, Whatever code is right here, is UPDATING 60 times per second.

Now here is how the magic works. We go above the start of a script, right here we are going to create a reference to a GameObject, and we call it this, and this semicolon initiates the line of code.

We will now Control-S to save the script in VS code. 

Now, using Alt-Tab or Command–Tab, switch over to Unity.

Unity will Automatically sense that the file in the project folder was changed, and will refresh or recompile the project for you! 

So we see here that the script is different and we can now go over to this object and drag it in the place holder. We are assigning an object in the inspector. This is a key foundation of Unity. 

Going back to The script, we are going to now be able to manipulate this object with code! How cool is that? 

We can get different components of Objects and control it with code. We can make Objects appear or disappear.

There is an incredible amount of functionality building up in this program. This is a preliminary walkthrough to get your familiarized with the look of the system and how things can come together.

I hope you learned a lot from this first walkthrough, as in the next episode we are diving in and doing our first project together! See you there.

Unity virtual production episode 3 is all about controlling the lens! Its one of my favorite topics to discuss because there is so much to it! Lenses shape our cinematography, and I make some observations about how technology is going to change our visual language and lenses will be a huge forefront of this evolution.

Lets talk about the meaning of Virtual Production! What would you define VP? What techniques make up the contemporary meaning.