Month: May 2021

Our phones have powerful Gyroscopes built in, and Unity lets us harness that power for Virtual Production. In this episode we are going to write a simple script that connects the real world with the virtual world. 

The first step is to setup a mobile device. Plug your mobile device into your computer, whether it be mac or PC, android or iPhone. Note- you will need a Mac to control a iPhone. 

On your device, download an app called Unity Remote. Open the app and Unity, and play the Unity Scene.

Next follow along to the rest of the video.

Now, all that we are controlling here is a square, this is not creative enough, we should be more creative, so lets have this representation of a camera look more like a camera In the next episode we are going to create a 3D object right here without leaving Unity. Check it out! 

Why I switched to iPhone after 12 years of android:

1.​ The camera hole. Apple design is better.
2.​ The lidar camera, scanning and autofocus use.
3. The three camera design. Same as canon pioneered back in the day, hmm canon, should you look into this “revolutionary design” in front of your eyes?!
4.​ The ability to record multiple inputs at the same time.
5.​ I wanted to switch it up.
6. ​The screen is really really bright, and really really big, epic.
7.​ 10 bit on a phone is a first I think, beats everyone else.
8.​ AR tech is something apple cares about a lot right now.
9.​ Filmic pro is a good app.
10. Screen mirroring is easier and sometimes better with apple.
11. Sensor is optically stabilized and digitally stabilized- can turn on and off.
12. The best camera is the one on you.

In the future, there will be a perfect camera. But Jeff, how could you say such a thing? In 10 months there is going to be another camera with another new feature. I’m here to say that no, one day there will be a camera that will give you total creative control over the image.

Let me explain, the camera will be the size of a phone. Big cameras and big lenses will not be needed because better digital tech will replace it. There will be light field cameras that unlike cameras today, they record rays of light coming in as data, today cameras only record light intensity. So these cameras exist today, but making it happen 24 times a second is very, very hard. Once this is figured out, the idea of focus will become a decision in the editing room, not on set. No more buzzed focus takes, no more virtual focus. Also the light field camera gives you 3D with no additional lens needed.

Second, the zoom will be a decision in editing as well. Today it’s already possible to shoot 8k and punch in to 4k.

Cameras  in the future will have 1000 megapixels and no need to have a zoom. The unique characteristics of a zoom lens, flares, and breathing, that will be added into your edit software when you zoom, it will look the same with computer aided lens distortion.

Third, the iris will not be needed.

Cameras will have a ISO so sensitive, the brightest and darkest parts of the scene will not be lost. The file sizes will be- big. 

This is the same principle as the new 32 bit recorders. They record such a big array that you don’t need to adjust the gain. 

This perfect camera is a long time away, until then I am sticking with my iPhone 12 Pro Max.

This episode we are going to explore our first tutorial, setting up video feed into unity. As of now, We implement digital video into unity through code using the WebCamtexture. 

I believe in future versions of Unity this process may become easier, but we may still do it this way by code because of how much control we have. 

If you are new to coding, Don’t run away, I will help you here I will hold your hand through this. We do not need to do any abstract coding here, just follow and repeat the code.

If you are coming from the filmmaking world like myself and you think coding is too much, think of it as just like making a video project: ten thousand little steps, we are going to go one by one and before you know it we will have done it. 

Coding lets us explore options and ideas in the virtual world that is not possible from a program with a graphical user interface. We have to look at and manipulate raw information. 

I learned how to do this from a tutorial by sloan kelly and he is extremely kind to give me his permission to revise this tutorial for you today. Here is a link to his channel, follow his socials and subscribe to his channel.

The first step is we are going to have unity and our code editor of choice setup. If you are new to the series consider watching my unity starter tutorial here.

To setup a camera feed into Unity we are going to need a capture card piece of hardware. Today thanks to overseas manufacturing and Amazon’s competitive dominance, you can get a capture card for as low as 15 bucks. This does really good 720p. Perfect for simple capture and using a camera you might already have as a webcam. Here is a step up to 1080p. We are going to use this. This device also solves a problem with a hdmi in and out. 

This product costs about $35 dollars, it’s incredible because until now, this hardware was costing five times more. 

To capture HD-SDI, there are two easy options: First, get a good capture card like the time tested blackmagic hdmi and SDI capture card. Problem solved. Secondly, you can try a SDI to HDMI converter.

As of right now, the way to get video into Unity is webcam texture. We setup a webcam button, and an input switcher, and this tutorial is great because the button is built with context in mind. 

The first step in this process is creating the object that unity will show the camera video on. Everything in Unity is an object. We make a 2 dimensional plane called a rawimage. Raw means that it will be a plane continually manipulated during runtime. We are now taking the camera of Unity and looking at the 2D image.

In addition I have made my own little ideations. I have added a simple stopwatch in the corner. This is just for fun, it’s just an idea of what could be added as useful tools for a project. Where a new tool can be made from a few lines of code. 

When you plug a USB camera option into the PC, and you run the program, Unity will automatically find the camera. This is the same way that we are going to find the camera on a mobile device, whether Android Smartphone or iPhone.

This small line of code here is what Unity reads out as the active camera, 

So, important things I want to cover here, I think this system will be replaced entirely in the near future with updates. There are many other workflows to consider for your project, one that we will implement in different situations is sending the video and the unity background and compositing them in a third party computer and system. If you separate operations by computers you can gain more flexibility and versioning, which could alleviate problems.

So give this a shot, Play around with video inside of Unity. With this as the basis of our virtual operations, anything is possible after this.