What video format?
One of the early questions that comes up when people start using Painting With Light is “What video format should I use?”. Digital video is an complex, ever evolving subject but there is fortunately a simple answer to this question, which we will get to in a minute.
The complexity arises because video data is generally large – far too large to transfer the raw data over the Internet or contain a whole movie on a Blu-ray disc so this data requires processing and compressing.
Unfortunately there is no one video format that fulfils all possible uses and there are a bewildering amount of options available when choosing how to save your videos.
Because of this, it’s useful to know a little about how video files are constructed.
Video (and audio) files generally consist of a container that wraps one or more streams of data (video, audio, subtitles, etc) each of which might be in one of hundreds of formats.
Different operating systems and software applications can favour different container formats over others. Apple computers work well with QuickTime MOV files, while traditionally Windows computers favour AVI ones.
These days you will no doubt have encountered others such as MP4 and MKV, for video, and MP3, FLAC, OGG, for music.
Wikipedia has a much more detailed page on Digital Container Formats.
Each stream of data is chopped up into packets and added to the container so you might have several packets of audio data followed by a packet containing a video frame, followed by some more audio packets.
And each of these streams needs to be processed by a piece of code called a codec that knows how to read and write the actual data before it can be used by an application. A codec will generally add some level of compression to make the original data smaller.
Wikipedia has lots of information on Codecs
Choosing the right format
To cut a very long story short, there are some key aspects of video we need to keep in mind when choosing what formats to use with Painting With Light.
There are always exceptions to every rule, but here’s some to get you on the right track.
While high definition video is now commonplace on consumer computers, it doesn’t always make sense to use it for every application.
When you use a video projector, this has a native resolution which is the highest it can physically run at. While it might accept higher resolutions from your computer or Blu-ray, it’s still limited to its hardware, and will scale the source image down if needs be.
Rule #1: your video resolution never needs to be higher that your projector resolution
When video mapping, you may not actually cover the whole scene with a video. If you’re mapping a video to the side of a box and it’s only taking up a small part of the output (such as the example in the image below) there is no point using a large video resolution for such a small area.
Smaller resolutions means less processing power is required to decompress the video, meaning you can use more videos simultaneously.
Rule #2: always scale your video resolution down to match what you’re going to use it for
Depending on the video codec, there might also be some benefit or requirement to using video resolutions that are divisible by 8 or 4. So a video that is 320 pixels wide would be fine, but one that is 326 might not work so well (or at all).
Rule #3: make your video resolution divisible by 8 both horizontally and vertically
Painting With Light will load most video containers but some are better supported than others.
Rule #4: use MP4 videos, if you have the option
Compression can happen on a per-frame basis (spacial) and/or over time (temporal).
Spacial compression looks at each frame separately. MJPEG (Motion JPEG) is one example, where each frame is processed with the same compression as in JPG images.
Temporal compression is excellent for videos that are designed to be played forward at normal speed. It stores one whole compressed frame but follows it with frames that describe only the differences between it and the new frame.
If you try to decode one of these difference frames without the original key frame, you get this kind of effect. This can also happen if your video file is damaged.
When exporting videos you can sometimes specify how often key frames should be used.
Rule #5: export temporal videos with key frames every 1 frames (makes every frame a key frame)
Rule #6: optionally, it’s better to use a spacial compression codec such as MJPEG
The Best Format
There is a unique video format built into Painting With Light that will ensure the best performance.
You will still need to follow the rules regarding video resolution but once you have a video that you want to use, just follow these simple steps:
1. In the Textures window, click Load…
2. Select the video you want to convert
3. Click on the thumbnail of the video
4. Click on Convert…
5. Choose where to save the converted video
6. Choose DXT1 (the other formats will be documented later)
7. Choose the compression speed (the slower the speed, the better quality of the final output)
8. Wait for the conversion to complete
9. Click Remove to get rid of the original video
10. Click Load… again and choose the new DXT video file
11. Use the new DXT video as you would any other video file
Why is DXT a good choice?
Your graphics card supports DXT compression directly, which means that Painting With Light hardly has to do any processing on the video data other than loading it off your hard drive.
Spending less time decompressing video means Painting With Light will be able to use many more simultaneous video streams in your work, and at a higher resolution.
For even more speed, use a SSD hard drive to store your video.
There is a downside, of course – the output quality is not quite as good as more modern compression codecs on smooth gradients. You may see bands of colour across your video.