Multifiles
A multifile is a file that contains a set of files, similar to a .zip or .rar archive file. They are meant for containing multiple resources such as models, textures, sounds, shaders, and so on, and Panda can load them directly from the multifiles without having to unpack them first. Many games employ a similar concept of “data” file such as .upk for Unreal Engine and .pak for Quake Engine.
The multify program
The multify console program creates such files. You can get information about
the commandline parameters by running multify with the -h
option. This is
how program describes itself:
Usage: multify -[c|r|u|t|x] -f <multifile_name> [options] <subfile_name> ...
multify is used to store and extract files from a Panda Multifile. This is similar to a tar or zip file in that it is an archive file that contains a number of subfiles that may later be extracted.
Panda’s VirtualFileSystem is capable of mounting Multifiles for direct access to the subfiles contained within without having to extract them out to independent files first.
The command-line options for multify are designed to be similar to those for tar, the traditional Unix archiver utility.
Read Assets
If you want to prepare to read assets from a Multifile directly, you can “mount” it into the virtual file system:
from panda3d.core import VirtualFileSystem
from panda3d.core import Multifile
from panda3d.core import Filename
vfs = VirtualFileSystem.getGlobalPtr()
vfs.mount(Filename("foo.mf"), ".", VirtualFileSystem.MFReadOnly)
If you want to read assets, you can mount a whole directory structure from a webserver.
If your webserver hosts:
http://localhost/mydir/models/myfile.bam
http://localhost/mydir/maps/mytexture.png
Put this in your config.prc:
vfs-mount-url http://myserver/mydir /mydir
model-path /mydir
Or, equivalently, write this code at startup:
vfs.mount(VirtualFileMountHTTP('http://myserver/mydir'), '/mydir', 0)
getModelPath().appendDirectory('/mydir')
and then you can load models like this in your Python code:
model = loader.loadModel('models/myfile.bam')
texture = loader.loadTexture('maps/mytexture.png')
If you want to prepare for reading and writing assets to a Multifile do the following.
from panda3d.core import VirtualFileSystem
from panda3d.core import Multifile
from panda3d.core import Filename
mf = Multifile()
mf.openReadWrite("models.mf")
vfs = VirtualFileSystem.getGlobalPtr()
if vfs.mount(mf, ".", VirtualFileSystem.MFReadOnly):
print('mounted')
If you want to prepare for reading and writing assets to a ‘subdirectory’ Multifile do the following. Note “mysys” must always be literally written in any python code. E.g. “mysys/memfdir/mfbar2.txt”
from panda3d.core import VirtualFileSystem
from panda3d.core import Multifile
from panda3d.core import Filename
mf = Multifile()
mf.openReadWrite("models.mf")
vfs = VirtualFileSystem.getGlobalPtr()
if vfs.mount(mf, "mysys", VirtualFileSystem.MFReadOnly):
print('mounted')
If you are having problems loading from multifiles you can list the complete contents of your .mf file with a command like:
multify -tvf mymultifile.mf
Doing a sanity inspection like this can be useful to ensure that your assets are in the right place within the multifile.
Multifile objects
The Multifile
class is designed for opening, reading and
writing multifiles. You can open a new multifile by creating an instance of the
class and calling the openRead()
method:
from panda3d.core import Multifile
mf = Multifile()
mf.openRead("foo.mf")
The openRead()
method opens the multifile as read-only.
If you want to make changes to it and write it back to disk, you will need to
use the openReadWrite()
method.
Also, there exists openWrite()
to create a new multifile.
If you have made important structural changes to a Multifile, it is recommended
to rewrite the multifile using the repack()
method.
(This won’t work if you’ve opened it using openRead()
.)
If you are uncertain about whether it has become suboptimal, you can call
neesd_repack()
which returns True if the Multifile is
suboptimal and should be repacked.
To write it back to disk, you can use the flush()
method
which flushes the changes you’ve made to the multifile back to disk, or the
close()
method if you’re done with the file.
To mount Multifile objects into the VirtualFileSystem without writing them to disk first, here’s an example on how to mount them:
mf = Multifile()
#... now do something with mf
vfs = VirtualFileSystem.getGlobalPtr()
vfs.mount(mf, ".", VirtualFileSystem.MFReadOnly)
Subfiles
Files that are added to a multifile are called subfiles. You can add existing
files to a multifile object using the addSubfile()
method.
This method takes three arguments: the target filename, the existing source file
and the compression level (1-9).
There is also updateSubfile()
, which does the same thing
but if the file already exists, only updates it if the content is different.
There are several other methods which operate on subfiles, which you can find on
the Multifile
page in the API Reference.
Here are a few examples of working with subfiles:
from panda3d.core import VirtualFileSystem
from panda3d.core import Multifile
from panda3d.core import Filename
m = Multifile()
# Add an existing real os file with compression level 6
m.openReadWrite("foo.mf")
m.addSubfile("bar.txt", Filename("/tmp/bar.txt"), 6)
m.flush()
# Destroy the contents of the multifile
# Add an existing real os file to be the first multifile
m.openWrite("foo.mf")
m.addSubfile("bar.txt", Filename("/tmp/bar.txt"), 6)
m.flush()
# Permanently re-order in ascending order the
# directories and files in the multifile
m.openReadWrite("foo.mf")
m.repack()
m.flush()
# Open a multifile and replace the contents of the mulifile file
# with new contents
m = Multifile()
m.openReadWrite("foo.mf")
m.updateSubfile("bar.txt", Filename("/tmp/bar2.txt"), 9)
m.flush()
# Open a multifile and extract all files smaller than 3kb
# New real os files are created with the contents of the multifile data
m = Multifile()
m.openRead("foo.mf")
for i in range(m.getNumSubfiles()):
if m.getSubfileLength(i) < 3 * 1024:
m.extractSubfile(i, Filename("/tmp/" + m.getSubfileName(i)))
# Find, print and remove a file named bar.txt
barIdx = m.findSubfile("bar.txt")
if barIdx != -1:
# It returns -1 if it doesn't exist
print(m.readSubfile(barIdx))
m.removeSubfile(barIdx)
m.flush()
m.close()
If the foo.mf file were to have a contained bar.egg.pz file, load the egg and use it similar to other model loading methods.
nodepath = loader.loadModel("foo/bar")
Stream-Based
Multifile algorithms are stream-based and not random-based. In a running game,
from the output, if a message is received saying something similar to the words
seek error for offset
then a file in the multifile is trying to be accessed
by a random-based method. For multifiles and fonts, an example of a random-based
file is an .rgb file. An alternative different from the use of an .rgb file is
the use of a .ttf instead. An example follows.
# models is the original directory
# models.mf it the new target multifile
multify -c -f models.mf -v models
In the game, from the multifile models.mf, load the .ttf file.
font = loader.loadFont("models/arial.ttf")
Encryption
Multifiles can also encrypt your files with a password.
To do so, you need to set the encryption flag and password using the
setEncryptionFlag()
and
setEncryptionPassword()
methods, before adding, extracting
or reading multifiles.
At the OS prompt, to create a password protected multifile and print out the contents do the following.
# models is the original directory
# models.mf it the new target multifile
multify -c -f models.mf -ep "mypass" -v models
This code creates a multifile and adds an encrypted file to it:
m = Multifile()
m.openReadWrite("foo.mf")
m.setEncryptionFlag(True)
m.setEncryptionPassword("foobar")
# Add a new file to the multifile
m.addSubfile("bar.txt", Filename("/tmp/bar.txt"), 1)
m.flush()
m.close()
You can read encrypted multifiles the same way:
m = Multifile()
m.openRead("foo.mf")
m.setEncryptionFlag(True)
m.setEncryptionPassword("foobar")
# Prints the contents of the multifile
print(m.readSubfile("bar.txt"))
At the OS prompt, to see the contents of a password protected multifile perform
multify -tvf models.mf -p "mypass"
You can test the reading in a of password-protected multifile, followed by the mounting of the file using the following code.
from panda3d.core import Multifile
mf = Multifile()
mf.openRead("models.mf")
mf.setEncryptionFlag(True)
mf.setEncryptionPassword("mypass")
from panda3d.core import VirtualFileSystem
vfs = VirtualFileSystem.getGlobalPtr()
if vfs.mount(mf, ".", VirtualFileSystem.MFReadOnly):
print('mounted')
When running the application, the following should be seen:
mounted
You can check if a certain subfile is encrypted or not using the
isSubfileEncrypted()
method, which takes the subfile index
as parameter.
It is possible to have a multifile where different subfiles have different
encryption, but you will not be able to mount it with the VirtualFileSystem or
use it with the multify tool. To mount an encrypted file using the virtual file
system, pass the password as parameter to the
mount()
method:
from panda3d.core import VirtualFileSystem, Filename
vfs = VirtualFileSystem.getGlobalPtr()
vfs.mount(Filename("foo.mf"), ".", vfs.MFReadOnly, "foobar")
To use encryption with the multify tool, run it with the -e
option, which
will prompt for a password on the command line. Alternatively, if you also
specify the -p "password"
option, you can specify it in the command instead
of typing it at the prompt.