Creating Multifiles
Multifiles archives are archive files that store game resources. Think of it as a giant zip file that stores, optionally zips and encrypts your data files, but does not need to be extracted.
The multify program
The multify console program creates such files. You can get information about
the commandline parameters by running multify with the -h
option. This is
how program describes itself:
Usage: multify -[c|r|u|t|x] -f <multifile_name> [options] <subfile_name> ...
multify is used to store and extract files from a Panda Multifile. This is similar to a tar or zip file in that it is an archive file that contains a number of subfiles that may later be extracted.
Panda’s VirtualFileSystem is capable of mounting Multifiles for direct access to the subfiles contained within without having to extract them out to independent files first.
The command-line options for multify are designed to be similar to those for tar, the traditional Unix archiver utility.
Read Assets
If you want to prepare to read assets from a Multifile directly, you can “mount” it into the virtual file system:
VirtualFileSystem *vfs = VirtualFileSystem::get_global_ptr();
vfs->mount("./foo.mf", ".", VirtualFileSystem::MF_read_only);
If you want to read assets, you can mount a whole directory structure from a webserver.
If your webserver hosts:
http://localhost/mydir/models/myfile.bam
http://localhost/mydir/maps/mytexture.png
Put this in your config.prc:
vfs-mount-url http://myserver/mydir /mydir
model-path /mydir
Or, equivalently, write this Python code at startup:
vfs.mount(VirtualFileMountHTTP('http://myserver/mydir'), '/mydir', 0)
getModelPath().appendDirectory('/mydir')
and then you can load models like this in your Python code:
model = loader.loadModel('models/myfile.bam')
texture = loader.loadTexture('maps/mytexture.png')
If you want to prepare for reading and writing assets to a Multifile do the following.
from panda3d.core import VirtualFileSystem
from panda3d.core import Multifile
from panda3d.core import Filename
mf = Multifile()
mf.openReadWrite("models.mf")
vfs = VirtualFileSystem.getGlobalPtr()
if vfs.mount(mf, ".", VirtualFileSystem.MFReadOnly):
print('mounted')
If you want to prepare for reading and writing assets to a ‘subdirectory’ Multifile do the following. Note “mysys” must always be literally written in any python code. E.g. “mysys/memfdir/mfbar2.txt”
from panda3d.core import VirtualFileSystem
from panda3d.core import Multifile
from panda3d.core import Filename
mf = Multifile()
mf.openReadWrite("models.mf")
vfs = VirtualFileSystem.getGlobalPtr()
if vfs.mount(mf, "mysys", VirtualFileSystem.MFReadOnly):
print('mounted')
If you are having problems loading from multifiles you can list the complete contents of your .mf file with a command like:
multify -tvf mymultifile.mf
Doing a sanity inspection like this can be useful to ensure that your assets are in the right place within the multifile.
Multifile objects
The Multifile class is designed for opening, reading and writing multifiles. You
can open a new multifile by creating an instance of the class and calling the
openRead
method:
from panda3d.core import Multifile
mf = Multifile()
mf.openRead("foo.mf")
The openRead
method opens the multifile as read-only. If you want to make
changes to it and write it back to disk, you will need to use the
openReadWrite
method. Also, there exists openWrite
to create a new
multifile.
If you have made important structural changes to a Multifile, it is recommended
to rewrite the multifile using the repack()
method. (This won’t work if
you’ve opened it using openRead
.) If you are uncertain about whether it has
become suboptimal, you can call needsRepack()
which returns True if the
Multifile is suboptimal and should be repacked.
To write it back to disk, you can use the flush()
method which flushes the
changes you’ve made to the multifile back to disk, or the close()
method if
you’re done with the file.
To mount Multifile objects into the VirtualFileSystem without writing them to disk first, here’s an example on how to mount them:
yourMF = Multifile()
#... now do something with yourMF
vfs = VirtualFileSystem.getGlobalPtr()
vfs.mount(yourMF, ".", VirtualFileSystem.MFReadOnly)
Subfiles
Files that are added to a multifile are called subfiles. You can add existing
files to a multifile object using the addSubfile
function. This function
takes three arguments: the target filename, the existing source file and the
compression level (1-9). There is also updateSubfile
, which does the same
thing but if the file already exists, only updates it if the content is
different.
There are several other methods which operate on subfiles, which you can find on
the Multifile
page in the API Reference.
Here are a few examples of working with subfiles:
from panda3d.core import VirtualFileSystem
from panda3d.core import Multifile
from panda3d.core import Filename
m = Multifile()
# Add an existing real os file with compression level 6
m.openReadWrite("foo.mf")
m.addSubfile("bar.txt", Filename("/tmp/bar.txt"), 6)
m.flush()
# Destroy the contents of the multifile
# Add an existing real os file to be the first multifile
m.openWrite("foo.mf")
m.addSubfile("bar.txt", Filename("/tmp/bar.txt"), 6)
m.flush()
# Permanently re-order in ascending order the
# directories and files in the multifile
m.openReadWrite("foo.mf")
m.repack()
m.flush()
# Open a multifile and replace the contents of the mulifile file
# with new contents
m = Multifile()
m.openReadWrite("foo.mf")
m.updateSubfile("bar.txt", Filename("/tmp/bar2.txt"), 9)
m.flush()
# Open a multifile and extract all files smaller than 3kb
# New real os files are created with the contents of the multifile data
m = Multifile()
m.openRead("foo.mf")
for i in range(m.getNumSubfiles()):
if m.getSubfileLength(i) < 3 * 1024:
m.extractSubfile(i, Filename("/tmp/" + m.getSubfileName(i)))
# Find, print and remove a file named bar.txt
barIdx = m.findSubfile("bar.txt")
if barIdx != -1:
# It returns -1 if it doesn't exist
print(m.readSubfile(barIdx))
m.removeSubfile(barIdx)
m.flush()
m.close()
std::ostringstream os (std::ios::in | std::ios::out);
std::istream is (os.rdbuf ());
os.write((char*)&stuff, sizeof(stuff));
PT(Multifile) mf = new Multifile();
mf->open_write(fileName);
mf->add_subfile("foo.mf", &is,6);
mf->flush();
mf->close();
If the foo.mf file were to have a contained bar.egg.pz file, load the egg and use it similar to other model loading methods.
nodepath = loader.loadModel("foo/bar")
Stream-Based
Multifile algorithms are stream-based and not random-based. In a running game,
from the output, if a message is received saying something similar to the words
seek error for offset
then a file in the multifile is trying to be accessed
by a random-based method. For multifiles and fonts, an example of a random-based
file is an .rgb file. An alternative different from the use of an .rgb file is
the use of a .ttf instead. An example follows.
# models is the original directory
# models.mf it the new target multifile
multify -c -f models.mf -v models
In the game, from the multifile models.mf, load the .ttf file.
font = loader.loadFont("models/arial.ttf")
Encryption
Multifiles can also encrypt your files with a password. To do so, you need to
set the encryption flag and password using the setEncryptionFlag
and
setEncryptionPassword
methods, before adding, extracting or reading
multifiles.
At the OS prompt, to create a password protected multifile and print out the contents do the following.
# models is the original directory
# models.mf it the new target multifile
multify -c -f models.mf -ep "mypass" -v models
This code creates a multifile and adds an encrypted file to it:
m = Multifile()
m.openReadWrite("foo.mf")
m.setEncryptionFlag(True)
m.setEncryptionPassword("foobar")
# Add a new file to the multifile
m.addSubfile("bar.txt", Filename("/tmp/bar.txt"), 1)
m.flush()
m.close()
You can read encrypted multifiles the same way:
m = Multifile()
m.openRead("foo.mf")
m.setEncryptionFlag(True)
m.setEncryptionPassword("foobar")
# Prints the contents of the multifile
print(m.readSubfile("bar.txt"))
At the OS prompt, to see the contents of a password protected multifile perform
multify -tvf models.mf -p "mypass"
You can test the reading in a of password-protected multifile, followed by the mounting of the file using the following code.
from panda3d.core import Multifile
mf = Multifile()
mf.openRead("models.mf")
mf.setEncryptionFlag(True)
mf.setEncryptionPassword("mypass")
from panda3d.core import VirtualFileSystem
vfs = VirtualFileSystem.getGlobalPtr()
if vfs.mount(mf, ".", VirtualFileSystem.MFReadOnly):
print('mounted')
When running the game, the following should be seen:
mounted
You can check if a certain subfile is encrypted or not using the
isSubfileEncrypted
method, which takes the subfile index as parameter.
It is possible to have a multifile where different subfiles have different
encryption, but you will not be able to mount it with the VirtualFileSystem or
use it with the multify tool. To mount an encrypted file using the
VirtualFileSystem, pass the password as parameter to the mount
method:
VirtualFileSystem *vfs = VirtualFileSystem::get_global_ptr()
vfs->mount("./foo.mf", ".", VirtualFileSystem::MF_read_only, "foobar");
To use encryption with the multify tool, run it with the -e
option, which
will prompt for a password on the command line. Alternatively, if you also
specify the -p "password"
option, you can specify it in the command instead
of typing it at the prompt.