We use blender to create game assets; we also use blender as level editor. Blend files are exported to our proprietary format using py scripts.
Exporting files manually is okay… to a point. Soon we find out not only that exporting files manually is a chore, it contributes to creating a beautiful mess.
How to automate the conversion? With XCode, it can be done using build rules. I’m covering the following points:
- Build integration – add files to your build, automatically translate them using utilities and include the products in your app bundle.
- Defining useful build rules.
- Invoking blender from the command line.
- Modifying blender scripts so that they won’t crash in a windowless environment.
- Signaling export errors (notably, errors generated by a script running inside blender) using XCode.
While we have a mixed workflow (Blender 2.49 / Blender 2.61), we’re dominantly using 2.49b, so I started with the older version. I’ll be covering 2.61… later.
I used the following articles as a starting point:
This article is not concise; I’ll summarize my findings in a coming article.
The experiment
The first step is to configure build rules to process *.blend files I guess, so in a first approximation I used this:
- Source files with names matching *.blend
- Custom Script: /Applications/blender/blender.app/Contents/MacOS/blender ${INPUT_FILE_PATH} (bad)
- Output Files:
${TARGET_BUILD_DIR}/../../${INPUT_FILE_BASE}.scg (bad)
I just thought it would be interesting to try this way although it’s obviously not complete. And indeed, it was.
First surprise, Blender opens in a window. Good for testing (notably because it’s displaying errors already; for a nice smooth build I’ll want to disable this).
Both Blender & XCode are displaying messages in the XCode errors & warnings panel, which is nice.
The initial problem is path related. The file won’t open because spaces in the path are incorrectly interpreted. So the correct form to open the blend is like this:
- /Applications/blender/blender.app/Contents/MacOS/blender “${INPUT_FILE_PATH}” (good)
Running the script is the next obvious step. For now blender just opens and idles happily until I quit. So I modify my command, like this:
- /Applications/blender/blender.app/Contents/MacOS/blender -b “${INPUT_FILE_PATH}” -P qwagga.py (bad)
-b to tell blender to ‘render in background’. -P qwagga.py is the wishful step to run my script hoping that scripts are being run from the blender scripts folder. qwagga.py is my export script. There will be a problem with that but let’s try anyway.
Indeed, -b prevents the blender window from opening. Blender is reporting (via XCode) that qwagga.py wasn’t found. Interestingly it would appear that -P could also refer a script included inside the blend file (text node). For now, however, I’ll just provide the full path.
- /Applications/blender/blender.app/Contents/MacOS/blender -b “${INPUT_FILE_PATH}” -P /Applications/blender/blender.app/Contents/MacOS/.blender/scripts/qwagga.py (good)
Yea this works. As you’d expect since the exporter is trying to pop a file selection panel the script crashes with a fairly explicit message:
File “/Applications/blender/blender.app/Contents/MacOS/.blender/scripts/qwagga.py”, line 213, in <module> Blender.Window.FileSelector(exportAssets,’Export to File’,PATH)
RuntimeError: the file/image selector is not available in background mode
This error can be patched easily:
try:
Blender.Window.FileSelector(exportAssets,’Export to File’,PATH)
except RuntimeError:
OUTPUT=Blender.sys.makename(ext=’.scg’)
exportAssets(OUTPUT)
Telling XCode where the ouptut is located turned out to be a nag. The docs aren’t overly explicit about what environment variables can be used. As it turn out, however, whenever running a custom shell script (like, what we are doing now) the messages window displays a whole batch of setenv commands. That is all we need to know. I adjusted my entry in output files accordingly:
- ${INPUT_FILE_DIR}/${INPUT_FILE_BASE}.scg
Immediately after I got this work, it seemed there was ‘a bit of weirdness’ in XCode (an error I couldn’t make sense of). For whatever reason this cleared after cleaning the build and re-saving my blend file.
It appears that each file is getting processed/converted whenever building (even if the input file hasn’t changed). I’m a bit worried about how this will turn when I have 200 blend files linked in this way – since blender loads almost instantly I’ll just wait and see.
Single input => multiple output
Now, the above looks quite useful but there are issues:
- My py scripts don’t output just one file.
- In fact I have two py scripts used in blender. One outputs a single file, the other outputs a whole lot of files. The names of the files match object names in blender
I would hope that both Blender and XCode can handle this situation gracefully.
For proof of concept, I just hacked a different output file path and created a file hierarchy looking like this :
I added a ‘build’ folder under blends because we don’t want to have output files sprawling around. Then the question is to know whether car_park.blz will be exported along with all its content (for sake, foo.rtf, bar.rtf). The answer is YES.
Not perfect
Using this method I don’t know how to modify the location of the output in the resulting package. Before I would link a ‘blue folder’, so all my media 3D assets would go under [package]/3D/. Controlling the structure of this package is useful in various ways.
We just demonstrated that we can link a whole folder given a single input file; this means we could input a config file describing which blend files we want to translate and how; then we’d run blender once , translate all the files and producing the correct output.
In the end I think this will turn out to be a better solution, notably because it will avoid translating the same files other and other even if they weren’t modified. Scene files process fast; not the case with files containing large meshes and animations.
For whatever reason, the next steps turned out to be troublesome.
My second py script was symlinked. I didn’t think it would matter (strictly speaking, still don’t think it does) but I found it hard to get the script to refresh (meaning, I would edit the py script and run the build again, find the same error as previously. And again. And again.
An error in the scene export script caused the wrong path to be used.
Passing several scripts on blender command line (-P foo.py -P bar.py) doesn’t work so I ended calling blender twice which is a lesser evil. Maybe.
Complications
- I have a mixed workflow using both Blender 2.4x and Blender 2.5x. In theory this means that I would have to go through the motions again, finding out how I can run 2.5x from the command line (likely as not, it won’t work in the same way).
- Scene files contain level edits and assets. Asset files do not contain level edits. Both blender 2.4x and 2.5x files are just annotated as *.blend. How do I know which version of blender to run and which scripts to call?
The answer to (1) will be simple for now. Don’t do that. I mainly use 2.4x; the only 2.5x file that really matters so far is the one containing the PC mesh and animations. So I’ll just pass and use the manual workflow for that.
The answer two question (2) will be equally simple. Use a convention. For starters:
- level file: level.s.blend
- asset file: asset.a.blend
( and likewise when I get 2.5x scripts working, different conventions may be used )
This is a bit lame. Maybe a better convention might be something like:
- *2.4x/scenes*.blend
- *2.5x/assets*.blend
Works – although the build rule does say something like “files with name matching”, the matching rule actually uses the path.
A nag here is that I don’t want to have 2 or 4 ‘build’ folders for my assets. Just one folder (that I can trash as a quick and dirty way to make sure no out of data lingers in the system) would be best. This should be straightforward, matching the output like so:
- ${INPUT_FILE_DIR}/../../build/${INPUT_FILE_BASE}.blz
Conflating the output from various files (e.g. 2.4x/assets/foo.blend and 2.4x/scenes/foo.blend ) is possible. Collisions are unlikely, however, so I’ll keep it this way.
Polish?
Silencing the output?
When running blender directly from the desktop, getting console output to show up can be tricky. XCode surely doesn’t have this problem. All debug output appears in the messages window. Too much of it, in fact. Worse, something forces the item matching the translation script to expand.
No luck. I was assuming (maybe wrongly) that blender output is displayed because it is sent to stderr. Either way standing messages to stderr or stdout will yield the same result. Until further notice if I want to avoid script messages blotching the output, I need to disable debug output manually.
Displaying errors during the export process
An issue serious enough that putting it under ‘polish’ may be foolish : how to ensure that XCode will alert us when export errors occur? And by the way, what is an export error?
I tried my luck again, this time by looking at the unit test framework output. I thought XCode may be detecting a simple pattern to identify errors, and it is. The pattern is something like this:
- error: description of error
The space after the colon in error : is required. In python (~2.7), use:
print (“error: something went wrong”)
And XCode will detect the error and display a flamboyant exclamation mark. Neat.
This won’t stop the build from completing but a red exclamation mark is difficult to ignore so it will be enough.
What is an error, then?
A py script failing while running inside blender does NOT cause an error to appear in XCode. This is because blender returns with exit code ‘OK’ regardless. Now that we know how to signal errors to XCode, we can insert validation related errors. But we also want to report an error when our script fails unexpectedly:
try:
exportAssets(PATH)
except:
print “error: in blender script: “, sys.exc_info()[0], “(see messages window)”
raise
raise causes the error to propagate, allowing detailed information to appear in the log.
caveat: If the script contains a syntax error, it won’t run. In this case blender will report the error and exit, however we cannot alert XCode since the script didn’t even start running.
(how to cope with) limitations
Limitations with the approach I described derive from the atomicity of the conversion. It has to be done like this:
input file => output file or directory
For level edits, it is fine.
I see a problem when several files want to output to the same deployment directory. For example, it often makes sense to define actors as shared assets (same creatures used in different levels). Although it is a bit messy, it is often convenient to keep several actors in the same blend file. But then what we’ll end up doing is this:
- myCreatures1 => targetFolderA
- myCreatures2 => targetFolderA
This is bad in at least two ways:
- We should cleanup targetFolderA before exporting (to avoid outdated files lingering in the build)
- When exporting myCreatures2, we’ll end up copying over again data output by myCreatures1.
If we export to separate folders (myCreatures1Out/, myCreatures2Out/) the deployment structure becomes inconvenient. We may end up manually adding every output folder to the path used by a loader, and if we do so retrieving assets will take more time.
For shared assets, we have to choose. Either we start from a file listing all shared asset files and this becomes the input – processed by a special script – or we agree that each file will contain one actor / item. While the first solution is somewhat constraining it does help keeping things neat and requires less work.
So, do what?
Although not ideal, a solution is to modify the way the loader retrieves data blocks. I want to do this conservatively.
Before, a data block would be retrieved by creating a file name against the resource paths (e.g: rabbit => retrieve the mesh called rabbit.dbk)
After, a data block can also be retrieved by matching a file name against a path in a designated shared resource folder (this would have to be the bundle’s root I’m afraid) : rabbit => rabbit.blz/rabbit.dbk
Additionally, the name of the blend file for a shared resource should match the name of the data it contains.