Wall Worm Forums

It has come to our attention that a number of Anti-Virus solutions out there are picking up WallWorm.Stubs.dll as malware. I wish to assure all users that this is a false-positive due to overly generic signatures. The stubs DLL is tiny, and is needed for one and exactly one function, as it has to get around the fact that you cannot create a pointer to a managed class in C#. I need to be able to create that pointer in order to work around a bug in the C# interface for 3ds Max that makes it impossible to get an IGameMesh. For this reason, I wrote a tiny stub function in MSIL (which is what C# compiles to) to do an upcast to an arbitrary class, and that is the function that is put in WallWorm.Stubs.dll.

We are currently contacting the AV companies effected, but this will take time.

To help ease any other suspicions, I present to you the entire 80 lines of code that make up WallWorm.Stubs.dll:


.assembly extern mscorlib
{
.publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
.hash = (CD 95 65 AD 55 34 B6 AD 72 9D 3F BB 80 DA 0A 6C 49 E5 F2 5B)
.ver 4:0:0:0
}
.assembly extern Autodesk.Max
{
.hash = (2F 54 6D B7 43 34 A0 B6 DB 14 BA 8D AD 40 23 DF 4C 11 B0 09)
.ver 16:3:0:0
}

.assembly WallWorm.Stubs
{
    .ver 1:0:5419:28332
    .hash algorithm 0x00008004
    .custom instance void [mscorlib]System.Security.SecurityRulesAttribute::.ctor(valuetype [mscorlib]System.Security.SecurityRuleSet) = { uint8(1) }
    .custom instance void [mscorlib]System.Runtime.Versioning.TargetFrameworkAttribute::.ctor(string) = { string('.NETFramework,Version=v4.0') property string 'FrameworkDisplayName' = string('.NET Framework 4') }
    .custom instance void [mscorlib]System.Reflection.AssemblyProductAttribute::.ctor(string) = { string('WallWorm Stubs') }
    .custom instance void [mscorlib]System.CLSCompliantAttribute::.ctor(bool) = { bool(true) }
    .custom instance void [mscorlib]System.Runtime.InteropServices.ComVisibleAttribute::.ctor(bool) = { bool(false) }
    .custom instance void [mscorlib]System.Reflection.AssemblyTrademarkAttribute::.ctor(string) = { string('') }
    .custom instance void [mscorlib]System.Reflection.AssemblyCopyrightAttribute::.ctor(string) = { string('Copyright (c) WallWorm.com 2014') }
    .custom instance void [mscorlib]System.Reflection.AssemblyTitleAttribute::.ctor(string) = { string('WallWorm Stubs') }
    .custom instance void [mscorlib]System.Reflection.AssemblyCompanyAttribute::.ctor(string) = { string('WallWorm.com') }
    .custom instance void [mscorlib]System.Reflection.AssemblyConfigurationAttribute::.ctor(string) = { string('') }
    .custom instance void [mscorlib]System.Reflection.AssemblyDescriptionAttribute::.ctor(string) = { string('') }

.permissionset reqmin = {
[mscorlib]System.Security.Permissions.SecurityPermissionAttribute = { property bool 'SkipVerification' = bool(true) },
[mscorlib]System.Security.Permissions.SecurityPermissionAttribute = { property bool 'UnmanagedCode' = bool(true) }
}
}

.class public abstract sealed auto ansi beforefieldinit WallWorm.Stubs.Activator extends [mscorlib]System.Object
{
.method public hidebysig static !!T  Upclass<([Autodesk.Max]Autodesk.Max.INativeObject) T>(class [Autodesk.Max]Autodesk.Max.INativeObject val) cil managed
{
ldtoken    !!T
call       class [mscorlib]System.Type [mscorlib]System.Type::GetTypeFromHandle(valuetype [mscorlib]System.RuntimeTypeHandle)
callvirt   instance class [mscorlib]System.Reflection.ConstructorInfo[] [mscorlib]System.Type::GetConstructors()
ldc.i4.0
ldelem.ref

ldc.i4.2
newarr     [mscorlib]System.Object
dup
dup

ldc.i4.0
// Unfortunately, the API changed between 2014 and 2015 and we can't use this directly.
ldtoken [Autodesk.Max]Autodesk.Max.INativeObject
call class [mscorlib]System.Type [mscorlib]System.Type::GetTypeFromHandle(valuetype [mscorlib]System.RuntimeTypeHandle)
ldstr "Handle"
call instance class [mscorlib]System.Reflection.PropertyInfo [mscorlib]System.Type::GetProperty(string)
dup
brtrue.s Pre2015
pop
ldtoken [Autodesk.Max]Autodesk.Max.INativeObject
call class [mscorlib]System.Type [mscorlib]System.Type::GetTypeFromHandle(valuetype [mscorlib]System.RuntimeTypeHandle)
ldstr "NativePointer"
call instance class [mscorlib]System.Reflection.PropertyInfo [mscorlib]System.Type::GetProperty(string)
Pre2015:
ldarg.0
ldnull
callvirt instance object [mscorlib]System.Reflection.PropertyInfo::GetValue(object, object[])
stelem.ref

ldc.i4.1
ldc.i4.0
box        [mscorlib]System.Boolean
stelem.ref

call       instance object [mscorlib]System.Reflection.ConstructorInfo::Invoke(object[])
unbox.any  !!T
ret
}
}


Just a note as-to why WW wasn't working at all in 2012, about 1 release ago I updated wwBuild (the tool I wrote to package WW builds) so that it was able to encrypt the MaxScript files itself, without relying on 3ds Max, because, due to the time it takes to start 3ds Max, it would take anywhere from 1-5 minutes to do a single build. With the tool doing the encryption, a full build of WallWorm, including all plugins, takes 3-5 seconds, which is *way* better than the 1-5 minutes it took previously.

My initial implementation worked just fine in 2013 and up because they were written in the same way as the original decryption code I was looking at; They ignore any extra trailing bytes. 2012 on the other-hand expects that there is both an end of file byte, and that the trailing padding bytes are present. After adding the EOF byte and determining how the trailing padding size is determined, I was able to get it working correctly with 2012.

2.53 and on will continue to use the encryption mechanism I wrote for wwBuild.

Now that I think about it, the brushes that failed to import in my fairly simple map were also mitered (and I was only able to get them in that shape after some *very* careful adjustment of the verts, otherwise hammer would try to build the sides wrong...)

The first thing to understand when using the Pro implementation of the VMT importer is that it does actually parse the VMT files, which the original doesn't do. This means that the Pro version has to deal with a lot of absurdity that makes it into the VMT files that valve produces. If you find a VMT in a Source game that doesn't parse due to a syntax error, then post it here, so that a work around can be hard-coded. We will only do this for the main Source games themselves; we won't (except in special cases, or if you ask nicely) add support for syntax errors that are located in the materials for custom maps produced by users. If that map is then integrated into the game however, we will support it.

The VMT parser in Source itself simply ignores values that fail to parse, but for various reasons, the Pro implementation is not designed that way.

And on a bit of a proud note, the Pro parser parses the combined TF2 and CS:GO materials, totalling 361,492 lines across 26132 VMTs in 3.2 seconds. Unfortunately, the vast majority of the time the VMT importer spends is in creating the actual materials in 3ds Max, which it isn't possible to get much faster than it already is.

So, to give you an idea of the sort of messes that the Pro parser has to deal with, here are a few of the things that has to be handled specially by Pro for some games.


Counter-Strike: Global Offensive:

materials\asphalt\4wb_asphalt_b1.vmt:
$detailscale is officially documented to be a Scale, and makes sense as a Vec2, but this provides a Vec3 value. The Vec2 parsing was modified to ignore the 3rd value if provided.

materials\de_cbble\grass_alt\grass_b_blend.vmt:
This file has an extra closing curly brace at the end. The root node file parser was updated to try to consume all closing curly braces before expecting the EOF.

materials\models\effects\urban_puddle01a.vmt:
$envmapcontrast is documented to be a Normal. A Vec3 value, not enclosed in square brackets is instead provided. The $envmapcontrast parsing was updated to forward this value to $envmaptint instead, which I believe is what was originally intended.

materials\models\props\de_aztec\aztec_door_no_sign.vmt:
$phongexponent is set to "11:26 AM 3/4/20115", which is definitely not the Float that's expected. $phongexponent parsing was updated to specifically ignore this value.

materials\models\props\de_train\hr_t\train_car_a\train_a_numbers.vmt:
$AlphaTestReference is expected to be a Normal value, but instead contains the value "4". Normal parsing was updated to clamp values to within the 0 to 1 range.

materials\models\props_c17\pottery01a_09a.vmt:
$basealphaenvmapmask is expected to be a Bool, but is instead used as a texture reference. Bool parsing was updated to assume that the presence of a value means that the flag is set.

materials\models\props_unique\subwaycardoors.vmt:
$EnvMapTint is expected to be a Vec3, and one is provided, the only problem though is that there are no spaces separating the decimal components. Vec3 parsing was updated to assume that there is a space before each dot in this particular case.

The entirety of the WallWorm codebase has just been reformatted, all 80k lines (after formatting, I have no idea what it was before) spread across 180 different files. It was formatted automatically by a tool I've been working on to deal with the very inconsistent formatting used in WallWorm's codebase. Up until now, I've simply run it on the file I'm working on at the time, but it's finally gotten to the point where it's stable enough to run on the whole codebase. It's not in the current dev link, but will be in 2.42. While I don't believe I broke anything while doing it, it's perfectly possible that I did, because of the sheer size of the changes. When it takes 2.5 hours just to review the changes, you know they probably break something somewhere.

Don't forget about making it fast! I've already cut export time by 1/3rd, and that's just with a small modification! (it should provide a noticeable performance boost to SMD exporting as well)

The one thing you're failing to mention about DMX is that, apart from the plaintext version, it is not documented in it's current form anywhere. The current version of DMX, as used in DOTA 2 is not even supported by any of the DMX libraries I looked at. While I did manage to hack support into it for most of the DMX forms, but the form used by the maps is even more different than the rest. Why is it not supported? Because the DMX format is, and has been pretty much since it was created, in a state of flux, constantly changing. How much in flux is it? Well, dmxconvert, the tool provided in the DOTA SDK for converting between the different DMX formats, does not even support the current version of DMX used in DOTA 2.

What we can, and already do partially support (admittedly it's incomplete and mostly broken), is exporting to the plain-text form of DMX.

Lastly, of the games you listed, only half of them (Dota, and SFM) actually support DMX as a model format, the rest use SMDs exclusively. The DMX format supported by source since orangebox is supported exclusively by the particle engine. These PCF files do use a form of DMX, but are limited to the particle system, they are not used for models.

Edit:
After checking, it appears I was actually mistaken in the last statement I made.

Next update.

The test map that we've been working with originally exported to a vmf in a little over 180 seconds. That same map now exports with WW Pro in ~36 seconds, and I'm not done yet. I hope to be able to get at least another 5 seconds out of it, if not 10.

Enabling the "Break Non-Planar Polygons" option in the VMF exporter should also be enough, as it causes the exporter to add that modifier to a duplicate of the object at compile-time. Just be aware that this can slow down the export.

Next update, but first I have to ask, how many ways can you misspell $baseMapAlphaPhongMask?

Well, as it turns out, Valve can spell this undocumented property 4 ways in TF2, none of which are correct. (basealphaphongmask, basemapalphaphonemask, basemapalphaphonmask, and basemapalphphongmask)

Even while dealing with the large number of misspellings, and deciding that the English language spells recieve wrong, I've managed to implement the rest of the properties supported by the original VMT importer, although I've not implemented support for blend textures (usually used on displacements). The good news is that it has barely effected the total import time, still sitting at about 4 minutes for all 17k textures. When I implement blend texture importing, I do expect this time to go up at least a moderate amount, due to the fact most of the time is currently being spent by 3ds Max creating the actual materials. (Parsing all 17k materials only takes ~2.5 seconds with a hot cache) I've also added support for importing a few properties that were previously only exported for VMTs. The importer is still far from complete in my eyes though, as there are still another 364 properties that the parser knows nothing about.

Next update.

The initial implementation of the VMT parser is pretty much complete, and, after hooking it up, it imported those same 17k materials in just under 4 minutes. Please note that, while this time is impressive, it will likely go up (hopefully not too much) as support for more of the properties that the original importer supported is added.

It's also worth noting that the VMT parser in WallWorm Pro takes a slightly different approach to parsing than the current importer, because the one in WW Pro performs a best-effort style parse, meaning that it attempts to account for the errors present in various VMT files in TF2 (really, who sticks a "TODO:" in the middle of the file?!?!), this includes going to some pretty extreme lengths for a tokenizing parser to be able to deal with the places where someone decided they didn't really need to quote that color value, which would normally produce multiple tokens, and break parsing for the rest of the file, but that the new parser deals with as a single token, and parses it as if it were properly quoted. Another good example of where the new parser differs from the old in parsing style is that the new parser realizes that a developer trying to pass a color to $envmapsaturation was actually wanting to make it the $envmaptint, and accounts for that fact.

Unfortunately, I have yet to figure out whether passing 0.5 as an argument to $phong (which is supposed to take a single boolean value) was intended to represent true, or false, and have, for now, assumed that the presence of a value is intended to represent true. Sometimes you've just gotta wish VMTs were generated, not written by humans :)

And another update on this.

Most of the bugs in the FGD parser have now been resolved, and the memory used by the entity script has also been reduced by a little over 1/3rd.

Next up is the material library importer, because 45 minutes to import a meager 17k materials (All of the materials in TF2) is just too long.

For the fairly simple model that I tried, WallWorm was able to correctly export the model that I created in Inventor and imported into 3ds Max then exported with WallWorm.

If I'm working on independent geometry I usually do it in Inventor (it's because it works very well with how I think about creating things), and I know that can be imported directly into 3ds max as a model, but I don't know how well it works if you try to export it... Lets find out  ;D

And when you do so, if you use the "Export Non-Renderable as Bone" option, make sure that your collision hulls aren't set to be non-renderable.