Foundry Performance Issues to Consider

[Commodore]

Hello Foundry authors! This is an informational post for those looking to get the best performance from their Foundry quests. This post will get very technical, but I recommend reading it for any avid Foundry user, as it will give you a better understanding of performance in Neverwinter.

Draw Calls

Draw calls are one of the biggest bottlenecks for performance in Neverwinter as well as many other modern games. A draw call is when your GPU has to look up a new material for a given set or strip of polygons.

Facts About Draw Calls

1. Every material per-object constitutes a separate draw call, so one object can have multiple draw calls. This also means less materials equals less draw calls.
2. Lighting can affect draw calls as well by breaking object instancing. More info on this below.
3. In general, draw calls are more impactful to performance than polycount, so a high poly object can potentially be cheaper than a low poly, multi-material object.
There is no way for you to tell exactly how many draw calls an object has in the editor, but the budget costs in the foundry were partially calculated from draw calls, so an object with a high cost probably has multiple draw calls. Sometimes you can tell if an object has multiple materials just by looking at it. A building object may have separate materials for the walls, trim, windows, roof, Ivy foliage, and any extra details that couldn’t be worked into a single texture sheet for a single material.

How Lighting Affects Draw Calls

1. In an unlit environment, all identical objects will “instance”. This means they will draw in a single pass on the GPU, all at once. This makes them much cheaper to draw, and you can have many of them with very little cost.
2. Adding a dynamic light will “break” instancing, meaning it will force some of the objects to draw on a separate pass on the GPU. This is generally more expensive and has an impact on draw calls and the amount of time the GPU has to draw stuff.
For example, let’s say you have a row of identical tables in a rectangular room, and one side of the room was lit with a dynamic light while the other side was completely unlit. The tables on the lit side would be in a separate instance group than the ones on the unlit side of the room, effectively increasing the number of draw calls in the room. Each table that the light affects are instanced with each other, but not with the unlit tables.
3. Static lights, or as we say internally “vertex lights”, break instancing as well, but in a worse way. Any object touched by a static light becomes unique, because the lighting data is stored per-vertex, and each vertex is duplicated in memory. Using the same rectangular room with tables example above, if one side of a room is lit with a static light, each table within that light’s radius is now a unique draw call. However, once an object is no longer instanced due to vertex lighting, you can place as many vertex lights on that object as you want and it will not decrease performance. This kind of light can work fine for smaller scope environments, but can be a real problem for larger complex ones.

AI Path Nodes

When you publish a quest, the bulk of the time spent in the publishing process has to do with AI pathing. Our server takes what you’ve built and organizes and compacts the data to a more efficient and readable format. The server then creates a layer of path nodes throughout your map, covering any surface that can possibly be reached by the player or an enemy critter. Essentially, the server starts from your map’s player spawn point position and creates a branching network of nodes that represent the topology of your map. These nodes are required for the AI to function properly, but they also use significant resources on the server.

Facts about Path Nodes

1. Any areas that are completely sealed off with collision from the spawn point will not be populated with path nodes, however areas like the tops of buildings and such do get path nodes. This is because the server isn’t very aware of the intricate flow of your map, so it doesn’t know if there’s a teleporter that takes you to a rooftop or if the player can use a power to knock an enemy into a previously unreachable area.
2. Dense geometry, especially on the ground, attracts more path nodes. For example, a large map with relatively smooth terrain may have far fewer path nodes than a small map with very rugged and rocky terrain. The more geometrically complex your map is, the more complex the path nodes have to be (and the longer they take to publish).
3. Most AI behavior requires path nodes to function properly with the exception of patrols, which only require path nodes when their patrol path is obstructed. This is why AI pathing does not work well in preview mode… There is simply no data for the AI to use yet.
4. Path nodes eat up resources on the server. If they get out of hand, they can eat up quite a tremendous amount of resources. For this reason, there are different categories of maps that the server handles in different ways. An adventure zone is allowed more resources than your typical instanced quest map, but zones also allow many more players in the map at the same time. They are also able to create multiple optimized instances of that zone when the player cap is reached, and some data and resources are shared between these instances. Quest maps do not have these resource-saving features enabled, nor do they have the larger resource allocation from the server that zones enjoy.
Path nodes typically do not affect things like the framerate on the client, but they can cause stalls on the server if they are over budget limits. As an example, in previous beta weekends, there were some quests that used entire zone exteriors for some maps. Since those zones were built with a different budget cap in mind, the server wasn’t expecting such complex maps to be used for instanced quests. As a result, every time those maps were opened by someone playing the quest, the server would have to quickly allocate additional resources to those maps. This was causing a solid 3-5 second freeze for all players on the server every time one of those maps loaded, which just made the entire game significantly less stable.

Optimizing Your Map for AI Path Nodes

You don’t really have a lot of control over path nodes, but there are some things you can do to help your map will work well with them.
1. Avoid excessive amounts of objects with complex geometry in walkable areas. A pile of rocks will have many more path nodes covering it than flat ground. If your entire map is one big debris field with extremely uneven rough topology, it will attract significantly more path nodes than a map with simpler topology.
2. Seal off non-playable areas with collision walls. If you don’t intend for a player to be able to reach a spot on your map, take the time to completely seal it behind invisible walls to ensure it does not get path nodes.
3. Objects that do not have collision, such as ground decals and grass, do not add to the path node count.
4. Placing a collision ceiling as low as possible on your map will reduce the number of nodes. This works best on maps where you have high places and rooftops that the player never gets to and that can be excluded from the playable area.
5. Break your map into multiple maps if you can. This not only improves how the server handles your content, but can also make the AI function better in some cases.
6. Interiors tend to use far fewer path nodes than exteriors, and are also quicker to load for the player. All of the above tips still work for interiors.

This is a lot of information to consider, but hopefully now you have a better understanding of map performance both on the client and on the server.

[Argantis]

I would have probably never looked at things this way, thanks for the perspective Aandre.