Resource file conversion problem

We encountered some bugs in the work above the game engine last week, involving some of the designs issues. Maintain a solution for this problem in a few days. Today, I finally determined the final plan and worthy of a record.

BUG is on the conversion of the game resource file.

The resources used in the game typically require a process of importing a repository. For example, your original map is a PNG file, but the engine needs to correspond to the compression format of the corresponding running platform. Windows is DXT, the phone is KTX, etc. This process, in the business engine such as Unity, is placed in the resource import process.

Our engine puts this conversion process at this level of virtual file system. This design decision is because I feel unified importance is a pain point, and people usually need to wait for the import process. Unity used Cache Server to solve this pain point, but I think Cache Server also has some design problems, which will be opened again.

I prefer the conversion process is inert until the resource that eventually needs to be required.

In our design, all resources that need to be converted have a suffix for .LK’s same name file in the file system. It describes how to process raw materials, its role and Unity. Meta file is basically consistent.

Our virtual file system is discovered that a file is .LK, call the build module to convert the source file when requested the file.

The premise of our assumption is that a source file plus the corresponding LK file, in the plus platform parameters, the content of the three determines what the final generated file is. Therefore, the top of the top three can be used for the Cache of the conversion process. Recently, this premise is not established, resulting in the generation of bugs.

The reason is: For Shader files, it is actually a code, similar to C / C ++ code. Compiling a shader is actually relying on a lot of files. So the light has a shader source file that cannot be accurately cache results: For example, we modified another file in the SHADER in CaDe, but FileServer did not know that the virtual file system returned the cache result without recompilation.


I want to give up the inert build mechanism when I want to solve this problem. That is, resource conversion and FILESERVER are separated. In this way, the source file is modified, and the resource file is built by the resource conversion module. The client always thinks that the resource is already constructing in runtime (unity).

This solution is the most simpler, but it is vetoed in the group. Because this is reversed, there is no solve the pain that originally wants to solve. After several discussions, we finally found a relatively reasonable way.

Take Shader as an example, assume that there is a shader file called A.sc, there is a A.SC.LK indicates how the SC is built. When we receive the Build request, our Fieserver will unconditionally recompile A.SC’s result on the specified platform and return the HASH of the result file. This step is not to do any cache.

However, in the new solution, if you request a version of A.SC in iOS, the build module adds a build script file called A.sc.lk.ios in the virtual file system, and records A. SC build methods on iOS, and dependencies during the construction process, including dependent files, and the current dependency files.

Then, this a.sc.lk.ios file is actually the only identified target file. Because any one dependent file is modified, it will lead to changes in the content of the file (the Hash value changes).

This file is invisible to the clientver client in this running session. This is because we assume that all files are in a Merkle Tree during operation. But the new session can see this new .ios file.

Once the client can get the .ios file, it can use this .ios file’s Hash to request the results of compilation. Since each .ios file content can only determine a compilation result, so FileServer can make Cache on the results.

The .ios file is not the result of resource compilation, but the compiled parameters. It can have multiple copies in the data warehouse, and the results of compilation only need to save the final one. In most cases, users will only request the latest compile results, but in case the user requests expiration versions, FILeServer can also regenerate.

This is a bit like GitHub’s large file storage scheme, not in the Git main warehouse, and saves a unique URL and puts the file in another service. Here, this another service is a module that compiles resources. This mechanism has achieved inert compile resources and can reasonably compile results. Unlike Unity’s Cache Server, it is used to index Cache’s key. It is actually a complete process of compiling resources, including compilation dependencies.


Therefore, compilation modules, which can fully cross the project Cache. That is, if you use a map in a project, it is compiled once; when you copy this map to a new project, it is not necessary to recompile. The same map in Unity will cause the GUID to change even in the same project, so that the Meta file changes, causing resources to recompile.

The file service also knows all the needs, so it is fully capable of expecting the Andriod version later. So File Server has the ability to use idle time to generate those versions that have not yet requested. And Unity’s Cache Server is a pure Key / Value service, it is not possible to do this.