AI learns control robot arm, finds the cube, grabs it and carry it on another platform. Trained virtually over 30h and 40mln steps. In this in-depth review we take the new M1 powered MacBook's out against the 16' MBP king with your favourite development platforms: from iOS, to 3D gaming d. 1 every application works with data. Both unity and tensorflow work with data. Tensorflow is just a machine learning library that basically is made for python language. You dont need to find a wrapper to let you work with tensorflow but i think there is some. The sample is written in TensorFlow 1.0 API style. Gradient supports TensorFlow 1.15, which has newer Keras APIs. Try converting the old style code into the new style. Try on other environments. Try other Unity ML Agents environments, and see how actor-critic will perform there. You might need to tweak observation and action processing for that.
I’m extremely excited about the new Unity3D Machine Learning functionality that’s being added. Setting it up was a little painful though, so I wanted to share the steps I followed, with the specific versions that work (I tried a whole lot and nothing else worked). In this guide, I’ll show you everything you need to get setup and ready to start with the 3D ball example. There’s also a video version at the end..
You’ll need to download CUDA 8.0.61 for this to work.
You can view the CUDA download archive here: https://developer.nvidia.com/cuda-toolkit-archive
Select and downloadCUDA Toolkit 8.0 GA2
Close any open Unity and Visual Studio instances (you’ll have to restart the installer if you don’t do this first)
Run the Installer
ChooseExpress
Unity Tensorflow Tutorial
Next we need to grab the CUDA Deep Neural Network library aka cuDNN
Visit the CUDNN page here: https://developer.nvidia.com/cudnn
You’ll need to create an NVIDIA account and log in to download the library. It’s easy to do and free though.
Once you’re logged in, click the download button.
Choose the download link for v6.0 for CUDA 8.0 – Download cuDNN v6.0 (April 27, 2017), for CUDA 8.0
Open the cuDNN zip file.
Copy the 3 folders (bin, include, and lib) from the zip file into your CUDA 8.0 folder.
The default folder path you’d copy into is C:Program FilesNVIDIA GPU Computing Toolkitcudav8.0
Into Here
Now you need to add an environment variable and two path entries.
Open the environment variable dialog (hit the windows key and start typing envir…. and it’ll auto popup for you)
Click the Environment Variables Button
Click the New button to add a new System Variable
Set the Variable Name to CUDA_HOME
Set the Value to C:Program FilesNVIDIA GPU Computing Toolkitcudav8.0
Find the Path in the Environment Variables Dialog
Make sure you select the System variables version, not the user variables!
Click Edit
Add the following twofolders to the path.
C:Program FilesNVIDIA GPU Computing ToolkitCUDAv8.0libx64
C:Program FilesNVIDIA GPU Computing ToolkitCUDAv8.0extrasCUPTIlibx64
Click Ok a couple times to close out the dialogs.
Next, we need to install Anaconda to configure our python environment.
Visit the download page here: https://www.anaconda.com/download/
Download the 3.6 version (I went with the 64 bit installer, not sure if it matters but I’d go with that one).
Run the Anaconda Installer and choose the default options all the way through.
After the installation completes, open the Anaconda Prompt
Creating the Conda Enviroment
Next, we need to create an environment with python 3.5.2.
Run the conda create command in the prompt like this:
conda create -n tensorflow-gpu python=3.5.2
Next, activate the newly created environment with this command:
activate tensorflow-gpu
And finally, install tensorflow with this command
pip install tensorflow-gpu
Once the installation completes, you can test that it was successful by launching python (still from that anaconda prompt) by typing:
python
Then use the command:
import tensorflow as tf
To try out the ML agents, you’ll want to download the sample project from the GitHub page here: https://github.com/Unity-Technologies/ml-agents
You can download the zip or use GIT and clone the repository.
Next, open another anaconda prompt as an administrator.
Change directory into the ‘python’ subfolder or the location you cloned or unzipped the project into.
For example, my folder is: C:Gitml-agentspython because I cloned the repository into my c:git folder.
Fixing Html5lib
Next, type the following command: conda install –force html5lib
Now type: pip install .
Make sure you include that period, it’s part of the command.
If there are no errors, you’re good to go and ready to start setting up your Unity project with tensorflow and machine learning!
Next, you’ll want to build the Unity Environment. The steps for this are very clearly laid out here so I won’t repeat them: https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Getting-Started-with-Balance-Ball.md
One issue I ran into though was the ENABLE_TENSORFLOW define being cleared out after I installed the tensorflowsharp library. When this happened, the “Internal” option disappeared. Simply re-add it and hit play so it re-compiles, then the internal option should re-appear.
First I wanted to say thanks to the guys at Unity for building this all out. I’m excited to start integrating machine learning into projects for my AI.
I also wanted to thank Nitish Mutha for this awesome blog post that got me 90% of the way through this setup.
I’ve been thinking for a while about how best to combine machine learning knowledge I’ve built up and my other hobby - making video games. To this end I’ve been looking into using Tensorflow with Unity3d. I forsee a lot of issues around performance at runtime, along with cross platform issues issues down the road.
As a start here is a quick rundown of compiling Tensorflow to run a trained graph from a C# Unity script using the C++ api.
1. Getting a Graph to Use
The idea here is to keep it to a bare minimum for what you need. The goal here is to make sure we can get TF will run at all rather than spending time making it do something useful. For these first couple of steps I borrowed heavily from Jim Flemming’s excellent Medium post on the basics of using the C++ api, which we’ll need. Using a simple python script you can generate a protobuf file to store your graph, I used:
which does the very impressive task of finding the max of two numbers and storing it in a variable c.
2. Compiling Tensorflow to a shared libaray (.so)
We can use the C++ api for TF to read and excetue graphs we’ve already saved pretty easily. In addition the Google documentation for using their build tool, Bazel is pretty good so compiling the shared library isn’t really too much of an issue. Here I used Jim Flemming’s build script with a few changes.
Debugging
When we start using this with Unity it’s not the same as running a script internally, we have to recompile to add debug messages. There’s not a simple method of accessing the Unity editor console to print useful debug messages anyway since we’re constrained by the return type of our function. This is a problem even with this very simple example as we’ve hard coded the path to the protobuf file which will cause problems down the road. Unity isn’t handling the paths so it won’t put the files in sensible places when building for instance. To give us some useful output on errors I’ve added logging to a file instead of standard out and known return values so it’s obvious where in the code we hit an error.
Shared library
The function is incased in an extern 'C' {}. Since we’re not building an exceutable we can’t just have a main function that returns 0, we want to return our max number. The C++ compiler however doesn’t preserve function names so calling our function after compilation won’t work, we have to tell the compiler we want to declare our funtion with C linkage. The Unity documentation does a fine job of explaining why we need to do this.
Here’s the code put inside the Tensorflow repo at /tensorflow/loader/loader.cc
We’ve hard coded here that it should output 3, which is the max of 3.0 and 2.0 cast to an int
We can compile using Bazel to either an exceutable or a shared library. The former is good to make sure the above code works, you can just throw the function into a main function and compile it. Once that’s working we can make our shared library. The Bazel BUILD file for that looks like:
If you keep this inside the loader folder with the cc file we can build from that folder using bazel build :loader.so. The actual so we need will end up
3: Running in the Unity Editor
Unity Tensorflow
You can access the shared library using the [DllImport ...] statement for C#, it doesn’t matter this isn’t a dll! This works the same as any Unity plugin, I have it in assets/plugins Here’s my very simple unity script:
All this is doing is:
- Loading our library
- On scene start running the
runfunction from the library - Putting the resulting integer as the string on an attached UI element.
Because of the way we hardcoded the model loading in the C++ and how the Unity editor does paths the protobuf needs to be in the root of the Unity project itself, not the plugins folder, not /assets. Now we can run this from the editor…
… Very impressive Unity, 3 is the right answer…
4: Running in a build
The shared libaray we’ve made as far as I know will only work for Linux. We need to compile different plugins for other platforms. Within the Unity editor we can set plugins to be included in different builds from the inspector for the asset. By default everything gets included which is fine if we’re only bulding for Linux but our project size will get pretty out of control if we have 3 or 4 TF libraries always being included in each build.
Unity Tensorflow Lite
The default settings will work for the build; however this is where our lazy hard coding of paths is a problem. In this case the root path is going to be whereever your excutable is run from, so the /models folder needs to go there, not anywhere within the /data folder!
It’s not particularly impressive but it works - we can access Tensorflow from a built Unity3d project! Now to run some more exciting graphs…