Plugins | MANUS Core 2.3 SDK | Client

Introduction

This article will describe the more important functions of the SDK client and where needed go into detail about the code behind the function. It will however not describe the complete program layout as it is not representative of a normal client program using the SDK, nor always compatible with different program architectures. In fact, it is written to demonstrate the function usage instead of being the optimal way of using those functions. Due to this there is no graphical UI beyond the simple console output as that would require more complexity that may obfuscate the SDK usage.

The inner functions of the SDK are outside of this document's scope and the sample implementation is only used as a demonstration on how to use the SDK. What you will find in this document are the most common functions to set up a (temporary) skeleton, process the landscape, handle haptic feedback, handle trackers, gestures, and handle time tracking.

If you want a simple quick start guide for the basics required to get data, the SDK minimal client quick start guide might be a better choice, but for more in-depth details look at this documentation.

‍

SDK Callbacks

All the data streams MANUS Core provides are structured as callbacks. A callback is a function that you can pass into a CoreSdk_RegisterCallback... function. These functions call them at another point (on another thread) when a certain event occurs. This way users do not need to poll the SDK to figure out if there is new data available.

It is good practice to register all the callbacks you require after initializing the SDK and before connecting to MANUS Core. You only need to register the callbacks which you intend to use, if you do not plan to use certain data streams this could decrease the traffic from MANUS Core.

An example of a callback is the Ergonomics stream callback:

This callback is called whenever the SDK receives new ergonomic data from MANUS Core. A best practice for data being received on another thread (which happens with the SDKs callback mechanism) is to save the data somewhere and then process whatever you want to do with the data on your own thread. This way you do not block/delay the callback thread, which could lead to delayed/laggy data transfer. In several of our streams, we add timestamps to show when the data was captured. To get a more readable timestamp you need to call theCoreSdk_GetTimestampInfo` on the timestamp value, this will translate the timestamp value to a date & time format.

‍

Client Connection

There are several options for connecting to MANUS Core. You usually have MANUS Core running on your local machine, which makes connecting simpler than via a network connection. Selecting local by pressing the L key will connect you locally. You may potentially have certain network security settings that may block certain connections , this may be a first indicator that Manus Core could have trouble making a connection.

If you opt for a remote connection, press the H key. It will now search all available networks for MANUS Core hosts and display them in a simple selection menu.

Once a connection is made the Main Menu screen will be displayed which allows you to look at the various kinds of data available via the SDK.

To dive into the code behind our example client a bit, when looking at the code for finding hosts, you will see something like the following:

To find all the available MANUS Core instances, the CoreSdk_LookForHosts function needs to be called. This function call is a blocking function and therefore will block the calling thread for the specified duration. We advise not running any time sensitive functions or UI in the same thread as the Manus SDK connection. After this function has completed you can get the number of hosts found by calling the CoreSdk_GetNumberOfAvailableHostsFound function. If this function returns 0 hosts, either no MANUS core is available, or the network is being blocked by a security system or another program. The next step is creating an array at least the size of the number of found hosts and passing this array into the CoreSdk_GetAvailableHostsFound function. This function will put information about all the hosts that are found on the network into the array.

Once you have at least one host to connect to you can connect to it with code like this:

The chosen host structure needs to be put into the CoreSdk_ConnectToHost function. This function then attempts to connect to the given host and will trigger the OnConnectedCallback callback when it is successful. This callback is registered via the CoreSdk_RegisterCallbackForOnConnect function. If a connection to MANUS Core is lost the OnDisconnected callback is invoked. This one can be registered via the CoreSdk_RegisterCallbackForOnDisconnect function.

‍

Main menu

To make the example client easier to navigate and our data easier to observe, we made a menu system to view various parts of the data available. None of the code used to display or navigate through the menus are needed to use the SDK, all data can be accessed simultaneously.

‍

Landscape

The landscape callback is one that happens every second, this callback gives information about various parts of MANUS Core, for example which gloves and dongles are connected, which users exist, gestures, etc. Our sample client does not use all the information given via the landscape to keep the example a bit easier to follow.

If you plan to use the Gestures and need the landscape data related to these make sure to call the CoreSdk_GetGestureLandscapeData to retrieve all the data needed for the gestures from within the callback to ensure that the entirety of the landscape is stored correctly.

An overview of most of the landscape is displayed in our landscape viewer tool:

The entire Landscape structure is self-explanatory, the gloves and dongle information can be found under the gloveDevices variable. You can find the battery information, the pairing state of the glove, the type of glove, haptics, etc. inside the GloveLandscapeData structure. In the DongleLandscapeData structure you can find information such as the dongle type, firmware versions, which gloves are paired to it, etc. In the users you can find information such as which dongle and gloves are assigned to which user.

‍

Gloves & Dongles

The code in the DisplayingDataGlove function handles the console logging, the HandleHapticCommands function shows the way haptics are handled when using a glove ID to identify which glove needs to vibrate. In this example 2 gloves are connected, and their ergonomics data (received via the callback) is shown with a timestamp. If the gloves are haptic enabled, you can use the number keys to trigger individual haptic actuators on the fingers. This view also shows which dongles are present and the licenses on them.

In the ergonomics callback we filter on the first user`s gloves so as not to bloat the sample code, in normal cases you usually want all the information contained in this callback when using ergonomics.

‍

Skeletons

MANUS Core uses something weve namedskeletonsto animate the hands and bodies. An important part of the skeleton setup are nodes, which you can consider as being the joints (the point around which your finger bone rotates) of your fingers. Another important part of the skeleton setup are the chains, these define which nodes belong to which part of the skeleton. For example, your thumb chain would contain all the nodes that make up your thumbs bones. There are three different skeletons in MANUS Core, Temporary Skeletons, Retargeted Skeletons and Raw Skeletons. Temporary Skeletons are the skeletons that you create and send to the development tools for verification, but you do not load them into MANUS Core`s retargeting system. Retargeted Skeletons are usually called Skeletons in the SDK, which are loaded into MANUS Core with the animation data applied. The Raw Skeletons are hand models on which MANUS Core applies the hand data without doing any retargeting, these are not necessarily using the same bone hierarchy structure as the skeletons that the user loads into MANUS Core.

‍

Skeletal Data

MANUS Core supplies two streams of skeletal data. The first being retargeted skeletal data, this is the glove data applied onto a skeleton of the users choosing. The second stream is the raw skeletal data, this is the glove data applied onto a standardized MANUS hand skeleton. Depending on your applications needs you can use one stream or the other, or in some cases even both. The retargeted skeletal data is useful when you have your own skeleton or hand model, and you would like things like pinches to look better than they would when simply applying rotations from the raw skeleton stream. Using retargeted skeletal data allows you to create hands that have strange dimensions, or a different number of fingers compared to a real-life hand. The raw skeletal data is generally the more accurate representation of the glove wearers hand. The rotations and positions output in the raw skeleton stream are usually closer to thereal world` hand of a person, but applying this data directly to your desired skeleton or hand model can be a much more challenging task and will not guarantee lifelike animation . Another important detail to keep in mind is that our raw skeletal data does not contain any wrist positioning or orientation, if these are needed then it is advised to use the retargeted skeletal data. The closer the dimensions are to a real-life hand the smaller the difference is between the retargeted and the raw skeletal data, at which point it tends to be much easier to just use the retargeted skeletal data for your skeleton or hand model. Both skeletal stream callbacks function in a similar way, and can be seen in the example client:

Getting the basic skeleton information such as the ID and node count ca n be done via the CoreSdk_GetSkeletonInfo or the CoreSdk_GetRawSkeletonInfo respectively. In this info you can get the ID of the skeleton, which is either the id of the retargeted skeleton or of the glove ID , depending on which of the two streams you are looking at. Using the CoreSdk_GetSkeletonData or the CoreSdk_GetRawSkeletonData you can get all the node data for the skeleton. The ID of the node is either the ID you gave it when setting up the skeleton for the retargeting or the ID that we gave the raw skeleton in MANUS Core. The difficult part about using the Raw Skeletons compared to the Retargeting Skeleton is that you will need to reconstruct the hierarchy of that skeleton to be able to apply the data gotten from the stream. In the PrintRawSkeletonData function we demonstrate how to get the hierarchal information for the raw skeleton.

The CoreSdk_GetRawSkeletonNodeCount function can be used to get the number of nodes that exist in the raw skeleton of a certain glove. Using the CoreSdk_GetRawSkeletonNodeInfo you can get all the information for every node in the Raw Skeleton. This NodeInfo gives you information such as the parent`s ID and the side. Using this information you can reconstruct the skeleton. When using a relative coordinate system (a not world space coordinate system) you will need to keep the hierarchy into account to be able to accurately reconstruct the skeleton.

‍

Skeletons in the Sample Client

This menu shows a few options. The first two are to load or unload a skeleton to MANUS Core, which we will talk about later in this article.The other options are to activate haptics based on the skeleton id instead of a glove id. This is an easier way to do haptics since you do not have to manually keep track of which glove belongs to what skeleton.

‍

Setup a skeleton

To setup a hand skeleton and make sure it animates correctly, the following steps are required:

• Setup the basic skeleton information
• Create and add nodes for the skeleton
• Create and add chains for the skeleton

The first step is creating a SkeletonSetupInfo structure and setting the data you wish to adjust. For a hand skeleton the type needs to be set to the SkeletonType_Hand enumerator.

To have the skeleton move according to the hand movement of the first user you can set the settings.targetType to SkeletonTarget_UserIndexData and the settings.SkeletonTargetUserIndexData' to 0. This means it will look at all the users in MANUS Core and take the first users hands to animate the skeleton.

There are several different target types which allow the user to choose how the skeleton is animated. The simplest is the user index, which we explained above, it requires very little knowledge of which gloves or users are available. When changing the target type make sure to match the targetType with the structure of the data, for example the SkeletonTargetType_GloveData enumerator requires the use of the SkeletonTargetGloveData structure.

The user data target allows you to specify a user ID which will enable the skeleton to be animated by the user specified, this does require you to know which users are available. Please keep in mind that the ID is not the same as the index, and you would need to find a certain user`s ID in the landscape. The glove data target allows you to specify a glove ID. This ID will be used to find the correct glove and animate your skeleton according to the given glove. If this glove does not exist, the skeleton will not be animated.

To start adding the nodes and the chains to the skeleton setup we need to make it into a Temporary Skeleton by calling the CoreSdk_CreateSkeletonSetup function, which returns an index value which we can use to modify and load the skeleton.Every skeleton needs a root node to define a point where everything originates from. The ID of the root node should be 0, and the parent of every other node should have a parent id.

In our example we setup our skeleton nodes with the following positions:

We use these positions when creating a NodeSetup for each of the nodes, as seen in the CreateNodeSetup function. The resulting structure is then passed into the CoreSdk_AddNodeToSkeletonSetup function which adds it to the Temporary Skeleton. To have a functional hand skeleton we need to have a Hand chain which contains the ids of all finger chains, and it should have the node ID of a wrist along with which side the hand is from. The handMotion setting in the hand chain specifies if the wrist should move via IMU data or via tracker data, or not at all. Each of the needed chains can be added with the CoreSdk_AddChainToSkeletonSetup function. It is also possible to assign nodes to chains in the Dev Tools, we will explain this in more details during the Temporary Skeletons part of this article. In our example we add a wrist and all 5 fingers:

Adding the finger chains goes in a similar process to the hand chain. We need to make sure to add all the node ids and set the handChainId to the same id as the one set in the hand chain, which is 0 in our example. Do not forget to set the correct side for the finger chains as well. At this point we have a fully functioning Temporary Skeleton. This skeleton can either be loaded into the retargeting system of MANUS Core or sent to the Dev Tools for verification via the CoreSdk_SaveTemporarySkeleton function, which we will get more into in the Temporary Skeletons section of this article.

‍

Retargeted Skeletons

After setting up a Temporary Skeleton via the method explained above the user can call the CoreSdk_LoadSkeleton function. This will load the Temporary Skeleton into MANUS Core`s retargeting system and remove it from the list of Temporary Skeletons (because it is no longer temporary at this point). The ID returned from the load function is the one you should keep track of to match the skeleton stream data to the skeleton you loaded in. It is also possible to unload a skeleton when you no longer need it to be animated. This prevents unnecessary calculations being done for a skeleton that may no longer be in use.

In this example, if there are skeletons loaded, we can unload a skeleton by calling CoreSdk_UnloadSkeleton and passing the skeleton id. When a client application is closed, the session will be closed, and MANUS Core will automatically unload all skeletons that belong to that session. This can also happen when a connection is lost for any reason, but in such a case you can simply reload the skeletons.

‍

Temporary Skeletons

When a skeleton is still being defined it cannot be animated fully yet, but it can be sent via Manus Core to the Dev Tools for ease of verification. The Dev Tools will automatically launch and load the temporary skeleton. For in-depth information about how to use the Dev Tools, you can look at the Dev Tools article in our knowledge center.

To do this a Temporary Skeleton should be set up, which is like a regular skeleton, however not all the chains need to be defined.

The Dev Tools can send an updated Temporary Skeleton back (depending on what the user did in the Dev Tools) which can be used to create a regular skeleton for animation.

You can see this functionality in both the Unreal Engine and Unity plugins, where this is used to verify the animation.

If the Temporary Skeleton is not destroyed, it can be updated easily by the Dev Tools. If a network connection with MANUS core is lost, the temporary skeleton needs to be resent for the plugins to be able to work on it.

If the skeleton is well defined, you can also try to automatically allocate the chains with the CoreSdk_AllocateChainsForSkeletonSetup function, however this can be quite opaque without a visualization, so i t is not advised to do this outside the Dev Tools.

See the BuildTemporarySkeleton function in the sample client for an example of this process. A temporary skeleton can also be transformed into a file and vice versa for ease of handling, these files are saved as an MSKL file. These binary files can be loaded into MANUS Core, this might make it easier for you to adjust the data later. An example of saving and loading MSKL can be found in the SaveTemporarySkeletonToFile and GetTemporarySkeletonFromFile functions.

The example first sets up a temporary skeleton and saves it to Manus Core via the CoreSdk_SaveTemporarySkeleton function. After that it can get the number of bytes needed to save the data from CoreSdk_CompressTemporarySkeletonAndGetSize and you can call the CoreSdk_GetCompressedTemporarySkeletonData function to get the actual data into the byte array. In the end it saves it to file like a normal byte array. In the example the temporary skeleton is also cleared and removed, but this is only for illustration purposes and not required if you plan on using the skeleton.Loading a temporary skeleton from a file is like saving one, in the example function GetTemporarySkeletonFromFile you can see the process of reading a file out and sending it to MANUS Core.

The byte array you read out of the file can be sent to MANUS Core via the CoreSdk_GetTemporarySkeletonFromCompressedData function, which will create a new temporary skeleton and return the ID of that skeleton.

‍

Trackers

Manus Core is able to ingest positional and rotation data from various tracking systems (for example ART and SteamVR) and can be used to set up a skeleton for the whole body. For this, they must be assigned to a user and configured to represent a limb or body part location. For more details on this, please check https://www.manus-meta.com/resources/knowledge-base. However, getting the data for trackers can still be useful for props etc. You may also want to add your own tracker system and be able to stream the tracker data back into MANUS core.

You can access tracker information via the data gotten via the landscape callback or via a few functions we made specifically for the tracker information.

Showing trackers per user or globally:

In this code example all users are first gathered and then the tracker IDs per user are counted and displayed. For normal body tracking the trackers are usually assigned to a user. Unassigned trackers will need to be found with the global function CoreSdk_GetNumberOfAvailableTrackers followed by the CoreSdk_GetIdsOfAvailableTrackers function.

No actual tracker data or meta data is being shown here, but it can be retrieved with the CoreSdk_GetDataForTracker_UsingTrackerId function. Using the meta data, you can determine if it is an unassigned tracker or not in case you want to use a tracker for something else than body tracking.

‍

Adding and removing custom trackers

In this code example a simple TestTracker is set up and passed into MANUS Core by passing an array of trackers into the CoreSdk_SendDataForTrackers function. As a test, the position is slightly altered every update to show it moving and is visualized in the MANUS Core Dashboard if it is also running. When you make your own trackers make sure every tracker has a unique ID and is trackable. In this test only one tracker is being sent, but it is advisable for synchronization to send all custom trackers in the same array at the same time.

‍

Timecode

A timecode generator is best configured via the MANUS Core Dashboard . However, you might still be interested in the timecode settings in your own client. For this you can use the functions shown in the PrintLandscapeTimeData function of the example client.

If the landscape callback is registered, you can read out the landscape data to retrieve the time landscape with timecode information. Due to thread safety, this is copied during a mutex to a local copy of the data and eventually printed in the UI thread via the PrintLandscapeTimeData function. For more information on timecode, please consult your timecode device manual.

‍

Gestures

MANUS Core has a built-in gesture system. Data from this gesture system can be accessed via the Gesture Stream. In our example in the function OnGestureStreamCallback you can see how to receive the data:

The stream info gives you the number of gestures available per glove. Using the CoreSdk_GetGestureStreamData function you can get the specific gesture information for a given glove. Due to the amount of potential gesture data in the future, it is advisable to call this function in the callback to gather all the gesture data. This way it becomes impossible for the data to be modified by the SDK during this function call. In our example we gather the gesture data for the first left and right glove, we display this data to the console:

The gesture data you receive contains ID and normalized percentage numbers (from 0.0 to 1.0). The gesture ID can be matched to the gesture name shown in the landscape data for gestures, which will allow you to see what gesture it is.Currently it is not supported to add custom gestures to MANUS Core , when this does become possible it is best to simply remember what IDs are assigned to which gesture. In predefined gestures, the IDs will remain the same throughout the application runtime, so you will only need to match the gesture`s name to its ID once if you wish to use those names for identification.

‍