Gameheim

Install the Gameheim Perforce extension: every submit reaches your project

For the Perforce administrator of a studio that uses Gameheim. You install one signed extension on the server, once, and from then on every submitted changelist reaches the Gameheim project it belongs to, whatever tool submitted it. The guide covers the installation from Gameheim with your agent, the same steps from the command line, more projects on one server, updates and the checks to run when something does not arrive.

By Gianmarco Virzì, co-founder and lead developer of Virsion · Updated · 22 min read

Submits made outside Gameheim are invisible to the project

People submit from many places: P4V, the source control panel of Unreal Editor, the p4 command line, build machines. A tool that only sees the submits it made itself misses most of the history of the project, and anything built on that history, such as tasks closed by a changelist, lags behind what is really in the depot.

The usual answer is a trigger script on the server. It works, and it brings its own chores:

  • The script needs an account that can read every changelist, and its ticket or password usually ends up in a file on the server.
  • If the script calls a web service while the submit runs, a slow network makes every submit slow.
  • A changelist sent while the service was down is often lost, unless someone writes a queue and a retry loop.
  • Each server needs its own copy of the script and its own line in the triggers table, kept up to date by hand.

How the Gameheim extension sends your changelists

The Gameheim extension is a Perforce server extension: a small Lua program that the server installs, runs and replicates itself. Perforce recommends extensions over trigger scripts for this kind of integration. The package is signed by Gameheim, so a server that only accepts signed extensions can still install it once you trust the Gameheim certificate.

Nothing runs during a submit. A background task inside the server wakes up every ten seconds, reads the changelists submitted since the last one it sent, and sends each one, oldest first, to the Gameheim projects whose depot paths it touches. One counter per project on the server remembers how far that project got, and it moves only after Gameheim has answered for it. If Gameheim cannot be reached, the changelists simply wait in Perforce and leave as soon as it answers again, in order. A project Gameheim cannot take for a while waits alone: the others go on.

Sent for each changelistNever sent
Number, author, date, description and streamThe content of any file
Whether the changelist is restrictedFiles outside the depot paths of the project
Action, file type and depot path of up to 500 files, and how many files there arePasswords, tickets, the activation code or the signing secret in any log
For workspace events: the command name (such as sync), its time and the names of the workspaces Gameheim already knows for the projectThe files a command touched, its arguments, or workspaces Gameheim does not know

The extension reads as a service user of your choice, named in its configuration (ExtP4USER). The server signs it in by itself: the password of that user is not stored in any file, neither on the server nor in Gameheim. The extension only reads changelists and sets its own counters. It never changes files, changelists, users or the protections table.

For the review your security team may ask for, with every request the extension sends, every command it runs and what it writes on the server, see Gameheim Perforce extension: security and IT review.

One extension serves every Gameheim project on the server: each project is one line of its configuration, with the project's activation code and depot paths. A changelist that touches two projects is sent to both, and each project receives only its own files.

How to install the Gameheim extension on a Perforce server

The steps are done once, by the Perforce administrator. People who submit change nothing in their tools.

Before you start
  • Helix Core Server 2019.2 or later on Linux, or 2021.2 or later on Windows. With replicas or edge servers, install the extension on the commit server: it reaches the edges by itself, and its background task also runs on each edge (see Large servers and edge servers below).
  • A Perforce user with super access, to trust the certificate, install the extension and start its background task.
  • Outbound HTTPS (port 443) from the Perforce server to www.gameheim.com. No inbound port is needed.
  • Perforce super level on the Gameheim project, to generate its activation code: whoever holds the project's key speaks for the Perforce server. A Perforce admin sees the Gameheim trigger tab but cannot generate the code.
  • A ServerID on the Perforce server (p4 serverid), so that the background task starts again with the server.
  • For the installation from Gameheim: your Gameheim agent 1.3.1 or later, online and connected to the project's commit server. The command line needs only p4.
  1. Install from Gameheim with your agent (recommended)

    In Gameheim, open Project Administration > Gameheim trigger. Under Install with the agent, your own Gameheim agent runs every Perforce step for you: no files to download and no terminal. You need Perforce super access on the project and an agent connected to the project's commit server (or standalone server). Each step waits for your click:

    • Check the server: your agent reads the server version, its role, your access, the extension in place, the Perforce licence and the edge servers. Nothing is changed.
    • Service user: create a new standard user (it takes a licence seat; its password is random, never shown and needed by nobody), or choose an existing standard user, whose password is never touched. The review access to the project's path is added if missing, through the same protections write as Perforce administration > Permissions, with its snapshot.
    • Gameheim certificate: compare the fingerprint your server reads with the one Gameheim publishes, and confirm. Your agent checks it again before trusting the certificate.
    • Install: installs or updates the extension, sets the service user, adds the project with a new activation code that nobody has to copy, starts the background task now and at every server start, and runs the connection test. The activation is approved automatically when the fingerprint your agent read matches the key Gameheim received; the test then runs again and should read gameheim-test: OK.
    • Done: for each edge server, whether its task is already running or starts at the next restart of the edge, with the command to start it now if you prefer.
    Note:An update keeps the configuration and the activation of your projects: no new code. On a Perforce server running on Windows, use the extension 1.2.1 or later: 1.2.0 cannot create its keys file there and never activates.

    The steps below do the same from the command line, if you prefer it or your agent cannot reach the commit server.

  2. From the command line: generate the activation code

    In Gameheim, open Project Administration > Gameheim trigger and click Generate activation code. Copy the code right away: Gameheim shows it only this once and keeps only a fingerprint of it. It works once, within 30 minutes. With it the extension binds a key pair it creates on the Perforce server to your project: only the public key travels, and the extension and Gameheim each compute the same signing secret from their own keys, so the secret never travels at all, not even at the activation. Every request is signed with it, and every answer of Gameheim is signed back. A proxy that inspects TLS traffic, or anyone who changes the extension's CaFile or HttpsProxy, can read the traffic but cannot sign a request or change an answer. You paste the code when the install script asks for it.

    From the same page, download the four files: gameheim.p4-extension, gameheim-certificate.txt, install-gameheim-extension.sh and install-gameheim-extension.ps1.

  3. Trust the Gameheim certificate

    Run the command without --yes first: p4 only shows the fingerprint of the certificate. Compare it with the fingerprint shown on the Gameheim trigger tab, then run it again with --yes.

    As a super user
    p4 extension --install gameheim-certificate.txt --cert
    p4 extension --install gameheim-certificate.txt --cert --yes
  4. Create the service user

    Create a standard user for the extension, for example gameheim-svc, give it a password and log in once as that user to change it. At security level 3 or higher a password set by an administrator starts expired, and the extension cannot work with an expired password.

    Then give it review access to the depot paths of your Gameheim projects in the protections table (p4 protect). review lets it read changelists and file lists and set its counters, nothing more. Without it, the files of a changelist are hidden from the extension. On a server with replicas or edges it needs this access on the commit server too, where it also writes the key of the Refresh mark; without it Gameheim keeps working, and every Refresh reads everything again.

    Protections table
    review user gameheim-svc * //Ocean/...

    Optional: super instead of review. Each person sees in Gameheim only the changelists and files Perforce shows them. By default each person's Gameheim agent works this out while it runs. With super, the extension works it out for every person whose Perforce user this server confirmed, also while their agent is off, and updates it within five minutes of a change to the protections or the groups. It only asks Perforce what each user may see (p4 protects -M): it still never changes files, changelists, users or the protections table. The connection test says which of the two is in use. With super, a member without a Perforce user can also follow the access of a Perforce group or user you choose in Perforce administration > P4 Users; a group or user with admin or super access is refused.

    Protections table, with the super option
    super user gameheim-svc * //...
  5. Run the install script

    Run the script as a super user, on any machine with p4 that can reach the server. It installs the extension (or updates it, keeping its configuration), sets the service user and the Gameheim address, adds the project, starts the background task now and at every server start, and runs the connection test. It asks for the activation code without showing it: paste the code from Gameheim. The code never goes on the command line, which stays in the shell history and shows in the process list; for automation it can come on standard input.

    Linux or macOS
    ./install-gameheim-extension.sh --package gameheim.p4-extension \
      --service-user gameheim-svc --path //Ocean/...
    Windows (PowerShell)
    Unblock-File .\install-gameheim-extension.ps1
    .\install-gameheim-extension.ps1 -Package gameheim.p4-extension `
      -ServiceUser gameheim-svc -Path //Ocean/...

    On Windows, Unblock-File removes the mark Windows puts on downloaded files, which the default execution policy of servers (RemoteSigned) refuses. If PowerShell still refuses the script, your execution policy is Restricted, the default on Windows desktops: run Set-ExecutionPolicy -Scope Process RemoteSigned in the same window first. It applies to that window only.

    A project with several roots takes several paths: --path //Ocean/... --path //Shared/Ocean/... on Linux, -Path //Ocean/...,//Shared/Ocean/... on Windows. Paths must end with /.... If the server reaches the internet through a proxy, add --https-proxy or -HttpsProxy with its address.

    Important:The script warns when the server has no ServerID. The task then runs until the next restart of the server and does not start again by itself: set one with p4 serverid <name> and restart the server when convenient.
  6. Approve the activation

    The first time, the connection test at the end of the script reads gameheim-test: WAITING FOR APPROVAL and prints the fingerprint of the extension's key, such as 32e5-5b5b-e726-c2f9-a50c-6c8d. Copy it, open Gameheim trigger in Project Administration and paste it under Approve the activation. A Perforce super user approves it. The fingerprint is computed on the Perforce server from the key the extension holds, so it proves that Gameheim received that key and not another one.

    Note:Until the activation is approved, Gameheim accepts nothing from the extension, and the changelists wait on the server: none is lost. If Gameheim says the fingerprint is not the one it received, someone else presented the code first with a key of their own. Do not approve it: generate a new code and run the script again.
  7. Check the extension

    The connection test runs at the end of the script, and you can run it at any time. Every line should end well, and the last one reads gameheim-test: OK.

    Connection test
    p4 extension --run gameheim test-connection

    Then submit a changelist under the project's path from any tool. Within about ten seconds Last changelist received on the Gameheim trigger tab shows its number. The first run of the task starts from the current changelist: older history is not imported.

More projects on the same server, and updates

To add another Gameheim project, run the install script again with that project's paths, and paste its activation code when asked. The other projects stay as they are, and a line with the same paths gets the new code, which is also how you put a new code in place after generating it again.

To update the extension, run the script with the new package. It moves the configuration to the new version and removes the old one. It also replaces the background task: a running task keeps the code and the configuration it started with, so it must be restarted after every update or change. The script does it for you.

If you change the configuration by hand (p4 extension --configure Gameheim::gameheim-perforce --name gameheim), run the connection test afterwards. The old task then stops sending, so that nothing leaves with the old list of projects, and the test tells you to start a new one.

Large servers and edge servers: how Gameheim keeps its queries light

The project page shows, for each person, the highest changelist their workspace is synced to. Perforce has one way to compute it, p4 changes -m1 @<workspace>, and the server reads about as many rows as there are revisions of the synced files, plus three per file. A workspace of 100,000 files with five revisions each costs about 800,000 rows; one of 2 million files with seven revisions costs about 20 million. The cost depends on the size of the workspace, so Gameheim chooses how often to ask for it.

ModeWhen Gameheim asksWhat people see
Every minuteSmall and medium workspaces, where the query takes less than a second. This is how Gameheim always worked.The value, as always.
On eventsWith the extension installed: only after a command that can change the value (sync, flush, update, submit, reconcile, clean, unload, reload, switch, a deleted or renamed workspace) and after changes that may concern everyone (protect, group, ldapsync -g, obliterate -y). ldapsync run as a startup command leaves no event: the extension compares the members of every group every five minutes and asks then. Also once every six hours, as a safety net.The value, and Updating for the few seconds after a sync.
On requestWithout the extension, when the query takes a second or more three times in a row, or when a group limit refuses it: when someone opens the workspace page (if the value is older than two minutes), when they press Refresh, when a crash arrives, and once an hour.Updated at HH:MM and a Refresh button. A crash takes the most recent value, at most one hour old.

Gameheim moves a workspace to On request when the query takes a second or more three times in a row, so that one slow moment of the server moves nothing, and back to Every minute when it drops under 300 milliseconds. The Perforce administrators of the project get one notice the first time it happens, suggesting the extension. With the extension, a large workspace costs one query per sync instead of one per minute.

The Perforce administrators of the project can choose the behaviour in Project Administration > Gameheim trigger > Synced changelist refresh: Automatic (the default, as described above), Always on request (on request for every workspace, also with the extension: the least load on the server) or Always up to date (never on request because of cost: on events with the extension, otherwise every minute whatever the query costs). The tab also shows how many workspaces are in each mode. In every choice a query refused by a group limit stays on request.

A manual Refresh also uses the extension. From version 1.2.0, each command that can change what a workspace has synced (sync, flush, update, submit, client, reconcile, unload, reload, renameclient, clean, switch) leaves a mark in a small file of the extension's own folder on the server, at its start and at its end. Commands that can concern every workspace (such as obliterate, protect, group, stream or typemap), and commands that name another workspace, increase the Perforce key gameheim-global-seq instead. When someone presses Refresh and nothing changed their workspace since the last full reading, less than an hour before, the agent reuses that reading and the Refresh is almost immediate. Nothing is written in the depot, nothing travels on the network, previews such as sync -n leave no mark, and the extension never refuses or delays a command: we measured about 1 to 3 ms per command on a standard server and about 4 ms on an edge. The Refresh reads everything again, as before, on a forwarding replica, on a workspace owned by another Perforce user, after the agent restarts, or with an earlier version of the extension.

Gameheim runs at most four of these computations at once on each Perforce server, the others wait their turn, so that a protections change or the morning syncs never reach the server as one burst. The limit counts the whole server, every project of your organization on it, and the Perforce administrators can change it in the same tab, under Computations at once on the Perforce server: 1, 2, 4 or 8. It is a trade-off. While a computation reads the synced files of a workspace, a sync or submit on that server may wait up to about a second; in our tests on a 2 000 000-file workspace with four computations always running, one of those commands in twenty took about 1.5 seconds instead of tens of milliseconds. Fewer at once means fewer cores and fewer waits, but the value arrives later when many people sync together. A server counts once the Gameheim extension has reported its ServerID; on a server without the extension, or without a ServerID, it stays four at once per project: install the extension, and give the server a ServerID with p4 serverid if it has none, to choose.

On edge servers the background task also runs, started by the same startup.N at their next restart, or at once with the command the install script prints for each edge. There it only hands the workspace events of the edge's users to the commit server, in Perforce keys gameheim-ws.<ServerID>.*, and marks itself alive in gameheim-ws-alive.<ServerID>. The commit server sends them to Gameheim with the changelists. Until the task runs on an edge, the workspaces on that edge stay on Every minute or On request: nothing is missed, it only costs more.

Group limits. The query runs with the Perforce user of each person, so the MaxScanRows, MaxResults and MaxLockTime of their groups apply. When a limit refuses it, the page says why, the workspace moves to On request and Gameheim tries again once an hour instead of every minute. To have the value for very large workspaces, raise MaxScanRows for the groups of those people above the rows estimated above, or leave the limit as it is: the rest of Gameheim works without the value.

Configuration reference

WhereFieldValue
Global configurationExtP4USERThe service user the extension reads as
Global configurationGameheimUrlhttps://www.gameheim.com: the only address the extension talks to, written in its signed code
Global configurationHttpsProxyProxy address, or - for none
Global configurationCaFileCA bundle of the machine, or - to let the extension find it
Global configurationServerFingerprintThe SSL fingerprint of this server, written by the Gameheim wizard (1.2.3): the background task trusts only this certificate on its own connection. - with the script: trusted at the first connection
Instance gameheimProjectsOne line per project: <activation code> //depot/path/..., more paths separated by spaces
Serverstartup.Nbgtask -t gameheim -i 10 -m 2000000000 -b 2000000000

The instance must be called gameheim: the Gameheim agent runs p4 extension --run gameheim to confirm identities. The extension keeps these counters on the server: one gameheim-feed.<id> per project (the last changelist sent to it; the id is a fingerprint of the activation code, never the code), gameheim-feed-owner, gameheim-feed-config and gameheim-feed-heartbeat. A server that ran an earlier version also has gameheim-feed: each project starts from it, then the extension deletes it. On a server with edges the extension also keeps the keys gameheim-ws.<ServerID>.* (workspace events waiting to leave, deleted once sent) and gameheim-ws-alive.<ServerID> (when each edge task last ran); the service user needs review access to read and delete them. The counter gameheim-ws-groups holds a fingerprint of who is in which group (p4 groups, members and subgroups only), compared every five minutes. The keys of the activated projects are in the file gameheim-keys, in the extension's folder under server.extensions.dir on the Perforce server: beside the folders of each installed version, so that an update keeps them. Only the account p4d runs as can read or write it, and no p4 command reads it. From version 1.2.0 the extension also keeps the key gameheim-global-seq (increased by the commands that can concern every workspace) and, in the same folder, one file ws-<16 hexadecimal characters> per workspace with the marks of its commands, one run-… file for each command still running (deleted when it ends) and a marker-epoch file. They hold random marks, times and the process id of a running command: no workspace names, paths, users or command arguments.

Troubleshooting: changelists do not reach Gameheim

To see what the extension knows about a workspace, its owner can run p4 extension --run gameheim state <workspace>. It prints one line starting with gameheim-state:, with the workspace's mark, the key, the server and whether the extension can write its files and the key (keys=ok, no or error). Anybody but the workspace's owner gets gameheim-state: denied.

What you seeWhat to do
Installation failure: extension package is signed by untrusted certificateTrust the Gameheim certificate first, then run the script again.
Gameheim does not recognise this activation code, or it expiredThe code works once, within 30 minutes. Generate a new one in Gameheim trigger and run the script again with it.
This activation code was already used, or someone else presented itThe code works once, for one Perforce server. Generate a new one in Gameheim trigger and run the script again with it.
Gameheim refused the signature of this serverCheck that the server's clock is right (within 5 minutes). If the code was generated again in Gameheim, run the script again with the new one.
An answer arrived without Gameheim's signatureSomething between the server and Gameheim answered in its place, or changed the answer: the extension ignored it and will try again. Check the proxy in HttpsProxy and the certificates in CaFile.
No secure random sourceThe extension creates its key from /dev/urandom on Linux and from PowerShell on Windows. On Windows, allow the Perforce service to run %SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe, then run the connection test again.
The extension's data directory is not where this version expects itThe extension keeps no key and does not activate. Send the output of p4 extension --list --type extensions to Gameheim support.
Notice: the Gameheim extension connects differentlyIts CaFile or HttpsProxy changed. If nobody in your team changed them, find out who did. Since admin users can configure signed extensions unless server.extensions.allow.admin is 0, consider p4 configure set server.extensions.allow.admin=0.
Gameheim could not be reachedAllow outbound HTTPS to www.gameheim.com, by host name or with the AzureFrontDoor.Frontend service tag, or set HttpsProxy.
The server could not verify Gameheim's certificateOn Linux distributions that keep certificates in an unusual place, set CaFile to the CA bundle of the machine.
Service user cannot run Perforce commandsIts password has expired or was never set. In Project Administration > P4 Users, a Perforce super user can choose Set password > Nobody knows it for that user; or log in once as that user and change the password.
Cannot set countersGive the service user review access in the protections table.
Warning: a group of the service user sets MaxResults (or MaxScanRows, MaxLockTime, MaxMemory)With protected paths, a changelist too large for these limits cannot be checked, and the extension lets it through and tells Gameheim why. Put the service user in a group that sets these limits to unlimited: when a user is in several groups, unlimited in any of them wins.
Background task: not runningStart it with p4 bgtask -t gameheim -i 10 -m 2000000000 -b 2000000000 -d. If it stops at every restart, set a ServerID.
Background task: stopped, because this server's SSL certificate changedThe certificate was renewed, or changed. If you expected it, run the installation again from Project Administration > Gameheim trigger: Gameheim reads the new certificate through your agent and gives it to the extension. If nobody renewed it, find out who changed it first.
Background task: running, but each run failsThe line carries the first line of the error. A Perforce error names the cause (protections, password, limits); send the line to Gameheim support otherwise.
Background task: running an older version or configurationStart a new task with the same command: the old one stops sending by itself.
Workspaces on an edge still refresh every minute, or on requestStart the task on that edge: p4 -p <edge address> bgtask -t gameheim -i 10 -m 2000000000 -b 2000000000 -d, or restart the edge. Check that it has a ServerID.
The workspace page says the server limits refuse the queryA group limit of that person (MaxScanRows, MaxResults or MaxLockTime) refuses p4 changes -m1 @<workspace>. Raise it for their groups, or leave it: only that value is missing.
Notice: the Gameheim extension did not send some changelistsRun p4 extension --run gameheim test-connection on the server. If it says OK, check that the depot paths of the project in the extension configuration cover every stream the team submits to.

Remove the extension

Unset the startup command the script added, stop the task (its process id is in p4 monitor show -al, on the commit server and on each edge), delete the extension, its counters and its keys:

As a super user
p4 configure unset startup.<N>
p4 monitor terminate <process id>
p4 extension --delete Gameheim::gameheim-perforce --yes
p4 counters -e "gameheim-feed.*"
p4 counter -d gameheim-feed.<id>
p4 counter -d gameheim-feed-owner
p4 counter -d gameheim-feed-config
p4 counter -d gameheim-feed-heartbeat
p4 counter -d gameheim-ws-groups
p4 key -d gameheim-global-seq
p4 keys -e "gameheim-ws*"
p4 key -d <each key listed>

Frequently asked questions

Does a submit wait for Gameheim?

No. The extension does nothing while a submit runs. A background task on the server reads the submitted changelists every ten seconds and sends them afterwards, so a slow network or an outage never slows down P4V, Unreal Editor or the command line.

What happens to changelists while Gameheim cannot be reached?

They wait in Perforce. The extension moves a project's counter only after Gameheim has stored the changelist for that project, so when Gameheim answers again the waiting changelists leave in the order they were submitted.

Does Gameheim receive the content of our files?

No. For each changelist it receives the number, author, date, description, stream, whether it is restricted, and the action, type and depot path of up to 500 files under the project's paths. File content never leaves the server.

Can one Perforce server feed several Gameheim projects?

Yes. One extension serves them all: each project is a line with its activation code and depot paths. Run the install script once per project. A changelist that touches two projects reaches both, each with its own files only.

How much load does Gameheim put on a large Perforce server?

Locks cost about two rows per opened file, read once a minute while someone uses the PC and every five minutes once it is idle. The synced changelist is the expensive query on large workspaces: with the extension Gameheim computes it only after a sync or a change of protections, and without it, on large workspaces, only when someone looks. See Large servers and edge servers above.

Where does the password of the service user live?

Nowhere outside Perforce. The server signs the extension in as the service user by itself, so the password is not written in any file on the server and Gameheim never sees it.