[SOLVED] Trouble starting a game server at boot with Systemd

[sebsensegreen]

Hi, I made a game server with Steam (The game is Wurm unlimited but it's not really relevant.) and I have some trouble make it start on boot time with Systemd. Just to be clear, I'm absolutely not familiar with Systemd.

I'm using Linux Mint 19.2 Cinnamon.

So, everything works fine up until I reboot the server. At boot time, the WU server starts without errors but it doesn't show up in the server list and when I try to connect directly via IP with a client, the login process stops, the window closes and I get this error:

Code: Select all

Execution aborted at connection 0, iteration 0
Run time 0s, local time Thu Oct 10 13:55:54 EDT 2019
Destroying game window
====== CONNECT DENIED ======
Steam could not authenticate the user

Now, if I restart the server manually with Systemd (Even a few seconds after the actual server booted up), everything works okay and is back to normal. This only happens at boot time.

I've tried to put some delays in the Systemd script, even up to a minute, waiting for an actual functioning internet connection, etc. But to no avail. The first start at boot time doesn't work because apparently Steam can't authenticate the clients connecting to it.
I've tried with After=network.target nss-lookup.target or After=NetworkManager.service, ExecStartPre=/bin/sleep 60 without success.

My current Systemd service for the server:

Code: Select all

# Wurm Unlimited Dedicated Server systemd service file

[Unit]
Description=Wurm Unlimited Server
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=sebsensegreen
WorkingDirectory=/home/sebsensegreen/.steam/steam/steamapps/common/Wurm Unlimited Dedicated Server/
ExecStart="/home/sebsensegreen/.steam/steam/steamapps/common/Wurm Unlimited Dedicated Server/startserver.sh"
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
Restart=on-failure

[Install]
WantedBy=multi-user.target

I'm at a loss and can't find how to fix this. Any help would be greatly appreciated .

[gm10]

Usually things that run through steam need you to be logged in to Steam, which you are not because you are not even logged in at the time this runs.

[sebsensegreen]

Usually things that run through steam need you to be logged in to Steam, which you are not because you are not even logged in at the time this runs.

This is a server, it logs in anonymously to Steam when starting. Steam isn't actually installed on the machine, only SteamCMD.

[gm10]

I don't know how that server works specifically but since you're getting the error message about Steam failing to authenticate it seems that nevertheless something is wrong with the Steam interaction. Check the server's logs I guess (I assume you already checked service logs to make sure it ran at all).

[sebsensegreen]

I don't know how that server works specifically but since you're getting the error message about Steam failing to authenticate it seems that nevertheless something is wrong with the Steam interaction. Check the server's logs I guess (I assume you already checked service logs to make sure it ran at all).

Basically, from what I understand about this stuff, Steam or SteamCMD doesn't start at boot. The game server calls the Steam API to connect anonymously i.e. The servers are free tools anybody can use, you don't need a steam account or anything. You just need the right API librairy.

I did look at the service's logs and indeed, I missed this error:

Code: Select all

Oct 15 22:21:36 Sophie startserver.sh[1335]: [S_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.
Oct 15 22:21:36 Sophie startserver.sh[1335]: dlopen failed trying to load:
Oct 15 22:21:36 Sophie startserver.sh[1335]: steamclient.so
Oct 15 22:21:36 Sophie startserver.sh[1335]: with error:
Oct 15 22:21:36 Sophie startserver.sh[1335]: steamclient.so: cannot open shared object file: No such file or directory
Oct 15 22:21:36 Sophie startserver.sh[1335]: [S_API FAIL] SteamAPI_Init() failed; unable to locate a running instance of Steam, or a local steamclient.so.
Oct 15 22:21:36 Sophie startserver.sh[1335]: [10:21:36 PM] INFO com.wurmonline.server.steam.SteamHandler: Could not start server
Oct 15 22:21:36 Sophie startserver.sh[1335]: Could not start server

It can't find the steamclient.so librairy but it is present:

Code: Select all

Wurm Unlimited Dedicated Server$ ls steamclient.so
steamclient.so

and it is also in another sub-folder:

Code: Select all

Wurm Unlimited Dedicated Server/linux64$ ls
steamclient.so

It's weird because the actual service is started and active and the server is up (but not accessible)
This is the status of the server:

Code: Select all

● wurm.service - Wurm Unlimited Server
   Loaded: loaded (/etc/systemd/system/wurm.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-10-15 22:21:26 EDT; 7min ago
 Main PID: 1335 (startserver.sh)
    Tasks: 59 (limit: 4915)
   CGroup: /system.slice/wurm.service
           ├─1335 /bin/bash /home/sebsensegreen/.steam/steam/steamapps/common/Wurm Unlimited Dedicated Server/startserver.sh
           └─1344 ./WurmServerLauncher-patched HOMESERVER=true HOMEKINGDOM=4 LOGINSERVER=true MAXPLAYERS=200 QUERYPORT=27017 SERVERNAME=The Test Server SERVERPASSWORD=1234 START=05102019-01 INTRASERVERADDRESS= INTRASERVERPASSWORD= EXTERNA

Oct 15 22:21:39 Sophie startserver.sh[1335]: ==================================================================
Oct 15 22:21:39 Sophie startserver.sh[1335]: Wurm Server launcher finished at Tue Oct 15 22:21:39 EDT 2019
Oct 15 22:21:39 Sophie startserver.sh[1335]: ==================================================================
Oct 15 22:21:39 Sophie startserver.sh[1335]: [10:21:39 PM] INFO com.wurmonline.server.gui.WurmServerGuiMain: WurmServerGuiMain finished
Oct 15 22:21:39 Sophie startserver.sh[1335]: [10:21:39 PM] INFO com.wurmonline.server.console.CommandReader: Starting command reader for console input
Oct 15 22:21:39 Sophie startserver.sh[1335]: [10:21:39 PM] INFO com.wurmonline.server.console.CommandReader: Console reader exiting.
Oct 15 22:26:32 Sophie startserver.sh[1335]: [10:26:32 PM] INFO com.wurmonline.server.Server: current mem in use: 1214M free mem: 515M Max mem: 2010M
Oct 15 22:26:32 Sophie startserver.sh[1335]: player count: 0
Oct 15 22:26:32 Sophie startserver.sh[1335]: bytes in: 0 bytes out: 0 total in: 0 total out: 0
Oct 15 22:26:32 Sophie startserver.sh[1335]: Server uptime: 300 seconds. Unanswered questions:0

[sebsensegreen]

Also, like I said earlier, if I restart the service, even a few seconds after the computer is rebooted, everything is alright and Systemd, or the script that starts the WU server, do find the library.

[sebsensegreen]

Ok, I did an strace on the script that starts the server and I found that is was looking for the library in another folder than what it is supposed to look in.

Code: Select all

openat(AT_FDCWD, "./nativelibs/x86_64/steamclient.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

This sub-folder x86_64 doesn't exist...

So I created the folder and I copied the library in that folder and now everything seems to work. I didn't spot the error in the Systemd logs at first because the logs are so long, I probably passed it while trying to find something.

Anyways, thanks for your help, it's really appreciated.

[gm10]

Had to be something like that. I suppose the alternative would have been amending the LD_LIBRARY_PATH environment, I suppose something like that happens in your user's environment for it to work once you log in.