Runner troubleshooting
What to do when a SourceLace Runner does not start, stays Quiet, cannot sign people in, has no license, or a run fails: every message the runner can show, what it means, how to confirm the cause and how to fix it, plus what to collect before you contact support so that one message is enough.
The installation is in Installing a runner; upgrades, backups and moves in Looking after a runner.
Where to look first
| Where | What it tells you | How |
|---|---|---|
| The Runners page in SourceLace | Status (Healthy, Quiet, Out of date, Revoked), last seen, version, license, health and the type of the last error | SourceLace app, Runners |
| The runner's Settings page | Last contact with SourceLace or the last problem, license, runs in progress, reports waiting, errors of the last 24 hours by type | https://<runner page>/settings |
| Check connectivity | Whether the runner reaches SourceLace (and through which proxy), whether SourceLace accepts its token, its database and each source, with what to do for each problem | Button on the Settings page |
| The runner's log | Start-up problems, contact problems with SourceLace, runs that could not be run | docker compose logs --since 1h runner, or kubectl -n <namespace> logs deployment/<release>-sourcelace --since=1h |
| The health address | Whether the runner answers and reaches its database | curl -sS http://127.0.0.1:8000/health on the machine, or https://<runner page>/health |
Log lines from the runner itself start with INFO:sourcelace:, WARNING:sourcelace: or ERROR:sourcelace:, followed by Runner: for its work with SourceLace. Lines starting with INFO: are the web server's own.
Check connectivity
An admin signs in to the runner's page, opens Settings and presses Check connectivity. Within about 10 seconds a table shows, for each line, OK, Not reached or Not checked, what happened and what to do:
- SourceLace: whether its health address answers, and the proxy used ("Through the proxy ... (HTTPS_PROXY)" or "Directly: no proxy is set").
- Runner token: "SourceLace accepts this runner's token", or why not.
- Database: whether the runner's own database answers.
- Each source: only whether its address answers. No sign-in is used and nothing is read. Not checked:
mongodb+srvaddresses, sources with no address yet, and sources reached through a SourceLace Agent for SAP ECC.
The check is recorded in the runner's own audit trail, with counts only.
When the runner does not start
The container stops right after starting, or restarts again and again (docker compose ps shows Restarting; Kubernetes shows CrashLoopBackOff). Confirm with docker compose logs runner (or kubectl logs ... --previous): the last lines name the problem.
| Log line | Likely cause | Fix |
|---|---|---|
SOURCELACE_RUNNER_TOKEN is required in runner mode: the token (slr_...) an admin created on the Runners page at sourcelace.ai. |
The token is missing, or does not start with slr_ (often quotes or a space copied with it). |
Put the token in SOURCELACE_RUNNER_TOKEN exactly as copied, without quotes. |
SOURCELACE_PUBLIC_URL must be the runner page's https:// address inside your network, such as https://runner.corvanta.internal. |
The page address is missing or starts with http://. |
Set the https:// address of the runner's page. |
SOURCELACE_CONTROL_URL must be an https:// address, such as https://sourcelace.ai. |
The SourceLace address was changed, or has a user name in it. | Set https://sourcelace.ai, or the address your SourceLace contact gave you. |
A runner needs its own database (DATABASE_URL) and master key (SOURCELACE_ENCRYPTION_KEY) when SOURCELACE_ENV=prod, ... |
One of them is not set. In Docker Compose, DATABASE_URL is set in docker-compose.yml; in Kubernetes, in the secret. |
Set both. |
SOURCELACE_ENCRYPTION_KEY must be a ... key or a random secret of at least 32 characters. |
The master key is not the generated value (cut short, or a made-up password). | Use the line sourcelace-keygen printed (Step 3), from your secret store. |
A tenant key could not be unwrapped with SOURCELACE_ENCRYPTION_KEY. (at start or on the first page) |
The master key is not the one this database was used with. | Put back the original key from your secret store. If it is lost, nothing in this database can be read again: start with a new, empty database and a new key; people reconnect their sources and admins enter the settings again. |
SOURCELACE_DEV_TOKEN cannot be used in runner mode: a runner serves no MCP clients. |
A development setting is set. | Remove it. |
SOURCELACE_RUNNER_LICENSE_SIGNING_KEY belongs on SourceLace's cloud only, never on a runner. |
A setting that does not belong on a runner is set. | Remove it. |
A validation error naming agent_parallel_runs, engine_memory_mb or another setting |
A number out of range (runs at a time must be 1 to 100). | Correct the value. |
Errors with password authentication failed, does not exist, Connection refused, could not translate host name or a timeout, while the database schema is being set up |
The runner cannot use its database: wrong address, password, database name, or the firewall. | Check DATABASE_URL with psql "<the same address>" -c "select 1" from the runner machine; fix the address or the firewall. |
permission denied for schema public or permission denied to create |
The runner's user does not own its database. | ALTER DATABASE runner OWNER TO runner; as an administrator (Step 2). |
Read-only file system or Permission denied naming /var/lib/sourcelace |
The folder for large results is not mounted, or not owned by user 10001. | Mount the folder and sudo chown 10001:10001 it (4.4), or remove SOURCELACE_ENGINE_DIR. |
no basic auth credentials, authorization token has expired, or Error response from daemon: ... unauthorized when pulling |
Docker is not signed in to SourceLace's registry, or the registry login has expired (it works for one hour). | Get a fresh login with Pull from registry (Get the runner) and run the docker login command it shows again; the user name is AWS. |
denied: ... not authorized or manifest unknown / not found when pulling |
The image name is not your organization's (a login reaches only your own images), or that version is not there (a typo, or a version whose support ended over a month ago). | Copy the image name exactly from Get the runner, and pick a version it lists. |
| Pull from registry says "Your organization's runner image of version ... is not ready yet" | SourceLace makes your images within a day of runners being switched on for you and of each new version. | Try again later, pick another supported version, or download the image file if it is offered. If it lasts more than a day, ask your SourceLace contact. |
A pull stops with a timeout, i/o timeout or TLS handshake timeout, often after Pulling fs layer |
The machine reaches the registry host but not Amazon's storage for image layers, or the reverse. | Allow outbound HTTPS to the registry host and to prod-us-west-2-starport-layer-bucket.s3.us-west-2.amazonaws.com (Every connection), or download the image as a file. |
ErrImagePull or ImagePullBackOff on Kubernetes |
The cluster cannot pull the image from the registry image.repository names. |
Use a copy in your own registry (Step 5, item 2), never SourceLace's registry login, which expires after one hour; check the pull secret for your registry. |
set SOURCELACE_IMAGE to the runner image SourceLace gave you |
docker compose up was run without the image. |
export SOURCELACE_IMAGE=<image>:<version> first, or put it in .env (4.6). |
The container is killed with exit code 137 (OOMKilled in Kubernetes) |
It ran out of memory. | Give it more memory, lower SOURCELACE_AGENT_PARALLEL_RUNS, or set SOURCELACE_ENGINE_MEMORY_MB lower (Sizing). With Docker Compose, also move large results to disk (4.4). |
The database schema is at version ..., newer than this release's ... (a rollback?). Leaving it as it is. (a warning, it still starts) |
You started an older version than the database was last used with. | Nothing, if you meant to go back a version (Going back a version). Otherwise start the newer version again. |
The runner stays Quiet
Quiet on the Runners page means the runner has not asked SourceLace for work in the last 90 seconds, or never has. Admins see a notice when a runner with agents stays quiet longer than your alert limit (10 minutes unless changed), and runs due on it wait with "Waiting for the runner ...: it has not asked SourceLace for work since ..., so this run has not started."
| Symptom | Likely cause | How to confirm | Fix |
|---|---|---|---|
| Last seen "Never" | The runner is not running, or has never reached SourceLace. | docker compose ps; then the log. |
Start it (When the runner does not start); then the rows below. |
Log: ERROR:sourcelace:Runner: SourceLace refused this runner's token: it is wrong or was revoked. An admin can create a new one on the Runners page at sourcelace.ai. Status Revoked, or Quiet |
The token is wrong, mistyped, from another runner, or the runner was revoked. The runner tries again every minute. | Check connectivity: "SourceLace refused this runner's token". | If the runner was revoked, or the token is lost: add a new runner on the Runners page and put its token in the settings (Changing the runner token). If it was mistyped: correct it and restart. |
Log: WARNING:sourcelace:Runner: could not ask SourceLace for work (Could not reach SourceLace (...)); trying in 15s. The brackets name the kind of failure. The wait grows (30s, 60s, ...) up to 300s |
The runner cannot reach sourcelace.ai: DNS, firewall, proxy or TLS. |
Check connectivity, line SourceLace; or from the machine curl -sS https://sourcelace.ai/health. |
Follow the line's advice; see Network problems below. |
| Same, naming a timeout | A firewall drops the traffic, or a proxy is required. | curl -sS --max-time 10 https://sourcelace.ai/health hangs. |
Allow outbound 443 to sourcelace.ai, or set HTTPS_PROXY. |
| Same, naming the proxy | The proxy refused the connection. | Check connectivity: "The proxy ... refused or could not make the connection." | Ask your network team to allow sourcelace.ai:443 through the proxy. |
| Same, and the connectivity check says the certificate could not be verified | TLS inspection, or a certificate the runner does not trust. | `curl -v https://sourcelace.ai/health 2>&1 | grep issuer` shows your company's authority instead of a public one. |
Log: Runner: could not ask SourceLace for work (SourceLace answered 503: ...) or (SourceLace answered 429: ...) |
SourceLace is updating or busy for a moment. | It passes by itself. | Nothing. The runner tries again and keeps every report until it is taken. |
Log: Runner: could not ask SourceLace for work (SourceLace sent an answer the runner could not read.) |
Something between the runner and SourceLace (a proxy or captive portal) answered instead of SourceLace, or the runner is much older than SourceLace. | curl -sS https://sourcelace.ai/health from the machine shows something other than {"status":"ok", ...}. |
Fix the proxy so it passes the traffic unchanged; otherwise upgrade the runner. |
Log: Runner: could not ask SourceLace for work (SourceLace Runners are not turned on for your organization. Ask your SourceLace contact.) |
Runners were switched off for your organization. | The Runners page says runners are not turned on. | Ask your SourceLace contact. |
| Status was Healthy, then Quiet, and the container is running | The machine lost its network, or its clock jumped. | The log at that time; timedatectl for the clock. |
Restore the network; synchronize the clock. |
Network problems
| Message (connectivity check or log) | Likely cause | How to confirm | Fix |
|---|---|---|---|
| "The name ... could not be found." | DNS cannot resolve the address (often an internal name the container's DNS does not know). | docker compose exec does not work (no shell); from the machine: getent hosts <name>. In Kubernetes, check the cluster's DNS forwards to your internal DNS. |
Fix the name, or the DNS servers the machine or cluster uses. |
| "No answer from ...:... within 8 seconds." | A firewall drops the traffic, or a proxy is needed. | curl -sS --max-time 10 https://<host> from the machine. |
Allow the runner's outbound connection to that host and port, or set HTTPS_PROXY. |
| "Could not connect to ...:...." | The connection was refused (nothing listens there, or a firewall rejects it). | curl -v https://<host>:<port> from the machine. |
Check the address and port; allow the connection. |
| "The proxy ... refused or could not make the connection." | The proxy does not allow this address, or needs a user name and password. | Ask your network team for the proxy's log. | Allow the address in the proxy, add the credentials to HTTPS_PROXY, or list internal hosts in NO_PROXY. |
| "The connection to ... was made, but its certificate could not be verified." | TLS inspection, a self-signed or internal certificate, or a certificate for another name. | `openssl s_client -connect <host>:443 -servername <host> </dev/null 2>/dev/null | openssl x509 -noout -issuer -subject` shows who issued it and for which name. |
| A source works through the proxy in your browser but not from the runner | That kind of source always connects directly (databases, REST and SOAP sources, outside MCP servers). | The connectivity check shows it tried directly. | Allow the runner to reach it directly in your firewall. |
| The connectivity check shows "Directly: no proxy is set (HTTPS_PROXY)" although you set one | The setting is not reaching the container. | docker compose config shows the environment the runner gets (it also prints secrets: do not share its output). |
Put HTTPS_PROXY in runner.env (or Helm env) and restart. |
Signing in to the runner's page
| What people see | Likely cause | Fix |
|---|---|---|
| "Cannot sign in yet" with "This runner has not reached SourceLace yet (...). Check its token and network." | The runner has not completed its first contact with SourceLace. | Fix the cause shown in brackets (The runner stays Quiet); then sign in again. |
| "The runner ... has not connected to SourceLace yet. Try again in a minute." (on SourceLace) | SourceLace has not heard the runner's page address yet. | Wait a minute after the first start. If it lasts, see The runner stays Quiet. |
| "This runner is not one of your organization's runners, or you are not a member of its organization. Sign in with your work account." or "You are not a member of this runner's organization." | The person is signed in to SourceLace with another account, or is not in your organization. | Sign in to SourceLace with your work account, then again on the runner. |
| "You are no longer a member of this runner's organization." | The person was removed in SourceLace. | An admin adds them again in SourceLace, if that is right. |
| "The runner ... was revoked. Ask your admin." | Nobody can sign in to a revoked runner. | Use the runner that replaced it. |
| "This sign-in has expired or was already used. Sign in again.", "This sign-in is not valid. Sign in again.", "This sign-in could not be verified. Sign in again." or "SourceLace sent no sign-in code. Sign in again." | The sign-in took more than a few minutes, the page was reloaded or opened twice, or a proxy changed the address. | Sign in again. If it keeps happening, check that your TLS proxy passes the whole address, including everything after ?, unchanged. |
| "This sign-in was for another runner." | Two runners' pages were open, or a bookmark pointed at another runner. | Open the right runner's address and sign in again. |
| "This sign-in link is not valid. Start again from the runner's page." | A sign-in address was opened directly or cut short. | Start from the runner's page. |
| "Sign-in was cancelled." | The person cancelled at SourceLace. | Sign in again. |
After signing in, the browser lands on an http:// address or another host |
SOURCELACE_PUBLIC_URL is not the address people use, or the proxy does not send the original host. |
Set SOURCELACE_PUBLIC_URL to the address people open, and the proxy's Host and X-Forwarded-Proto headers (nginx). |
| "Signed out" / "You are signed out. Sign in again." after a while | A sign-in lasts 12 hours; a change of master key also signs everyone out. | Sign in again. |
| "This form expired. Go back and try again." | The page was open in another session, or the sign-in ended. | Reload the page and try again. |
| "Only your organization's SourceLace admins can do this." | A member who is not an admin opened Settings or an admin action. | An admin does it. |
License
| What you see | Likely cause | Fix |
|---|---|---|
| "This runner has no license, so it takes no new work. It gets one from SourceLace on its next contact ..." | A new runner before its first contact, or one that has not reached SourceLace. | Fix the connection to SourceLace (The runner stays Quiet); the license arrives with the next contact. |
| "This runner's license expired on ... UTC, so it takes no new work. Runs in progress finish and nothing is deleted. ..." | The runner could not reach SourceLace for a long time, or your contract no longer includes runners of its label. | If Check connectivity shows SourceLace OK, ask your SourceLace contact about your contract (When runners get no new work). |
| "This runner's license expires on ... UTC and has not been renewed. Check that the runner can reach SourceLace, or ask your SourceLace contact about the contract." | Renewals have stopped: no contact, or the contract changed. | Same as above. |
| "This machine's clock is behind SourceLace's time; the runner uses the later one." | The machine's clock is behind. | Turn on time synchronization (timedatectl set-ntp true on most Linux systems). |
| The license looks expired although the runner is in touch and the contract is fine | The machine's clock is far ahead. | Synchronize the clock; check with timedatectl or date -u against a reliable time. |
| "The license is for another organization." or "The license is for another runner." | The database belongs to another runner or organization. | Use the database that belongs to this runner, or a new empty one. |
Log: Runner: ... runs handed out without a valid license; not run. |
SourceLace handed out runs to a runner without a license. | The runs are handed out again later. Fix the license as above. |
| The Runners page: "License expired ...: no new work until it is renewed." | As above. | As above. |
| "Your runner licenses end on ..." (Runners page, notice at the top of the app) | Your organization's runner license end date is within 30 days: your contract ends then and does not renew by itself, or a date agreed with SourceLace. | Ask your SourceLace contact to extend the contract (License end dates). |
| "Your runner licenses ended on ..., so your runners take no new work." / the runner's Settings: "No new work: the organization's runner licenses ended on ..." | The date has passed. | Once the contract is extended, the runners take work again at their next contact; nothing to install. |
A run fails or waits
| What you see | Likely cause | How to confirm | Fix |
|---|---|---|---|
| The run waits with "Waiting for the runner ...: it has not asked SourceLace for work since ..." | The runner is quiet. | The Runners page. | The runner stays Quiet. |
| Runs wait although the runner is Healthy | The runner is busy with as many runs as it may run at once, or your organization's limit on runs at once is reached, or your contract's terms lapsed for its label. | The Runners page health ("... in its queue"); the notice at the top of the app. | Wait, raise SOURCELACE_AGENT_PARALLEL_RUNS (with memory), or add a runner. For contract notices, see When runners get no new work. |
| A run ends with "This run continues on another instance of this runner." | The runner runs as several instances, and this one lost touch with SourceLace for more than three minutes, so another instance took the run over (Failover). Nothing was written twice. | The run's page shows the instance that finished it; the Runners page shows which instance was quiet. | Nothing, if the run finished on the other instance. Check the network of the instance that lost touch (Network problems). |
An approval stays waiting after Approve, and the runner's log says AccessNotHeld |
The instance you approved on could not reach SourceLace to get the organization's current rules for that run, and has not run one of that person's agents since it started. | The runner's Settings page: last contact with SourceLace. | Fix the instance's connection to SourceLace, then approve again; or wait for that person's next run. |
| "The AI model is not set up yet. Ask your admin to set it up in Settings." | No AI key on the runner. | The runner's Settings page shows the AI key "not set up". | Enter it on the runner's Settings page (Step 8). |
| "Choose Anthropic or OpenAI: a runner uses your organization's own AI key." (when saving settings) | Another provider was chosen. | Choose Anthropic or OpenAI. | |
| A run fails with an error from the AI provider about the key (such as invalid or unauthorized) | The key is wrong, revoked, or for another provider than the one chosen. | The provider's console. | Enter a valid key for the chosen provider. |
| "This runner could not read the data protection rules, read rules or access by group SourceLace sent with this run (the runner is older than SourceLace), so it did not run it and nothing was read. Update the runner to its latest release." | The runner is older than SourceLace. | Out of date on the Runners page. | Upgrade, then run the agent again. |
| "SourceLace sent this run without your organization's data protection rules, read rules and access by group, so the runner did not run it and nothing was read. ..." | A problem on SourceLace's side. | Run the agent again; if it repeats, send SourceLace support the run's link (Before you contact support). | |
A step fails with an error type such as AccessDenied or a sign-in error, although the source works in the SourceLace app |
The owner has not connected the source on the runner (sign-ins in SourceLace do not count there), or the sign-in expired. | The runner's Your sources page for the owner. | The owner connects the source on the runner's Your sources page. |
| A step or the runner's page says the person may not use a source or object (the same message as in the SourceLace app) | Access by group: none of the groups of the person the run is for allows that source or object, or allows changes to it. The runner applies your admins' access by group as SourceLace sent it with the run. | The person's groups in SourceLace (Groups). | An admin adds the person to a group that allows it, in SourceLace; the agent's next run brings the new access. |
| "This runner does not have your access settings from SourceLace yet: they come with each of your agent runs here, and a restart clears them. Nothing was read. ..." | Someone used the runner's page (for example approving a change) before any of their agent runs had reached the runner since it last started. | Whether the runner restarted recently (its log). | Wait for that person's next agent run on the runner, then try again. A change waiting for approval keeps waiting meanwhile; nothing is lost. |
| A source fails, and the runner's Settings shows its secret option "not set" | A secret option, such as a client secret, is missing on the runner. | Settings, Secret source options. | An admin enters it. SourceLace never has it. |
| Connecting a source on the runner fails at the vendor with a redirect or callback error | The OAuth app at the vendor does not list the runner's callback address. | The vendor's error names the address. | Add https://<runner page>/connect/callback to the app at the vendor, and enter its client ID and secret on the runner's Settings page. |
| "It uses the source ..., which may not be used now: ... is switched off for your organization. Ask your SourceLace contact." (the run ends as stopped) | SourceLace switched that system off for your organization, or it needs a feature your organization does not have. The run did not start; nothing was read. | The same message when connecting the source in the SourceLace app. | Ask your SourceLace contact. Once it is switched on, run the agent again; an admin re-enters the source's secret settings on the runner's Settings page if they show "not set". |
| "Interrupted after it had made drafts or changes, so it was not run again." | The runner restarted or lost its connection after the run had written something. | Check what the run made on the runner's page, then run it again yourself if needed (If the runner restarts or loses its connection). | |
| "The run did not finish on the runner." | The run ended on the runner without finishing (for example it was stopped by a restart). | The runner's log around that time. | Run it again. |
| "SourceLace asked the runner to stop this run (the agent was paused or stopped)." | Someone paused or stopped the agent, or used the stop switch. | The agent's page in SourceLace. | Start the agent again when wanted. |
| "This runner no longer holds the run on SourceLace." or "SourceLace handed this run out again after it lost touch with the runner; this copy stopped so that the run never runs twice." | The runner lost its connection long enough for SourceLace to hand the run out again. | Log lines Runner: could not renew the lease of run .... |
Nothing: the other copy is the real one. Fix the network if it repeats. |
| "The runner it was to run on was revoked." | The runner was revoked while the run was queued or running. | Choose another runner under Where it runs. | |
On the runner's page: "SourceLace refused this runner's token (the runner was revoked, or its token is wrong), so the runner stopped this run and wrote nothing more. ..." Log: ERROR:sourcelace:Runner: SourceLace refused this runner's token (revoked or wrong); ... runs in progress stopped, no new work is taken. |
The runner was revoked (or its token is wrong) while the run was going. The runner stopped it at its next contact with SourceLace. A network problem never does this. | The Runners page shows the runner Revoked. | Check on the runner's page what the run made before it stopped. Choose another runner for the agent, or, if the token was wrong, see The runner stays Quiet. |
| Approving a change on the runner: "SourceLace refused this runner's token (the runner was revoked, or its token is wrong), so it makes no more changes. You can still reject this change here." | The runner was revoked. | Reject it here. If the change is still needed, run the agent again on another runner. | |
| An approval on the runner: "An approval rule says someone in a group must approve this change, which cannot be checked here. ..." or "A decision routed to groups cannot be decided here. Decide it in SourceLace." | Approvals that need a group cannot be decided on the runner. | Reject it on the runner, or ask an admin to change the approval rule. | |
Log: Runner: run ... could not be run. followed by details |
An unexpected error in that run. | The lines after it (tokens and passwords are removed from the log). | Run it again; if it repeats, send support the run's link and those log lines. |
Log: Runner: SourceLace did not take the report of run ... (...) or Runner: could not tell SourceLace that run ... writes (...); kept to send. |
A report could not be delivered yet. | The Settings page shows "Reports waiting to be sent". | Nothing, if the connection works again: kept reports are sent at the next contact, also after a restart. |
Log: Runner: source ... is of a kind this version does not know. |
Your organization uses a kind of source newer than this runner. | Upgrade the runner. | |
Log: Runner: source ... could not be used. |
A source's settings are not valid on the runner. | Check the source's settings in SourceLace; if they are right, send support the source's name. |
Disk, memory and database limits
| Symptom | Likely cause | How to confirm | Fix |
|---|---|---|---|
| Free disk on the Runners page is low, or runs with large results fail | The folder for large results is full. The Runners page shows the free space of that folder: SOURCELACE_ENGINE_DIR (/engine with Helm), or the temporary folder when it is not set. |
df -h /srv/sourcelace-runner/engine on the machine (or the folder you gave it); in Kubernetes, pod events about ephemeral storage. |
Give it more disk, keep results fewer days, or lower runs at a time. Results there expire by themselves. |
| The pod is evicted with a message about ephemeral storage | The node has less local storage than ephemeralDisk and runnerTmpSize ask. |
kubectl describe pod ... shows the eviction. |
Use nodes with enough local disk, or lower the sizes. |
| Free memory on the Runners page is near 0; the container restarts with exit code 137 | Out of memory. | `docker inspect <container> | grep OOMKilledprints"OOMKilled": true,`. |
| The log at start warns that an engine setting is set higher than fits | SOURCELACE_ENGINE_MEMORY_MB (or another engine setting) is more than the machine has. |
The warning names the setting. | Remove or lower it; the runner uses what fits. |
/health answers 503 with "database":"unreachable" |
The runner cannot reach its database right now. | psql "<DATABASE_URL>" -c "select 1" from the machine. |
Start the database, fix the network, or check the password. The runner recovers by itself once it reaches it. |
Log errors about too many connections or remaining connection slots are reserved |
The database's connection limit is reached. | select count(*) from pg_stat_activity where usename = 'runner'; |
Allow at least 20 connections for the runner's user, or raise the server's max_connections. |
| The database grows faster than expected | Many large runs kept for a long retention period. | Your database monitoring. | Shorten Keep run results for (days) on the runner's Settings page; give the database more storage. |
Statuses on the Runners page
| Status | Means | What to do |
|---|---|---|
| Healthy | It asked for work in the last 90 seconds, and its version is SourceLace's or newer. | Nothing. |
| Quiet | It has not asked for work in the last 90 seconds, or never has. | The runner stays Quiet. |
| Out of date | It works, but its version is behind SourceLace's (shown next to it). Runs that use newer rules may be refused. | Upgrade. |
| Revoked | Its token no longer works. | Use or add another runner. |
| "No new work." with a reason | Your contract's terms are not met for its label. | When runners get no new work. |
| "1 of 2 instances asking for work" (and an instance "quiet since ...") | The runner runs as several instances and one has stopped. The others take its runs. | Start the stopped one again, or ignore it if you removed it on purpose: an instance not seen for a day leaves the list. |
| The runner's Settings page: "No new work: this runner already has 10 instances taking work; this one (...) takes none until one of them stops" | More than 10 instances of one runner were started. The 10 that started first take work. | Run at most 10 instances of a runner, or add a second runner (Failover). |
| Update available / Upgrade due / Unsupported under the version | Its version's support (How long each version is supported). Unsupported: "Runner version ... is no longer supported (support ended on ...), so the runner takes no new work." | Upgrade: it takes work again at once. |
| "Licensed until ..." / "License expired ...: no new work until it is renewed." | Its license. | License. |
| Health "Last error: ..." | The type of the last error, never its message. ControlUnavailable: could not reach SourceLace. ControlRefused: token refused. LeaseLost: a run was handed out again. TakenOver: another instance of the runner took a run over. |
The rows above for that problem. |
Before you contact support
Most problems are solved with the tables above. If you still need SourceLace support, one message with the following is usually enough:
- Check first:
- Is the runner's container running, and does
/healthanswer"status":"ok"and"database":"ok"? - What does Check connectivity show? Fix any line with advice you can follow.
- Is the machine's clock right (
date -u)? - Is there free disk and memory?
- Is the runner on the version your SourceLace contact last gave you?
- Is the runner's container running, and does
- Collect:
- The support bundle: the runner's Settings page, Download support bundle. It holds the version, the connection to SourceLace, counts of runs, the settings with every secret left out, the types of recent errors and the connectivity check, and no tokens, keys, passwords, data or results. You can open the zip and read it before sending.
- The runner's name and id (on the Runners page and the runner's Settings page, such as
rnr_3f9a2c1b7d4e) and its version. - The status, last seen and health shown on the Runners page.
- When it started, with your time zone, and what changed just before (an upgrade, a new firewall or proxy rule, a new certificate, a moved machine).
- For a run: the link to its run page in SourceLace.
- The log lines around that time:
docker compose logs --since 2h runner > runner-log.txt, orkubectl logs ... --since=2h > runner-log.txt. The runner removes tokens, keys and passwords from its log, but read it before sending. - How you install it: Docker Compose or Kubernetes, and whether you use a proxy or TLS inspection.
- Never send: the runner token, the master key, your AI key, a registry login, database passwords,
runner.env, Kubernetes secrets, or the output ofdocker compose config(it prints secrets). Support never needs them. If one was sent by mistake, change it (Looking after a runner).
Send it to support@sourcelace.com or through the support channel your Premium support gives you.