Skip to main content
System StatusContact Support
VersionOne Community

Troubleshooting

Overview

No system is perfect, and Continuum will have it's occasional glitches just like any other software system. In the following sections we aim to cover the most typical scenarios encountered and the proper resolutions.

 

If you're having a problem, please feel free to contact us. Of course, the bravest of you will first read through this document and try to fix it yourself first!

Before We Begin

Continuum runs on Linux, and most of the following troubleshooting tips will require the ability to 'ssh' into the Continuum server(s). Before proceeding any further, make sure you know:

  • the hostname or IP address of the Continuum server(s)
  • username and password (or 'identity' file for public key authentication)
  • sudo capabilities in some cases

General Procedures

Not to point fingers, but 90% of the time, Continuum 'issues' are the result of negligence maintaining the OS itself. (Disk space filled up, out of memory, network errors, etc.)

So, the first thing to do is a quick check of the health of the server.

 

Comprehensive OS troubleshooting is beyond the scope of this document - see the official documentation for your flavor of Linux for details.

Disk Space

Every server is configured differently, but quickly check and make sure the volumes aren't full. Here, our main volume is at 82%.

df -k

Filesystem                 1K-blocks     Used Available Use% Mounted on
/dev/mapper/ccl01--vg-root  59487996 46447968  10541936  82% /
udev                         2012772        4   2012768   1% /dev
tmpfs                         404804      516    404288   1% /run
/dev/sda1                     240972   239802         0 100% /boot

Processes and Memory Utilization

All Continuum supported flavors of Linux implement the top command. This will show at a glance if any processes are runaways or memory hogs. This example shows MySQL hogging a whopping 0.7% CPU and 5.1% Memory.

top

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
999 mysql     20   0 2404092 207112   5328 S   0.7  5.1 264:12.27 mysqld

Continuum Logs

Checking the Continuum logs is another must-do on your troubleshooting checklist. Continuum logs can be accessed in several ways: either via the Linux shell or in the Continuum UI.

 

If the Continuum UI is unavailable, you'll have to use the Linux prompt.

Via the Prompt

In a default installation, logs are stored in /var/continuum/log.

There are many log files, one for each Continuum service. Additionally, log files 'rotate' - occasionally backing themselves up using a timestamp. The current logs are those with no timestamp in the filename. To list all the log files:

ls -l *.log

-rw-rw-r-- 1 ubuntu ubuntu       468 Apr 17 15:30 ctm-autoscheduler.log
-rw-rw-r-- 1 ubuntu ubuntu       480 Apr 17 15:30 ctm-deployscheduler.log
-rw-rw-r-- 1 ubuntu ubuntu       450 Apr 17 15:30 ctm-depmarshall.log
-rw-rw-r-- 1 ubuntu ubuntu  38476863 Apr 17 17:42 ctm-jobhandler.log
-rw-rw-r-- 1 ubuntu ubuntu      4452 Apr 17 16:26 ctm-messenger.log
-rw-rw-r-- 1 ubuntu ubuntu    499861 Apr 17 17:42 ctm-metrics.log
-rw-rw-r-- 1 ubuntu ubuntu      2191 Apr 17 16:37 ctm-msghub.log
-rw-rw-r-- 1 ubuntu ubuntu       420 Apr 17 15:30 ctm-poller.log
-rw-rw-r-- 1 ubuntu ubuntu    868659 Apr 17 17:41 ctm-ui.log

Depending on the type of issue being researched, look in the appropriate (and hopefully self-explanatory) log file. For example, researching UI problems - ctm-ui.log. Make note of anything unusual in the log files, and make copies to send to us when you contact Continuum Support.

Via the UI

Logs can be viewed in the Continuum UI by navigating to a special url:

http://server:8080/getlog

On this page, you can select any log file and view it. The number of lines returned is limited, so change that value to get more. Be aware that very large logfiles may slow down your browser!

 

The /getlog view is a helper feature, but is limited - it can only view logs on the same server the Continuum UI is running on. In high-availability or distributed installs, use the Linux shell.

Restarting the Services

Sometimes, the best thing to do is start fresh. You can easily restart all the Continuum services.

 

 Restarting the Continuum services is invasive! Any Users will get booted, and all running automation will get aborted.

ctm-restart-services

Restarting MySql

Sometimes the database server may act up for various reasons - network instability or other causes. It's easy to quickly restart MySql.

 

The following command works on a typical Continuum install on Ubuntu. See the OS/MySql documentation for different Linux flavors.

sudo service mysql restart

Mongo Issues

MongoDB is a great service, but like any database it occasionally hiccups. When MongoDB is having trouble, this will manifest in Continuum (either in the UI or in the log files) as some variation of a message like Couldn't create a Mongo connection to database.

 

The following command works on a typical Continuum install on Ubuntu. See the OS/MongoDB documentation for different Linux flavors.

sudo service mongodb restart
  • Was this article helpful?