Troubleshooting
Overview
No system is perfect, and Continuum will have it's occasional glitches just like any other software system. In the following sections we aim to cover the most typical scenarios encountered and the proper resolutions.
If you're having a problem, please feel free to contact us. Of course, the bravest of you will first read through this document and try to fix it yourself first!
Before We Begin
Continuum runs on Linux, and most of the following troubleshooting tips will require the ability to 'ssh' into the Continuum server(s). Before proceeding any further, make sure you know:
- the hostname or IP address of the Continuum server(s)
- username and password (or 'identity' file for public key authentication)
- sudo capabilities in some cases
General Procedures
Not to point fingers, but 90% of the time, Continuum 'issues' are the result of negligence maintaining the OS itself. (Disk space filled up, out of memory, network errors, etc.)
So, the first thing to do is a quick check of the health of the server.
Comprehensive OS troubleshooting is beyond the scope of this document - see the official documentation for your flavor of Linux for details.
Disk Space
Every server is configured differently, but quickly check and make sure the volumes aren't full. Here, our main volume is at 82%.
df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/ccl01--vg-root 59487996 46447968 10541936 82% /
udev 2012772 4 2012768 1% /dev
tmpfs 404804 516 404288 1% /run
/dev/sda1 240972 239802 0 100% /boot
Processes and Memory Utilization
All Continuum supported flavors of Linux implement the top command. This will show at a glance if any processes are runaways or memory hogs. This example shows MySQL hogging a whopping 0.7% CPU and 5.1% Memory.
top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
999 mysql 20 0 2404092 207112 5328 S 0.7 5.1 264:12.27 mysqld
Continuum Logs
Checking the Continuum logs is another must-do on your troubleshooting checklist. Continuum logs can be accessed in several ways: either via the Linux shell or in the Continuum UI.
If the Continuum UI is unavailable, you'll have to use the Linux prompt.
Via the Prompt
In a default installation, logs are stored in /var/continuum/log.
There are many log files, one for each Continuum service. Additionally, log files 'rotate' - occasionally backing themselves up using a timestamp. The current logs are those with no timestamp in the filename. To list all the log files:
ls -l *.log
-rw-rw-r-- 1 ubuntu ubuntu 468 Apr 17 15:30 ctm-autoscheduler.log
-rw-rw-r-- 1 ubuntu ubuntu 480 Apr 17 15:30 ctm-deployscheduler.log
-rw-rw-r-- 1 ubuntu ubuntu 450 Apr 17 15:30 ctm-depmarshall.log
-rw-rw-r-- 1 ubuntu ubuntu 38476863 Apr 17 17:42 ctm-jobhandler.log
-rw-rw-r-- 1 ubuntu ubuntu 4452 Apr 17 16:26 ctm-messenger.log
-rw-rw-r-- 1 ubuntu ubuntu 499861 Apr 17 17:42 ctm-metrics.log
-rw-rw-r-- 1 ubuntu ubuntu 2191 Apr 17 16:37 ctm-msghub.log
-rw-rw-r-- 1 ubuntu ubuntu 420 Apr 17 15:30 ctm-poller.log
-rw-rw-r-- 1 ubuntu ubuntu 868659 Apr 17 17:41 ctm-ui.log
Depending on the type of issue being researched, look in the appropriate (and hopefully self-explanatory) log file. For example, researching UI problems - ctm-ui.log. Make note of anything unusual in the log files, and make copies to send to us when you contact Continuum Support.
Via the UI
Logs can be viewed in the Continuum UI by navigating to a special url:
http://server:8080/getlog
On this page, you can select any log file and view it. The number of lines returned is limited, so change that value to get more. Be aware that very large logfiles may slow down your browser!
The /getlog view is a helper feature, but is limited - it can only view logs on the same server the Continuum UI is running on. In high-availability or distributed installs, use the Linux shell.
Restarting the Services
Sometimes, the best thing to do is start fresh. You can easily restart all the Continuum services.
Restarting the Continuum services is invasive! Any Users will get booted, and all running automation will get aborted.
ctm-restart-services
Mongo Issues
MongoDB is a great service, but like any database it occasionally hiccups. When MongoDB is having trouble, this will manifest in Continuum (either in the UI or in the log files) as some variation of a message like Couldn't create a Mongo connection to database.
The following command works on a typical Continuum install on Ubuntu. See the OS/MongoDB documentation for different Linux flavors.
sudo service mongodb restart