Summary:	ASTERISK-26025: core: Asterisk doesn't start correctly in kickstart post-install script
Reporter:	Michael Newton (miken32)	Labels:
Date Opened:	2016-05-16 14:04:03	Date Closed:
Priority:	Major	Regression?
Status:	Open/New	Components:	Core/General
Versions:	13.9.1	Frequency of Occurrence	Constant
Related Issues:
Environment:	Scientific Linux (RHEL) 6.7	Attachments:	( 0) full
Description:	Our provisioning process includes a Kickstart post script that compiles, installs, and starts Asterisk before installing FreePBX. This works very well under Asterisk 11 but fails under Asterisk 13. The FreePBX installer attempts to run {{asterisk -rx 'core show version'}} before installation, and complains that it cannot. Investigating this failure, I found that Asterisk starts up, but does not create {{/var/run/asterisk/asterisk.ctl}} so no remote connections can be created. I've verified that {{/var/run/asterisk/asterisk.pid}} is created, and the output from {{ps}} shows that Asterisk continues running. Setting the Asterisk logging level to debug shows no errors occurring at startup. Once the server is rebooted, and no longer in Kickstart, the control file is created without issue.
Comments:	By: Asterisk Team (asteriskteam) 2016-05-16 14:04:03.777-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Jared Smith (jsmith) 2016-05-16 14:12:27.120-0500 Just out of curiosity, how are you installing Asterisk?  Are you compiling from source, or using a pre-compiled package (such as an RPM, etc.)?  if you're using an RPM, which RPM are you using? By: Michael Newton (miken32) 2016-05-16 16:18:02.706-0500 Asterisk is being compiled from source from within the post script. By: Jared Smith (jsmith) 2016-05-16 16:24:59.294-0500 How is Asterisk being started after compilation/installation?  is it being started manually?  Or with an initscript?  Or via systemd? By: Michael Newton (miken32) 2016-05-16 16:44:13.814-0500 Using the supplied initscript; {{service asterisk start}} is run after {{make install}} and creation of configuration files. By: Michael Newton (miken32) 2016-05-17 14:55:40.206-0500 Also possibly of note, Asterisk is configured to run as user and group {{asterisk}} via the {{safe_asterisk}} script. Permissions aren't a problem though, as PID file is created fine, and CTL file is created after a reboot. SELinux is also disabled. By: Rusty Newton (rnewton) 2016-05-18 08:42:12.265-0500 bq.  Setting the Asterisk logging level to debug shows no errors occurring at startup. Where are you setting the level for debug and verbose channels, and which logging channels did you direct to your log file? By: Michael Newton (miken32) 2016-05-18 11:07:43.955-0500 I looked into anything I could set at compile time, but could only find debugging options related to crashes. So I altered the initscript to start Asterisk with -vvvdd, used the following {{logger.conf}}, and didn't observe any errors at all, except expected ones due to missing config files, etc. I can provide those logs, or if you have a method that will generate more detail, please advise. {code:title=logger.conf} [general] queuelog = no ; uncomment this if you actually want queue logging! [logfiles] messages = notice,warning,error full     = notice,warning,error,debug,verbose,dtmf,fax console  = notice,warning,error,debug,verbose,dtmf,fax {code} By: Rusty Newton (rnewton) 2016-05-18 16:22:46.117-0500 You could bump it up to -vvvvvvvvvvdddddddddd , so, level 10 for verbose and debug. However, I doubt we'll see anything useful. Worth a shot. If you get *anything* in the logs, probably want to attach it regardless. By: Michael Newton (miken32) 2016-05-19 00:17:51.292-0500 Full log file when Asterisk started with {{-vvvvvvvvvvdddddddddd}} attached. No errors appear related to this issue. By: George Joseph (gjoseph) 2016-05-24 10:01:03.677-0500 I've seen the ctl file problem before and it has to do with running in the chrooted environment from the post script.  I'm trying to remember the exact issue. You are creating the asterisk user and group before doing make install correct? By: Michael Newton (miken32) 2016-05-24 11:02:31.566-0500 I'm creating the user after {{make install}} but before starting Asterisk. I can try moving the user creation up and see if it helps. The PID file is created with correct ownership and permissions in the same directory. By: Michael Newton (miken32) 2016-05-24 13:09:20.508-0500 No change when creating the user before building Asterisk. By: George Joseph (gjoseph) 2016-05-24 15:04:20.869-0500 I should be able to test this tonight but IIRC it had something to do with /dev or /sys not being fully populated.  The ctl file is a socket unlike the pid file which is a plain file, so there's more to it than just having permissions to create an entry in the directory. By: Michael Newton (miken32) 2016-05-31 15:51:19.550-0500 One small update; this didn't work in Asterisk 11, as I previously stated. It has actually stopped working for us because of an update to the FreePBX installer, which now runs {{asterisk -rx 'core show version'}} instead of {{asterisk -V}} to verify Asterisk is installed. This doesn't change the underlying problem of not being able to connect to a running instance of Asterisk, however. By: George Joseph (gjoseph) 2016-06-12 14:31:38.780-0500 I finally got a chance to test this in a kickstart environment and I didn't have any problem starting asterisk.   My kickstart installs asterisk from RPM but I was able to ssh into the box before the reboot, chroot, and start asterisk as the asterisk user.  I then ssh'd again and ran asterisk -R. What was the exact command used to create the asterisk user and group. What's the exact command used to start asterisk. What are the permissions on /var/run/asterisk or /run/asterisk, whichever you use? Can you "su - asterisk" and do a "mksock /var/run/asterisk/test.ctl"? A few things you can check... permissions on var/run/asterisk   By: Michael Newton (miken32) 2017-03-21 00:27:04.581-0500 Sorry for dropping this, I meant to debug and never got to it. We're still using 11 in production so this has not been high priority. User was created with {{adduser -d /var/lib/asterisk -M -r -s /sbin/nologin asterisk}} Asterisk was started with {{service asterisk start}} (using the RHEL init script provided with source) which looks like it should run {{daemon /usr/sbin/safe_asterisk -U asterisk -G asterisk}} {{/var/run/asterisk}} is 755, owned by asterisk:asterisk {{mksock}} is not available on this system, can't find a package that provides it either. By: Corey Farrell (coreyfarrell) 2018-02-23 22:45:00.209-0600 Is it possible you are experiencing an SELinux issue?  What happens if you modify your kickstart to disable selinux?  For testing only, I'm definitely not suggesting that you should disable selinux for production.