| Summary: | ASTERISK-23265: Preloading Certain Modules in Asterisk 12 causes a core dump | ||||||
| Reporter: | Andrew Nagy (tm1000) | Labels: | |||||
| Date Opened: | 2014-02-07 12:50:44.000-0600 | Date Closed: | 2014-02-21 14:43:16.000-0600 | ||||
| Priority: | Major | Regression? | No | ||||
| Status: | Closed/Complete | Components: | Core/General | ||||
| Versions: | 12.1.0 | Frequency of Occurrence | |||||
| Related Issues: |
| ||||||
| Environment: | CentOS 6.5 | Attachments: | ( 0) backtrace.txt ( 1) pre-load-reorder.diff | ||||
| Description: | Preloading pbx_config.so and chan_local.so causes asterisk to Core Dump. The original module.conf is below:
{code} [modules] autoload=yes preload = pbx_config.so preload = chan_local.so noload = pbx_gtkconsole.so noload = pbx_kdeconsole.so noload = app_intercom.so noload = chan_modem.so noload = chan_modem_bestdata.so noload = chan_modem_i4l.so noload = app_trunkisavail.so noload = chan_alsa.so noload = chan_oss.so noload = app_directory_odbcstorage.so noload = app_voicemail_odbcstorage.so noload = chan_modem_aopen.so load = format_wav.so load = format_pcm.so load = format_mp3.so load = res_musiconhold.so {code} Removing the two preloads allows asterisk to function normally {code} [modules] autoload=yes noload = pbx_gtkconsole.so noload = pbx_kdeconsole.so noload = app_intercom.so noload = chan_modem.so noload = chan_modem_bestdata.so noload = chan_modem_i4l.so noload = app_trunkisavail.so noload = chan_alsa.so noload = chan_oss.so noload = app_directory_odbcstorage.so noload = app_voicemail_odbcstorage.so noload = chan_modem_aopen.so load = format_wav.so load = format_pcm.so load = format_mp3.so load = res_musiconhold.so {code} | ||||||
| Comments: | By: Andrew Nagy (tm1000) 2014-02-07 12:52:41.473-0600 Backtrace By: Andrew Nagy (tm1000) 2014-02-07 13:02:51.981-0600 IRC logs for clarification {code} [10:36am] tm1000: file: mjordan http://pastebin.com/sZXDbFK6 [10:36am] mjordan: hm. [10:36am] file: hrm [10:37am] file: yup, we initialize presence stuff afterwards [10:37am] mjordan: gr. Preload. [10:38am] tm1000: mjordan: do I need to reload pjsip correctly [10:38am] mjordan: we've got to do something about the module loader. [10:38am] mjordan: tm1000: turns out, it isn't PJSIP's fault. [10:38am] mjordan: this is just pure Asterisk guts being wrong. And stupid. [10:39am] tm1000: mjordan: anything I can do? [10:39am] mjordan: tm1000: do you have a preload directive in your modules.conf? [10:39am] tm1000: mjordan: yes [10:39am] mjordan: tm1000: which module(s)? [10:40am] tm1000: http://pastebin.com/HhJd5Lqw [10:40am] file: pbx_config.so. [10:40am] mjordan: what happens if you remove pbx_config and chan_local? [10:40am] mjordan: chan_local no longer exists... [10:40am] mjordan: so there's that. [10:41am] tm1000: mjordan: tonyclewis is the one who said pbx_config needs to be there [10:41am] mjordan: he may not be wrong, but right now, let's just see if it removes the crash [10:41am] tonyclewis: We had to preload before for a issue with queues [10:42am] mjordan: tm1000: the short story is that Asterisk's module loader is antiquated and needs an overhaul. [10:42am] mjordan: tonyclewis: Device state not showing up for Queues on load? [10:42am] tonyclewis: Yes thats it [10:43am] mjordan: tonyclewis: how long ago was that? [10:44am] tonyclewis: Couple years [10:44am] tm1000: mjordan: ok that works [10:44am] tm1000: it starts fine now [10:44am] tm1000: and lets me register [10:44am] mjordan: k. We might have fixed that, since pbx_config has a module load priority before app_queue now [10:44am] mjordan: tm1000: that's a work around, the long answer is we have to go fix the module loader {code} By: Rusty Newton (rnewton) 2014-02-07 14:34:54.437-0600 Can't reproduce with your modules.conf on SVN-branch-12-r407423. pbx_config loads fine, and chan_local doesn't get loaded as it doesn't exist, debug confirms. No crash. Upgrading to latest SVN to try again. By: Rusty Newton (rnewton) 2014-02-07 14:39:26.648-0600 Can't reproduce on SVN-branch-12-r407750. Same deal. {noformat} == Registered channel type 'Surrogate' (Surrogate channel used to pull channel from an application) == Parsing '/etc/asterisk/codecs.conf': Found Asterisk Dynamic Loader Starting: == Parsing '/etc/asterisk/modules.conf': Found [Feb 7 14:36:40] NOTICE[25072]: loader.c:1323 load_modules: 2 modules will be loaded. > Registering module pbx_config > Unregistering module pbx_config [Feb 7 14:36:40] WARNING[25072]: loader.c:523 load_dynamic_module: Error loading module 'chan_local.so': /usr/lib/asterisk/modules/chan_local.so: cannot open shared object file: No such file or directory > Registering module pbx_config > Unregistering module pbx_config > Registering module pbx_config [Feb 7 14:36:40] WARNING[25072]: loader.c:523 load_dynamic_module: Error loading module 'chan_local.so': /usr/lib/asterisk/modules/chan_local.so: cannot open shared object file: No such file or directory [Feb 7 14:36:40] WARNING[25072]: loader.c:1032 load_resource: Module 'chan_local.so' could not be loaded. Loading pbx_config.so. == Parsing '/etc/asterisk/extensions.conf': Found {noformat} So, getting it to crash may be more complicated. Perhaps attach output showing what modules are in your /usr/lib/asterisk/modules directory? Also maybe, asterisk.conf. By: Andrew Nagy (tm1000) 2014-02-07 15:06:57.909-0600 Rusty try it on a FreePBX machine with all the configurations. Or I can send you my complete configurations By: Andrew Nagy (tm1000) 2014-02-07 15:21:19.609-0600 {code} [root@freepbxdev1 a12]# ls /usr/lib/asterisk/modules app_adsiprog.so app_transfer.so format_g729.so func_vmcount.so res_pjsip_acl.so app_agent_pool.so app_url.so format_gsm.so func_volume.so res_pjsip_authenticator_digest.so app_alarmreceiver.so app_userevent.so format_h263.so pbx_ael.so res_pjsip_caller_id.so app_amd.so app_verbose.so format_h264.so pbx_config.so res_pjsip_diversion.so app_authenticate.so app_waitforring.so format_ilbc.so pbx_dundi.so res_pjsip_dtmf_info.so app_bridgewait.so app_waitforsilence.so format_jpeg.so pbx_loopback.so res_pjsip_endpoint_identifier_anonymous.so app_cdr.so app_waituntil.so format_ogg_vorbis.so pbx_lua.so res_pjsip_endpoint_identifier_ip.so app_celgenuserevent.so app_while.so format_pcm.so pbx_realtime.so res_pjsip_endpoint_identifier_user.so app_chanisavail.so app_zapateller.so format_siren14.so pbx_spool.so res_pjsip_exten_state.so app_channelredirect.so bridge_builtin_features.so format_siren7.so res_adsi.so res_pjsip_header_funcs.so app_chanspy.so bridge_builtin_interval_features.so format_sln.so res_ael_share.so res_pjsip_log_forwarder.so app_confbridge.so bridge_holding.so format_vox.so res_agi.so res_pjsip_logger.so app_controlplayback.so bridge_native_rtp.so format_wav_gsm.so res_ari_applications.so res_pjsip_messaging.so app_dahdiras.so bridge_simple.so format_wav.so res_ari_asterisk.so res_pjsip_mwi_body_generator.so app_db.so bridge_softmix.so func_aes.so res_ari_bridges.so res_pjsip_mwi.so app_dial.so cdr_adaptive_odbc.so func_audiohookinherit.so res_ari_channels.so res_pjsip_nat.so app_dictate.so cdr_csv.so func_base64.so res_ari_device_states.so res_pjsip_notify.so app_directed_pickup.so cdr_custom.so func_blacklist.so res_ari_endpoints.so res_pjsip_one_touch_record_info.so app_disa.so cdr_manager.so func_callcompletion.so res_ari_events.so res_pjsip_outbound_authenticator_digest.so app_dumpchan.so cdr_odbc.so func_callerid.so res_ari_mailboxes.so res_pjsip_outbound_registration.so app_echo.so cdr_pgsql.so func_cdr.so res_ari_model.so res_pjsip_path.so app_exec.so cdr_radius.so func_channel.so res_ari_playbacks.so res_pjsip_pidf_body_generator.so app_externalivr.so cdr_sqlite3_custom.so func_config.so res_ari_recordings.so res_pjsip_pidf_eyebeam_body_supplement.so app_festival.so cdr_syslog.so func_curl.so res_ari.so res_pjsip_pubsub.so app_flash.so cel_custom.so func_cut.so res_ari_sounds.so res_pjsip_refer.so app_followme.so cel_manager.so func_db.so res_calendar_caldav.so res_pjsip_registrar_expire.so app_forkcdr.so cel_odbc.so func_devstate.so res_calendar_ews.so res_pjsip_registrar.so app_getcpeid.so cel_pgsql.so func_dialgroup.so res_calendar_exchange.so res_pjsip_rfc3326.so app_ices.so cel_radius.so func_dialplan.so res_calendar_icalendar.so res_pjsip_sdp_rtp.so app_image.so cel_sqlite3_custom.so func_enum.so res_calendar.so res_pjsip_session.so app_jack.so chan_alsa.so func_env.so res_clialiases.so res_pjsip.so app_macro.so chan_bridge_media.so func_extstate.so res_clioriginate.so res_pjsip_t38.so app_milliwatt.so chan_console.so func_frame_trace.so res_config_curl.so res_pjsip_transport_websocket.so app_minivm.so chan_dahdi.so func_global.so res_config_ldap.so res_pjsip_xpidf_body_generator.so app_mixmonitor.so chan_iax2.so func_groupcount.so res_config_odbc.so res_realtime.so app_morsecode.so chan_mgcp.so func_hangupcause.so res_config_pgsql.so res_rtp_asterisk.so app_mp3.so chan_motif.so func_iconv.so res_config_sqlite3.so res_rtp_multicast.so app_nbscat.so chan_multicast_rtp.so func_jitterbuffer.so res_config_sqlite.so res_security_log.so app_originate.so chan_oss.so func_lock.so res_convert.so res_smdi.so app_page.so chan_phone.so func_logic.so res_crypto.so res_snmp.so app_playback.so chan_pjsip.so func_math.so res_curl.so res_sorcery_astdb.so app_playtones.so chan_sip.so func_md5.so res_fax.so res_sorcery_config.so app_privacy.so chan_skinny.so func_module.so res_fax_spandsp.so res_sorcery_memory.so app_queue.so chan_unistim.so func_odbc.so res_format_attr_celt.so res_sorcery_realtime.so app_readexten.so codec_adpcm.so func_pitchshift.so res_format_attr_h263.so res_speech.so app_read.so codec_alaw.so func_pjsip_endpoint.so res_format_attr_h264.so res_stasis_answer.so app_record.so codec_a_mu.so func_presencestate.so res_format_attr_opus.so res_stasis_device_state.so app_sayunixtime.so codec_dahdi.so func_rand.so res_format_attr_silk.so res_stasis_mailbox.so app_senddtmf.so codec_g722.so func_realtime.so res_http_websocket.so res_stasis_playback.so app_sendtext.so codec_g726.so func_sha1.so res_limit.so res_stasis_recording.so app_sms.so codec_gsm.so func_shell.so res_monitor.so res_stasis_snoop.so app_softhangup.so codec_ilbc.so func_sprintf.so res_musiconhold.so res_stasis.so app_speech_utils.so codec_lpc10.so func_srv.so res_mutestream.so res_statsd.so app_stack.so codec_resample.so func_strings.so res_mwi_external_ami.so res_stun_monitor.so app_stasis.so codec_ulaw.so func_sysinfo.so res_mwi_external.so res_timing_dahdi.so app_system.so format_g719.so func_timeout.so res_odbc.so res_timing_pthread.so app_talkdetect.so format_g723.so func_uri.so res_parking.so res_timing_timerfd.so app_test.so format_g726.so func_version.so res_phoneprov.so res_xmpp.so {code} {code} [root@freepbxdev1 a12]# cat /etc/asterisk/asterisk.conf [directories] astetcdir => /etc/asterisk astmoddir => /usr/lib/asterisk/modules astvarlibdir => /var/lib/asterisk astagidir => /var/lib/asterisk/agi-bin astspooldir => /var/spool/asterisk astrundir => /var/run/asterisk astlogdir => /var/log/asterisk [options] transmit_silence_during_record = yes languageprefix=yes execincludes=yes {code} By: Matt Jordan (mjordan) 2014-02-20 20:43:51.252-0600 Rusty: The key here is the existence of the dialplan hint. A dialplan hint that references a presence state provider should cause the crash. This is because we don't initialize the presence state engine - or really, the device state engine either - before we pre-load modules. Pre-loading {{pbx_config}} in particular is problematic for this, as it will go ahead and create all of the hints - but creating the hints before the entities that can service those hints have been initialized is rather explosive. By: Matt Jordan (mjordan) 2014-02-20 20:53:00.944-0600 Andrew - attached is a patch (pre-load-reorder.diff) which I think will fix the problem. It should initialize things that access the dialplan before modules that are preloaded. By: Rusty Newton (rnewton) 2014-02-21 13:35:35.832-0600 Matt, thanks! I verified the patch resolves the crash. By: Rusty Newton (rnewton) 2014-02-21 14:42:54.292-0600 Closing this issue out since it duplicates ASTERISK-23320 and we know the patch resolves the issue. The commit and such can close out ASTERISK-23320 when it happens. By: Andrew Nagy (tm1000) 2014-02-21 14:59:39.695-0600 Confirming that Matt's patch resolves the issue at hand. Thanks Matt! | ||||||