Summary:	ASTERISK-28587: Asterisk crash when answering a call
Reporter:	Dan Cropp (daninmadison)	Labels:	fax pjsip
Date Opened:	2019-10-16 16:09:38	Date Closed:	2019-10-22 12:09:46
Priority:	Major	Regression?
Status:	Closed/Complete	Components:	Channels/chan_pjsip
Versions:	16.4.1	Frequency of Occurrence	Constant
Related Issues:
Environment:	Ubuntu 16.04 running in a VMWare ESXi environment	Attachments:	( 0) core.tar.gz
Description:	Everything is fine with asterisk 16.3.0 (been running it for some time). We went to upgrade to 16.6.0 and encountered a crash.  I went back to 16.4.1 and replicated the crash. I start up Asterisk. Connect up to AMI socket using telnet session. Place a call into asterisk and see everything in telnet, and receive control of the call through AsyncAGI. Using telnet, I send a command to answer the call and it crashes. Following the getting a backtrace steps, I discovered that if I enable DONT_OPTIMIZE the crash does not happen.  I repeated the steps and recompiled verifying I don't see this if DONT_OPTIMIZE is enabled. I will submit the coredump once I know the issue number
Comments:	By: Asterisk Team (asteriskteam) 2019-10-16 16:09:39.178-0500 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. By: Dan Cropp (daninmadison) 2019-10-16 16:14:04.294-0500 coredump By: Joshua C. Colp (jcolp) 2019-10-16 16:21:34.662-0500 This appears to have crashed as a result of an illegal instruction and not a segmentation fault. Are you building on one platform/CPU and deploying on another? If so then the NATIVE_BUILD option in menuselect needs to be disabled. By: Dan Cropp (daninmadison) 2019-10-16 16:24:32.730-0500 Good question.  I have used this same exact environment for building versions for some time. However, it is possible the other VM I'm installing and testing on has changed. I will disable NATIVE_BUILD option and retest. Dan By: Dan Cropp (daninmadison) 2019-10-17 09:51:39.927-0500 I looked and I am disabling BUILD_NATIVE.        menuselect/menuselect --disable-all \            --enable-category MENUSELECT_APPS \            --enable-category MENUSELECT_UTILS \            --enable-category MENUSELECT_CHANNELS \            --enable-category MENUSELECT_CODECS \            --enable-category MENUSELECT_BRIDGES \            --enable-category MENUSELECT_CHANNELS \            --enable-category MENUSELECT_FORMATS \            --enable-category MENUSELECT_FUNCS \            --enable-category MENUSELECT_PBX \            --enable-category MENUSELECT_RES \            --enable-category MENUSELECT_CORE_SOUNDS \            --enable-category MENUSELECT_CFLAGS \            --disable CORE-SOUNDS-EN-WAV \            --enable CORE-SOUNDS-EN-ALAW \            --disable CORE-SOUNDS-EN-ALAW \            --disable CORE-SOUNDS-EN-GSM \            --disable CORE-SOUNDS-EN-G729 \            --disable CORE-SOUNDS-EN-G722 \            --disable CORE-SOUNDS-EN-SLN16 \            --disable CORE-SOUNDS-EN-SIREN7 \            --disable CORE-SOUNDS-EN-SIREN14 \            --disable CORE-SOUNDS-EN_AU-WAV \            --disable CORE-SOUNDS-EN_AU-ULAW \            --disable CORE-SOUNDS-EN_AU-ALAW \            --disable CORE-SOUNDS-EN_AU-GSM \            --disable CORE-SOUNDS-EN_AU-G729 \            --disable CORE-SOUNDS-EN_AU-G722 \            --disable CORE-SOUNDS-EN_AU-SLN16 \            --disable CORE-SOUNDS-EN_AU-SIREN7 \            --disable CORE-SOUNDS-EN_AU-SIREN14 \            --disable CORE-SOUNDS-EN_GB-WAV \            --disable CORE-SOUNDS-EN_GB-ULAW \            --disable CORE-SOUNDS-EN_GB-ALAW \            --disable CORE-SOUNDS-EN_GB-GSM \            --disable CORE-SOUNDS-EN_GB-G729 \            --disable CORE-SOUNDS-EN_GB-G722 \            --disable CORE-SOUNDS-EN_GB-SLN16 \            --disable CORE-SOUNDS-EN_GB-SIREN7 \            --disable CORE-SOUNDS-EN_GB-SIREN14 \            --disable CORE-SOUNDS-EN_NZ-WAV \            --disable CORE-SOUNDS-EN_NZ-ULAW \            --disable CORE-SOUNDS-EN_NZ-ALAW \            --disable CORE-SOUNDS-EN_NZ-GSM \            --disable CORE-SOUNDS-EN_NZ-G729 \            --disable CORE-SOUNDS-EN_NZ-G722 \            --disable CORE-SOUNDS-EN_NZ-SLN16 \            --disable CORE-SOUNDS-EN_NZ-SIREN7 \            --disable CORE-SOUNDS-EN_NZ-SIREN14 \            --disable CORE-SOUNDS-ES-WAV \            --enable CORE-SOUNDS-ES-ULAW \            --disable CORE-SOUNDS-ES-ALAW \            --disable CORE-SOUNDS-ES-GSM \            --disable CORE-SOUNDS-ES-G729 \            --disable CORE-SOUNDS-ES-G722 \            --disable CORE-SOUNDS-ES-SLN16 \            --disable CORE-SOUNDS-ES-SIREN7 \            --disable CORE-SOUNDS-ES-SIREN14 \            --disable CORE-SOUNDS-FR-WAV \            --enable CORE-SOUNDS-FR-ULAW \            --disable CORE-SOUNDS-FR-ALAW \            --disable CORE-SOUNDS-FR-GSM \            --disable CORE-SOUNDS-FR-G729 \            --disable CORE-SOUNDS-FR-G722 \            --disable CORE-SOUNDS-FR-SLN16 \            --disable CORE-SOUNDS-FR-SIREN7 \            --disable CORE-SOUNDS-FR-SIREN14 \            --disable CORE-SOUNDS-IT-WAV \            --disable CORE-SOUNDS-IT-ULAW \            --disable CORE-SOUNDS-IT-ALAW \            --disable CORE-SOUNDS-IT-GSM \            --disable CORE-SOUNDS-IT-G729 \            --disable CORE-SOUNDS-IT-G722 \            --disable CORE-SOUNDS-IT-SLN16 \            --disable CORE-SOUNDS-IT-SIREN7 \            --disable CORE-SOUNDS-IT-SIREN14 \            --disable CORE-SOUNDS-RU-WAV \            --disable CORE-SOUNDS-RU-ULAW \            --disable CORE-SOUNDS-RU-ALAW \            --disable CORE-SOUNDS-RU-GSM \            --disable CORE-SOUNDS-RU-G729 \            --disable CORE-SOUNDS-RU-G722 \            --disable CORE-SOUNDS-RU-SLN16 \            --disable CORE-SOUNDS-RU-SIREN7 \            --disable CORE-SOUNDS-RU-SIREN14 \            --disable CORE-SOUNDS-JA-WAV \            --disable CORE-SOUNDS-JA-ULAW \            --disable CORE-SOUNDS-JA-ALAW \            --disable CORE-SOUNDS-JA-GSM \            --disable CORE-SOUNDS-JA-G729 \            --disable CORE-SOUNDS-JA-G722 \            --disable CORE-SOUNDS-JA-SLN16 \            --disable CORE-SOUNDS-JA-SIREN7 \            --disable CORE-SOUNDS-JA-SIREN14 \            --disable CORE-SOUNDS-SV-WAV \            --disable CORE-SOUNDS-SV-ULAW \            --disable CORE-SOUNDS-SV-ALAW \            --disable CORE-SOUNDS-SV-GSM \            --disable CORE-SOUNDS-SV-G729 \            --disable CORE-SOUNDS-SV-G722 \            --disable CORE-SOUNDS-SV-SLN16 \            --disable CORE-SOUNDS-SV-SIREN7 \            --disable CORE-SOUNDS-SV-SIREN14 \            --disable chan_mobile \            --disable chan_ooh323 \            --disable app_ivrdemo \            --disable app_jack \            --disable app_skel \            --disable app_saycounted \            --disable app_statsd \            --enable app_macro \            --disable app_voicemail \            --disable chan_dahdi \            --disable chan_iax2 \            --disable chan_misdn \            --disable chan_motif \            --disable chan_nbs \            --disable chan_oss \            --disable chan_skinny \            --disable chan_vpb \            --enable chan_bridge_media \            --enable chan_pjsip \            --enable chan_rtp \            --enable chan_alsa \            --enable chan_console \            --enable chan_mgcp \            --enable chan_unistim \            --enable chan_phone \            --enable chan_sip \            --enable codec_speex \            --enable codec_opus \            --enable codec_silk \            --enable codec_siren7 \            --enable codec_siren14 \            --enable codec_g729a \            --enable codec_gsm \            --enable format_g729 \            --enable format_ogg_vorbis \            --enable func_speex \            --disable pbx_ael \            --disable pbx_dundi \            --disable pbx_lua \            --enable streamplayer \            --enable res_agi \            --disable res_ari \            --disable res_ari_applications \            --disable res_ari_asterisk \            --disable res_ari_bridges \            --disable res_ari_channels \            --disable res_ari_device_states \            --disable res_ari_endpoints \            --disable res_ari_events \            --disable res_ari_mailboxes \            --enable res_ari_model \            --disable res_ari_playbacks \            --disable res_ari_recordings \            --disable res_ari_sounds \            --enable res_clialiases \            --enable res_clioriginate \            --enable res_config_curl \            --disable res_config_odbc \            --disable res_config_sqlite3 \            --enable res_convert \            --enable res_crypto \            --enable res_curl \            --disable res_fax \            --enable res_format_attr_celt \            --enable res_format_attr_g729 \            --enable res_format_attr_h263 \            --enable res_format_attr_h264 \            --enable res_format_attr_ilbc \            --enable res_format_attr_opus \            --enable res_format_attr_silk \            --enable res_format_attr_siren14 \            --enable res_format_attr_siren7 \            --enable res_format_attr_vp8 \            --enable res_http_media_cache \            --enable res_http_post \            --enable res_http_websocket \            --enable res_limit \            --enable res_manager_devicestate \            --enable res_manager_presencestate \            --enable res_musiconhold \            --enable res_mutestream \            --disable res_mwi_external \            --disable res_mwi_external_ami \            --disable res_odbc \            --disable res_odbc_transaction \            --enable res_parking \            --enable res_pjproject \            --enable res_pjsip \            --enable res_pjsip_acl \            --enable res_pjsip_authenticator_digest \            --enable res_pjsip_caller_id \            --disable res_pjsip_config_wizard \            --enable res_pjsip_dialog_info_body_generator \            --enable res_pjsip_diversion \            --enable res_pjsip_dlg_options \            --enable res_pjsip_dtmf_info \            --enable res_pjsip_empty_info \            --enable res_pjsip_endpoint_identifier_anonymous \            --enable res_pjsip_endpoint_identifier_ip \            --enable res_pjsip_endpoint_identifier_user \            --enable res_pjsip_exten_state \            --enable res_pjsip_header_funcs \            --enable res_pjsip_logger \            --enable res_pjsip_messaging \            --enable res_pjsip_mwi \            --enable res_pjsip_mwi_body_generator \            --enable res_pjsip_nat \            --enable res_pjsip_notify \            --enable res_pjsip_one_touch_record_info \            --enable res_pjsip_outbound_authenticator_digest \            --enable res_pjsip_outbound_publish \            --enable res_pjsip_outbound_registration \            --enable res_pjsip_path \            --enable res_pjsip_pidf_body_generator \            --enable res_pjsip_pidf_digium_body_supplement \            --enable res_pjsip_pidf_eyebeam_body_supplement \            --enable res_pjsip_publish_asterisk \            --enable res_pjsip_pubsub \            --enable res_pjsip_refer \            --enable res_pjsip_registrar \            --enable res_pjsip_rfc3326 \            --enable res_pjsip_sdp_rtp \            --enable res_pjsip_send_to_voicemail \            --enable res_pjsip_session \            --enable res_pjsip_sips_contact \            --enable res_pjsip_t38 \            --enable res_pjsip_transport_websocket \            --enable res_pjsip_xpidf_body_generator \            --enable res_realtime \            --enable res_resolver_unbound \            --enable res_rtp_asterisk \            --enable res_rtp_multicast \            --enable res_security_log \            --enable res_sorcery_astdb \            --enable res_sorcery_config \            --enable res_sorcery_memory \            --enable res_sorcery_memory_cache \            --enable res_sorcery_realtime \            --enable res_speech \            --enable res_srtp \            --enable res_stasis \            --enable res_stasis_answer \            --enable res_stasis_device_state \            --enable res_stasis_mailbox \            --enable res_stasis_playback \            --enable res_stasis_recording \            --enable res_stasis_snoop \            --disable res_stasis_test \            --enable res_stun_monitor \            --disable res_timing_dahdi \            --enable res_timing_timerfd \            --enable res_xmpp \            --enable res_ael_share \            --enable res_calendar \            --disable res_calendar_exchange \            --disable res_calendar_caldav \            --disable res_calendar_ews \            --disable res_calendar_icalendar \            --disable res_chan_stats \            --disable res_config_ldap \            --disable res_config_pgsql \            --disable res_corosync \            --enable res_endpoint_stats \            --disable res_fax_spandsp \            --disable res_hep \            --disable res_hep_pjsip \            --disable res_hep_rtcp \            --disable res_phoneprov \            --disable res_config_sqlite3 \            --enable res_pjsip_history \            --disable res_pjsip_phoneprov_provider \            --enable res_pktccops \            --disable res_remb_modifier \            --enable res_smdi \            --disable res_snmp \            --enable res_statsd \            --disable res_timing_kqueue \            --enable res_timing_pthread \            --enable res_adsi \            --disable res_config_sqlite \            --enable res_monitor \            --disable res_digium_phone \            --enable DONT_OPTIMIZE \            --enable BETTER_BACKTRACES \            --disable BUILD_NATIVE menuselect.makeopts I am going to try a brand new Ubuntu 64 VM on the QA server.  Perhaps that VM has become corrupted. By: Dan Cropp (daninmadison) 2019-10-18 07:56:21.786-0500 We installed a brand new Ubuntu 16 VM installed on the VMWare ESXi having problems. Then we installed my compiled 16.4.1 asterisk and it encountered the same problems. My thought is to start with asterisk 16.3.0 (which we have compiled and installed on this VM server without issue). Then, step through the list of changes from 16.3.0 to 16.4.1 patching the code with each change one at a time until I run into the change where things stop working. By: Joshua C. Colp (jcolp) 2019-10-18 08:10:22.959-0500 You can try using git bisect[1] to do this quicker, as long as you track down the commit hash where it works in 16. [1] https://flaviocopes.com/git-bisect/ By: Dan Cropp (daninmadison) 2019-10-22 12:08:31.840-0500 I can't explain it, but the VM server where this was not working last week is now working. I went through the git source commits 16.3.0 through 16.6.1 and everything worked. Recompiled using the tarball (which I did several times last week).  It always failed on that box, but worked on 4 other VM servers. No explanation to the cause, but I'm closing this issue.