Summary:	ASTERISK-29149: res_pjsip may crash on load_module (two times)
Reporter:	Alexander Traud (traud)	Labels:
Date Opened:	2020-11-02 04:17:00.000-0600	Date Closed:
Priority:	Minor	Regression?
Status:	Open/New	Components:	Resources/res_pjsip
Versions:	16.14.0 18.0.0	Frequency of Occurrence
Related Issues:
Environment:		Attachments:
Description:	In the configuration file {{/etc/asterisk/modules.conf}} go for: {code}noload = res_sorcery_astdb.so require = res_pjsip.so{code}then Asterisk crashes with:{code}  == Manager registered action PJSIPShowEndpoints  == Manager registered action PJSIPShowEndpoint  == Manager registered action PJSIPShowAuths ERROR: sorcery.c:886 __ast_sorcery_object_type_insert_wizard: Wizard 'astdb' could not be applied to object type 'contact' as it was not found ERROR: res_pjsip/pjsip_configuration.c:2164 ast_res_pjsip_initialize_configuration: Failed to register SIP location support with sorcery ERROR: res_pjsip.c:5722 load_module: Failed to initialize SIP configuration. Aborting load munmap_chunk(): invalid pointer Thread 1 "asterisk" received signal SIGABRT, Aborted. #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49 #1  0x00007ffff72a1864 in __GI_abort () at abort.c:79 #2  0x00007ffff7304af6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff742c128 "%s\n")    at ../sysdeps/posix/libc_fatal.c:155 #3  0x00007ffff730d46c in malloc_printerr (str=str@entry=0x7ffff742e0f0 "munmap_chunk(): invalid pointer") at malloc.c:5389 #4  0x00007ffff730d83c in munmap_chunk (p=<optimized out>) at malloc.c:2845 #5  0x00005555555c80a2 in __ast_free (ptr=0x7ffff49f8dd0, file=0x5555558001ad "cli.c", lineno=2363,    func=0x555555802b70 <__PRETTY_FUNCTION__.11> "ast_cli_unregister") at astmm.c:1588 #6  0x000055555563be40 in ast_cli_unregister (e=0x7ffff4a0cb20 <cli_commands>) at cli.c:2363 #7  0x000055555563c45f in ast_cli_unregister_multiple (e=0x7ffff4a0cb20 <cli_commands>, len=1) at cli.c:2464 #8  0x00007ffff49da845 in ast_sip_destroy_distributor () at res_pjsip/pjsip_distributor.c:1336 #9  0x00007ffff49b7a37 in unload_pjsip (data=0x0) at res_pjsip.c:5558 #10 0x00007ffff49b814e in load_module () at res_pjsip.c:5762 #11 0x000055555568ecd5 in start_resource (mod=0x55555597ea30) at loader.c:1711 #12 0x000055555568f7a2 in start_resource_attempt (mod=0x55555597ea30, count=0x7fffffffb0e8) at loader.c:1887 #13 0x0000555555690189 in start_resource_list (resources=0x7fffffffb140, mod_count=0x7fffffffb0e8) at loader.c:1984 #14 0x0000555555690f36 in load_resource_list (load_order=0x7fffffffb1b0, mod_count=0x7fffffffb170) at loader.c:2166 #15 0x0000555555691996 in load_modules () at loader.c:2379 #16 0x00005555555c7bcd in asterisk_daemon (isroot=1, runuser=0x0, rungroup=0x0) at asterisk.c:4158 #17 0x00005555555c7086 in main (argc=2, argv=0x7fffffffe4f8) at asterisk.c:3925{code}The cause is {{ast_res_pjsip_initialize_configuration(…)}} → {{goto error}} → {{unload_module()}} → {{ast_sip_destroy_distributor()}} → {{ast_cli_unregister_multiple(…)}}. That is called with one (statically defined) CLI command. However, that command has not been registered yet. For a quick test, I commented that out. Which gave the next crash:{code}Thread 1 "asterisk" received signal SIGSEGV, Segmentation fault. #0  __GI___pthread_mutex_lock (mutex=0x0) at ../nptl/pthread_mutex_lock.c:67 #1  0x0000555555692891 in __ast_pthread_mutex_lock (filename=0x555555820674 "sched.c", lineno=412,    func=0x555555820ad0 <__PRETTY_FUNCTION__.15> "ast_sched_clean_by_callback", mutex_name=0x555555820669 "&con->lock", t=0x0) at lock.c:326 #2  0x000055555571173d in ast_sched_clean_by_callback (con=0x0, match=0x7ffff49ed1b9 <idle_sched_cb>,    cleanup_cb=0x7ffff49ed2e3 <idle_sched_cleanup>) at sched.c:412 #3  0x00007ffff49edbbf in ast_sip_destroy_transport_management () at res_pjsip/pjsip_transport_management.c:427 #4  0x00007ffff49b7a3c in unload_pjsip (data=0x0) at res_pjsip.c:5559 #5  0x00007ffff49b814e in load_module () at res_pjsip.c:5762 #6  0x000055555568ecd5 in start_resource (mod=0x55555594a2e0) at loader.c:1711 #7  0x000055555568f7a2 in start_resource_attempt (mod=0x55555594a2e0, count=0x7fffffffb0e8) at loader.c:1887 #8  0x0000555555690189 in start_resource_list (resources=0x7fffffffb140, mod_count=0x7fffffffb0e8) at loader.c:1984 #9  0x0000555555690f36 in load_resource_list (load_order=0x7fffffffb1b0, mod_count=0x7fffffffb170) at loader.c:2166 #10 0x0000555555691996 in load_modules () at loader.c:2379 #11 0x00005555555c7bcd in asterisk_daemon (isroot=1, runuser=0x0, rungroup=0x0) at asterisk.c:4158 #12 0x00005555555c7086 in main (argc=2, argv=0x7fffffffe4f8) at asterisk.c:3925{code}Again, something is destroyed which has not be inited, yet.
Comments:	By: Asterisk Team (asteriskteam) 2020-11-02 04:17:01.786-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur. Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/]. By: Alexander Traud (traud) 2020-11-02 04:25:44.647-0600 As a side-effect, the solution for ASTERISK-29148 is going to prevent why this crash happened for me. However, the init/destroy logic is broken in general. I have no idea what parts/layers must be fixed exactly: * For example, why can {{ast_cli_unregister(…)}} crash at a non-registered command at all? That would prevent the first crash. * For example, why is the CLI command not dynamically but statically defined in {{pjsip_distributor}}? That would prevent the first crash, too. * Is it correct, that {{res_pjsip}} calls destroy even on not yet inited parts? That would prevent both crashes. These are architectural decisions, I cannot do. Furthermore, because I do not support the PJProject in Asterisk, I do not plan to work at it. Perhaps this is even not an issue at all: “Use {{res_pjsip}} correctly and this never happens!” Therefore, I marked this issue not as major but minor. By: Benjamin Keith Ford (bford) 2020-11-05 13:41:00.784-0600 Just doing a {{noload => res_pjsip.so}} does not replicate the issue for me on 18. I get error messages saying other PJSIP modules failed to load (missing the res_pjsip dependency, of course), but no crash. There may be some other moving parts involved. # Does it crash every time you start Asterisk? # Is that the only {{noload}} in your {{modules.conf}}? Anything else of importance? By: Alexander Traud (traud) 2020-11-06 03:40:49.579-0600 That was a vanilla Asterisk 18.0.0 in Ubuntu 20.10 with {{./configure && make && sudo make install config samples}}. However, I did not noload {{res_pjsip}}, instead I require it and noload the module {{res_sorcery_astdb.so}}. If you cannot reproduce with that either, force the crashes manually: Instead of checking the result of {{ast_res_pjsip_initialize_configuration}} do {{goto error}} from there always (in other words, in res_pjsip.load_module, remove the if condition which checks {{ast_res_pjsip_initialize_configuration}} but keep the stuff inside the brackets). Then, Asterisk is going to crash as described above. By: Benjamin Keith Ford (bford) 2020-11-06 10:21:10.604-0600 My apologies, I misread that. I do get the crash you outlined above. I'll open a ticket for this.