Summary: | ASTERISK-25744: res_pjsip: Segfaults in ssl3_write_bytes, pj_ssl_sock_send, tls_send_msg | ||
Reporter: | Dmitriy Serov (Demon) | Labels: | |
Date Opened: | 2016-02-04 03:32:48.000-0600 | Date Closed: | |
Priority: | Major | Regression? | |
Status: | Open/New | Components: | Resources/res_pjsip |
Versions: | 13.7.0 | Frequency of Occurrence | |
Related Issues: | |||
Environment: | openssl-1.0.1q-x86_64 pjproject 2.4.5 | Attachments: | ( 0) 2016_02_03__17_00_07.backtrace-threads.txt ( 1) 2016_02_03__17_00_07.full.tail.txt ( 2) 2016_02_03__23_38_07.backtrace-threads.txt ( 3) 2016_02_03__23_38_07.full.tail.txt ( 4) 2016_02_03__23_52_07.backtrace-threads.txt ( 5) 2016_02_03__23_52_07.full.tail.txt ( 6) 2016_02_04__00_42_07.backtrace-threads.txt ( 7) 2016_02_04__00_42_07.full.tail.txt ( 8) 2016_02_04__10_36_07.backtrace-threads.txt ( 9) 2016_02_04__10_36_07.full.tail.txt (10) 2016_02_10__00_04_07.backtrace-threads.txt (11) 2016_02_10__00_04_07.full.tail.txt (12) 2016_02_10__06_18_07.backtrace-threads.txt (13) 2016_02_10__06_18_07.full.tail.txt (14) 2016_02_13__02_54_07.backtrace-threads.txt (15) 2016_02_13__02_54_07.full.tail.txt (16) 2016_02_14__21_44_07.backtrace-threads.txt (17) 2016_02_14__21_44_07.full.tail.txt |
Description: | Regular segfaults in the same place of the program.
Asterisk faults, 3-4 times per day Several logs attached. | ||
Comments: | By: Asterisk Team (asteriskteam) 2016-02-04 03:32:50.416-0600 Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report. Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process]. By: Dmitriy Serov (Demon) 2016-02-04 03:34:22.437-0600 Log each fault is represented by two files: backtrace and full log tail. By: Joshua C. Colp (jcolp) 2016-02-04 06:43:34.580-0600 We're going to need more information in addition to this to try to narrow it down more. How many TLS connections are active? Are there few, are there a lot? Are they connecting/disconnecting a lot? Is there a lot of traffic going on? By: Dmitriy Serov (Demon) 2016-02-04 07:08:22.319-0600 cmd "pjsip show contacts like transport=TLS" / grep -c Contact Result: 18. But this number may change as you connect and disconnect devices. By: Rusty Newton (rnewton) 2016-02-04 17:56:46.804-0600 Dmitriy is this a new install or did it start after an upgrade? By: Dmitriy Serov (Demon) 2016-02-05 06:04:58.720-0600 This error was before, but much less frequently. But it is difficult to say, because the server kept falling over for a different reason: ASTERISK-25439 After eliminating one of the causes started to fall with TLS. Most likely this error is a consequence, a manifestation of some other situation. I'm ready to put in the right place assert in order to catch the situation earlier. For example, I don't understand how in backtraces can be such an IP address (despite the fact that the length of the value 12 bytes): {noformat} #7 0x00007fb2d40c5eab in pjsip_endpt_respond_stateless (endpt=0x1fb8a28, rdata=0x7fb2900c23c8, st_code=501, st_text=0x0, hdr_list=0x0, body=0x0) at ../src/pjsip/sip_util.c:1869 status = 0 res_addr = {transport = 0x7fb2902b9928, addr = {addr = {sa_family = 2}, ipv4 = {sin_family = 2, sin_port = 53779, sin_addr = {s_addr = 624595237}, sin_zero = "\000\000\000\000\000\000\000"}, ipv6 = {sin6_family = 2, sin6_port = 53779, sin6_flowinfo = 624595237, sin6_addr = {s6_addr = '\000' <repeats 15 times>, u6_addr32 = {0, 0, 0, 0}}, sin6_scope_id = 0}}, addr_len = 28, dst_host = {flag = 3, type = PJSIP_TRANSPORT_TLS, addr = {host = {ptr = 0x7fb28ae31708 "37.145.58.3710)", slen = 12}, port = 5074}}} tdata = 0x7fb28ae30bb8 {noformat} {noformat} #11 0x00007f3d4a0baeab in pjsip_endpt_respond_stateless (endpt=0x1fb7d38, rdata=0x7f3ce442c158, st_code=501, st_text=0x0, hdr_list=0x0, body=0x0) at ../src/pjsip/sip_util.c:1869 status = 0 res_addr = {transport = 0x7f3ce41cdc28, addr = {addr = {sa_family = 2}, ipv4 = {sin_family = 2, sin_port = 53779, sin_addr = {s_addr = 624595237}, sin_zero = "\000\000\000\000\000\000\000"}, ipv6 = {sin6_family = 2, sin6_port = 53779, sin6_flowinfo = 624595237, sin6_addr = {s6_addr = '\000' <repeats 15 times>, u6_addr32 = {0, 0, 0, 0}}, sin6_scope_id = 0}}, addr_len = 28, dst_host = {flag = 3, type = PJSIP_TRANSPORT_TLS, addr = {host = {ptr = 0x7f3ce4a4ced8 "37.145.58.37", slen = 12}, port = 5074}}} tdata = 0x7f3ce4a4c388 {noformat} {noformat} #8 0x00007fca268f5eab in pjsip_endpt_respond_stateless (endpt=0x1fb7df8, rdata=0x7fca08303be8, st_code=501, st_text=0x0, hdr_list=0x0, body=0x0) at ../src/pjsip/sip_util.c:1869 status = 0 res_addr = {transport = 0x7fca08253eb8, addr = {addr = {sa_family = 2}, ipv4 = {sin_family = 2, sin_port = 50451, sin_addr = {s_addr = 624595237}, sin_zero = "\000\000\000\000\000\000\000"}, ipv6 = {sin6_family = 2, sin6_port = 50451, sin6_flowinfo = 624595237, sin6_addr = {s6_addr = '\000' <repeats 15 times>, u6_addr32 = {0, 0, 0, 0}}, sin6_scope_id = 0}}, addr_len = 28, dst_host = {flag = 3, type = PJSIP_TRANSPORT_TLS, addr = {host = {ptr = 0x7fca5810dea8 "37.145.58.37\312\177", slen = 12}, port = 5061}}} tdata = 0x7fca5810d358 {noformat} By: Dmitriy Serov (Demon) 2016-02-05 13:37:06.879-0600 pjproject config_site.h {noformat} #define NDEBUG #define PJ_ENABLE_EXTRA_CHECK 1 #define PJ_MAX_HOSTNAME 1024 #define PJSIP_MAX_URL_SIZE 1024 #define PJ_IOQUEUE_MAX_HANDLES 5000 #define PJSIP_MAX_TSX_COUNT ((640*1024)-1) #define PJSIP_MAX_DIALOG_COUNT ((640*1024)-1) #define PJSIP_UDP_SO_SNDBUF_SIZE (24*1024*1024) #define PJSIP_UDP_SO_RCVBUF_SIZE (24*1024*1024) #define PJSUA_MAX_CALLS 512 {noformat} By: Dmitriy Serov (Demon) 2016-02-10 01:47:21.588-0600 Was the idea that reason is shared with res_xmpp (iksemel-1.5). xmpp uses TLS and I've met a lot of references to xmpp in backtraces. I disabled this module and was even happy when the server then has worked for more than a day. But after that, still segfaults, the log of which is attached (2016_02_10). By: Dmitriy Serov (Demon) 2016-02-12 01:28:03.680-0600 The server crashes at least 2-3 times a day with this error. Logs already accumulated for the publication of separate books. What's wrong with the libraries or the server settings? No one is using res_pjsip TLS? By: Dmitriy Serov (Demon) 2016-02-13 08:33:16.435-0600 2016_02_13__02_54_07.backtrace-threads.txt Other stack of segfault in SSL. By: Dmitriy Serov (Demon) 2016-02-14 13:17:48.640-0600 once more backtrace with more deep stack. 2016_02_14__21_44_07.backtrace-threads.txt |