[Home]

Summary:ASTERISK-25727: RPM build requires OPTIONAL_API cflag due to PJSIP requirement
Reporter:Gergely Dömsödi (doome)Labels:
Date Opened:2016-01-26 10:57:21.000-0600Date Closed:2016-02-16 14:40:42.000-0600
Priority:MajorRegression?Yes
Status:Closed/CompleteComponents:Resources/res_pjsip
Versions:13.7.0 Frequency of
Occurrence
Related
Issues:
Environment:Fedora 23Attachments:( 0) asterisk_hardened_debug.txt
( 1) asterisk_not_hardened_debug.txt
( 2) build.log.txt
( 3) modules.dir.txt
Description:I built 13.7.0 on Fedora 23 (gcc 5.3.1) and res_pjsip fails to load with the message: {{Error loading module 'res_pjsip.so': /usr/lib64/asterisk/modules/res_pjsip.so: undefined symbol: ast_statsd_log_full_va}}

Vanilla configure, make, make install: works
rpm build with hardening enabled (fedora default): asterisk starts up, but res_pjsip is not loaded, and cannot be loaded
rpm build with hardenind disabled (%undefine _hardened_build): asterisk doesn't even starts with the message above

update:

Made a successful build with rpmbuild:
The problem was with the OPTIONAL_API cflag was not enabled in my build. As it turned out, it is not optional for PJSIP in 13.7.0 (it uses ast_statsd_log_full_va which is exported only if this flag is set), thus it must be turned on.
If it is enabled, asterisk starts properly both with _hardened_build enabled and disabled. However when _hardened_build is enabled, there are a bunch of warnings during startup about various undefined symbols, but at the end everything loads just fine. If it is disabled there are no such messages.
Comments:By: Asterisk Team (asteriskteam) 2016-01-26 10:57:22.795-0600

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

By: Gergely Dömsödi (doome) 2016-01-26 11:33:33.096-0600

Could this be related to ASTERISK-25651 ? I had similar problems with 13.6.0, only there I could remedy the module loading issues by manually loading the failed modules in modules.conf. Here, it didn't help.

By: Rusty Newton (rnewton) 2016-01-26 17:55:18.533-0600

We require additional debug to continue with triage of your issue. Please follow the instructions on the wiki [1] for how to collect debugging information from Asterisk. For expediency, where possible, attach the debug with a '.txt' file extension so that the debug will be usable for further analysis.

Thanks!

[1] https://wiki.asterisk.org/wiki/display/AST/Collecting+Debug+Information


Following the instructions linked above - please provide two logs,

* one with (warning,error,notice,verbose)
* and the other with (warning,error,notice,verbose,debug)

When capturing the log simply start up Asterisk and stop it. We want to see what happens during startup.

NOTE: Makes sure verbose and debug are both turned up to 5.

* Attach your modules.conf
* Attach ls -l output for your modules directory,

Attach everything as .txt so we can read it quickly. Thanks!

By: Gergely Dömsödi (doome) 2016-01-27 09:55:23.540-0600

As it turned out during further investigation, the problem could be with fedora rpmbuild's default build options.

Originally, when I reported the issue, I tried to build asterisk from an updated version of Fedora's asterisk .spec file with rpmbuild, and that produced the error above. But when I do a normal {{./configure; make; make install}}, everything seems to start up fine.

Firstly I suspected Fedora 23's default [hardening|https://fedoraproject.org/wiki/Changes/Harden_All_Packages] rules, and I turned it off from the spec file. But this resulted in asterisk not even starting up with the message: {{asterisk: symbol lookup error: /usr/lib64/asterisk/modules/res_pjsip.so: undefined symbol: ast_statsd_log_full_va}}

I attached the debug logs of both the hardened, and the not hardened startup, and the build logs of the not hardened rpm build. Please notice that {{-z relro}} is still in effect. I also tried to turned that out (and other gcc and ld option added by rpmbuild), without success, yet.

So, to summarize:

Vanilla configure, make, make install: works
rpm build with hardening enabled (fedora default): asterisk starts up, but res_pjsip is not loaded, and cannot be loaded
rpm build with hardenind disabled (%undefine _hardened_build): asterisk doesn't even starts with the message above

By: Rusty Newton (rnewton) 2016-01-27 10:00:09.445-0600

Linking to ASTERISK-25726 - may be some relation there.

By: Gergely Dömsödi (doome) 2016-01-28 05:30:28.008-0600

Made a successful build with rpmbuild:

The problem was with the OPTIONAL_API cflag was not enabled in my build. As it turned out, it is not optional for PJSIP in 13.7.0 (it uses {{ast_statsd_log_full_va}} which is exported only if this flag is set), thus it must be turned on.

If it is enabled, asterisk starts properly both with _hardened_build enabled and disabled. However when _hardened_build is enabled, there are a bunch of warnings during startup about various undefined symbols, but at the end everything loads just fine. If it is disabled there are no such messages.

By: George Joseph (gjoseph) 2016-02-16 11:37:35.589-0600

You would think that OPTIONAL_API turns on and off the optional apis, but it's a little more complicated than that.  When it's on, the optional apis are indeed optional but when it's off, the optional apis become REQUIRED.  In this case with OPTIONAL_API turned off, ast_statsd_log_full_va (and a whole bunch of other things) became required instead of optional.  Unfortunately, res_pjsip tried to load before res_statsd and failed.  You *could* fix this at runtime in modules.conf by loading res_statsd before res_pjsip but that's probably not the best solution.

Hardened build makes this even a little more complicated.   When the Asterisk module loader opens each module, it does it with lazy symbol resolution meaning that it defers lookup of functions until the function is actually called.  This way if one module depends on another, they'll still both load.  Hardened build adds the "-z now" flag to the linker which requires that those symbols be resolved at load so many modules will fail.  OPTIONAL_API helps again here.  With it turned on, the api will be available as a stub until the module providing the api loads. This satisfies the system loader (but still spits out messages).



By: George Joseph (gjoseph) 2016-02-16 12:20:42.125-0600

Oops, there was actually bug in res_statsd that was causing the crash without OPTIONAL_API.  Fix in progress.