[Home]

Summary:ASTERISK-05862: [patch] implement AEL via bison/flex.
Reporter:Steve Murphy (murf)Labels:
Date Opened:2005-12-17 16:06:13.000-0600Date Closed:2006-07-06 11:16:20
Priority:MinorRegression?No
Status:Closed/CompleteComponents:PBX/pbx_ael
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) ael_lex.c_v2
( 1) ael_structs.h_v2
( 2) ael.flex
( 3) ael.flex_v2
( 4) ael.tab.c
( 5) ael.tab.h
( 6) ael.y
( 7) ael.y_v2
( 8) ael2.loader.patch
( 9) ael2.Makefiles.patch
(10) AEL2.patch.bz2
(11) AEL2-1.2.x-v0.14.patch
(12) AEL2v0.10.patch.bz2
(13) AEL2v0.11.patch.bz2
(14) AEL2v0.12.patch.bz2
(15) AEL2v0.13.patch.bz2
(16) AEL2v0.14.patch.bz2
(17) AEL2v0.2.patch.bz2
(18) AEL2v0.3.patch.bz2
(19) AEL2v0.4.patch.bz2
(20) AEL2v0.5.patch.bz2
(21) AEL2v0.6.patch.bz2
(22) AEL2v0.7.patch.bz2
(23) AEL2v0.8.patch.bz2
(24) AEL2v0.9.patch.bz2
(25) applist
(26) extensions.ael
(27) mm
(28) mm_v2
Description:From bug 0006013:

Begin rave.

I just sat down and started mapping out a BNF (in bison syntax) for AEL.
We could use flex's ability to build contextual lexers. At the very
mininum, a bison parser could be built, that would not only generate the
information that the compiler would need, but would also do a very
rigorous check of the syntax in the file, right down to the innards of the
$[ ] expressions, so probs can be found at compile time instead of at run
time.

Maybe I'm a fool for proposing this, but it would make the detection and
reporting of syntax errors easier, make the compiler more robust and
easier to maintain and upgrade, get rid of useless/needless line
restrictions, provide a clear definition of exactly what the parser
expects everywhere. The added benefit would be that the parser and
compiler would be separated from each other, which doesn't appear to be
the case in pbx_ael.c, but is a better practice in compiler building.

The parser would generate a linked data structure, and the compiler could
traverse it to generate code (asterisk extension.conf syntax, or straight
to internal structures). The code would be simpler, as all parsing details
would be separated out into the parser. And you won't have to exhaustively
test for all possible syntax problems and report each in the code-- the
parser has mechanisms to detect and recover from errors, which can be
hand-tuned. Right now, I think you'd have to considerably add code (double
the size?) to pbx_ael to do proper error checking, and still miss a ton of
cases.



End of rave.


****** ADDITIONAL INFORMATION ******

From bug 0006013:

Corydon76 - 12-17-05 11:47
----------------------------------------------------------------------
In that case, I invite you to open a Feature Request with your suggestion
on bison and flex, and let us close this one out, since the problems would
be resolved with such an implementation.
----------------------------------------------------------------------

OK, so I'll put my "money", where my mouth is :)
Comments:By: Steve Murphy (murf) 2005-12-19 23:33:46.000-0600

I've cobbled together a grammar and a scanner for ael. It's barebones, but it does successfully parse my extensions.ael file.

the mm file is a makefile. Better have flex 2.5.31 and the latest 2.x version of bison.

The mm (via make -f mm) will build aelparse, which is a standalone exec, that reads /etc/asterisk/extensions.ael, and reports the first syntax error it finds, if any. If it ends in silence, you've got a "clean" file.

The extensions.ael file that I started with, yielded no messages from the ael compiler, and a cursory examination of the dialplan showed that all extensions seemed to exist, with all priorities within them.

But this parser showed that I had a missing }; at the end of one extension,
three Goto statements instead of "goto" statements. Several missing semicolons at the end of application calls.

It did not detect that I had a macro call to a macro that doesn't exist. It didn't spot that I had a few $[] expressions with bad characters. But, I hope to improve it so it will spot these kinds of problems.

It didn't spot that I had a few application calls on the same line as the extension declaration (ie, t => {Hangup();...}, but my parser will allow that kind of syntax, by design.

The parser messages are pretty crude at the moment, just the line number, and a terse explanation of what it sees wrong. I'll improve this. It stops at the first error. I can sprinkle error tokens here and there to allow the parser to recover and continue.

To Do:
1. Have the parser generate data structures to represent the input.
2. Use the data structs to check things, like macro calls are legal references,
  goto's reference existing labels, etc.
3. Modify the $[] parser to provide a mode where evaluation will not be
  attempted, just syntax checking, and then feed all $[] exprs to the $[]
  parser (ast_expr2.y) for checking.
4. Write a routine to compile the ael, perhaps using the existing pbx_ael code,
  at least adapted to the structs in this parser...
5. Clean it up, test the living daylights out of it, etc.

Sound good?

By: Steve Murphy (murf) 2005-12-21 15:14:03.000-0600

OK, completed steps 1 and 2 from above.

The parser now builds a linked data structure that it returns. I have written a check function which crawls it and generates errors for:

1. goto some location (without varibles embedded in the name) that doesn't exist.
2. Macro calls with the wrong number of arguments.
3. Macro calls missing the &.
4. Macro calls to contexts.
5. Macro calls to non-existent macros.
6. If I can find out how to access the list of applications available within the asterisk runtime, I'll add code to at least verify that the applications referenced by the dialplan exist... it would be cool if the args to the apps could be verified, also... to some extent.

To demonstrate the structures, I've implemented a pretty printer. It's output, of course, will not be able to be used for input to the current ael parser. The curlies aren't placed right for it's needs, but could be, if this becomes necc. The output should be OK for this parser to read in again, tho.

I've also thrown in a template for complete traversal of the data structures. I used it to do the check code.

The mini-makefile will build a stand-alone parser, as before. Now, it generates a pretty printed version of the extensions.ael file, and then runs the check. In my "clean" extensions.ael file, it turned up 8 bad goto references, one of which resulted from a typo in the context name, the others because the label was referenced out of its scope.

So, If you are monitoring this, and you have any .ael files, please try them out, and if they generate syntax errors or semantic errors that aren't correct, please send me the test case.

I'll upload the new source files. One note: I'm using flex 2.5.31, which is apparently too cutting edge... I had to fix a bug in the scanner generator to get output that would compile. So, I'm including the output of flex instead of its input... I reported the bug to the sourceforge site, but honestly, it sure looks like flex is pretty dead over there... we'll see.

By: Steve Murphy (murf) 2005-12-28 10:19:21.000-0600

I hope no-one objects to my using the bug database as a development diary! ;^)

OK, I've finished the item list of 12-19-05. I've filed bug 0006072 which includes a patch to add two funcs to the ast_expr2* stuff, to allow it to be used as a $[] expression syntax checker.

I've "finished" the code generator.

I've gone thru the calls to ast_add_extension2, and the ast_context_add_*() func calls, one by one, making sure the right stuff is in the right place.

I've done memory leak analysis, and got rid of all the memory leaks I could, but the ast_add_extension2() call seems to require that the data arg be strdup'd... I hope the pbx* funcs will free it when the dialplan is unregistered...! At any rate, all the allocs done by the parser, compiler, and syntax checker are otherwise freed by the time the pbx_ael2 terminates.  EXCEPT one small leak in the parser. I'm still doing code reviews, trying to figure out where the darn stuff could get leaked. It's a tiny leak, so not to panic.

I've added error tokens to the bison grammar, so the parser doesn't die on the first syntax error, but more work could be done here to minimize the granularity of errors, and reduce the cascade of error messages that may result from the tokens left laying around when the error was hit.

I've formed it into the module pbx_ael2, which reads in the /etc/asterisk/extensions.ael2 file, and enters it into the dialplan, if there are no syntax errors.

I've built some stand-alone parser/checker/compiler execs for both the pbx_ael module, and the pbx_ael2 module, in the utils directory. It short circuits all the ast_* funcs, including the log and verbose calls. It prints all this to stdout, as what function is called, and what its arguments are. This is for debug. You can get syntax errors for a file without having to restart asterisk, or reload the module. And you can debug the mess without involving asterisk at all.

I'm running asterisk at home using the .ael2 dialplan, and so far, so good.
Before I attach the source, I need to run a lot more test cases!

SEND ME TESTCASES!!!

Thoughts: There is no "return" keyword in the original ael parser. Right now, I have it as a keyword, but it just generates a Return() app call, which is disastrous at the moment (expecially if you don't have any gosub calls previously!). I'm thinking of implementing it in macros as a true return directive, which would generate a goto to the end of the macro. I'm also thinking of adding a check to SWITCH constructs if any CASE should have control flow off the end of the statement list, without a break, or a return as the last element in the case statement list. At least, failing this check would generate warnings about poor programming practice...

If anyone can think of common errors made in dialplans, that could be checked for in the dialplan at compile time, and at least warnings and advise could be generated at compile time, let me know. It will increase the value of the module, and help everyone increase the quality of their dialplans.

Really, I'm excited about this. Most of the runtime errors I encounter could have been caught at compile time! While testing still needs to be done, this module will decrease the cost of dialplan errors considerably!!! The days of writing dialplans in "assembler" (extensions.conf) format are over!

OK, that just gave me another idea. I'll rewrite all extensions.conf examples in the asterisk source as extensions.ael(2) examples, and add them to the patch.

And, if folks want to use python or perl or something to generate dialplans, they could either generate calls to ast_add_extension2(), ast_context_create(), and the ast_context_add_*() calls themselves, or build the pval tree, and use the existing semantic checks/compiler to generate the dialplan.

And, I'm still thinking of implementing a check for Application calls, to make sure any typos of application commands are caught. It might even be nice to check the args to the application commands, as discussed earlier.

By: Steve Murphy (murf) 2005-12-30 00:11:29.000-0600

1. I've implemented the "return" keyword, so it does a goto to a NoOp at the the of the macro/extension.
2. I looked around for ugly, ugly function-rich variable expressions, and made an improvement to the ast_expr2 stuff that will allow it to handle the ${} better for the sake of syntax checking $[] expressions.
3. I checked 3 level deep for/while loops with continue, break, and return statements at each level. generated code looks good.
4. The vaiable init and increment in the for loop: I wrap the RHS of these assignments in $[].
5. Worked on translating the extensions.conf.sample file fully into extensions.ael.sample. I have some questions:

the Jump command. I see it implemented in the existing pbx_ael.c file. Why isn't it in the README? what EXACTLY is it supposed to do, that goto doesn't? Why isn't it documented in the Wiki? Is it something brand new?

lswitch. I've coded eswitch, and switch, but lswitch? It's in the extensions.conf.sample.

Context variables... What do we do with the [general] section? Turn it into a context? And how do we set the variables that are set there? The AEL syntax allows you to set globals, but you can't put variable initializations in a context, only in an extension.

Timed contexts-- I assume the time stuff is just part of the name...?

Application names-- at the time pbx_ael2 is loaded, few of the applications are registered, so calls to pbx_findapp(name) are useless. Looks like a list will have to be loaded. Is there any other way to get a list of the application names, before they are loaded?

Idea: it might be simple to allow extension generation, via a 'macro' language, and put that in a loop at the context level... That way, you could automatically generate extensions based on variables, etc.

By: Sergey Okhapkin (sokhapkin) 2005-12-31 08:46:16.000-0600

murf, see http://bugs.digium.com/view.php?id=5472 for the description of "jump" command (see the file attached to the bug). Also you parser is missed "pattern" keyword used in switch statement.

By: Steve Murphy (murf) 2006-01-01 01:10:55.000-0600

Thanks. I noticed also that the extensions.conf file features the

#include "filepath"

directive. It also goes on to boldly state that ALL config files supply this feature. OK. I implemented it in AEL2. Seems to work fine at the moment. I did not restrict its occurrence to the root level... it could be used almost anywhere in the file... anywhere but where $[] contents are being gathered, but the safest place to use them is at the root level, I'd imagine.

I also note that the current pbx_ael.so doesn't handle the #include directive yet. I assume it is intended to, one day...

I've also added code to detect and allow '1' as a priority label reference. The check will verify that the extension has at least one instruction in it. Seems better than forcing users to insert a "begin:" label on every reference. I can see why 'jump' is such a brainstorm.

Now, as the patterns of the "pattern" form of case statement: I plugged such a "pattern" into a test case, and ran it against the existing AEL compiler. It simply generated extensions with the pattern in the name.... along with the _ to indicate that a pattern matching extension name is present...

This:

       switch(${BOOBOP}) {
       case BUSY:
               Voicemail(b${ext});
               break;
       pattern 777[6-8]:
               NoOp(You called invalid ${EXTEN});
       default:
               Voicemail(u${ext});
       };

Generates these calls via the current pbx_ael.so:

Executed ast_add_extension2(con, rep=0, exten=s, priority=10, label=(null), callerid=(null), appl=Goto, data=sw-10-${BOOBOP}|1, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=s, priority=11, label=(null), callerid=(null), appl=NoOp, data=Finish switch-10, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=sw-10-BUSY, priority=1, label=(null), callerid=(null), appl=Voicemail, data=b${ext}, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=sw-10-BUSY, priority=2, label=(null), callerid=(null), appl=Goto, data=s|11, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=_sw-10-777[6-8], priority=1, label=(null), callerid=(null), appl=NoOp, data=You called invalid ${EXTEN}, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=_sw-10-777[6-8], priority=2, label=(null), callerid=(null), appl=Goto, data=sw-10-.|1, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=_sw-10-., priority=1, label=(null), callerid=(null), appl=Voicemail, data=u${ext}, FREE, registrar=pbx_ael);

[ hint: look for the "exten=_sw-10-777[6-8]" in the above jumble ]

While I see that the pattern inserts the '_' in the extension name, we could get the same affect without it, if there is any 'pattern' chars in the extension name... For instance, if there's a '.', a '!', or a '[', for sure it's a pattern. If there's an X,Z,or N, and the pattern isn't one of the result words returned by any of the apps, like USEREXIT, NOTFOUND, etc, then it's a pattern.
OR, maybe a better route is if the first char of the case var is an underscore, then we can force it forward... at any rate, these are fairly simple options...

Does anyone care? Currently, AEL2 now accepts these "patterns" with the case alone, and generates pretty much the same code...

This code, then, under pbx_ael2.so:

       switch(${DIALSTATUS}) {
       case BUSY:
               Voicemail(b${ext});
               break;
       case 777[6-8]:
               NoOp(You called invalid ${EXTEN});
       default:
               Voicemail(u${ext});
       };

Is then compiled, and generates these calls to ast_add_extension2 in AEL2:

Executed ast_add_extension2(con, rep=0, exten=s, priority=10, label=(null), callerid=(null), appl=Goto, data=sw-3-${DIALSTATUS}|1, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=s, priority=11, label=(null), callerid=(null), appl=NoOp, data=Finish switch-exten-gen-3, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=_sw-3-., priority=1, label=(null), callerid=(null), appl=Voicemail, data=u${ext}, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=_sw-3-777[6-8], priority=1, label=(null), callerid=(null), appl=NoOp, data=You called invalid ${EXTEN}, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-3-BUSY, priority=1, label=(null), callerid=(null), appl=Voicemail, data=b${ext}, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-3-BUSY, priority=2, label=(null), callerid=(null), appl=Goto, data=s|11, FREE, registrar=pbx_ael2);

Let me know what's best... it's best not to introduce big language changes from one version of a language to the next, but this stuff is "undocumented" so far (the bug fix to introduce this stuff into the readme was/is in limbo right now, it appears!). And, in my world, the documentation is like a contract. And if it's not in the contract, it's not in the language.


And as to the jump, it looks useful... I will code it soon. Coding the extensions.ael.sample to follow more closely what is done in extensions.conf.sample, has been quite revealing! extensions.conf.sample is kind of a demonstration of what you can do in the "assembly" code...

I've upgraded the semantic checker to check included contexts for goto targets, if they are not in the context specified.

Comments were giving me a headache, so I strip them in a 'pre-processor' phase.

I've obtained the coding standards, and have spruced up the source a bit, need to read it finer and clean things up one more time, and I'll upload the patch.

My biggest problem at the moment is that "default" is both a keyword in AEL, and a very popular context name! Sheeee- wizzzz!!!

My next goal is to come up with an application name list, and an arg description of all the applications. I will then be able to verify that application calls are to existing applications, and that the args are sane. I will also be able to  issue errors if a 'j' appears in any arglists, which would be real bad behavior for AEL code. Seems a lot of grief in programming the extensions comes from messing up the args to applications, especially the "Dial". I will also be able to critique the switch statements: are all the possible cases for a particular variable accounted for in the case/default statements? I don't have my notes right now, but it seems that there is one particular variable name used, that has different possible values, between different apps that set it... Ah, yes, SYSTEMSTATUS, when set by System(), can be set to: FAILURE or SUCCESS. But, TrySystem() sets SYSTEMSTATUS to FAILURE or SUCCESS or APPERROR... the only inconsistency I've noticed in my research. Not a big deal, but checking is easier when things are consistent, and so is programming, in the end...

By: Steve Murphy (murf) 2006-01-01 07:58:18.000-0600

After thinking more on pattern, I implemented it in the parser. Looking for pattern chars is foolish, the N is in all sorts of return results. The _ prefix might be OK, but could anyone guarantee that no possible pattern would ever want or need the '_' at the beginning? Since I can't say for sure, 'pattern' is now part of the AEL2 syntax.

By: Steve Murphy (murf) 2006-01-01 08:00:51.000-0600

Another note. Falling thru case statements: I had misinterpreted this previously. I got it into my head that control didn't actually fall thru from one case to another. But, I see where pbx_ael.so currently inserts a goto to the next case if there is no terminating goto, or break. I'll do the same, then.

By: Sergey Okhapkin (sokhapkin) 2006-01-01 09:50:59.000-0600

I see no reason in automatic parsing application names, arguments and return status. First, the applications could be loaded/unloaded dynamically after parsing AEL file, second, a lot of people use non-standard or home-brew applications the parser will have no knoweledge about.

I would suggest to compile AEL dialplan on module load the same way like now (without applications parsing) AND implement an additional CLI command "ael check" which will run all proposed tests and complain about all potential problems found.

By: Russell Bryant (russell) 2006-01-01 10:24:29.000-0600

As far as the [general] section goes, I would not worry about it.  I think it's fine to let those options stay in 'extensions.conf'.  I feel like it makes more since like that, anyway.

To expand upon the concerns about checking Application names and arguments, I would be worried about the maintainance required to keep this information up to date.  Options are added or changed on a pretty regular basis.  What were your thoughts about how you were going to store all of this information for your parser?

Given the concerns, I do think we could come up with a way to make this work.  There have been some talks about making some serious changes to the configuration parsing.  We have talked about moving information about available options into an XML file.  We would then have an external tool that would read this information, generate a header file with hashes for all of the options, generate sample configuration, etc.  (This part is the moral to this story -->) The reason we would do it that way is so the information about available options is easily available to other applications that would be interested in this information as well.

I have wanted to redo the storage of application descriptions but hadn't thought of a way to do it until just now.  What if we take a similar approach for all of the information about an application?  We could have a file (such as an XML file) that contains all of the information about an application - its description, and all of the possible options.  We would then write a tool to automatically generate documentation about applications in a *consistent* format for built-in documentation or other formats as well.   You could use the information from this file for AEL as well.

Thoughts?

By: Steve Murphy (murf) 2006-01-02 12:11:52.000-0600

Sokhapkin-- The checks on apps will issue only one kind of error, the use of a 'j' option. The rest will be warnings/notes. If the application arg description cannot be found, no checking will be done, beyond a note saying information on the command cannot be found (there might be a misspelling of the name)... and, of course, if an option is passed as a variable reference, we have to skip it, as variables won't be evaluated until run-time.

I do agree about allowing the checks to be done by a command, but I don't agree to not perform semantic checks by default. The whole idea here is to provide the user with information about their dialplan at compile time, that may lead to errors. If they want to ignore it, such is their privilege. But the basic principle here is that catching problems earlier (at compile time) is about an order of magnitude cheaper than catching them at run time.


Drumkilla--

As to providing a database of applications and associated arguments, I'll start with a file in /var/lib/asterisk, perhaps. I'm almost done with a simple parser for the information.

I'm halfway thru all the commands presented by "show applications". My initial approach will not be sufficient, I can see that now, but maybe could serve as a demo of what such a check could do. I'm realizing a linear description of args will not work. It's got to offer forks and dependencies. And some of the descriptions in the "show applications" are either wrong or ambiguous. Take the args for "read". If all the args so marked are truely optional, how can you tell timeout from attempts?  I see a lot of dependant optional items like [|options[|timeout[|...]]], but queue does a set of optional items if one optional item is provided. So, at the very minimum, a nested sort of syntax (which xml can provide) will be required. I will probably rework my parser to provide a more tree-like description.

And, of course, if you provide a language for providing application commands, you could also provide a compiler, and an API for accessing the arguments would be nice, like getopt, popt, or whatever.

Whatever you do or provide, will be a step up from a possibly buggy situation that now exists, where each app gets a raw string of all the arguments, which each application parses "by hand". It would be better/easier if a linear list of data structures were presented (a path down the argument "tree"), with fields that are already broken down from the raw input string, and any errors already reported. A simple set of funcs to access each item, and loop thru the list, would make option processing easier and less buggy, and more uniform in description.

And of course, the arg processor should be able to process unevaluated ${} stuff as part of the input.

I could then use this interface to access the application descriptions and compare them to the provided input, and issue warnings at compile time. That is, if the arg processor itself doesn't generate all the errors I'd be interested in showing!

By: Steve Murphy (murf) 2006-01-02 23:22:31.000-0600

Progress!

I've had the wrong idea about switch behavior; I got the impression from looking at generated code, that there was no "fall-thru" behavior, which on recent investigations, was false. So, I fixed my code, to supply the "fall-thru" feature.  Where there are no goto/return/break at the end of a case/default/pattern block, it will supply a goto to the either the next case statement block/exten, or if there are no following case/default/pattern blocks, then a goto to the end of the switch. All warnings about implicit returns have been removed.

I also added code, to supply any single-element goto's in a switch's case/default/pattern blocks with the extension of the location of the switch. There were notes about this in the wiki. So, if there's a "goto label" in a case statment, and the switch is in the "s" extension, then the goto is modified to "goto s|label".  So, to the user, the switch appears to be all in one extension.

I finished the application arg checking code, at least I put in a few elementary sanity checks. It will have to be re-written, but will serve as a good demo for the current moment. It reads in the file "applist", which parses, and I'll attach it to give you a taste of what I include in the data.

The "applist" file includes the variables that an application sets, and what values each variable can take. I intend to write some code to check switch variables case/pattern/default values against the vars that the applications set. Of course, warnings where switch statements don't cover all possible values of a known variable, can be generated. And, if a switch is written against a variable that isn't known, a note will be generated about that, just in case the variable name is a typo. And, of course, we can verify that if a known variable is used, that the app that sets it should be called in the same block, or a warning will be generated. Sometimes little oversights can be pointed out...

By: Steve Murphy (murf) 2006-01-04 22:09:56.000-0600

OK, I'm attaching my first "alpha" release of the AEL2 module.

The patch includes the following files:

Index: pbx/Makefile
Index: ast_expr2f.c
Index: ast_expr2.fl
Index: ast_expr2.c
Index: ast_expr2.h
Index: ast_expr2.y
Index: CREDITS
Index: configs/extensions.ael.sample
Index: utils/Makefile
Index: pbx/argdesc.tab.c
Index: pbx/argdesc.tab.h
Index: pbx/argdesc.l
Index: pbx/argdesc.y
Index: pbx/argdesc_lex.c
Index: pbx/pbx_ael2.c
Index: pbx/applist
Index: pbx/ael.y
Index: pbx/ael.flex
Index: pbx/ael.tab.c
Index: pbx/ael.tab.h
Index: pbx/ael_lex.c
Index: pbx/testfor.ael2
Index: utils/ael_main.c
Index: include/asterisk/argdesc.h
Index: include/asterisk/ael_structs.h
Index: doc/README.ael2
Index: flexpatch

All patches are made against svn head version 7771, so far.

The ast_expr2 files are patched to provide:
 1. fixes for all memory leaks.
 2. two added funcs for setting a phrase to include in any syntax
    error messages.
 3. The lexer has some added rules to handle unevaluated
    ${ } expressions in the input, so we can use the existing
    parser to check the validity of $[ ] expressions.

I submitted these patches under bug 6072. It's anyone's guess how
long it will take for the fixes to be merged into the release... hopefully,
not long. Until they are merged, you will have them here.

The argdesc* files are all having to do with reading in, or working
with, the "applist" file, with all the application information.

Want the Exact, fully elaborated BNF for AEL2? check out the file
pbx/ael.y; the scanner is in pbx/ael.flex. Don't like the complexity
of ael.flex? Then insist on non-contextual grammars in the future.

The flex scanner generator needs a patch to generate the _lex.c files
that are included here, so I include these flex output files in the patch,
so you don't have to mess with flex at all to build and test.
The fix to flex is simple, I have provided the file "flexpatch" for those of
masochistic tendencies, who want to patch the source to flex-2.5.31, and generate
their own ..._lex.c files. Have at it!

I also did the same for the bison input/output files, so you don't have to have
a bison-2.0 version floating around to test and build AEL2.

WARNING---WARNING---WARNING---WARNING---WARNING---WARNING---WARNING---WARNING---

This is an "alpha" version!!!!!!

Don't get all steamed up if it totally bombs with your dialplan!

Not one soul has provided any test cases. I've generated my own, and very few
of them at that. Report the bug. Include your dialplan file. I'll see what I can do.

Speaking of test data, I include the files "testfor.ael2", and "extensions.ael.sample".

Both these files should be acceptable input to both AEL and AEL2. In general, all AEL files SHOULD be acceptable input to AEL2, but because of the freer format, not all AEL2 files will be parsed by AEL. Caveat Emptor, etc., etc.

murf

By: Steve Murphy (murf) 2006-01-04 22:12:10.000-0600

Oh, and to the lords of the bug database: you can delete EVERTHING submitted before the AEL2.patch.bz2 file. Either it's in that patch file, or obsolete.
Thanks...!

By: Steve Murphy (murf) 2006-01-05 21:56:15.000-0600

OK, I've uploaded the AEL2v0.2patch.bz2 file. All previous files are obsolete.
I've fixed the command to install applist, to use the right variable.

I've added the time related functionality, you can use the ifTime() statement, which acts exactly like the if() statement, and also allows an "else" clause. The ifTime() expression is the 4 time fields (like GotoIfTime). You'd specify:

        ifTime(14:00-23:00|sat-sun|*|*) {
               BackGround(Hello);
        } else
               BackGround(Sorry);

And the other time-related feature I've added, is the ability to attach a time spec to the includes entries:

       includes {
               other|16:00-23:59|mon-fri|*|*;
       };

The time specification in both constructs are checked for the exact same problems as the asterisk code checks for. The time range has have two times, that are realistic 24-hour clock times. The day-of-week has to have the 3-digit day abbrevs that are embedded in the code. Same for the months field. The day-of-month field has to be a number from 1 to 31. Stuff like that. Any errors are logged as warnings, referencing the file, and line.

OK, can't think of anything useful to add at this point. Can anybody give it a spin and help me with criticism/advice?

By: Sergey Okhapkin (sokhapkin) 2006-01-06 05:12:21.000-0600

Cool! Now I see only 2 features of extensions.conf not supported by AEL2 - hints and an ability to start the extension definition with a priority other than 1 (it's useful in conjunction with regcontext/regexten parameters in sip.conf and iax.conf).

AEL extension definition syntax is

 pattern => list of operators;

I would propose to extend the syntax (keeping it AEL compatible) to

 [regexten][hint(channel)] pattern => list of operators;

with the following semantics:
- if "regexten" keyword is present, compile the list of operators as usual, but set start priority to 2 instead of 1.
- if "hint(channel)" keyword and parameter are specified, compile the list of operators as usual but emit "pattern,hint,channel" do dialplan.

Here are some examples:

123 => Noop; compiles to 123,1,Noop

regexten 123 =>Noop; compiles to 123,2,Noop

hint(SIP/1) 123 => Noop; compiles to
123,hint,SIP/1
123,1,Noop

regexten hint(SIP/1) 123 => Noop; compiles to
123,hint,SIP/1
123,2,Noop

What to you think about?

By: Steve Murphy (murf) 2006-01-06 09:55:42.000-0600

Sokhapkin-- no problem. I've just attached the v0.3 patch. It includes the syntax you suggested. I also loosened up the if() statements a little.

Here's from the ael2 README:

14. There is no restriction about using {}'s in the if or ifTime statements. In
   other words, you can say this:

               ifTime(14:00-25:00|sat-sun|*|*)
                       BackGround(Hello);
               else
                       BackGround(Sorry);

   (you don't have to wrap a single "true" statement in curly braces, as in the
   orignal ael). This creates a conflict in the grammar, but the parser
   "shifts" in this case, and the result is that an "else" is attached to the
   closest if.
   So, stating something like this:

               ifTime(14:00-23:00|sat-sun|*|*)
                       ifTime(18:00-23:00|sat-sun|*|*)
                           BackGround(Hello);
                       else
                           BackGround(Sorry);

       results in the else being associated with the second "ifTime". As usual,
       be careful about nested if statements! When in doubt, use curlies!

15. Added the syntax [regexten] [hint(channel)] to preceed an extension
   declaration.
   You can now say things like this:
                               regexten hint(SIP/1) 123 => {
                                               NoOp(hello there);
                               };

   The regexten keyword will cause the priorities in the extension to begin
   with 2 instead of 1. The hint keyword will cause its arguments to be
   inserted in the extension under the hint priority. They are both optional,
   of course,
   but the order is fixed at the moment-- the regexten must come before the
   hint.



By: Steve Murphy (murf) 2006-01-06 10:09:02.000-0600

Hey, you "lords of the bug database", tell me, would it spoil some vast, eternal plan, if you allowed the Reporter of a bug to delete attachments they uploaded?

By: Steve Murphy (murf) 2006-01-06 14:11:26.000-0600

Just added info about this project to:

http://www.voip-info.org/wiki/view/Asterisk+AEL2

By: capouch (capouch) 2006-01-07 02:37:12.000-0600

I just built this from the latest patchfile above.  I'm running this on a MIPSEL box, if that makes any difference.

I cut and pasted the example file from the wiki into extensions.ael2

The demo context doesn't get merged.  Here are the errors it throws:

CLI> reload pbx_ael2.so
   -- Reloading module 'pbx_ael2.so' (Asterisk Extension Language Compiler v2)Jan  7 04:20:47 NOTICE[13701]: pbx_ael2.c:2817 pbx_load_module: Starting AEL2 load process.
Jan  7 04:20:47 NOTICE[13701]: pbx_ael2.c:2824 pbx_load_module: AEL2 load process: calculated config file name '/etc/asterisk/extensions.ael2'.
Jan  7 04:20:47 NOTICE[13701]: ael.flex:538 ael2_parse: Removed 0 comments
Jan  7 04:20:47 ERROR[13701]: ael.y:426 ael_yyerror: ==== File: /etc/asterisk/extensions.ael2, Line 44, Cols: 24-24: Error: syntax error, unexpected EQ, expecting SEMI
Jan  7 04:20:47 ERROR[13701]: ael.flex:290 ael_yylex: File=/etc/asterisk/extensions.ael2, line=53, column=33: Mismatched '}' in expression!
Jan  7 04:20:47 ERROR[13701]: ael.flex:290 ael_yylex: File=/etc/asterisk/extensions.ael2, line=79, column=9: Mismatched '}' in expression!
Jan  7 04:20:47 NOTICE[13701]: pbx_ael2.c:2827 pbx_load_module: AEL2 load process: parsed config file name '/etc/asterisk/extensions.ael2'.
Jan  7 04:20:47 ERROR[13701]: argdesc.l:66 argdesc_parse: File /usr/lib/asterisk/applist could not be opened
Jan  7 04:20:47 NOTICE[13701]: pbx_ael2.c:2830 pbx_load_module: AEL2 load process: checked config file name '/etc/asterisk/extensions.ael2'.
Jan  7 04:20:47 NOTICE[13701]: pbx_ael2.c:2832 pbx_load_module: AEL2 load process: compiled config file name '/etc/asterisk/extensions.ael2'.
Jan  7 04:20:47 WARNING[13701]: pbx.c:3674 ast_merge_contexts_and_delete: Requested contexts didn't get merged
Jan  7 04:20:47 NOTICE[13701]: pbx_ael2.c:2835 pbx_load_module: AEL2 load process: merged config file name '/etc/asterisk/extensions.ael2'.
Jan  7 04:20:47 NOTICE[13701]: pbx_ael2.c:2838 pbx_load_module: AEL2 load process: verified config file name '/etc/asterisk/extensions.ael2'.

Just in case, here are the contents of extensions.ael2:

context demo {
   s => {
        Wait(1);
        Answer();
        TIMEOUT(digit)=5;
        TIMEOUT(response)=10;
restart:
        Background(demo-congrats);
instructions:
        for (x=0; ${x} < 3; x=${x} + 1) {
             Background(demo-instruct);
             WaitExten();
        };
   };
   2 => {
        Background(demo-moreinfo);
        goto s|instructions;
   };
   3 => {
        LANGUAGE()=fr;
        goto s|restart;
   };
   500 => {
        Dial(IAX2/guest@misery.digium.com);
        goto s|instructions;
   };
   600 => {
        Playback(demo-echotest);
        Echo();
        Playback(demo-echodone);
        goto s|instructions;
   };
   # => {
hangup:
        Playback(demo-thanks);
        Hangup();
   };
   t => goto #|hangup;
   i => Playback(invalid);
};

By: Steve Murphy (murf) 2006-01-07 09:36:59.000-0600

capouch-- Two problems overlapped here. First, the line counters don't reset after the first load of pbx_ael2.so; I've fixed this. Second, the parser sees

TIMEOUT(digit)=5;

and thinks it's an application call with an erroneous "=5" pasted onto the end. I'm still thinking about how to best handle this. Right for the moment, you'd best wrap variable assignments like this involving function calls, with a Set() application call. I've updated the configs/extensions.ael.sample already (you have it in the patch), but I didn't spot this in the wiki... Let me consider how I can make this work without resorting to Set().

By: Steve Murphy (murf) 2006-01-07 13:30:19.000-0600

[added 0.4 patch]

Ok, I've entered the syntax of func(args)=val; as a statement type in the syntax,
and the wiki stuff should be OK to parse.

I also added some fixes to the application check, it was going wild on options.

I added code reset the error counters, but now I crash asterisk on reload. If you have the same prob, use restart until I get a handle on the problem.

By: Steve Murphy (murf) 2006-01-08 22:13:57.000-0600

Just added the 0.5 patch. Main addition: upgraded both the AEL2 and ast_expr2 parsers with code to transform the syntax error messages, which can contain cryptic refrences to TOK_COLON, or AMPER, or RP, which are internal token names.
Now, in the text of syntax error messages, instead of seeing TOK_COLON, you'll see ':' instead. Instead of AMPER, you'll see '&'; instead of RP, you'll see ')', and so on. I thought the improvement in clarity would justify a new patch.

Also-- I added a check that will issue a warning should the user needlessly wrap an expression in $[ ]. And another warning that issues a warning if an expression has operators (like +,-,~,!, etc), but no ${var} references. It is possible that users might forget to wrap a variable name in ${}, or to do a $var instead, and think they will be OK, when they will not be. I see some instances where the message is not very useful, but think it is well worth the inclusion.

Anybody see other subtle/obvious pitfalls I should check for? What kind of mistakes have you guys made in your dialplans, that took more than a minute to spot and fix?

Or do you guys ENJOY answering endless newby requests for help involving these same errors?

The time will soon come, where it will be everybody's advantage to strongly discourage new users from touching extensions.conf files; they should only be playing with AEL2. This code needs to be more solid. Find me problems!

By: Steve Murphy (murf) 2006-01-08 22:18:48.000-0600

Another note on crashing asterisk by doing an "ael2 reload".

It only happens when I issue the command from an "asterisk -r" process. If I issue the "ael2 reload" from a console (asterisk -cgvv), there is no problem. I've done it several times with no problems.

My code is malloc-free perfect, as far as testing shows. No leaked memory. No accesses to non-initialized memory. No array bounds violations. At least, so says purify, in the standalone stuff... anybody got any ideas? I'll keep plugging away...

By: Steve Murphy (murf) 2006-01-09 15:23:25.000-0600

Just uploaded 0.6 patch...

This one had a restriction, that case statements couldn't be empty. This
restriction is removed now. And, unlike the current AEL, won't silently
drop the empty cases.

And, I added code, such that a trailing label automatically will have a NoOp instruction inserted after it.

This code:

context something {
       somewhere =>
       {
               switch(${DIALSTATUS})
               {
               case BUSY:
                       NoOp(wow);
               case NOANSWER:
               case TORTURE2:
               case TORTURE3:
               case TORTURE:
                       NoOp(woow);
               };
               NoOp(woohoo);
       TrailingLabel:
       };
};

Turns into this with the "original" AEL compiler:

Executed ast_context_create(conts, name=something, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=somewhere, priority=1, label=(null), callerid=(null), appl=Goto, data=sw-1-${DIALSTATUS}|1, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=somewhere, priority=2, label=(null), callerid=(null), appl=NoOp, data=Finish switch-1, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=sw-1-BUSY, priority=1, label=(null), callerid=(null), appl=NoOp, data=wow, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=sw-1-BUSY, priority=2, label=(null), callerid=(null), appl=Goto, data=sw-1-NOANSWER|1, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=sw-1-NOANSWER, priority=1, label=(null), callerid=(null), appl=case, data=woow, FREE, registrar=pbx_ael);
Executed ast_add_extension2(con, rep=0, exten=somewhere, priority=3, label=(null), callerid=(null), appl=NoOp, data=woohoo, FREE, registrar=pbx_ael);


The same code, with AEL2, emits:

Executed ast_context_create(conts, name=something, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=somewhere, priority=1, label=(null), callerid=(null), appl=Goto, data=sw-3-${DIALSTATUS}|1, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=somewhere, priority=2, label=(null), callerid=(null), appl=NoOp, data=Finish switch-something-3, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=somewhere, priority=3, label=(null), callerid=(null), appl=NoOp, data=woohoo, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=somewhere, priority=4, label=TrailingLabel, callerid=(null), appl=NoOp, data=A NoOp to follow a trailing label TrailingLabel, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-3-TORTURE, priority=1, label=(null), callerid=(null), appl=NoOp, data=woow, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-3-TORTURE, priority=2, label=(null), callerid=(null), appl=Goto, data=somewhere|2, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-3-TORTURE3, priority=1, label=(null), callerid=(null), appl=Goto, data=sw-3-TORTURE|1, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-3-TORTURE2, priority=1, label=(null), callerid=(null), appl=Goto, data=sw-3-TORTURE3|1, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-3-NOANSWER, priority=1, label=(null), callerid=(null), appl=Goto, data=sw-3-TORTURE2|1, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-3-BUSY, priority=1, label=(null), callerid=(null), appl=NoOp, data=wow, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-3-BUSY, priority=2, label=(null), callerid=(null), appl=Goto, data=sw-3-NOANSWER|1, FREE, registrar=pbx_ael2);

By: Steve Murphy (murf) 2006-01-10 09:24:15.000-0600

Just uploaded the 0.7 patch.

First of all, fixed a crash caused by using the wrong index, in the code that tries to prevent infinite recursion in #include "" directives.

Next, I expanded the syntax to allow the use of "default" as a context name. Since "default" is a keyword, this involved extra rules for context, goto targets, jump targets, and includes { } entries.

I also fixed a typo in the configs/extensions.ael.example, where it included the wrong context (demo vs ael-demo). None of this should affect the wiki page.

By: Steve Murphy (murf) 2006-01-10 09:47:12.000-0600

OK, guys, I don't FORESEE any more language syntax additions, unless it's to provide similar capability to the current AEL compiler. (why does this remind me of the Monte Python phrase, "no-one EXPECTED the Spanish Inquisition"?)

Anyway, all of us need to cast our eyes to the future (figuratively, of course!), and make some decisions now on certain issues, so they can be accounted for and things done in proper order. For instance:

1. Will AEL2 forever remain AEL2 and confuse users as to which to use? Provide two  maintenance paths? Or, is it possible that AEL2 will someday replace AEL as it now exists?
2. If AEL2 will replace AEL, when? In what future release? And, will it coexist before then with AEL in what release(s)?

While the AEL as it exists is a marvelous implementation, and a benchmark to compare against, and was a great vehicle to show us what AEL looks like, and does, it has a series of "built-in" problems, that may or may not be worth repairing... this is up to those who wrote it and have been maintaining it, to judge. At this point, because of its silence in reporting problems, and dropping input at times, I really can't refer new users to it, as it stands. It wouldn't necessarily make their lives easier.

If you/they think you/they can duplicate what I've done so far, and thereby make AEL2 useless, and think the current parser a better approach, then I'll close this bug, and we'll let the issue drop here. (pardon me if I use my copy privately until you/they do!).

If you want to move to AEL2, we/I need to make some changes, to docs, etc, to prepare for the move. It's up to you. If there's no interest, and nobody reads this stuff, I'll stop submitting patches. Edison swore he'd never invent something nobody would use, and I feel the same way about software. Right now, I'm doing it for me, mostly, because if I'm going to work with dialplans, I didn't want to waste time and effort on finding bugs at runtime. The rest I did as a "service project", because it looked like it was needed.

So tell me, and don't be afraid to be brutal. What should I do?

By: Beau Hargis (beaufrog) 2006-01-11 16:38:33.000-0600

I have been using AEL for a while now, trying to get it to work and sometimes it does. I submitted a patch or two, but decided that AEL needed a new parser all around. I am glad you did one because I started it, but got swamped with other things. I have dug around in the source for a while and have tried to used it so I will offer a couple of opinions.

The semi-colon at the end of braces needs to go. It's pointless, adds nothing to the structure of the language itself, even caused weird problems in the original parser and apparently makes this new grammer more complicated. Get rid of it. If getting rid of it makes the grammar simpler, then do it.

Comments... This has been a serious problem for me when wanting to use the CURL fuinction or app. '//' is part of the url, but AEL treats it as a comment, and escaping only works some of the time. Since nobody seems to have decided that quotes are necessary to enclose strings and such, comments should only be allowed on a line by itself, after a semi-colon ending a statement, or after a closing or opening brace for a block. Start conditions in the lexer should be able to facilitate that.

I am going to try the AEL2 with some of the stuff I have and see how I can break it.

By: Steve Murphy (murf) 2006-01-11 17:15:46.000-0600

I've uploaded the 0.8 patch-- a few fixes generated from my own testing...

1. A rule to allow * instead of clock times in the ifTimes and includes context time notation.
2. The line count was getting thrown off in multiline conditional expressions in an if() expression. fixed.
3. Added the paren checking to to $[] strings terminated by a semicolon. The gatherer will stop when an unbalanced closing bracket or semicolon is found. These can take in large chunks of the file if they go wild, hopefully this will reduce the damage.

beaufrog --

OK, that's it. Your the third person who hated the semicolons after '}'. Consider them gone. I hope tomorrow to make a patch upload without them. I'll have to rework the wiki examples and the README, and the config/ samples. Such is life.

Maybe, we could do /* comments */ instead. That way, they can be stripped by the scanner instead of pre-processed out of the input. And I won't have to set up ugly conditions on where comments can exist. That would change a lot of example code, but heck...

Please, have at the parser. It's starting (a little) to look more solid. My dialplan is in it, and I've been trying to make it behave better on syntax errors.

By: Steve Murphy (murf) 2006-01-12 08:04:38.000-0600

Uploaded the 0.9 patch.

1. The ';' after a closing brace '}' is now optional. All extraneous ';' at ALMOST every level will be silently ignored. Consider the ';' after '}' obsolesced. At some point in the future, we might do away with them entirely. All AEL2 docs will stop mentioning them, or using them. Users will not have to edit them out of .ael files to make them .ael2 files, at the moment.

2. Comments-- // is still used as comments, but comments are not recognized inside areas where $[ ] strings are gathered, nor in application call arguments. This way, we get rid of a situation where //'s are removed unexpectedly, as part of the arguments to the CURL or AGI applications. Comments will not be recognized and removed in assignment statements, before the terminating semicolon. So, AGI(agi://xxxx) will not have to have the // escaped. A problem could occur, if the // is the first two chars inside an application call...
let me know.

And, the above two changes chopped out some duplicate code, and a few functions, and hopefully saved a bit of processing time! I wish all minor corrections could do that!

By: Matt O'Gorman (mogorman) 2006-01-12 08:48:25.000-0600

murf you rock my world ^_^

By: Steve Murphy (murf) 2006-01-12 09:02:54.000-0600

mogorman-- thanks! Anything else I can do to, er, I mean 'for', you, let me know!

I forgot to mention I also included a little code in the #include "" stuff, that inserts the CONFIG_DIR, if the path is not absolute. That way, you can

                   #include "roto-rooter.ael2"

and it will find it in /etc/asterisk, if that's your config file dir.

Uh, does this then complete all the wishes anyone ever had? I mean, excluding things like millions of $$, etc?

By: Steve Murphy (murf) 2006-01-12 15:23:15.000-0600

Uploaded patch 0.10.

I apologize for the constant stream of updates.

I found a stupid omission of an index increment in a while loop, causes an infinite loop in the code that checks for a 'j' option. I fixed this. Sorry, if anybody fell into that hole.

I added ONE MORE CONSTRUCT. I noted that the Random() construct provides some flow of control, and thought it might be useful to have a construct for it in AEL2. So, I introduce the random(<int>) <statement> [ else <statement> ]  construct. It works like "if" does, and provides a "true" and "false" part, where the expression is meant to resolve to an integer between 0 and 100, which determines the probability of the "true" statements getting executed. The false statement, if present,  will get executed (100-<int>) percent of the time. See the Random() function, which is used underneath, for details. While a random number will determine whether the "if" or "else" statements is run, either one or the other will be executed every time flow of control comes thru the 'random' block.

By: Steve Murphy (murf) 2006-01-12 21:06:00.000-0600

I've been thinking on this issue for a while, and have meant to remark on it earlier, but I've seen some complaints about the lack of double-quotes to wrap strings.

And I may not be real popular for saying this, but AEL does delineated its strings. Maybe there's a few loopholes in the logic, but for the most part, it has been, and does work. Basically, the args to applications are delineated by the commas or vertical bars, and the opening and closing parenthesis.

As the parser writer, I curse and mutter about the fact that contextual lexical token forming changes substantially from one area to the next, but really, at the grammar level, you can tell exactly when to make the shift, and to what. The fact I had so much trouble moving from one context to another, is due more to the fact that bison and flex don't REALLY interoperate at a fine-grained level. Someday, the two camps will join heads, and smooth out the problems...

I hope.

Anyway, I'm really dry now. I can't think of anything more to add to AEL2. I need as many folks as possible to sit down, build it on their machine, and either invent an AEL2 dialplan, or convert their extensions.conf to an AEL2 format, and use the aelparse to debug it.

I'd love to hear what problems you encountered, how you found them, how you fixed them, and what maybe AEL2 could do, or could have done, to save you some pain and agony.

By: Leif Madsen (lmadsen) 2006-01-12 21:27:31.000-0600

murf, this has got to be my favourite bug of all time. Keep up the good work!

By: Steve Murphy (murf) 2006-01-14 16:48:18.000-0600

Thanks, blitzrage! I hope not to do anything that will lower me in your ratings!

I have given the Wiki some attention, and found the Function descriptions very weak, so I went to my late-model asterisk, and did a "show functions", and updated the wiki to describe them all. Then I came back to AEL2 and added the new functions to my internal list (now at 47 functions, I think). That will show up in v0.11, which I will upload when I solve some bug or problem. I also looked at and updated the wiki pages for Expressions, which was a little out of date, and checked the Variables page. I added links to the AEL2 wiki page to these other pages. I expanded the AEL2 page to include a little explanation about the basic structure of AEL2, and explained that the user programming in AEL2 has to know the syntax for Expressions. that is the $[...] format, and also has to know the format for the variable references, that is the ${...} stuff.

And I added a short blurb about the fundamental objects in AEL2, like contexts, macros, extensions, etc.

Hopefully, the Wiki additions will help answer newcomer's questions before they are asked. Kind of sad, because over 1200 page hits on that page in just over a week, and all those folks won't have seen all the neat new stuff.

By: pjc (pjc) 2006-01-14 17:04:41.000-0600

Let me start off by saying that I *really* like AEL2.  The error checking itself is one heck of a relief, to say nothing of the other features added.

I have to say, I'm one of the folks in the quoted-strings camp, although I've got this inkling that supporting them properly might be problematic at this point.  Things like this, though, make me think that eventual quoted-strings support would  result in less surprising results for things like this:

 GROUP(callcount)=vpconnect-t02;

This results in a warning that there are operators in the expression, but no variables.  Sure, my fault for putting a dash in the string, but it's a valid context name, and it's easily worked around by wrapping it in a Set().

More surprising, though, is this:

 intextmap=${DB(intextmap/${CALLERIDNUM})};

Which, when the key in question doesn't exist in the database, assigns 0 (zero) to intextmap.  Again, the fix is to wrap it in Set().  This is more an issue with the expression parser than AEL2, I suspect, simply assigning one variable to another blank variable also results in an assignment of zero, rather than the empty string, and yields a complaint of an unexpected $end from the expression parser.

I've noticed a few quirks of arg checking (although I really like the feature, all the same).

Using a comma in the args to NoOp results in a complaint of too many arguments; LookupBlacklist(); and NoOp(); both result in coredumps, apparently because when it tries to print the error message complaining about the arguments, it finds out the hard way that the arglist is NULL, not empty.

LookupBlacklist() with no args is, in fact, valid, as is NoOp(), although I no longer have a reason to use the latter, since I don't have to worry about line numbering.  The old extensions.conf may remind you of assembler, but it reminds *me* of BASIC. :)

It might be nice to be able to define an argument to a macro as optional; currently, I've got a couple macros that rely on the fact that macro arguments that are omitted are simply left unassigned.  Leaving them blank with Foo(arg,) yields a complaint of a ) found where a word was expected, so I've taken to feeding them bogus variables (${BLANK}) to work around this.

Along a similar vein, things like Dial(Device/Number,,ARG) also result in a similar complaint.  Spaces for the arguments in such cases result in a space being passed for the argument in question; sometimes that's okay, and sometimes it confuses things.

The following context causes aelparse to coredump -- I think it has something to do with a return inside of a switch statement.

macro testdial(number, timeout) {
   Dial(IAX2/vpconnect-t02/${number},${timeout},${OG_DIAL_FLAGS});
   switch (${DIALSTATUS}) {
   case CHANUNAVAIL:
       goto dial-trunk2;
break;
   default:
       NoOp(t02 Unavailable - ${DIALSTATUS});
return;
   }

dial-trunk2:
   Dial(IAX2/vpconnect-t01/${number},${timeout},${OG_DIAL_FLAGS});

}


The goto target checking appears to not recognize extensions when they're "sister" includes, so to speak:

context test {
       includes {
test1;
test2;
}
1003 => {
NoOp(test);
}
}
context test1 {
1001 => {
goto 1002|1;
}
}
context test2 {
1002 => {
NoOp(test);

}
}


Nor does it recognize patterns as a target for a goto:

context test {
_10XX. => {
NoOp(bah);
}
1100 => {
goto 1001|1;
}
}

In both cases, sneaking a variable in the label seems to make it happy.  The fact that I even found the two previous cases could be thought of as evidence of ugly dialplan design, to which I can only say... mea culpa.

All in all, these problems are a lot more trivial than the ones I saw when I played with the original AEL, and figuring out what was going wrong and how I could work around it was *much* easier with AEL2 than the original.

Thank you!

By: Steve Murphy (murf) 2006-01-14 22:26:07.000-0600

PJC--

Fantastic! Many thanks!

I've quickly reproduced, found and fixed the crashes:
1. the return was the culprit--I was lacking code to implement them. I should have caught this in a test case, but I failed to make a test case with return in switches. Right now, I did the easy thing-- the return lets you step off the end of the case/pattern/default extension. I figured stepping of the end of the case extension is just as good as stepping off the end of the extension the switch is declared in.

2. Crash on empty arglist-- I was using the arglist for the file and line numbers. Now I pass in both the appcall node, and its arglist, and use the appcall info if the arglist is null.


-- I can see that my concept of finding goto targets at build time was a bit naive. I'll be thinking about either accounting for the patterns in the search, or just flunking out of the search if patterns are spotted. Generalized pattern matching code, anyone? And while I took into account following the chain of context includes upwards, but the "sister" goto problem you present is fascinating. At worst, I could just downgrade goto reference problems to warnings instead of error. But it occurred to me that could introduce another language feature, that would drive everyone nuts: the concept of "abstract" contexts, that exist only to be included in other contexts. I'd skip trying to resolve goto's in such contexts, and it'd shut off the error messages. Heh, heh, wouldn't everybody love that?  I could even generate warnings about abstract contexts not included by any other context, in such an instance. The ol' unreferenced var...

The applist entries for lookupblacklist and noop were updated to make the one arg optional.

These issues yet remain, and I will work over the weekend to resolve them:
1. empty args in app call: Dial(Device/Number,,ARG)
2. multiple args because of comma in NoOp.
3. Macro calls with missing or null args OK... Foo(arg,) with possible addition of optional intention in the macro declaration.
4. Handle interesting possibilities of goto target matching.
5. Assignment of an empty string via $[]. The more I think of this, the more I want to reach down into the guts of ast_expr2.* and untwist it, and make it do the right, compatible thing. I spotted this behavior myself, and made a note in the Wiki. This isn't sufficient. People are expecting it to the do the right thing, and pass along the null value. And the fact that in current dialplans, people don't hand it stuff like this, because they get syntax errors when they do, or 0's... Hmmm.

More to come. Hang tight.





--

By: Steve Murphy (murf) 2006-01-16 12:00:47.000-0600

I uploaded the 0.11 patch.

It has a bunch of small tweaks, everyone (hopefully) will like.

1. First of all, the crashes pjc reported are fixed.
2. The syntax allows empty args in macro and Application calls (eg. Hello(,,j)
  -- the semantic checks won't complain about empty options, but will still
     complain about not enough, if you leave out the commas.
3. The applist was updated to have a ... entry for the NoOp args.
  Oops! No! Sorry,
  I'll include this in the 0.12 version.
4. The applist was updated with an OPTIONAL options entry for the
  LookupBlacklist entry.
5. The ast_expr2 stuff was updated to yield an empty string, if such was given
  to it in the first place. This will fix the DB() call behavior.
6. The ast_expr2 stuff was updated to take && as an alias of &, and || as an
  alias of |. and == as un alias of '=', to give the C/java/C++/whatever
  hackers some familiar operators. Doesn't look like the ast_expr2 stuff will
  ever be used to do assignments or bitwise operations, so this should OK.
7. Added the keyword "abstract", which can optionally precede the keyword
  "context" in an AEL2 file. Basically, all it does at the moment is prevent
  goto checking in that context. It's meant to express the intent that "this
  context is never used 'standalone'. It is purely here to be included in
  contexts.
8. The goto resolver/checker now can match against pattern extension
  names.

These items cover all of pjc's issues (I believe), except for #4. I'm still looking over the code, and may segment it better, but it seems clear I'll have to recursively descend included contexts, looking for goto's to resolve, and also do the same to resolve any of the goto's found. Shouldn't be too hard, but I think I'd best reorg the code I have to make this a bit clearer, and easier to follow. So, give me a few days to ponder the best approach.  In the meantime, you can declare any contexts with 'sister' extension goto's to be abstract, and get around the check. Right as it is, you just won't get an error/warning message about the goto targets. That shouldn't be a show-stopper by any means.

I'm still considering declaring macro arguments as "optional"... how best to do that for the best effect. In the meantime, just toss in enough commas to silence the check.

By: Steve Murphy (murf) 2006-01-18 17:01:04.000-0600

Uploaded the v0.12 patch.

This completes (except for optional macro args) all the items (as far as I could make out) that pjc brought up. The goto checker matches extension patterns, and the goto's are found via recursion down the includes. The check of each goto found is made by recursive search thru the inclusions. Seems to work for me. Y'all give it a spin, and see if it correctly finds any bad "sister" references.

I got the NoOp ... entry in the applist, the one I missed in .11.

I added a check for abstract contexts: is any context referencing them in the includes? If not, a warning is generated.

Also added a duplicate context name check. Prints a warning. Shows the file, and line(s) of both contexts, so you can find them easily. Hopefully, these will be flagged when you try to create a duplicate context. My check will find it a little earlier.

Any other possible problems or pitfalls that could checked for by the compiler? Let me know!

By: Steve Murphy (murf) 2006-01-19 07:09:25.000-0600

Last night, I put the AEL2 on a trunk copy of svn, at

asterisk/team/murf/AEL2

for those who don't like playing with patches.

By: Steve Murphy (murf) 2006-01-20 07:54:43.000-0600

Call For Testing!!!

C'mon, dudes! There's been over 1900 page hits on the AEL2 wiki so far, and it may top 2000 today. That's in 2 weeks of existence, roughly about 140 hits per day average. There must be interest on someone's part...

Build this, give it a spin! Tell me your joys/sorrows/angers/confusions. pjc was a great example!

I haven't put up a patch yet, but I've committed some changed to the ast_expr2 stuff (again) in AEL2's SVN tree. Built a test-harness for ast_expr2, standalone, and pumped in about a hundred test cases. Found some bugs with the 1.2 version. Fed the fixes back into the trunk version on AEL2.

By: Sergey Okhapkin (sokhapkin) 2006-01-24 09:39:35.000-0600

AEL allows to skip parens in application call without arguments like

Answer;

but AEL2 expects "Answer();" and generates an error:

LOG: lev:4 file:ael.y  line:663 func: ael_yyerror  ==== File: /etc/asterisk/extensions.ael2, Line 26, Cols: 14-14: Error: syntax error, unexpected ';', expecting '('

By: Sergey Okhapkin (sokhapkin) 2006-01-24 09:45:48.000-0600

What these errors could mean?

84        if("${prov}" = "${CPACKET1}" | "${prov}" = "${CPACKET2}") {                  
85                Set(MINUTES_LIMIT=${DB(Provider/${prov}/limit)});                    
86                &announce_minutes($[${MINUTES_LIMIT} - ${MINUTES_USED}]);            
87        };                                                                            
88        DeadAGI(dial.agi,${proto}/${ext}@${prov},${arg1},${arg2},${arg3},${arg4});    

LOG: lev:4 file:ael.flex  line:136 func: ael_yylex  File=/etc/asterisk/extensions.ael2, line=84, column=54: Mismatched ')' in expression!
LOG: lev:4 file:ael.y  line:663 func: ael_yyerror  ==== File: /etc/asterisk/extensions.ael2, Line 84, Cols: 55-55: Error: syntax error, unexpected '{', expecting ')'
LOG: lev:4 file:ael.y  line:663 func: ael_yyerror  ==== File: /etc/asterisk/extensions.ael2, Line 88, Cols: 81-81: Error: syntax error, unexpected ';', expecting '{'



By: Steve Murphy (murf) 2006-01-24 20:02:15.000-0600

sokhapkin --

The shortcut of making an appcall with no args, with no parens, either, was pretty trivial. I added the entry to the syntax. Will test b-4 committing tonight.

syntax errors: the statements you sent look clean, and indeed, if I paste them into a file, they yield no syntax errors. It's common to have "cascading" errors in this situation. I do careful bracket checking, but a missing closing bracket (paren, brace, etc), can propagate into innocent code. You might look backwards up thru the code and see if there are any unbalanced parens/bracks/braces anywhere before those lines. If you are really stumped, show it in a note, I'll help find the problem.



By: Steve Murphy (murf) 2006-01-28 22:03:34.000-0600

Sergey--

OK, I've found the problem causing the erroneous syntax errors. The fix has been committed to the teams/murf/AEL2 tree. It seems to work on your example. A glaring typo, with subtle effects!

Many thanks for me find this problem!

By: Steve Murphy (murf) 2006-01-28 22:28:18.000-0600

Just uploaded patch 0.13... See above for the things I added.
This patch is simply a diff between the trunk, and the teams/murf/AEL2 branch,
with some hand-editing to get rid of extra files thrown in by automerge or whatever.

By: pjc (pjc) 2006-01-29 03:27:26.000-0600

I've been up for far too long poking at this, so take this with a grain of salt, but I think I may have found a lead on the crash upon reload problem.

I threw gdb at asterisk, and did quite a bit of headscratching when it started claiming that the segfault was in pbx_ael2.c:2454 -- the very start of gen_prios().  I started wondering what goes on at the start, before the code, started wondering about automatic variable allocation and initialization, and, after initially thinking, "naaaah, that can't be it," went ahead and cranked the size of buf1, buf2, and new_label down considerably (to 8000, 8000, and 4096)... and it seems to reload nicely now.

The backtrace always seemed to be a few levels deep when it crashed, and just like you said earlier, it only seemed to crash on a remote connection.  I'm wondering if the overhead of that remote connection plus the parsing was just enough to push something over the edge.

I'm thinking if you wanted to keep the large buffers, using malloc might avoid allocating from the stack and give more breathing room for convoluted dialplans.  I can provide a patch against svn if you like, but what I've got right now is probably a little cheesy. :)

By: pjc (pjc) 2006-01-29 04:41:55.000-0600

I take this partially back -- although it reloads without crashing, it's not quite right, either.  It's only looking at the first include file, and ignoring each additional one.  Unloading and loading pbx_ael2.so seems to work fine, though.

By: Steve Murphy (murf) 2006-01-29 15:55:11.000-0600

pjc--

Great detective work! Even at 4k, mine still crashed. But it "seemed" to get further, as it is dying in the middle of a big if-else-else-else-... chain, which causes func calls maybe 20 levels deep or so... so I reduced ALL buffer sizes in pbx_ael2.c to 2K (2000) and, yes, my whole dialplan loaded without crashing asterisk from an "asterisk -r" command line "ael2 reload" command!

I've committed the 2k changes. I don't think it'll hurt anything. But it raises in me big questions about WHY this happening? Why is stack space so critical when the reload command comes from (asterisk -r) vs. an (asterisk -cgvvvvvvvvvvvvvvvvvvv)?

When issued from an (asterisk -r), the "ael2 reload" command causes a crash in the asterisk console process, and the (asterisk -r) disconnects and terminates. The "mother" (asterisk -cgvvv...) command echos the debug messages, reaches the critical moment, and dumps core, issueing "Segmentation fault (core dumped)" message and terminating...

But the same command would not yield a crash, if issued from the console process...!


What gives? Anyone around who can explain?

I still need to check out the "not loading #include files" during the reload.
Try my latest 2k buffer changes and see what happens...

By: p0l0 (p0l0) 2006-01-31 08:54:10.000-0600

It seems there is a problem with System(), i tried the following in extension.ael2:

System(/opt/asterisk-aix0.1/var/lib/asterisk/text2wave.sh ${name} ${text});

and in asterisk it executes:

   -- Executing System("SIP/57-b7c3", "/opt/asterisk-aix0.1/var/lib/asterisk/text2wave.sh prueba'  'Prueba de festival'") in new stack
/bin/sh: /opt/asterisk-aix0.1/var/lib/asterisk/text2wave.sh prueba'  'Prueba de festival': No existe el fichero o el directorio

And in AEL1, it works good:

   -- Executing System("SIP/57-decd", "/opt/asterisk-aix0.1/var/lib/asterisk/text2wave.sh prueba  Prueba de festival") in new stack

It seems to be problem with the single quote (')

By: p0l0 (p0l0) 2006-01-31 10:13:17.000-0600

I found the error, i called text2wave using ' ->

&text2wave('prueba', 'Prueba de festival', 0);

if i call it without quotes it works correctly.

By: Steve Murphy (murf) 2006-01-31 22:16:56.000-0600

POLO-- Glad you found the problem. In general, such questions are easier answered if you upload you extensions.ael2.

pjc-- Good News! I forgot to reset the include stack index at the start of the parse. Result: everything works on the initial load, but reloads will have probs with include files. THis is fixed in SVN. I guess, it will appear in 0.14 later.

By: p0l0 (p0l0) 2006-02-01 01:41:27.000-0600

Here is the extensions.ael2 that fails:

-------------------------------------------------------------------------
context test {
       1111 => {
               Answer();
               &text2wave('prueba','Prueba de festival',0);
               HangUp();
       };
};

macro text2wave(name, text, sob) {
       if ("${sob}" = "1") {
               System(/opt/asterisk-aix0.1/var/lib/asterisk/text2wave.sh -s ${name} ${text});
       } else {
                System(/opt/asterisk-aix0.1/var/lib/asterisk/text2wave.sh ${name} ${text});
       };
       Wait(1);
       Playback(text2wave-${name});
};
-------------------------------------------------------------------------

If I call text2wave, withouth quotes it works good. I think the problem could be parsing the arguments to the macro or executing the System() call.

By: Steve Murphy (murf) 2006-02-01 07:25:58.000-0600

POLO--

AEL2, so far, is a language that does not use any quoting. There *is* one single exception-- in $[..] expressions, you can use double quotes to wrap a token that may include spaces. So, the single quotes you put in your arguments did not mark the beginning or end of anything, and they are copied into the argument along with all the characters present. The only characters in your call that had any meaning to the parser and asterisk were the parenthesis (), and the commas, which split the arguments. All of the other characters in the arguments, including the spaces, are copied.

So, the fact that the single quotes showed up in the macro expansion is actually a feature, not a bug.

By: p0l0 (p0l0) 2006-02-01 07:38:39.000-0600

The problem isnt that the single quote is copied, the problem is that for the first argument, the single quote disappears. If I execute this script in asterisk it says the following:

Executing System("SIP/57-b7c3", "/opt/asterisk-aix0.1/var/lib/asterisk/text2wave.sh prueba' 'Prueba de festival'") in new stack

as you can see at "prueba'", there is only the last single quote, the first disappears, but with the second parameter there isnt any problem. I will use it withouth quotes anyways, so I will not fall in errors.

Thanks for your quick response, AEL2 is great ;o)

By: Steve Murphy (murf) 2006-02-01 09:13:13.000-0600

POLO--

Forgive me, I didn't understand the problem, but I do now!

A good bug! Many thanks!  I've checked a fix into the SVN archive at teams/murf/AEL2. I wan't counting a single quote in the contents of a "word"...
and oversight on my part. I've tested the fix, and it looks good...

Executed ast_add_extension2(con, rep=0, exten=1111, priority=2, label=(null), callerid=(null), appl=Macro, data=text2wave|'prueba'|'Prueba de festival'|0, FREE, registrar=pbx_ael2);

The single quotes are now included in the arguments, even at the beginning.

By: Gonz (gs) 2006-02-08 12:20:15.000-0600

murf,

First and foremost I'd like to thank you for taking the time to write this module. You seem to have spent a lot of time on it, and given the frequency of your notes on this bug report it seems you're a keen programmer wanting to get a real good module into Asterisk. I haven't yet used it, but looking at the benefits of using it over AEL it looks very promising!

Now, I'm sorry to bother you with this but I'm trying to get this module working with Asterisk 1.2.4. I've installed the Flex version as per your instructions (2.5.31) and installed bison 2.1. When I make clean && make on the Asterisk source, the error is as follows:

ast_expr2.o(.text+0x2686): In function `yyparse':
/usr/local/src/asterisk-1.2.4/y.tab.c:1666: undefined reference to `yyerror'
ast_expr2.o(.text+0x2853):/usr/local/src/asterisk-1.2.4/y.tab.c:1264: undefined reference to `yylex'
ast_expr2.o(.text+0x29a7):/usr/local/src/asterisk-1.2.4/y.tab.c:1655: undefined reference to `yyerror'
ast_expr2.o(.text+0x35dd):/usr/local/src/asterisk-1.2.4/y.tab.c:1660: undefined reference to `yyerror'
ast_expr2.o(.text+0x35ee):/usr/local/src/asterisk-1.2.4/y.tab.c:1781: undefined reference to `yyerror'
ast_expr2f.o(.text+0xe0f): In function `ast_expr':
/usr/local/src/asterisk-1.2.4/ast_expr2.fl:152: undefined reference to `ast_yyparse'
collect2: ld returned 1 exit status
make: *** [asterisk] Error 1

What I find strange (to my untrained eye), although it may be normal, is that y.tab.c is only 14 lines long, yet it mentions line 1781 in y.tab.c ??

Any suggestions?

Thanks in advance,
Gonzalo

By: Gonz (gs) 2006-02-08 12:22:38.000-0600

Actually, just realized the version for this bug/feature is SVN. Maybe that's why? I tried to use SVN, but I couldn't get it compiled (although that's another story on its own)

By: Steve Murphy (murf) 2006-02-08 13:56:25.000-0600

Gonzalo--

I've been using bison-2.0, and wasn't aware a newer version existed-- but I hadn't been looking for it, either! So I fetched it, to see if there is some problem. I compiled and installed it, and everything builds just as before.

So, I'll upload my ael.tab.* files, and you can copy them into the pbx dir, and try another build...

In the meantime, you need to try and figure out why your bison generated bad files. For your own sake, it may well be worth the investigation! The root problem may be causing you grief in other ways!

Perhaps there is some relation to why you can't build from SVN?

murf

By: Olle Johansson (oej) 2006-03-05 02:52:43.000-0600

murf - what's the current status of AEL2?

By: Sergey Okhapkin (sokhapkin) 2006-03-05 05:41:41.000-0600

oej - "it works for me":-) My 10K in size extensions.ael file designed for the original ael works OK with AEL2. The only note, it would be nice to have both res_ael.so and res_ael2.so marked as "noload" in the default modules.conf to let user select which one to use.

By: Steve Murphy (murf) 2006-03-05 09:06:56.000-0600

OEJ--

I've been keeping the team branch up to date automatically, and resolving conflicts when they occur, as quickly as possible, and getting the branch back on autoupdate.

My extensions.ael2 is 31K, and it includes my telemarket-torture.ael2 script of 28k. All known bugs have been corrected. A few brave testers have returned their input, sokhapkin among them (many thanks,  Sergey) (and pjc, Gonzalo, capouch, POLO, and everyone else!)

I have only a few things on my list, like updating the application args info against the current status of HEAD, updating the README so the Wiki isn't the better doc, and maybe putting in something to check the function args in ${} exprs. The arg checking will most likely change when there is some mechanisms incorp'd into asterisk to make the args uniform and the documentation accurate. None of it's critical, I think, I hope...

The Wiki page for AEL2 has over 4000 hits now. Haven't checked for a while... so there must be some interest out there....!

By: exomorph (exomorph) 2006-03-15 16:32:49.000-0600

Murf,

Can you add an updated patch file for this?  I'm able to compile your svn branch, but would like to apply this on top of my own branch for full testing.

Nice work on this, I like the changes included in here.

Thanks!

By: Russell Bryant (russell) 2006-03-15 16:53:24.000-0600

You should be able to create the patch yourself if the branch is up to date.  To do so, try the following ...

svn diff http://svn.digium.com/svn/asterisk/trunk http://svn.digium.com/svn/asterisk/team/murf/AEL2 > ael2.patch

By: exomorph (exomorph) 2006-03-15 17:00:03.000-0600

Murf,

Also, why are you using Flex 2.5.31?  I only ask because it seems like alot of the linux distro's still use 2.5.4a.  I had to download and install the latest version of it for things to work.

Thanks,
 --Greg

By: Steve Murphy (murf) 2006-03-15 17:54:29.000-0600

Exomorph-- Glad to have you testing! I've pretty much stopped generating new patches, because, as russell pointed out, it's pretty easy to roll your own patch with svn diff.

I use the latest bison and flex (or, at least, fairly late models). Flex has to be 31 or higher, because only these versions produce a re-entrant lexer. You don't have to fetch and build the late model flex or bison, if you don't have them, I should have the generated .c and .h files produced by flex/bison in svn. Look around. You might have to touch them to keep them from wanting to be regenerated.

murf

By: exomorph (exomorph) 2006-03-16 10:29:02.000-0600

Murf,

I've created an RPM package for the latest flex (now at 2.3.33) and they are available at the below address for peoples convenience.

http://www.digitalrealities.ca/rpms/fc4/

As for the pre-generated flex output… Maybe you can change the makefile to touch the dates on those files if they exists?  That way anyone svn’ing the latest copy won’t have to manually touch the files themselves.  I’m thinking of the future of when this goes into the trunk.

The other way to do this would be to setup a “make ael2-flex” command (or something of the like) to build the flex output, instead of including it in with the main “make all” command. You can also have the “make all” command check to see if the pre-generated files are their, and if not, then compile them.

Thoughts?
 --Greg

By: exomorph (exomorph) 2006-03-16 10:32:43.000-0600

Opps, thats a typo...  Flex is at 2.5.33 :)

By: exomorph (exomorph) 2006-03-16 10:52:12.000-0600

Murf,

You also make reference to your own large extensions.ael2... I'm not sure how diffrent it is from the extensions.ael file, but maybe you can include it in your svn tree as an example?

By: Steve Murphy (murf) 2006-03-16 13:50:16.000-0600

Exomorph--  Thanks for the link.

The way the existing trunk handles the expression parser (ast_expr2), is to include the rules to run bison/flex, but the dependencies are removed, much as you suggested. In my state of development, tho, it's easier to work with if they automatically rebuild when necc. I'm not as hot and heavy on development now, maybe I'll update the makefile.

As to examples from my personal dialplan, I'll think about it... There's a fair measure of cruft that only applies to my setup, and there's some confidential stuff I'd have to bleep out, too. But, there are substantial chunks of it already out there on the WIKI. I added a few AEL examples, and I provide the file for the TeleTorture scripts in AEL2. I need to rewrite the TeleTorture to take advantage of some language features I added later on, but it still works just fine! Someone actually went down into it this past week on my phone system... poor person.  ;^)

By: exomorph (exomorph) 2006-03-21 22:07:39.000-0600

Just a quick note, that I've been using this in production without any issues.

By: Steve Murphy (murf) 2006-03-21 22:28:14.000-0600

Exomorph--

Excellent news! Now that you've gone thru the "experience" of converting
your dialplan to AEL2, please, record here your impressions, both good and bad,
while they are yet fresh. Any confusions? Any frustrations? Anything you felt you
wasted time on? I'm looking for things that might make AEL2 more effective at
finding common problems earlier in the design cycle...

By: exomorph (exomorph) 2006-03-22 12:19:26.000-0600

Murf,

Well I'm a programmer, so moving the dialplan to AEL2 was quite a simple task for me. :)

I really only have two small "issues" with how AEL/AEL2 works... First would have to be the error output... When starting/reloading the server, its hard to pick out the AEL2 compile errors, and so I thought things were fine, but the dialplan just wasn't working. maybe we can come up with an easier to read error messages?  (Maybe even marking them as Errors and not Warnings would help some?)

The other thing that was odd was when I switched from the Goto app to the AEL2 goto, I had to use the "|" character to separate arguments... Can you also add the "," character as well?

That's about it right now.  I still have a bunch of work to do on the dialplan so if I come up with anything else I'll let you know!

By: Steve Murphy (murf) 2006-03-23 22:57:10.000-0600

Exomorph-- Sorry for the delay in replying.

For finding the results of parsing, I suggest using aelparse from the command line. It will read in the file /etc/asterisk/extensions.ael2, and run it thru the paces-- outside of asterisk. It's easier to use than picking thru the logs, and you don't need to feed possibly buggy input to asterisk to find the bugs.

I'll look into allowing commas in the goto notation.

By: walmike (walmike) 2006-03-30 01:40:57.000-0600

Is #include broken?

It seems that after a file is included with an #include "file" statement, the parser stops scanning the main file. The remainder of the file following the #include statement is not scanned or ignored. However, the included file is scanned.

Simple example:

extensions.ael2

context pstn-out {
_1NXXXXXXXXX =>
  {
  Set(CALLERID(number)=2121112222);
  Set(CALLERID(name)=unavailable);
  Dial(SIP/${EXTEN}@switch-out);
  }
#include "test"
}

test:
// This file as nothing but a comment

Results in: Error: syntax error, unexpected $end since the closing } is never seen.

Has anyone else experienced this?

- Mike

By: Steve Murphy (murf) 2006-03-30 07:18:53.000-0600

Mike--

I just set up situation you described on my machine, and ran aelparse (see utils/) and got the predicted results.

Tell you what-- you run aelparse on your machine, and send me (murf at parsetree dot com) the output generated by aelparse. It is possible that aelparse could work OK, but asterisk give some other result... if so, dive into your logs, and see if you can find anything relevant. Bump up your verbosity...

Let's see if we can get to the bottom of this

By: Steve Murphy (murf) 2006-04-04 09:54:50

OK, I just formed the 0.14 patch and uploaded it. Mike, it appears, may be suffering from bugs that were still in the 0.13 patch, and it looks like people are still downloading these patches, so I'm adding this new patch, to take in all fixes and updates since the 0.13 patch.

And, I've recently also added the team/murf/AEL2-1.2 branch to SVN, for those who want to experiment with 1.2 and AEL2.

Just a notice to everyone, to use the team/murf/AEL2  branch in SVN to make an up-to-date patch for your local asterisk source.

As russell pointed out previously, you can:

svn diff http://svn.digium.com/svn/asterisk/trunk  http://svn.digium.com/svn/asterisk/team/murf/AEL2  > ael2.patch,

or for 1.2, you can:

svn diff http://svn.digium.com/svn/asterisk/branches/1.2  http://svn.digium.com/svn/asterisk/team/murf/AEL2-1.2  > ael2-1.2.patch

Then apply with patch -p0 < ael2.patch

Hang onto the patch you apply to your source. If you want to update to a newer patch, then you'll have to use patch to revert; you may have to remove the extra files by hand. Once you reverse a patch, then you can apply the new one.

Just one caution: I'm finding that there is almost a constant stream of updates to the trunk, and even tho I've turned on automerge for the AEL2 branches, there is a good chance that some diffs will be captured by the above commands of things that were committed to the trunk, that weren't automerged yet into the AEL2 tree. What I do is grep for Index: in the patch, and look at the files in the list. If there are extra unrelated files, like stuff in channels, apps, etc, I simply edit the patch and remove those extra files... So maybe these periodic patch updates will be useful or necessary after all.

By: Steve Murphy (murf) 2006-04-08 17:14:56

Got this feedback today from Mike:

 YES! THANK YOU!

 The brand new Dell arrived and I just finished get everything installed.
 Works perfectly. Had to create a diff from the svn and applied it to 1.2.6.

 One thing to note, for people who get compile errors, is they might need
 to delete ast_expr2.c and st_expr2f.c. It seems the Makefile doesn't properly
 describe their dependencies.

 I'm looking forward to writing some asterisk functions now. I'm new to
 asterisk and was appalled by the extensions.conf format. AEL looked like a
 godsend (I'm a c++ programmer). I might email you some questions about
 "catch a".

 Thanks again for your great contribution to asterisk.

 - Mike

Many thanks for the feedback, Mike! Sorry you had any difficulties in the first
place!

As to the ast_expr2* stuff, the Makefile isn't faulty; it's that way on purpose.
There are many who don't have the latest/proper versions of bison and flex, and
rather than have a large number of people having trouble with their builds, they
opted instead to distributed the outputs of flex and bison instead, and have the
build use those by default. If you would rather use your own bison/flex, then
all you have to do is remove the ast_expr2*.c files.

Next, the "catch"...  Basically, the stuff inside a catch is placed in an extension with the name of the catch. Normally, in a macro, all the code normally goes into the 's' extension, so a 'catch' is a way to specify some
other extensions.

This code:

macro std-exten( ext , dev )
{
       Dial(${dev}/${ext},20);
       switch(${DIALSTATUS})
               {
       case BUSY:
               Voicemail(b${ext});
               break;
       case NOANSWER:
               Voicemail(u${ext});
               break;
       case ANSWER:
               break;
       default:
               Voicemail(u${ext});
       }
       catch a {
               VoiceMailMain(${ext});
       }
}

Note the 'catch a' and its VoiceMailMain call...

The above gets compiled into:

Executed ast_context_create(conts, name=macro-std-exten, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=s, priority=1, label=(null), callerid=(null), appl=Set, data=ext=${ARG1}, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=s, priority=2, label=(null), callerid=(null), appl=Set, data=dev=${ARG2}, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=s, priority=3, label=(null), callerid=(null), appl=Dial, data=${dev}/${ext}|20, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=s, priority=4, label=(null), callerid=(null), appl=Goto, data=sw-1-${DIALSTATUS}|1, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=s, priority=5, label=(null), callerid=(null), appl=NoOp, data=Finish switch-std-exten-1, FREE, registrar=pbx_ael2);

Executed ast_add_extension2(con, rep=0, exten=a, priority=1, label=(null), callerid=(null), appl=VoiceMailMain, data=${ext}, FREE, registrar=pbx_ael2);

Executed ast_add_extension2(con, rep=0, exten=_sw-1-., priority=1, label=(null), callerid=(null), appl=Voicemail, data=u${ext}, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=_sw-1-., priority=2, label=(null), callerid=(null), appl=Goto, data=s|5, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-1-ANSWER, priority=1, label=(null), callerid=(null), appl=Goto, data=s|5, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-1-NOANSWER, priority=1, label=(null), callerid=(null), appl=Voicemail, data=u${ext}, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-1-NOANSWER, priority=2, label=(null), callerid=(null), appl=Goto, data=s|5, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-1-BUSY, priority=1, label=(null), callerid=(null), appl=Voicemail, data=b${ext}, FREE, registrar=pbx_ael2);
Executed ast_add_extension2(con, rep=0, exten=sw-1-BUSY, priority=2, label=(null), callerid=(null), appl=Goto, data=s|5, FREE, registrar=pbx_ael2);

Note that I've separated out the line where the 'a' extension is declared. If there was more than one app call in the catch, they would all be declared under
the 'a' extension.

So, now, the next question that might be asked, is "WHY?, what good is a 'catch'? And my answer is 'heck if I know, but if you need to define an extra extension or two via a macro, this is the way to do it!'

By: Steve Murphy (murf) 2006-04-08 17:35:20

Hmmm. Pardon me for using this bug page as a diary. But yesterday, my wife
complained that she hasn't gotten any voicemail since I recoded the dialplan
into AEL2. So I debugged, and found it was because I had a typo in a variable
name in the stdexten macro!! AEL2 should have noticed that, don't you think?
I'm going to investigate having the semantic checker keep track of variables/scoping, and issue warnings when new and unseemly variables are used...

And, BTW, I'm going to upload a 1.2.x-0.14 patch also. I don't know how many folks are using 1.2.x; but I suspect a fair number.



By: Steve Murphy (murf) 2006-04-13 08:17:30

More diary stuff; I added code a few days back to added the -d option to aelparse, the stand-alone AEL2 compiler. The -d option overrides the config directory to be "." (the current dir). Thus, now, you can pull your extensions.ael2 file, and all the files it includes, into a separate directory, and syntax and semantic check them to your heart's content, and then, when they are clean and ready, you can "deploy" them into /etc/asterisk, and reload ael2.

Also, I changed the way syntax errors are counted... but, because of a bug in bison, which seems to always call yyerror with a NULL argument, I didn't do it the way it should be done. I'll register the bug with the bison folks, and fix my code as soon as they fix theirs.

And, I added some more error tokens to the AEL2 grammar, to improve its error handling. More could be done, that is certain; baby steps, I guess.

I haven't issued a new patch yet. I want to see if I can add the variable reference check, which I mentioned previously.

By: Russell Bryant (russell) 2006-04-13 10:16:13

We are going to begin having weekly developer conference calls once again.  The time we have set for these right now is at 10 AM Central Time.  I'm not sure where you are located, but would it be possible for you to make it to one of the calls?

I think this project would be a good candidate for live disucussion so that we can  all catch up on everything you have done as well as decide what remaining steps need to be taken to get this merged.

By: Steve Murphy (murf) 2006-04-13 15:19:39

I am in Mountain Time (Dailight Savings right now.) So, theoretically, when it's 10 am central, I'm at 9am here.

I am in the NW corner of Wyoming, between Cody, and Powell.

I should be available to participate in such a call. Give me a date, and I'll schedule it. Either I call you, or you call me? Whichever...

By: Russell Bryant (russell) 2006-04-13 15:55:37

Great!

Let's plan on talking about this during the upcoming developers' conference call on Tuesday, April 18th.  On my current list, I have it as roughly 10:45 - 11:00 AM CDT in case you want to only join around the time for your part.  The hosting is still being worked out but I'll have that available by Monday sometime, along with the finalized agenda.

Let me know if this will work out for you.

By: Steve Murphy (murf) 2006-04-13 18:58:26

Looks OK to me. You can send the gory details to murf at
parsetree period com, just in case some intelligent email harvesting
bot passes thru this data.

By: Steve Murphy (murf) 2006-04-18 20:33:16

;^)  Dear Diary--

Today I attended the developer conference call. They decided to make AEL the
replacement of AEL in the trunk. I am truly honored.

I was given two assignments:

1. in the pbx subdir, form an ael subdir, and move all the ael support files
  (except for pbx_ael.c) into the new subdir.
2. Get rid of the applist and the application argument checking stuff.

I have already pretty much accomplished #1 in svn. I have svn rm'd the ael stuff, and svn moved the ael2 stuff into the new dirs. I've upgraded the
two makefiles involved (in pbx/ and utils/). Make runs OK.

I also removed the ael.txt in the doc/ dir, and moved the ael2.txt file over
to ael.txt to replace it. I then edited the beast to sound more like the doc
for ael than a description of AEL2. The only mention of a "2" is some short
verbage about versions, and history.

The only prob so far is a set of 3 warnings during the compile, about incompatible pointer type initializations, from the STD_MOD() macro call.
Don't know if I did something wrong or not. I'll wait a day or two for
things to either calm down and the warnings disappear, or I'll investigate
and fix them... the module definition code is in upheaval right now, and
I'd like to wait for the storm to clear before going out to fix things... I
don't like getting hit by lightening!
--

Now to remove the stuff for checking the app args... But I have some Questions!

And I need the answers to finish properly...

Question 1: The application arg checking stuff I've done is a fair start to
           the AAL (Asterisk Argument Language) work. So, is it appropriate
           that I copy out a new svn trunk branch and move that work over,
           and start a new bug report (enhancement request?) for AAL. What's
           the best way to handle this?  A result of the new AAL, will be to
           return the code that I'm about to remove, but in it's "proper"
           incarnation... to the AEL semantic checker.

Question 2: I have a few AAL questions to ask, but it'd be better to blather
           on about these in the AAL 'bug' report. So, I guess I really only
           have one AEL question to ask, and this is not it...

By: Russell Bryant (russell) 2006-04-18 21:20:43

Waiting a few days for the module loader changes to calm down would be a good idea.  I still have no idea how the new stuff works and I don't plan to bother trying to figure any of it out until they done making the changes.

Response 1)  I think creating a new AAL branch for that work is a great idea.  Even though what you have done so far in this regard is not going into 1.4, I think it is definitely something we would like to continue to pursue.  It would be a great project for the development cycle in the second half of 2006.

Once we get the AEL2 branch merged, I think it would be fine to open up another issue to track your progress with AAL as you have done here for AEL2.

Response 2)  This response has been deferred just as the original question was for discussion in the new AAL issue.  :)

Thanks for all of your hard work!

By: Steve Murphy (murf) 2006-04-19 01:45:44

AS A SIDE NOTE:  an extensions.ael file (written for the original AEL)
 was uploaded as a test during the conference call. I didn't write down
 the kind fellow's name, that donated it. I ran it thru aelparse, and got
 a series of syntax and semantic errors/warnings. Here's the results:

First, the syntax errors, of which there were 7:

+LOG: lev:4 file:ael.flex  line:193 func: ael_yylex  File=./extensions.ael, line=165, column=21: Mismatched '}' in expression!
+LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 165, Cols: 23-23: Error: syntax error, unexpected '=', expecting ')'
+LOG: lev:4 file:ael.flex  line:317 func: ael_yylex  File=./extensions.ael, line=174, column=63: Mismatched '}' in expression!
+LOG: lev:4 file:ael.flex  line:317 func: ael_yylex  File=./extensions.ael, line=180, column=46: Mismatched '}' in expression!
+LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 184, Cols: 0-4: Error: syntax error, unexpected 'macro'
+LOG: lev:4 file:ael.flex  line:193 func: ael_yylex  File=./extensions.ael, line=222, column=21: Mismatched '}' in expression!
+LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 222, Cols: 23-23: Error: syntax error, unexpected '=', expecting ')'
+LOG: lev:4 file:ael.flex  line:317 func: ael_yylex  File=./extensions.ael, line=228, column=37: Mismatched '}' in expression!
+LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 235, Cols: 0-6: Error: syntax error, unexpected 'context'
+LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 344, Cols: 32-32: Error: syntax error, unexpected ';', expecting '{'
+LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 350, Cols: 32-32: Error: syntax error, unexpected ';', expecting '{'
+LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 461, Cols: 10-13: Error: syntax error, unexpected 'else'

Looking over the input, I discovered the following:

1st syntax error: In line 165, the text said:
                       if ("$(VMBOXEXISTSSTATUS}" = "FAILED") {
   but should have said:
                       if ("${VMBOXEXISTSSTATUS}" = "FAILED") {
   (My eyes are getting bad... it took me a minute to spot this!)

2nd syntax error:  on 165, one error begets another, the famous error
'cascade' affect seen in stuff like the c compiler.

3rd syntax error: on 174, the exact same typo as above.

4th syntax error: on 180, I think this another cascade problem caused by the
        previous error, a "cascade". But it involves the closing brace on the
        extension this time.

5th syntax error: on 184, the macro isn't expected... another "cascade"; because the error involved curly brackets, the parser is now officially lost...

It also correctly spotted a "Goto" call, that should be "goto" because AEL
is case-sensitive.

But it's not totally out of sync!! on line 222, the same typo is spotted and reported as on 165 and 174! But the rest seem to be cascade errors... so,
I fix the typos I've mentioned, and try again... and I get something very interesting!



LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 291, Cols: 21-21: Error: syntax error, unexpected ':', expecting ';'
LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 296, Cols: 8-18: Error: syntax error, unexpected word
LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 395, Cols: 20-20: Error: syntax error, unexpected ':', expecting ';'
LOG: lev:4 file:ael.y  line:660 func: ael_yyerror  ==== File: ./extensions.ael, Line 401, Cols: 8-18: Error: syntax error, unexpected word

while the errors on 296 and 401 are "cascades", the errors on 291 and 395 both involve a line that looks like:

               goto 9${EXTEN:3}|1;

Which is OK. The : is the choke point. I investigate the flex/bison description, and find that a variable ref with the colon in it is not covered, which I upgraded.

A semantic error generated because a goto target consisting of a pattern
could not be found, but it was there, an extension with an exactly matching pattern name. The pattern matching routine was a bit naive, so I upgraded it, and should function properly.

The problems I've solved have been committed back; I haven't added that particular test case yet to the repository,  tho. Many thanks for the test case, as it pointed out some weaknesses!!

By: Steve Murphy (murf) 2006-04-19 01:48:43

One other note on that testcase: There were a bunch of warnings about args to RealTime calls. The warnings are generated because the args to RealTime are separated with "|" instead of ",". I'd probably tend to upgrade the semantic checker to allow commas or vertical bars as arg separators, but since I'm dropping the argument checking for now, why bother?

By: Steve Murphy (murf) 2006-04-19 13:52:37

Oh, yes!!! NOW I remember what Question #2 should have been:

Will the replacement of AEL by AEL2 only apply to the trunk, or will it also
extend to 1.2? After all, it is a FIX... ;^)

By: Russell Bryant (russell) 2006-04-19 15:03:42

The replacement of AEL with AEL2 with only happen in the trunk.  AEL in 1.2 will remain unchanged.  Sorry :)

By: Russell Bryant (russell) 2006-04-19 16:29:29

It looks like you can't put a comment on the line of a Makefile target.  Or, at least it didn't work for me.  I made a few modifications to get everything to build correctly.  I have posted them in a patch in case it's helpful.

By: Russell Bryant (russell) 2006-04-19 16:35:32

I have also uploaded a patch that fixes your issues with the recent loader changes.

By: Steve Murphy (murf) 2006-04-20 22:18:00

OK, Russell--

I'll see you those patches, and raise you a couple more
fixes... ;^)

Your patch for the void *mod vs void args to load, unload and reload got rid
of the warnings. That felt good. And I didn't get run over or hit by lightening.

I removed the comments in the makefile as your patches directed. I'll figure it out when I finish with AAL again.

It's getting closer. Now I merge this stuff in with my personal build, and
see how it runs on a live installation. My wife hates this part!

By: Steve Murphy (murf) 2006-04-20 23:07:34

OK, I've had to commit another small change, removed the install
of applist... which is gone now...

I've got this stuff up and live on my home machine. It's read in my
dialplan, and I've got dialtone. Let's let it simmer for a day, and see
how it tastes tomorrow.

For those who are adventurous, you might try giving this AEL a spin.
C'mon, convert your extensions.conf to extensions.ael! Consider it a
challenge! It'll be something to brag about at breaks: "Well, ***I***
recoded my dialplan into the new AEL format!!! It was soooooo much fun!!"
(if only they could see you, cursing and muttering late into the night).

By: Steve Murphy (murf) 2006-04-21 18:46:40

OK, today I've done a little homework, implemented the %destructor directives that have been around a long time in bison (and I didn't realize it!), which seems to plug the memory leaks when there's syntax errors in the input.

I ran the code thru purify, and found that a few small problems had popped up since last I purified it. I attacked and conquered every problem.

I added sokhapkin's ael v1. test case to the regression suite, along with the extensions.ael that was attached to this some days ago.

All the above has been committed back into the repository teams/murf/AEL2...

Russell, it's there; I've done pretty much everything I wanted to do before the merge. You ready? I think the code is...



By: Russell Bryant (russell) 2006-04-22 09:07:08

Alright, well I'm going to begin trying to merge it.  However, Kevin tried to merge the autoconf branch yesterday and ran into some issues.  Hopefully it won't occur with this branch.  If it does, we should be able to get it figured out this week.

By: Steve Murphy (murf) 2006-04-22 09:42:06

One more note! (Just when you thought it was safe to...)

I have a little more experience with the %destructor directive.
I got rid of the conditional on the 'C' value. Got rid of the
start tokens in the %destructor lists. This will make the %destructor
code more upward compatible with future bison releases.

Moved up to bison 2.1a. It added a feature to report unused $$ and $n
tokens, which spotted some minor bugs in my grammar. Fixed those, too.

By: Russell Bryant (russell) 2006-04-22 12:36:49

Well, I ran into some problems with the merge.  As soon as we figure out what's wrong, I have my finger on the merge button for this branch.  :)

By: Steve Murphy (murf) 2006-04-22 13:29:38

OK... I see that yesterday, the automerge dropped out (again). So I just
hand merged it. app.c had a conflict... I don't understand all the cases where
this sort of happens, but it's not that unusual to have it happen. I wish I didn't have to enter my password 15 times or so over 15 minutes to do it. I wish the expect script I wrote to do it for me still worked. I wish a ton of things, but, luckily not all my wishes come true. Could the automerge dropout be what's holding you back, or is it something else?

By: Russell Bryant (russell) 2006-04-22 22:40:49

The problem isn't that.  After the merge completes, svn will not let me update or commit.  It complains about files missing from the repository or something.  Kevin is having the same problems trying to merge the autoconf branch.    He is discussing the problem with the subversion guys, so he should be able to get the issue resolved in the next few days.

As for typing your password 15 times, I know that is annoying.  Personally, I have added my password to my ~/.subversion/servers file so I don't have to type it at all:

[digium]
ssl-client-cert-file = ..............
ssl-client-cert-password = ..............

Corydon wrote an expect script for this as well.  You can grab it from subversion if you'd rather use that:

svn co http://svn.digium.com/svn/asterisk/repotools

By: Russell Bryant (russell) 2006-04-24 11:46:04

It looks like this will have to be merged by hand, instead of using "svn merge".

Can you provide a list of every file that has been changed, removed, or added?  I want to make sure I don't miss anything.

By: Russell Bryant (russell) 2006-04-24 13:02:38

Nevermind!  Kevin figured out the problem and has merged this branch into the trunk.  Thank you for all of your hard work!!!