diff options
Diffstat (limited to 'docs/COMPARISON.md')
| -rw-r--r-- | docs/COMPARISON.md | 89 |
1 files changed, 89 insertions, 0 deletions
diff --git a/docs/COMPARISON.md b/docs/COMPARISON.md new file mode 100644 index 0000000..94e8d3b --- /dev/null +++ b/docs/COMPARISON.md @@ -0,0 +1,89 @@ +# Comparison of SCXML Interpreters + +In this document, I will make an attempt to compare and measure the performance +of the various, freely available SCXML interpreters. If you are the author of +one of these interpreters and feel that I misrepresented your work, please do +not hesitate to contact me and I will reevaluate / correct any factual errors. + +This selection is based on the list of available implementations on the [SCXML +wiki](https://en.wikipedia.org/wiki/SCXML) page, plus a few interpreters I know +from the [W3C mailing list](https://lists.w3.org/Archives/Public/www-voice/) +and the [SCXML workshops](http://scxmlworkshop.de). I will list all +interpreters known to me, but only benchmark those that pass the W3C IRP tests +as is required for an actual SCXML interpreter. + +# Methodology + +The benchmarks were performed with a [huge +SCXML](https://github.com/tklab-tud/uscxml/tree/master/test/w3c/compound) file +containing all of the automatic W3C IRP tests that do not rely on timeouts to +pass. + +## scxmlcc + +| Platforms | Tested on | Tested | Website | +|---------------|---------------------|------------------|---------------| +| Linux only | Debian Jessie 64Bit | **No**: Subset of SCXML implemented is insufficient for SCXML IRP tests. | [http://scxmlcc.org](http://scxmlcc.org) | + +### How to build + + $ sudo apt-get install build-essential libboost-all-dev autorevision + $ git clone https://github.com/jp-embedded/scxmlcc.git + $ cd scxmlcc/src + $ make + +### Evaluation + + $ ./scxmlcc -i ./test-ecma-all.scxml |sort |uniq + warning: event asteriks not currently supported + warning: event tokens not currently supported + warning: parallel support is not fully implemented/tested + warning: unknown action type 'script' + warning: unknown item 'assign' in <onentry> or <onexit> + warning: unknown item 'assign' in <transition> + warning: unknown item 'cancel' in <transition> + warning: unknown item 'datamodel' in <state> + warning: unknown item 'donedata' in <state> + warning: unknown item 'foreach' in <onentry> or <onexit> + warning: unknown item 'if' in <onentry> or <onexit> + warning: unknown item 'invoke' in <state> + warning: unknown item 'send' in <onentry> or <onexit> + warning: unknown item 'send' in <transition> + +The subset of SCXML implemented in <tt>scxmlcc</tt> is insufficient to run the +performance / compliance benchmarks. + +## SCXML Lab + +| Platforms | Tested on | Tested | Website | +|---------------|---------------------|-----------|---------------| +| HTML5 | Safari 9.0.3 | **No**: Subset of SCXML implemented is insufficient for SCXML IRP tests. | [http://www.ling.gu.se/~lager/Labs/SCXML-Lab/](http://www.ling.gu.se/~lager/Labs/SCXML-Lab/) | + +I pasted the file above into their web-frontend and got a <tt>validation-error</tt> + + Unexpected attribute 'datamodel' in <scxml>. Hint: + Valid attributes are: 'id', 'initialstate', 'version', 'xmlns'. + +The <tt>datamodel</tt> is indeed not required for a compliant interpreter, but +we will not be able to pass any meaningful subset of the SCXML IRP tests +without support for a datamodel. + +## Legian + +| Platforms | Tested on | Tested | Website | +|---------------|---------------------|-----------|---------------| +| Java | Java(TM) SE (build 1.8.0_45-b14) | **No**: Subset of SCXML implemented is insufficient for SCXML IRP tests. | [https://github.com/pelatimtt/Legian](https://github.com/pelatimtt/Legian/) | + +Does not claim W3C compliance and will, therefore, not pass the compound IRP tests. + +## Qt SCXML Engine + +| Platforms | Tested on | Tested | Website | +|---------------|---------------------|-----------|---------------| +| Any | Mac OSX 10.11.3 with Macports 2.3.4 | **No**: Subset of SCXML implemented is insufficient for SCXML IRP tests. | [https://qt.gitorious.org/qt-labs/scxml](https://qt.gitorious.org/qt-labs/scxml/) | + +[No +support](https://qt.gitorious.org/qt-labs/scxml?p=qt-labs:scxml.git;a=blob_plain;f=doc/scxml.qdoc;hb=HEAD) +for <tt><donedata></tt>, <tt><finalize></tt>. Furthermore the +<tt>profile</tt> (now <tt>datamodel</tt> with finalized SCXML recommendation) +attribute is ignored. |