Planet Perl Six

Patrick Michaud: Where's the goal line?

2008-07-01T16:41:33+00:00

dstar said:
One thing I'm not clear on -- Rakudo is passing 1100 tests out of how many? IE, where's the goal line?

Well, the "goal line" is by definition a moving target. There's not a hard number we can cite -- even in Perl 5 the size of the test suite constantly changes as new features are added and new bugs are found. There will be a point at which we declare a particular set of tests as being the "official test suite" for Perl 6.0.0, but we're really not close to that point yet. Here's what I can say about the test suite thus far:

The "official test suite" is being built from the test suite that was created for Pugs -- the Pugs test suite has approximately 19,000 tests in it. But, the Pugs suite itself was somewhat disorganized and thus difficult to determine test coverage. It also has many tests that are incorrect, either because the language specification has changed since they were written or because they were incorrect in the original.

Thus, Adrian Kreher (mentored by Moritz Lenz and Jerry Gay) is in the process of conducting a complete review and reorganization of the Pugs test suite into the "official test suite", or what we call the "spectests" (in t/spec/ of the Pugs repository). We estimate that about 5,000 (25%-30%) of the tests in Pugs have been reviewed and migrated into t/spec/. And just because a test appears in t/spec/ doesn't mean it's necessarily correct -- we continue to find places where the test doesn't exactly match the language specification (or the language specification is ambiguous and needs clarification in the test).

Within the 5,000 or so spectests, there's a subset of 75 test files (~1,500 tests) that have been identified as the "spectest regression suite" for Rakudo. These are test files that we expect Rakudo to pass at any given moment in time, such that someone can do "make spectest_regression" and get back "All tests successful." The developers also use this to make sure that any changes to Rakudo don't break existing functionality. Of course, the spectest_regression suite is itself a rapidly moving target as Rakudo acquires more features and as the test suite evolves. For example, at the beginning of June there were 52 files (892 tests) in the spectest regression suite, today (July 1) we have 75 files (1522 tests) in the regression suite.

And beyond all of this have been comments from Larry and others that eventually the official test suite will likely be double or triple the current size of the Pugs suite -- I think I've heard estimates of 40,000-50,000 tests in the suite when we're "finished".

So, return to the original question of "Where's the goal line?", you can see that there's not really a straightforward answer. This is why I haven't been saying things like "passing 1100 out of N tests", because the exact value of N depends on what's being measured to depend on what one considers important, and until more of the official test suite is ready any useful N doesn't really correspond to the "goal line" of a released Perl 6. So, the number I tend to focus on at the moment is the absolute rate of increase in passing tests -- currently it's about 100 new passing tests per week. I expect that trend to continue or improve in the coming months. As long as the number of passing tests is increasing, we're making progress.

An obvious intermediate goal is to have Rakudo passing at least the tests in the Pugs test suite. This depends not only on Rakudo development progress, but also progress in reviewing the Pugs tests and migrating them into t/spec/ . This is also why the Perl 6 developers are constantly inviting and encouraging people to help with building and reviewing the test suite. As mentioned above, only about 25%-30% of the Pugs test suite has made it into t/spec/, and many of those have been simply moved into t/spec/ without significant review or verification against the Synopses. Once we have most of the Pugs test suite migrated to t/spec/ I'll probably start reporting passing tests relative to the size of overall suite. (But I'll also continue reporting the passing rate increase as well.)

Lastly, this is a good place for me to repeat an invitation I made at YAPC::NA: For those who are interested in helping with the test suite but don't know where to start, I suggest "adopting" a particular Perl 6 feature or Synopsis section and saying "I'm going to develop the tests for that feature." This doesn't require learning all of Perl 6, it only requires learning those parts of Perl 6 that are necessary for testing that particular feature. It also gives a sense of ownership for a part of Perl 6, as in "I did that part of the test suite".

I'll undoubtedly add more information about Perl 6 testing in the near future, as it's an area where many hands can help make light work.

Thanks to dstar for asking the question that prompted this post. :-)

Patrick Michaud: Rakudo (Perl 6 on Parrot) progress report

2008-07-01T04:59:40+00:00

This is the third "monthly" report for my development grant from the Mozilla Foundation and The Perl Foundation. As regular readers will have surmised by now, the definition of "month" has been stretched a bit for a variety of reasons, but as this report and the other reports will show, it's not been a hindrance to our progress.

To review, the primary goals of this grant are:

To have a Perl 6 on Parrot implementation that supports commonly-used Perl 6 constructs;
Improvements to the Perl 6 test suite;
To substantially complete the Parrot Compiler Toolkit, including documentation;
Increased community participation in Perl 6 and Parrot development, including development efforts on other languages utilizing Parrot and the Parrot Compiler Toolkit.

It's now clear that the work under this grant has been (or otherwise shortly will be) successful in meeting all of the above goals. As before, in this report I'll highlight the major events and milestones that have been reached since the previous report, and let my other article postings provide increased details.

Progress, April 2008 to June 2008

* Of course, one of the biggest news items is that in May 2008 The Perl Foundation received a $200,000 philanthropic donation from Ian Hague, roughly half of which will be used to support further development of Perl 6 and to build upon the work performed under this grant.

* In addition, Jonathan Worthington received a grant from Vienna.pm to continue his work on developing Rakudo; this grant led to implementations of type checking, multimethod dispatch, regex and grammar support, public and private methods, Ranges, scalar variables, runtime role composition, enums, and a lot more.

* During this period we continued to improve the documentation for the Rakudo (Perl 6 on Parrot) compiler, although more focus in this area is definitely needed. In April we published a list of Rakudo milestones as a basic "road map" to guide continued development. We also generated numerous articles and blog postings that describe the various features of the compiler and how it's all being put together.

* In May we started measuring progress on Rakudo by the number of passing tests from the official test suite. As of June 30, Rakudo Perl is passing 1126 tests from the official test suite, and we're averaging 100 new passing tests per week. I'm hoping this trend will continue.

* To facilitate development Moritz Lenz and Jerry Gay refactored the test harness to provide a "make spectest_regression" target. Now Rakudo developers can verify that changes to the compiler are not breaking any tests that were previously passing. Based on this we're able to to maintain a spectest-progress file, and Moritz created an excellent utility to display the progress as a graph:

http://www.pmichaud.com/perl6/rakudo-tests-2008-06-30.png

* Work on refactoring, improving, and updating the Perl 6 test suite is being handled by Adrian Kreher, Moritz Lenz, and Jerry Gay under a Google Summer of Code grant, along with test contributions and suggestions from many others. We now have a better process in place for reviewing and updating tests; this has enabled progress in other areas to also proceed more rapidly.

* Many of the basic Perl 6 statement types and constructs are now in place -- the primary notable exception being list assignment and lazy list operations. Implementing list assignment properly will require some modifications to the underlying grammar engine -- that work is expected to occur later this summer. Lazy lists and operators are awaiting some improvements to Parrot's exception subsystem (expected in early July).

* The Parrot Compiler Toolkit (PCT) and Not Quite Perl (NQP) tools developed in the first months of this grant continues to demonstrate its power and effectiveness. Most of the HLL translators for Parrot have either adopted or are planning to using PCT/NQP for their underlying code generation. In particular, both the Ruby and PHP implementations (Cardinal and Plumhead) have made significant progress by using the Parrot compiler tools. PCT now has support for basic "return" control exceptions -- other types of control exceptions will be added shortly.

* We continue to gather more active contributors to Parrot and Rakudo Perl. There has been a substantial increase in patch submissions -- so much so that we've held discussions about how we might improve our ability to respond to code contributions more quickly. I've given presentations about the recent improvements at FOSDEM 2008, The Texas Open Source Symposium, DFW Perl Mongers, and YAPC::NA 2008. Each of these presentations have increased participation and enthusiasm about Rakudo Perl and Parrot. More presentations about the project will be made at OSCON 2008 and YAPC::Europe 2008.

* At YAPC::NA 2008 Jim Keenan, Will Coleda, and others on the Parrot team organized a "Perl 6 and Parrot Workshop", in which we helped approximately 20 people download and build Parrot and Rakudo Perl and run through the basic test suite. This has further increased interest in the project; indeed, some of the participants came across some bugs (and filed bug reports and/or patches), and many are planning to hold similar workshops in their local user groups. We plan to repeat the workshop at other venues, including the Pittsburgh Perl Workshop, YAPC::EU, and likely other workshops and user groups.

* All of the "specific tasks" targeted in the end of the last grant report have been achieved.

Where things are headed next

The goal for the next few weeks will be to continue (and perhaps improve) the excellent development momentum achieved during the past couple of months. In particular, we will continue improving the test suite and Rakudo's ability to pass tests from the suite. In addition, as some new Parrot features become available (e.g., improved exception and lexical variable handling) we will be able to take advantage of them in the compilers and compiler toolkits. We will also begin identifying the specific individuals and tasks to be engaged under grants from the Ian Hague donation.

Specific tasks for the remainder of this grant:

Continue improving the official test suite
Develop a more complete implementation of Perl 6's exception model
Implement basic lazy lists and operators
More refactoring of basic operators, functions, and classes according to recent changes in the language specification
Allow compiler and builtin library components to be written in Perl 6
Continue convergence efforts with other Perl 6 implementations
Write the final grant report, documenting the work performed, quantifying results achieved, and outlining the next phases of development

As things stand today, I expect to publish the final report for this grant sometime around OSCON 2008 (July 2008). Of course, I will continue to post articles at use.perl, rakudo.org, and parrotblog.org.

Thanks for reading!

Jesse Vincent: ADMIN: rt.cpan.org and misdirected mail

2008-06-28T03:11:01+00:00

Just as a heads up, we've discovered that rt.cpan.org has started to send out mail about bugs to CPAN authors who aren't actually maintainers of the distribution.

The bug is the unintended consequence of some work we've been doing to make rt.cpan.org faster and easier to use.

I could claim that it's a cool new feature to help better publicize issues in CPAN distributions, but I can't quite bring myself to lie like that.

We're working on the issue and will get it sorted out as quickly as we possibly can. I'm sorry for the inconvenience.

Patrick Michaud: Rakudo Perl now passing over 1000 spectests!

2008-06-27T17:19:02+00:00

[Rakudo spec regression status: 75 files, 1080 passing tests]

Rakudo Perl is now passing over 1000 tests in the official test suite!

http://www.pmichaud.com/perl6/rakudo-tests-2008-06-27.png

For those who are interested, the history of test results is available in CSV format from http://svn.perl.org/parrot/trunk/languages/perl6/docs/spectest-progress.csv .

Thanks to Jonathan, Moritz, Auzon, bacek, particle, Coke, chromatic, and everyone else who is helping to make this happen!

Patrick Michaud: Parrot workshop at YAPC::NA 2008 -- success!!

2008-06-20T02:02:32+00:00

[Rakudo spec regression status: 65 files, 792 tests]

Yesterday at YAPC::NA 2008 we had a Parrot workshop, primarily organized by Jim Keenan. It went great -- Jim did a fantastic job. I think we had approximately 20 participants total, and several of the parrot folks came by to help.

During the workshop I think we managed to get nearly everyone with a working copy of Parrot and Rakudo Perl, and I even managed to take some notes on how we can improve things for others later.

We also found (and fixed) some bugs in the official test suite, and Deven Corzine even grabbed a Pugs commit bit and fixed a Windows bug in the fudgeall script!

I found the workshop to be very productive for me personally, and again my compliments and thanks to Jim Keenan and the other organizers for putting it together. (I was at the workshop to help, but I didn't do much of the work in putting it together. :-)

perl6.announce: Re: Parrot 0.6.3 "Beautiful Parrot" Released! by Alberto Simões

2008-06-18T18:38:46+00:00

Congratulations.

And nice timing. Right before Mozilla gets firefox.com working.

Cheers
Alberto

Nuno 'smash' Carvalho wrote:
> Greetings,
>
> On behalf of the Parrot team, I'm proud to announce Parrot 0.6.3
> "Beautiful Parrot." Parrot (http://parrotcode.org/) is a virtual machine aimed
> at running all dynamic languages.
>
> Parrot 0.6.3 is available via CPAN (soon), or follow the download
> instructions at http://parrotcode.org/source.html. For those who
> would like to develop on
> Parrot, or help develop Parrot itself, we recommend using Subversion on
> the source code repository to get the latest and best Parrot code.
>
> Parrot 0.6.3 News:
> - Specification
> + updated pdd09_gc.pod
> - Languages
> + Cardinal:
> - dramatically improved parsing speed
> - added support for defining and instantiating classes
> - started fleshing out the builtin class hierarchy
> - added support for hashes and arrays
> + Chitchat: added the start of a smalltalk compiler
> + Pheme: updated to match PGE changes
> + Pynie: return statement, other minor updates
> + Rakudo:
> - added working list and hash contexts
> - added 'return' statements
> - added => pair constructor
> - added ?? !! ternary
> - added Range, range operators, Complex
> - added common List, Hash methods
> - refactored base classes
> - added Mutable, Perl6Scalar classes
> - added type-checking, is readonly/rw/copy for parameters
> - added make localtest, docs/spectest-progress.csv
> - fix named unaries
> + Squaak: implement return statement
> + Tcl: updated control flow exceptions to use new builtin types
> - Compilers
> + All tools converted to P6object metamodel
> + PGE:
> - <?foo> is now a zero-width match
> - reduced backtracking to improve parsing speed
> + PCT:
> - added "return" PAST.op node type for subroutine returns
> - added "keyed_int" scoping to PAST::Var
> - fixed calls to closures
> - automatically transcode 7-bit unicode to ascii for faster processing
> + NQP: added "return" statement, ?? !! ternary operator
> - Configuration
> + expanded step gen::opengl
> - Implementation
> + updated function and macro names to match pdd09_gc.pod
> + removed Super PMC
> + add ".namespace []" as alternative to ".namespace"
> + "make codetest" target runs standard coding tests
> - Miscellaneous
> + added P6object class for Perl 6 interfaces to objects in Parrot
> + ported OpenGL/GLU/GLUT bindings to Win32, BSD, and more Mac OS X variants
> + generate OpenGL/GLU/GLUT bindings by parsing system headers
> + new OpenGL example shapes.pir, covering basic OpenGL 1.1 / GLUT 3 APIs
> + new float4.pir structure-packing benchmark
> + reduced memory use for variables
> + improved constant string caching
> + made code g++ and gcc (with optimizations) safe
> + the usual collection of bugfixes and optimizations
>
>
> Many thanks to all our contributors for making this possible, and our sponsors
> for supporting this project. Our next scheduled release is 15 July 2008.
>
> Enjoy!
> ./smash

--
Alberto Simões - Departamento de Informática - Universidade do Minho
Campus de Gualtar - 4710-057 Braga - Portugal

perl6.announce: Parrot 0.6.3 "Beautiful Parrot" Released! by Nuno 'smash' Carvalho

2008-06-17T16:14:24+00:00

Greetings,

On behalf of the Parrot team, I'm proud to announce Parrot 0.6.3
"Beautiful Parrot." Parrot (http://parrotcode.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 0.6.3 is available via CPAN (soon), or follow the download
instructions at http://parrotcode.org/source.html. For those who
would like to develop on
Parrot, or help develop Parrot itself, we recommend using Subversion on
the source code repository to get the latest and best Parrot code.

Parrot 0.6.3 News:
- Specification
+ updated pdd09_gc.pod
- Languages
+ Cardinal:
- dramatically improved parsing speed
- added support for defining and instantiating classes
- started fleshing out the builtin class hierarchy
- added support for hashes and arrays
+ Chitchat: added the start of a smalltalk compiler
+ Pheme: updated to match PGE changes
+ Pynie: return statement, other minor updates
+ Rakudo:
- added working list and hash contexts
- added 'return' statements
- added => pair constructor
- added ?? !! ternary
- added Range, range operators, Complex
- added common List, Hash methods
- refactored base classes
- added Mutable, Perl6Scalar classes
- added type-checking, is readonly/rw/copy for parameters
- added make localtest, docs/spectest-progress.csv
- fix named unaries
+ Squaak: implement return statement
+ Tcl: updated control flow exceptions to use new builtin types
- Compilers
+ All tools converted to P6object metamodel
+ PGE:
- <?foo> is now a zero-width match
- reduced backtracking to improve parsing speed
+ PCT:
- added "return" PAST.op node type for subroutine returns
- added "keyed_int" scoping to PAST::Var
- fixed calls to closures
- automatically transcode 7-bit unicode to ascii for faster processing
+ NQP: added "return" statement, ?? !! ternary operator
- Configuration
+ expanded step gen::opengl
- Implementation
+ updated function and macro names to match pdd09_gc.pod
+ removed Super PMC
+ add ".namespace []" as alternative to ".namespace"
+ "make codetest" target runs standard coding tests
- Miscellaneous
+ added P6object class for Perl 6 interfaces to objects in Parrot
+ ported OpenGL/GLU/GLUT bindings to Win32, BSD, and more Mac OS X variants
+ generate OpenGL/GLU/GLUT bindings by parsing system headers
+ new OpenGL example shapes.pir, covering basic OpenGL 1.1 / GLUT 3 APIs
+ new float4.pir structure-packing benchmark
+ reduced memory use for variables
+ improved constant string caching
+ made code g++ and gcc (with optimizations) safe
+ the usual collection of bugfixes and optimizations

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 15 July 2008.

Enjoy!
./smash

Patrick Michaud: Rakudo test suite progress

2008-06-16T06:34:15+00:00

[Rakudo spec regression status: 64 files, 779 tests]

In my previous posts I reported on Rakudo's progress in passing the test suite. For those I had been using the output of Test::Harness to estimate the number of passing tests, but ultimately decided that it wasn't really giving me the information I want in the form I want it. So I wrote a custom test summarizer for Rakudo to make it easier to measure our test passing rate in the spectest suite.

Here's how things have been progressing over the last three weeks:

Rakudo spectest regression daily results files test pass fail todo skip 2008-05-22 00:00 31 564 223 0 0 341 2008-05-23 00:00 32 569 228 0 0 341 2008-05-24 00:00 32 569 228 0 0 341 2008-05-25 00:00 39 666 310 0 0 356 2008-05-26 00:00 39 666 310 0 0 356 2008-05-27 00:00 39 666 310 0 0 356 2008-05-28 00:00 39 666 317 0 0 349 2008-05-29 00:00 43 774 394 4 15 361 2008-05-30 00:00 43 775 415 0 15 345 2008-05-31 00:00 43 775 415 0 15 345 2008-06-01 00:00 52 892 518 0 15 359 2008-06-02 00:00 55 1012 623 0 15 374 2008-06-03 00:00 55 1012 623 0 15 374 2008-06-04 00:00 55 1012 624 0 14 374 2008-06-05 00:00 58 1107 668 0 15 424 2008-06-06 00:00 58 1110 674 0 14 422 2008-06-07 00:00 59 1139 682 0 16 441 2008-06-08 00:00 59 1139 697 0 17 425 2008-06-09 00:00 59 1139 699 0 15 425 2008-06-10 00:00 59 1139 699 0 15 425 2008-06-11 00:00 59 1145 705 0 15 425 2008-06-12 00:00 59 1145 705 0 15 425 2008-06-13 00:00 60 1148 707 0 15 426 2008-06-14 00:00 60 1148 711 0 15 422 2008-06-15 00:00 63 1201 754 0 15 432 2008-06-16 00:00 64 1226 779 0 15 432

The 00:00 in the above table represents midnight U.S. Central Time. The 'test' column indicates the number of tests run, and the 'pass' column shows how many tests were passing (excluding 'todo' and 'skip'). So, as of June 16 Rakudo is passing 779 tests in its spectest_regression suite.

In order to get a feel for our overall trend, I like to look at week-over-week progress instead of just the daily numbers. So, in the week from June 9 to June 16 we added 779 - 699 = 80 new passing tests. We'll see how things go in the future -- it will all depend on a combination of how quickly spectests can be reviewed and features added to Rakudo.

At the top of my future posts I plan to include a one-line summary of Rakudo's passing rate so that people can continually monitor our ongoing progress without having to scan the article (see the top of this post for an example).

Update: Moritz has generated a PNG graph of the above data, and is updating the script to take advantage of the CSV version of the data that we'll be maintaining in the repository. (I don't know how often the graph will be updated yet.)

Patrick Michaud: YAPC::NA -- Parrot Hackathon report #1

2008-06-16T06:07:50+00:00

[Rakudo spec regression status: 64 files, 779 tests]

Throughout YAPC::NA I'm hoping to make frequent posts about various events taking place, including the hackathons and other discussions. The Parrot hackathon took place yesterday and today, and we've had a lot of contributors and some excellent progress. A brief summary of some things that have occurred during the hackathon:

Moritz noticed a post on Perl Monks about Rakudo not yet supporting %*ENV; I forwarded it to rakudobug@perl.org to be filed as an RT ticket, and just a few minutes after that Jonathan had a working implementation.

Scott Duff (PerlJam) tested and closed a long-standing patch from Zev Benjamin to supply the 'run' function in Rakudo.

Bacek made some fixes to S29-list/minmax.t and so we've been able to add that as another set of tests to our spectest_regression suite. Jerry Gay also noticed some additional tests for spectest_regression. As of this writing Rakudo is now passing 779 tests in the spectest_regression suite.

Jerry Gay (particle) updated the mk_language_shell.pl script to automatically include the optable rules.

I did more refactoring and updating of the Range operators, as well as added the ternary ?? !! operator to Rakudo.

Olivier Mengué (dolmen) supplied patches to add the .perl method to Bool and Range types.

Chromatic has been cleaning up a lot of IMCC issues, improving constant string handling, and otherwise applying tons of patches to Parrot. Right now Parrot has 658 outstanding tickets -- we're hoping that can go below 650 by Tuesday's Parrot release.

Allison, chromatic, and I had a discussion about adding new branch opcodes to Parrot that PGE can use in lieu of bsr/ret for local subroutines. We came up with the "push and branch/jump" and "pop and branch/return" opcodes. These work like bsr/ret, but instead of using Parrot's internal control stack they allow the user to provide a stack such as ResizableIntegerArray, which can be much more efficient at storing and processing addresses.

Coke struggled a bit with Tcl on Parrot, but managed to update it to use the new control exception types in Parrot (and that Rakudo and NQP are now using for their 'return' statements).

I spent most of Saturday looking at trying to make a precompiled form of Rakudo's Test.pm script, so that we aren't re-compiling it upon each test execution. But I ran into some snags having to do with :load/:init handling, so I've stepped back from that for a day or two to re-think it (and there's been plenty of other good things going on).

Everyone at the hackathon seems to be having a terrific time, and we're getting lots done. I know that this report misses a few people who were working on documentation -- I'll try to get details in my next post.