Planet Perl Six

Moritz Lenz (Perl 6): Mini-Challenge: Write Your Prisoner's Dilemma Strategy

Tue, 07 Feb 2012 16:41:42 +0000

Here is a small task we considered for the Perl 6 Coding Contest, but not chose to not pursue. But it's a nice little challenge for your leisure time.

In the Prisoner's Dilemma, two suspected criminals can chose to not betray each other (which we call "cooperate"), or betraying the other ("defecting"). If only one suspect betrays the other, the traitor gets released and the betrayed on gets a long sentence; if both betray each other, both get a rather long sentence. If both cooperate, both get rather short sentences.

It becomes more interesting when the dilemma is repeated multiple times. Now instead of prison sentences the contestants are assigned scores, which add up over multiple rounds.

I challenge you to write one or two strategies for the iterated prisoner's dilemma, and send them to moritz@faui2k3.org no later than Friday February 17.

You'll find some basic strategies and a harness here. It runs on both newest Rakudo and Niecza.

The scoring is as follows, where True means cooperate and False means defect:

my %scoring =
    'True True' => [4, 4],
    'True False' => [0, 6],
    'False True' => [6, 0],
    'False False' => [1, 1],

Your strategy should be a subroutine or block that accepts the named parameters mine and theirs, which are lists of previous decisions of your own algorithm and of its opponents, and total, which is the number of laps to be played. It should return True if it wishes to cooperate, and False to defect.

Here is an example strategy that starts off with cooperating, and then randomly chooses a previous reaction of the current opponent:

sub example-strategy(:@theirs, *%) {
    @theirs.roll // True;
}

Your strategy or strategies will play against each other and against the example strategies in the gist above. It is not allowed to submit strategies that commit suicide to actively support another strategy.

I too have written two strategies that will take participate in the contest. Here is the checksum to convince you I won't alter the strategies in response to the submissions:

6d4ba99b66e4963a658c8dcfc72922dd0f74e0ad  prisoner-moritz.pl

Carl Masak: A four-quarter plan for psyde

Sun, 05 Feb 2012 15:15:49 +0000

(If this blog had had a tagging system, this post would have been tagged "meta". You have been warned.)

My blogging engine, psyde, is written in Perl 6. It's been serving me well in the past year-and-a-quarter since use.perl went read-only and forced my hand to move out. It's not too powerful, and it's kinda idiosyncratic, but it has definitely made me happy.

Howver, I've increasingly been picturing where I want to take the engine. It could do more, and I know what bits I want it to do. But I never seem to get to the point where I sit down and tinker with the engine to make it better. You know, 'cus of tuits and yaks and worse-is-better and all that.

So, here goes. Now I'm changing that. Here's a four-quarter plan for 2012:

Q1: A way to create/edit posts online. All my posts sit in a repository, and theoretically I should be able to post from anywhere, just as one would expect from a blogging system. But in practice, I always forget to push the latest changes, which makes my local copy on my laptop at home the athoritative copy, and so I've no choice but to post from home. Better then to author all the posts online, to store the post sources on the server, and to have those be the authoritative copy.

Also, while writing posts in Emacs is hard to beat from a text-editing perspective, one advantage of authoring posts on the web is that I would be able to see the final HTML in real time. Better than having to wait through a compilation process to see how things will turn out.
Q2: A commenting system. Finally. This includes converting the old comments from my use.perl blog and retroactively adding them, too. I'd like to use something like openID as an authentication system, or more generally, something ready-made that makes it easy to just post a comment.

I've had Disqus recommended to me as a way to add a commenting system. While it would certainly work, it's slightly against the spirit of psyde, which is more or less NIH. I believe it's sometimes healthy to reinvent wheels, as much to learn how to make them round as to figure out how they can be made to work better. More importantly, I wouldn't be as happy with someone else's wheel, even a well-oiled one, as with my own.
Q3: A text-oriented graphic library. I often desire to inject an image/graph/diagram or other in my blog posts, explaining the relationship between some things. But there are too many steps involved:
1. open up Inkscape
2. painstakingly draw and position a bunch of rectangles and arrows and text fields
3. export to .png
4. upload the .png
5. insert an <img> tag linking to where I uploaded the .png
...and besides, I always feel like I'm doing work that the computer should be doing for me, drawing the SVG. I've drawn graphs like that a hundred times before, and they all look the same.

Imagine if I could just write a few symbols in my post saying "oh boy, here comes le graph!", a bit like \[ takes me into the displaymath environment in LaTeX. And, and! Because the groundwork I will have done in Q1 enables me to see things as I type in real time, and because the graphic library that I haven't written yet will be mostly implemented in JavaScript, those graphs will appear as I type them, too! Instant graphic!

I haven't seen any other blog system do this. Let me know if there's prior art.

The graphic library will support not just graphs, but any number of pre-set picture drawing modes: graphs, class diagrams, chess boards, hex boards, tables, Nim positions, pentomino pieces, sudoku boards, sparklines, basically anything. Adding another mode should be as simple (for a programmer) as adding a syntax parser and a generator of iamge elements. There will have to be an API for all of this; details pending.
Q4: Productize psyde. After all of this, I actually think I'll have something worth sharing with people. Not everyone will prefer to leave the cozy confines of Wordpress or Movable Type or whatever it is you kids use these days, but some people will like the combination of features in psyde, and will want to set up their own blog with it.

(This already happened once, by the way. I'm not at all prepared for this, so I just sent them a .tar.gz file with the minimal environment needed to get set up, and they succeeded. It was a slightly bumpy ride for them, of course, but they got it working.)

Now psyde isn't really a blogging system at its core, but a static web page generator. Making a product out of it probably means turning a bunch of things into modules, and have them play nicely with the Perl 6 ecosystem. And to make setup a one-click operation, or as close to it as possible. Probably psyde the static web generator and the yet-unnamed blogging software will separate into a module-client relation as a part of this.

When I say "product", by they way, I don't imagine I'll charge money for it. The source will remain free in all senses. I just want to make it easier to approach the software and get started with it. Removing various hard-coded things, as well as adding documentation and installation instructions, will be necessary.

Early on, I told myself that this would be a ten-year blog. We're now a few months into the second year. It's time to introduce some furniture and make this blog engine fit to live in. Fit to relax in after a hard day's work.

I want to be able to blog from my iPad. Or from a 40-inch TV screen. Or, in a pinch, from my Android phone. I want people to be able to comment. I want those graphs, oh nicely laid out, skinnable, auto-generated instant-gratification graphs! And I want to push psyde out into the wild, make it stand on its own legs.

I want to use Perl 6 more in production, because the time is here. Even though I know it's prudent to underpromise and overdeliver, I choose to publish my plans like this because so far I've only been scheming and dreaming — this somehow makes it all more concrete.

Onwards!

Carl Masak: t1: Expressing integers using four nines

Fri, 03 Feb 2012 22:43:28 +0000

(This is a guest post by Moritz Lenz. If you're wondering what this is all about, it's the aftermath of The 2011 Perl 6 Coding Contest. If you're not wondering, it's still about that.)

Let's consider the first task from the Perl 6 Coding Contest 2011.

Here is the description of the task once more:

What non-negative integers can you write as expressions containing exactly
four occurrences the number 9, and any of the binary operators *, /, %, +,
-, prefix negations, and any number of matching pairs of parentheses you
care to use?

The program should accept an upper limit N as a command-line argument.
It should then print all integers 0..N in increasing order, along with
an expression with four nines, if any such was found.

It was probably the easiest of all tasks, and the one we got the most submissions for. Yet there were still some things that could go wrong, and some submissions got some of them wrong:

non-negative implies that we start at 0 = 9 + 9 - 9 - 9, not at 1
integers means that the result must be an integer; it does not mean that intermediate results are automatically rounded or truncated
the expressions must consist of four times the number 9, not the digit nine. Thus 99 is not a valid expression of two nines.

All contestants solved this task with the following strategy: first find all possible expressions involving four nines (and possibly filter out those out of range), then iterate the numbers from 0 to N and check for each if an expression was found.

One possible approach to generate all expresions is to hard-code all expressions with one nine, namely 9 and -9. Then apply all operators to all combinations of those, and thus generate all expressions with two nines. This is a good time for filtering out duplicates, for example 18 == 9 + 9 == 9 - (-9). Then expressions of greater length can be generated from the shorter ones in the same manner.

There are just two small pitfalls: the first is that one has to remember that not all expressions can be generated by combining expressions of length (N-1) and 1. For example 324 == (9 + 9) * (9 + 9) can only be written as the combination of two expressions with two nines each. The second small pitfall is that three of the operators (%, / and -) are not commutative, so one has to explicitly try both combinations of $a op $b and $b op $a, or take care of that in some other way.

The task description allowed the prefix minus operator, but including it into the production does not produce any new numbers; it is sufficent to seed the expressions of length one with (9, -9).

As usual, we've published each contestant's code along with a review; feel free to have a look. This year, we're also publish solutions from people who sent in solutions but didn't sign up for the contest.

Isn't it noteworthy how, even with this narrow a task, people's solutions are all over the board in terms of the constructs used, code size, and style? We think so.

Stand by for the review on t2: "Sums of cubes".

perl6.announce: Announce: Niecza Perl 6 v14 by Stefan O'Rear

Mon, 30 Jan 2012 21:00:51 +0000

Announce: Niecza Perl 6 v14

This is the fourteenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. I'm on winter break now, which means lots
of time for improvements. Muahahaha. This month hasn't seen much of a
focus.

With this release, Niecza now passes a similar number of spectests as
Rakudo. (Note that they are not the same tests, and as such the test counts
are not completely comparable.) Solomon Foster has begun efforts to port
panda to niecza, with the goal of designing a multi-implementation ecosystem.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Breaking changes]

Multiple dispatch has been changed to reject all cases of ambiguity using
an explicit list of conflictors; some ambiguous cases were formerly accepted.

List iteration has been modified to throw an exception when the list generator
tries to access the unreified part of the list. The previous code exhibited
undefined behavior that some code actually relied on.

Parameters like @foo and %foo now insist that their arguments be Positional and
Associative respectively.

Rat and Complex stringification has been substantially changed.

Niecza now enforces "trusts".

&nextwith and CallFrame.args now conspire to hide the invocant parameter. If
you are calling nextwith directly, you no longer need to - and must no longer -
pass a self argument.

|$foo capture parameters now capture the logical "current" capture, rather
than the "initial" capture. In particular, 'method (|$foo)' no longer results
in $foo containing self.

[Major features]

END blocks are now suppported.

The subroutine entry process has been significantly modified to allow
signatures to reference variables and anonymous code blocks.

Roles have been overhauled to much more closely match the Rakudo nom behavior.
Importantly, roles now have a $?CLASS parameter, and role blocks are not run
until that is available. Role composition at compile time is now supported,
as is role summation, attributes in roles, role conflict detection, type
checking against roles, etc. &infix:<does> and &infix:<but> now support
much more of specced behaviors.

Accompanying that, the old tag and mixin classes CommonEnum, IntBasedEnum,
StrBasedEnum, Callable, Positional, Associative, Numeric, and Real have been
converted into roles.

Major signature improvements: Added support for sub-signatures, where blocks,
proper MMD with subtypes, values used as types. Parameter and Signature
objects are now reified into Perl 6 space and support a subset of the Rakudo
nom introspection API.

Niecza now supports constant folding! If you mark a sub 'is pure' and call
it with sufficiently constant arguments, it will be replaced at compile time
with the result of the call, provided said call does not throw an exception.
What constitutes "sufficiently constant" is not documented, poorly defined,
and subject to change.

Niecza now keeps attributes from different classes in different namespaces,
so you can have $!x in both a parent and a child class without issues.
Additionally, the sigil is part of the name, so you can now have both $!x
and @!x.

val() is now supported, and is used automatically on <> lists.

MAIN is now supported.

STD-imported syntax changes: \foo and |foo parameters no longer require a
sigil. my \foo = ... works to declare a "raw" variable. Initializer
assignment now binds tightly to the declarator, so that e.g. (5 + my $x = 3)
does something more useful.

.WHICH has been added, and === does the correct thing with value types now.

Perl 5 interop improvements: can now be used from any directory and builds
much more robustly. Also supports more pass and return cases. (Paweł Murias)

[Minor features]

No more pseudo-evaluators - all constructs which contain code that logically
is run at BEGIN time to produce a value, is now actually run rather than
attempting a static evaluation.

Binding to attributes is now supported.

Class attribute forms such as my $.foo, and the aliasing form has $foo, are
now supported.

Within a named 'anon sub', the name is visible, allowing for nicer recursion.

* now properly ignores assignments.

$obj.Foo::bar, $obj.::("Foo::bar"), and /<::("Foo::bar")>/ are now accepted.

You can now set 'is iffy' and 'is diffy' for fully custom operators.

Type adverbs :_ :U :D :T are now accepted.

Signatures like (Int, Int) are now allowed.

Phaser handling has been improved, and support added for CHECK.

Aliasing forms like <foo=$bar> are now allowed.

Version information is now embedded into the build, allowing for --version
and $?PERL support.

Printing of unhandled exceptions uses .gist.

Multiple dispatch now handles junctions.

Exporting multisubs from modules now approximately works.

Added ".Bridge" support. (Solomon Foster)

Attribute-binding parameters (:$!foo) now implemented.

:16() syntax is now supported.

Defaults and type constraints that are constants are now saved as such,
avoiding an unneeded block.

Str.perl now escapes special characters.

CLR interop now supports calls to shadowed and hidden methods, like
$obj.CLR::System::IDisposable.Dispose(). Note that this can NOT be used to
call overriden methods (callvirt semantics are used).

LTM processing ignores arguments and dispatchers forward the arguments to
multi regexes.

infix:<cmp> supports pairs, ±Inf (Solomon Foster)

Hash.perl sorts the output to be slightly more useful.

New setting things:
Cool.polar, Cool.roots, &roots, Array.delete, &infix:<minmax>, &rotate,
CommonEnum.pick, CommonEnum.roll, Any.min, Any.max, Any.minmax, Str.trans,
&elems, Any.reduce, &reduce, &shell, &categorize, &cwd, &chdir, $*CWD,
&printf, IO.copy, IO.chmod
(Solomon Foster)

Hash.push (Moritz Lenz)

Int.base, Array.splice, &splice (Will Coleda)

Set, Bag (Larry Wall, Solomon Foster)

&prefix:<sleep>, Mu.clone, &undefine, $*OS, Order, $*OUT, $*ERR

split and comb now support limits of Whatever (Solomon Foster)

Range coerces values to numbers according to spec (Solomon Foster)

&sort accepts a Callable first argument (Solomon Foster)

Added limit to &lines, :r and :rw to &open (Solomon Foster)

min and max support arbitrarily many arguments (Solomon Foster)

Changed sleep to return a value (Will Coleda)

[Other]

Daniel Ruoso is attempting an implementation of concurrent feeds.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

Next month is likely to see fewer tuits in general. The only concrete plan
I have is to continue with 6model convergence, hopefully reaching a point
where user-defined metaclasses are possible.

[1] https://github.com/downloads/sorear/niecza/niecza-14.zip
[2] https://github.com/sorear/niecza

Carl Masak: Macros progress report: interesting things

Sun, 29 Jan 2012 18:26:35 +0000

I'm here to report that the macros grant is coming along nicely. I've been busier with $dayjob than anticipated, and so the schedule in the grant application is slipping a bit. But I have a fairly good view of the obstacles and to-do items ahead.

When we last saw each other, I was saying that "variable lookups from inside of the quasiquote end up confused". That's still true, and it's the big thing I want to fix before merging my work so far into the master branch, er, into the nom branch.

What I've done since last time:

I wrote a thorough explanation of why ASTs need to carry around their original lexical environment. (Short explanation: they need to act like closures, but they aren't really so they need to fake it.)
I re-based the macros branch on the latest nom, and fixed a bunch of regressions in my code that fell out of that.
I started a test file with macro tests that are more adapted to the work that I'm doing than the tests that are already there. I hope to be able to absorb the ones that are there as we go along; right now they're not much use to me.

Now I'm well poised to go in and actually implement the lexical fixup I need for the ASTs to behave as if they're normal, honest-to-goblin closures. I just thought I'd blog this report in case I end up walking into the source code, never to return. :-)

The trick is this: .SET_BLOCK_OUTER_CTX. It lets you say "block, your OUTER is now this thingy". It's the kind of internal fixup that makes the cat walk by twice inside the Matrix.

Ok, I'm going in. See you on the other side.

Carl Masak: The Perl 6 Coding Contest (2011 edition) is now closed

Sun, 29 Jan 2012 12:04:28 +0000

"Thanks for organizing the contest. Staying awake until 3am trying to make my solutions work (and trying to write them in a perl6ish way) was big fun! :)"
        — one of the contestants

Ding! Pencils down.

So, here we are, five weeks later. Let's sum up:

35 people signed up as contestants this year.
6 contestants submitted solutions.
27 submissions passed a base-test run and were accepted.

Last year's figures for the same things are (18, 5, 26), so somehow we ended up with many more contestants and only one more submission this year. That's fine; we know that the step from signing up to actually sending something in is quite a big one to take. Very interested to get feedback on how to make that step easier, though.

Just as last year, we'll synchronously publish people's solutions for each task in sequence, along with a blog post about the task and its solutions.

Unlike last year, I won't promise that these posts will be forthcoming in "the next few days/weeks". Last year it took over two months to process everything and get to the winner. Let's aim for one month this time — we're as expectant as you are to see solutions, blog post explanations, and an eventual winner!

Let's do this! First up: "Four 9s."

Solomon Foster: Modules and Perl 6s

Sun, 29 Jan 2012 03:09:35 +0000

I’ve got my fork of Panda working on Niecza. There are two major issues which remain to be resolved, and they are both related. Panda assumes that the Perl 6 executable is named perl6, and modules should be installed to ~/.perl6. That’s great if you have just one Perl 6 on your system.

But in this crazy modern world, I really want both Rakudo and Niecza on my system. And so assuming that everything can be named “perl6″ is a big problem.

I’m hoping the community can put their heads together and figure out a solution. I guess the obvious one (which would only need sorear to buy in) would be installing Niecza modules to ~/.niecza, and creating a “real” niecza executable. (It might well be as simple as a one-line script “mono path-to-Niecza.exe”.)

Good idea or bad idea?

Jonathan Worthington (6guts): This month’s Rakudo Star release – and what’s coming next

Sun, 29 Jan 2012 00:41:57 +0000

So, we made it – a Rakudo Star release based on the “nom” development branch has landed. It’s based on the compiler release moritz++ cut earlier this week, and pmichaud++, masak++ and myself have been involved in getting the Star-specific build and installation scripts in shape.

Getting to this release has been a lot of work; in many sense this is a revolution in Rakudo’s development rather than a small evolutionary step over what we had before. It’s taken a while (indeed, longer than I’d first hoped) and has been a lot of work – but it was worth it. Not just because of the many improvements that are in this release, but because of the enormous future potential that we now have.

Here’s some of the things I’m happiest about in the release.

The performance improvements in many areas. Yes, we’ve plenty of work to do here – but this is a solid step forward for a wide range of scripts, and in some cases an order of magnitude improvement.
That 6model – something I started designing a year and a half ago – has not only cleanly supported all of the things we needed to do in Rakudo, but also opened up so many other doors. For example, the new NativeCall module uses its representation polymorphism support to great effect.
Protoregexes doing real NFA-driven Longest Token Matching rather than the cheating version we had before that only operated on literals.
The optimizer, along with the various extra compile time error reporting it gives. This will be an important future area for Rakudo.
Initial native type support, and bigint semantics for the Int type.
The POD6 support, thanks to tadzik++’s Google Summer of Code grant in summer.

So, what’s next? Currently I’m working hard on getting true bounded serialization support in place. This should further improve BEGIN time support (including constructs that depend on BEGIN time), greatly cut down resource consumption during CORE.setting compilation (both time and memory) and give us faster startup. It’s hard to guess at figures for the improvement, but I’m expecting it to be a noticeable improvement in all of these areas. I’m aiming at getting this landed for the next Rakudo compiler release (which I expect us to do a Star release based on too), though largely it depends on whether I can get it working and stable enough in time; while some parts are a simple matter or programming, other parts are tricky.

That aside, we’ve already got various other new features in the pipeline; even since last weekend’s compiler release, there are multiple new regex-related things in place, moritz++ has continued with his typed exceptions work, we’re catching a couple more errors at compile time rather than letting them slip through until runtime, and there’s various other miscellaneous bug fixes. Also, masak++ is working on macros in a branch, and I’m optimistic that we’ll have some initial macro support in place by the next release also. Busy times! :-)

rakudo.org: Rakudo Star 2012.01 released

Sat, 28 Jan 2012 14:34:19 +0000

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the January 2012 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the January 2012 release is available from http://github.com/rakudo/star/downloads.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release #48 of the Rakudo Perl 6 compiler [1], version 3.11 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

Significantly, this is the first distribution release based on the “nom” (New Object Model) development branch of Rakudo. This work has been carried out with the aim of increasing performance and correctness, as well as providing a better base for taking on a
range of missing features. Here are some of the major improvements in this release over the previous distribution release.

Greatly improved performance in many areas. For some scripts, this release offers an order of magnitude performance improvement.
POD6 support, including the $=POD variable to make the POD available at runtime and a –doc option to get at the POD
The Int type now has big integer semantics
Initial work on native types, which can be used to write much more efficient code
LTM-driven protoregexes
Meta-programming support, including custom meta-classes, overriding method dispatch and much more
Exception handling is much closer to the specification, and thus much more useful
Better package handling, including true separate compilation, lexical packages and better nested package handling
An optimizer, which improves generated code as well as catching a range of issues at compile-time that previously made it to runtime
Backslash sequences in character classes
Stubbed methods from roles are now required, providing interface style functionality
Typed arrays and hashes, as well as supporting for binding to array and hash elements

Due to improvements in the Perl 6 language specification, and changes to Rakudo to track them, some existing code will need changes. Here are some of the major differences to be aware of.

Attributes can no longer be initialized using “new” unless they are declared as having an accessor; either give them one or write a BUILD submethod
The proto keyword is no longer used to declare a multi-dispatch fallback
You may no longer do ‘filename’.lines; use ‘filename’.IO.lines

We have maintained backwards compatibility with some changed pieces of syntax, but will drop them in an upcoming release:

“<…>” in proto regex bodies; now this should be written “*”
The use of “**” with a separator in regexes; this is now done by using “%” or “%%” on another quantifier

While this release does contain a great number of improvements, unfortunately we have regressed in a few places. Of note:

Some cases of auto-vivification do not work
The binding of a capture against a signature literal is broken

We will be working to restore this functionality for future Rakudo Star releases; if you depend heavily on it, you may wish to stick with the previous Rakudo Star release for
another month.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:

pack and unpack
macros
threads and concurrency
Unicode strings at levels other than codepoints
pre and post constraints, and some other phasers
interactive readline that understands Unicode
non-blocking I/O
much of Synopsis 9

There is a new online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo Star 2012.01 and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at <rakudobug@perl.org>.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. An updated draft of a Perl 6 book is available as <docs/UsingPerl6-draft.pdf> in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see <http://rakudo.org/how-to-help>, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Jonathan Worthington (6guts): Looking back, looking forward

Sun, 15 Jan 2012 22:28:47 +0000

So, 2012 is here, and here’s my first Perl 6 post of the year. Welcome! :-)

Looking Back

2011 brought us a faster Rakudo with vastly improved meta-programming capabilities, the first work on exploring native types in Perl 6, the start of a powerful type-driven optimizer and many other bits. It also took me to various conferences and workshops, which I greatly enjoyed. I’d like to take a moment to thank everyone involved in all of this!

This was all great, but slightly tainted by not managing to get a Rakudo Star distribution release out based on a compiler release with all of these improvements. I’d really hoped to get one out in December. So what happened? Simply, there were a certain set of things I wanted to get in place, and while many of them got done, they didn’t all happen. While the compiler releases are time based – we do one every month – the distribution releases are more about stability and continuity. By the time I headed back to the UK to spend Christmas with family, we were still missing a some things I really wanted before a Star release was done. Given the first thing that happened when I started relaxing a little was that I immediately got unwell, I figured I should actually use my break as, well, a break – and come back recharged. So, I did that.

So, let’s try again

So, the new goal is this month. I’m happy to report that in the week since I’ve got back to things, one of the things I really wanted to sort out is now done: Zavolaj, the native calling library, now does everything the pre-6model version of it did. In fact, it does a heck of a lot more. It’s even documented now! It’s also far cleaner; the original implementation was built in the space of a couple of days with mberends++ while I was moving apartment, and was decidedly hacky in places. The missing bits of the NativeCall library were important because they are depended on by MiniDBI, and I really didn’t want to ship a Rakudo Star that can’t connect to a database. So, next up is to make sure that is in working order. I’m not expecting that to be difficult.

That aside, there were some things to worry about in Rakudo itself. I’ve dealt with some of those things in the last week, and perhaps the one important remaining thing I want to deal with before Star is a nasty regex engine backtracking related bug (I’ve been hoping pmichaud++, the regex engine guru, might appear and magic it away, but it seems it’s going to fall on my plate). But overall, we’re well on track to cut the Star release this month.

What’s the direction for the year ahead?

During 2011, Rakudo underwent a significant overhaul. It was somewhat painful, at times decidedly not much fun, but ultimately has been very much worth it: many long standing issues have been put to rest, performance has been improved and many things that were once hard to do are now relatively easy or at least accessible.

I think it goes without saying that we won’t be doing any such wide-ranging overhaul in 2012. :-) The work in 2011 has opened many doors that we have yet to walk through, and 2012 will see us doing that.

At a high level, here’s what I want:

Less bugs, more stability: mostly this is about continuing to work through the ticket queue and fix things, adding tests as bugs are fixed to ensure they stay fixed.
Better error reporting: there are things in STD, the Perl 6 standard grammar, that allow it to give much more informative error reports on syntax errors than we often can in Rakudo today. I want us to bring these things into Rakudo. Additionally, there’s plenty of improvements to be made in runtime errors. Furthermore, I want to expand the various bits of static analysis that I have started doing in the optimizer to catch a much wider range of errors at compile time.
Run programs faster: this process is helped by having decent profiling support these days. There’s a lot more to be done here; the optimizer will help, as will code generation improvements.
Compile programs faster: this will come from more efficient parsing and greatly improving the quality of the code NQP generates (NQP is the language we write much of the compiler in)
Shorter startup time: this mostly involves finishing the bounded serialization work up. I think the best way to describe this stuff is “fiddly”.
Use less memory: Rakudo has got faster in no small part thanks to being able to understand its performance through profiling. Our understanding of its memory consumption is much more limited, which makes it harder to target improvements. That said, I’ve some good guesses, and some ideas for analyzing the situation.
More features: while being able to do the things Rakudo can do today faster and with less bugs would give a very usable language for quite a lot of tasks, there’s still various features to come. In particular, Rakudo’s support for S05 and S09 needs work.
VM Portability: we’ve made good progress towards being able to make a serious stab at this over the last year, while at the same time also managing to perform vastly better on Parrot too. With help from kshannon++, I’m currently working on completing the switch to QRegex (a regex engine with NFA-powered LTM, and written in NQP rather than PIR), which should carry on a pattern of simultaneously increasing performance and portability. Beyond that will be an overhaul of our AST and code generation, with the same goal in mind.

So, lot’s of exciting things coming up, and I look forward to blogging about it here. :-)

A way to help

There are many ways to get involved, depending on what you’re interested in. One way is to take a look at our fixed bugs that need tests writing. At the time of writing, just short of 100 tickets are in this state. No guts-y knowledge needed for this – you just need to understand enough Perl 6 (or be willing to learn enough) to know what the ticket is about and how to write a test for it. Drop by #perl6 for hints. :-)

Solomon Foster: Last Piece of Pi

Sun, 15 Jan 2012 17:43:08 +0000

So, last spring I did a series of posts on making a Pi spigot. Unfortunately, the project foundered a bit, as it turned out that only Rakudo had the subsignatures needed to make the code pretty, but only Niecza had the big integers needed to make the code work.

Fast forward to today. Now both Rakudo and Niecza can handle the best version of the code with no problem!

Let me emphasize that again. A year ago, neither implementation had those two features. Eight months ago, each had one. Today both have both. What’s more, Rakudo seems to have made some pretty impressive speed improvements.

I think the awkward situation with Rakudo Star has helped obscure the fantastic good news in Perl 6. Right now we have two distinct Perl 6 implementations, each of which is markedly better than anything available a year ago. While there are still some rough edges, both implementations are making visible progress every single day.

Okay, enough cheerleading. Time to finish the Pi story. It turned out there was one last complication. I had never actually tried to use the code to generate an infinite stream of Pi. I always stopped at 40 digits. Confident that the algorithm must work, I tweaked the output to just continue generating digits until you stopped the program with control-C. And I ran it using Niecza, and it printed

3.14159265358979323846264338327950288419716
9399375105820974944592307816406286208998628
0348253421170679821480865

And then it just sat there, calculating. After a few minutes I stopped it and tried it in Rakudo, and got the exact same result.

Well, after adding a few debugging says here and there, I found the problem. The extr sub was returning NaN. It turns out that both compilers have issues with dividing huge numbers. Here’s the division operation that was causing the problem:

826855066209083067690330954944954674053
707782399091459328155002954168455127712
564546723209828068849110223672691692080
858850302237001093531862737473606364113
314687502675869281622802970765988449203
963736097729699655628829895255493809983
868753943269929165690008254816168624365
041070395716948346309925280258763697273
816643106559428329680316113883598846477
019844021876290510680558354153412094804
165563855909020631086890050609449881578
622437959410200560054513816596644762131
226627968813825552929967132893776980417
525678140579476414867767644626389410380
794467097761379794479928269796859019439
705966555011741254554959832606241504043
482378842096776403191455346497512084739
323724281071973237937801014210278895804
940475966938880398182275335278425442994
287812050560074302564177393567873480740
249095636709741437469651121924884638352
523975466249955052660310789169884060356
70777782797813415527343750 / 7640045443
915776552858245682965495041201868477244
923723289802372673948570989301189643881
881673616027696317656701576457227272117
067947294675092324411286583110995372015
785893970194452345100095389207557064515
618905243737091067059065039684867000766
465399984513882758095027633673968549038
659642193965599458094646356792444696562
299054844575814305259223023977803302735
242307789179027935107449661143998428584
590618630170775872761731454567203230484
311106708425828778192240791257477924515
937573923664112071637127786446287936043
637833776529791999568414593746973068229
015816020732598109749879566833692821582
816119454436978125677364775318707235283
939392855143663977524974469973442065855
128922452382372338686111634084367257050
255499210918328304009454606504169283500
652033488755998721320134300149134205253
095344802815192912405314659633341062491
389270940370884860337596933277382049709
5584869384765625

Turns out the bottom number is bigger than can be presented by a Num, so the division operation ends up becoming N / Inf, which is NaN. This is a bit obscured in Rakudo because the division operation actually returns a Rat (which should be illegal according to the spec!), but then .floor is called on the result, which tries to convert to Num before doing the floor operation.

This opens several questions, like: Should Perl 6′s Int / Int operator be modified to try to cope with this sort of case?

But for the Pi problem, the good news is this: every time the extr sub is called, its result is fed to .floor. That means we simply needed to replace / with div and get rid of the .floors.

And with that, the code easily produces 2,000+ digits of Pi using either Rakudo or Niecza!

Tadeusz Sośnierz (tadzik): State of Dancer on Perl 6

Fri, 06 Jan 2012 00:24:34 +0000

Bailador is growing better and bigger and starts to resemble a real tool more and more. Let’s see what new features it has gained recently.

Remember the first example in perldoc Dancer? It’s not really different in Bailador:

use Bailador;
get '/hello/:name' => sub ($name) {
    return "Why, hello there $name"
}
baile;

Aside of being a little Spanished, what did it give us? We have subroutine signatures in Perl 6 so we can pass the :name parameter to the sub; there’s no need to use param() now: it’s gone.

You don’t need to pass everything using GET of course. post keyword is also supported.

post '/' => sub {
    return request.params.perl
}

The above will print something like ("foo" => "bar").hash, if fed with appropriate request.

any() is a reserved keyword in Perl 6, and while you can use it, it means a completely different thing. Instead of any('get', 'post') you can just do it like this:

get post '/' => sub {
    if request.is_get {
        return "I am GET"
    } else {
        return request.params.perl
    }
}

post, as well as get return their arguments, so you can chain them like in the example above. It also shows the joy of request object, which you can use to inspect the request being processed. It’s not as cool as Dancer::Request, but it does the job, being quite small and simple.

What else do we have? Let’s show off a bit and write a simple-simple pastebin webapp.

use Bailador;

unless 'data'.IO ~~ :d {
    mkdir 'data'
}

get '/' => sub {
    template 'index.tt'
}

post '/new_paste' => sub {
    my $t  = time;
    my $c = request.params<content>;
    unless $c {
        return "No empty pastes please";
    }
    my $fh = open "data/$t", :w;
    $fh.print: $c;
    $fh.close;
    return "New paste available at paste/$t";
}

get /paste\/(.+)/ => sub ($tag) {
    content_type 'text/plain';
    if "data/$tag".IO.f {
        return slurp "data/$tag"
    }
    status 404;
    return "Paste does not exist";
}

baile;

Holy cow, what’s that! Let’s go there piece by piece. First, we’ll create a data directory if it doesn’t already exist. No black magic here, let’s proceed. What’s next? Templates! Here we just load index.tt, not passing any parameters, but that works too and some example apps use that in their example templates.

The handler of new_paste uses our well-known request object again, and creates a new file for a paste, identified by the current time.

The last get block uses some nifty features, so let’s take a look. It uses regexes, and you can see that they also cooperate with subroutine parameters without black magic. We then set a content_type as we’ll do in Dancer, and send status 404 if no paste have been found. Easy peasy? I suppose so. That’s it, it works like a charm.

Thus we’ve covered all the features in Bailador as for now. I don’t think it’s that poor, as for about 100 lines of code.

What’s next? What’s missing? You tell me. Or you contribute; the code is dead simple and implementing stuff like before(), after(), before_template() etc should be a matter of 3-5 lines, I think. Feel encouraged to look into the code and hack on it. If you have any questions, suggestions or criticism, don’t hesitate to tell, or poke me on #perl @ Freenode. Have fun!

Moritz Lenz (Perl 6): Perl 6 in 2011 - A Retrospection

Sat, 31 Dec 2011 18:00:00 +0000

The change of year is a good occasion to look back. Here I want to reflect on the development of Perl 6, its compilers and ecosystem.

At the start of the year, masak's Perl 6 Coding Contest continued from 2010, concluding in the announcement of the winner. I must admit that I still haven't read all the books I won :-)

Specification

2011 was a rather quiet year in terms of spec changes; they were a mixture of responses to compiler writer and user feedback, and some simplifications and cleanups.

Positional parameters used to be allowed to be called by name; this feature is now gone. That both makes the signature binder simpler, and removes accidental dependencies on names that weren't meant to be public. Read the full justification for more background.

A small change that illustrates the cleanup of old, p5-inherited features was the change that made &eval stop catching exceptions. There is really no good reason for it to catch them, except Perl 5 legacy.

say now uses a different stringification than print. The reasoning is that print is aimed at computer-readable output, whereas say is often used for debugging. As an example, undefined values stringify to the empty string (and produce a warning), whereas say calls the .gist method on the object to be said, which produces the type name on undefined values.

An area that has been greatly solidified due to implementation progress is Plain Old Documentation or Pod. Tadeusz Sośnierz' Google Summer of Code project ironed out many wrinkles and inconsistencies, and changed my perception of this part of the spec from "speculative" to "under development".

Rakudo

Rakudo underwent a huge refactoring this year; it is now bootstrapped by a new compiler called "nqp", and uses a new object model (nom).

It allows us to gain speed and memory advantages from gradual typing; for example the mandelbrot fractral generator used to take 18 minutes to run on a machine of mine, and now takes less than 40 seconds. Speedups in other areas are not as big, but there is still much room for improvement in the optimizer.

With the nom branch came support for different object representations. It makes it possible to store object attributes in simple C-like structs, which in turn makes it much easier and more convenient to interoperate with C libraries.

Tadeusz' work on Pod gave Rakudo support for converting Pod to plain text and HTML, and attach documentation objects to routines and other objects.

Rakudo now also has lazy lists, much better role handling, typed exceptions for a few errors, the -n and -p command line options, support for big integers, NFA-based support for proto regexes and improvements to many built-in functions, methods and operators.

Niecza

It is hard to accurately summarize the development of Niecza in a few sentences; instead of listing the many, many new features I should give an impression on how it feels and felt for the user.

At the start of 2011, programming in niecza was a real adventure. Running some random piece of Perl 6 code that worked with Rakudo rarely worked, most of the time it hit a missing built-in, feature or bug.

Now it often just works, and usually much faster than in Rakudo. There are still some missing features, but Stefan O'Rear and his fellow contributors work tirelessly on catching up to Rakudo, and it some areas Niecza is clearly ahead (for example Unicode support in regexes, and longest-token matching).

Since Niecza is implemented on top of the Common Language Runtime (CLR) (which means .NET or mono), it makes it easy to use existing CLR-based libraries. Examples include an interactive fractal generator and a small Tetris game in Perl 6.

Perlito

Perlito aims to be a minimal compiler with multiple backends, which can be used for embedding and experimenting with Perl 6. It had several releases in 2011, and has interesting features like a Javascript backend.

Ecosystem

The presence of two usable compilers (and in the case of Rakudo, two viable but very different branches) has led to many questions about the different compilers. The new Perl 6 Compiler Feature matrix tries to answer the questions about the state of the implemented features in the compilers.

With Panda we now have a module installer that actually works with Rakudo. It still has some lengths to go in terms of stability and feature completeness, but it is fun to work with.

The new Perl 6 Modules page gives an overview of existing Perl 6 modules; we hope to evolve it into a real CPAN equivalent.

Community

This year we had another Perl 6 Advent Calendar, with much positive feedback both from the Perl 6 community and the wider programming community.

We were also happy to welcome several new prolific contributors to the Perl 6 compilers and modules. The atmosphere in the community still feels relaxed, friendly and productive -- I quite enjoy it.

The year ends like it started: with a Perl 6 Coding Contest. This is a good opportunity to dive into Perl 6, provide feedback to compiler writers, and most of all have fun.

Carl Masak: The -c flag

Wed, 28 Dec 2011 20:49:17 +0000

The p6cc contest is underway. Yay.

[Coke]++ discovered on the channel that Rakudo nom didn't have a -c flag. The five base-test files all syntax-check the corresponding code files using the -c flag. Which made Rakudo nom and the base-test files incompatible. Oh noes.

<moritz> moritz-- # not reviewing the test harness properly

The fault is even more mine, of course, since I wrote the test harness. And I may be an "early adopter" with Perl 6, but I'm always very late at switching over to a new Rakudo branch.

I was late at switching over to ng, back when it was still called ng. And I'm late this time in switching over from b (the new name for ng, since the n stands for "new" and ng isn't new anymore) to nom (the new "new").

I'll take the leap any day now, I promise.

moritz++ was quick in patching up the -c omission.

<dalek> rakudo/nom: a9bead6 | moritz++ | src/ (2 files):
<dalek> rakudo/nom: reimplement -c command line option
<dalek> rakudo/nom: review: https://github.com/rakudo/rakudo/commit/a9bead6d48
<moritz> masak: there you go
<masak> moritz++

What this means in practice is: you can't use the latest Rakudo nom compiler release to solve the p6cc problems. Not without modifying the base-test files anyway. But you can use the bleeding-edge git checkout of the nom branch.

If you're on either Niecza or Rakudo b, things should be fine: those have a working -c flag already.

Those are the breaks. Perl 6 is evolving, and the mat is constantly being pulled out from under us. To keep up, one has to do a jig now and then.

We added a NOTES file to keep track of information of this kind that we didn't manage to get into the contest instructions.

On the channel, we also had some nice concluding discussion about the nature of the -c flag.

<moritz> to me it felt a bit like a cheat
<moritz> because there is already some mechanism for specifying the target stage
<moritz> but it's too tightly coupled to the output from the existing stages to
         be easily usable
<moritz> so I feel like hijacking an existing mechanism
<masak> I guess.
<masak> in some sense, "checking syntax" isn't so much of a compiler stage as...
        a decision not to go past a certain compiler stage.
<TimToady> in a sense, -c adds the final CHECK, that just exits with status
<masak> right.
<TimToady> it can even be implemented that way, since CHECKS do lifo order
<masak> "everything turns out to be yet another setting" :)
<TimToady> yes, it could also be done with a variant setting, but that seems
           a bit heavyweight
<TimToady> otoh, it would be possible to sneak CHECK pushes in before the -c,
           so maybe a setting is the cleaner way

perl6.announce: Announce: Niecza Perl 6 v13 by Stefan O'Rear

Mon, 26 Dec 2011 15:55:30 +0000

Announce: Niecza Perl 6 v13

This is the thirteenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. I'm on winter break now, which means lots
of time for improvements. Muahahaha. A large portion of the improvements
have been in regular expression support.

Will Coleda and Solomon Foster are working on fudging spectests so they run on
Niecza; between that and actual improvements, we've gained 5000+ working
spectests since the last release. See [4] for a dramatic visual.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Breaking changes]

/ <{ foo }> / is no longer accepted as a synonym for / <?{ foo }> /.

$0 is no longer allowed to mean $/ when there is no capture zero.

$/.ast no longer defaults to ~$/. Use $() if you want that.

[Major features]

New-style character class expressions like <:Uppercase & :Greek> are
now supported.

Unicode property access is now supported! In addition to the use in
regexes, there is also a minimal Niecza::UCD module which allows querying
the properties of characters. All non-Unihan properties defined in
Unicode 6.0.0 are available.

Runtime number parsing has been radically extended and now supports the
full gamut of Perl 6 number syntaxes.

[Minor features]

Supplementary characters are now generally supported, though StrPos-type
counting for chars, substr, etc is still in UTF-16 code units.

codes is now available, for when you need to actually count code points
(be aware that it is O(n)).

The regex infix operators & and && are now supported (currently treated
as synonyms, but don't rely on this).

&pow is a little bit smarter and needlessly returns NaN in fewer cases
(Solomon Foster, Stefan O'Rear).

Imported a few tweaks from STD, including a better message for say$_.

\h \v \s \w and \d now use the recommended UTS18[3] definitions.

\n, $$, and ^^ now match any vertical whitespace, including CRLF as
a single unit.

% is now supported for all quantifiers and %% is available too.

$() @() %() are now supported.

/$var/ now allows $var to be a Regex. Likewise /<$var>/, /@var/,
and /<@var>/ have been added. / <{...}> / now does the right thing.

The implementation of $<foo>=[...] has been considerably simplified
and depessimized.

/ <.$foo> / assertion syntax is now supported.

/ <foo> / will now call a lexical "my regex foo" if possible. To minimize
potential accidents, this applies ONLY to regexes, tokens, and rules;
despite using the same namespace, a "my sub foo" will not be called.

/ <&foo(...)> / now allows arguments.

@<foo> now correctly contextualizes.

Nontrivial regex protos, like "proto regex foo { "bar" {*} }", are now
implemented.

<( and )> are now supported.

Added Match methods: kv, keys, values, caps, chunks, prematch, postmatch.

Added \c[] syntax in strings and regexes.

Rat and FatRat now stringify as fractions.

Inf now stringifies as "Inf" rather than leaking C# Infinity.

Added predefined quasi-property rules like <alnum>.

Added Niecza::Benchmark, providing the barest minimum of functionality like
Perl 5's "Benchmark".

Inf, NaN correctly handled in rounding, Cool.truncate, &kv, Pair.invert,
&srand, allow :x(*) in subst, .chr, .ord, .chrs, .ords, &chrs, &ords,
trim methods, &roll, .roll(*), end, min, max, minmax (Solomon Foster)

Added .pick(*), corrected Str and Numeric for Range (Will Coleda)

classify (Moritz Lenz)

[Selected bug fixes]

Threads.pm6 is working again, now with exception-safe locking.

[Other]

There is now documentation on how to prepare releases.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

My current priorities are:
1. Make regexes much more feature-complete, including general Unicode
properties and grapheme mode
2. Prototype the debugger
3. 6model convergence work, including roles/native types
4. Figure out how modules and S11 stuff should work in Niecza. Do it.

[1] https://github.com/downloads/sorear/niecza/niecza-13.zip
[2] https://github.com/sorear/niecza
[3] http://www.unicode.org/reports/tr18/
[4] https://github.com/flussence/specgraphs/raw/master/impls.png

Perl 6 Advent Calendar: Day 25 – Merry Christmas!

Sun, 25 Dec 2011 10:52:41 +0000

The kind elves who spend the rest of the year working in Santa’s shop to bring you more of Perl 6 each year would like to wish you a very warm and fuzzy Christmas vacation. December is always a special time for us, because we get to interact with you all through the interface of the advent calendar. We think that’s wonderful.

Be sure to check out this year’s Perl 6 coding contest, where you can win €100 worth of books!

Merry Christmas!

Carl Masak: The 2011 Perl 6 Coding Contest

Sun, 25 Dec 2011 10:20:17 +0000

With slightly less gratuitous buildup of expectations this year, but with no less grandeur and excitement, we're proud to announce:

The 2011 Perl 6 Coding Contest

Today there's two of us arranging the contest: the esteemed Moritz Lenz, and me. Hello.

This is a contest for people who are aware that Perl 6 has been "officially released", and who want the perfect excuse to start playing around with the language. As the Perl 6 community steadily works its way towards production-readiness, we can already enjoy ourselves royally by solving interesting Computer Science problems!

Here's the deal:

do five tasks, and you might win Amazon books worth €100!

Addendum: Thanks to an anonymous donor, we're very happy to announce that there is now a second prize too: Amazon books worth 100 USD!

The contest starts now, today on 2011-12-25. It ends five weeks later, on 2012-01-29. Registration is open for two weeks, starting now. Just send an email to cmasak@gmail.com — saying "sign me up!" or even sending your Amazon wishlist so we'll know which books to buy you if you win.

You might be curious about what you need to do to win. Here's an extract from the file RULES:

Since "code quality" is a slightly subjective measure, let us provide a few
hints of what we'll be looking for:

* Correctness.
* Readability.
* Consistency.
* Clarity of intent.
* Algorithmic efficiency.
* Idiomatic use of Perl 6.
* Brevity.

In short, what we're looking for is top-quality code. That's how you win.

Here are the five tasks. Write Perl 6 programs to...

Find a way to express an integer as an expression containing four 9s. (more)
List the numbers which are sums of cubes in more ways than one. (more)
Calculate addition chains. (more)
Slide a hex-shaped piece across a board. (more)
List all possible trees of a certain type. (more)

We've chosen the problems so that they're easy to explain, but allow contestants quite a bit of freedom to play around with various solutions.

If you're curious about the details of the contest, I recommend the file RULES. For easy downloading, here's a .zip file of everything you need for the contest.

Addendum: For useful afterthoughts, we've decided to keep a file NOTES around. Think of it as useful clarifications to the existing files in the repo. We won't pass or fail people based on things found in NOTES, but the items may help you with the tasks.

The rest is up to you. Hope hear from you — we want you to win those books!

perl6.announce: Announce: Perlito version 8; Perl5 and Perl6 compilers by Flavio S. Glock

Sat, 24 Dec 2011 11:26:26 +0000

Perlito is a compiler collection that contains both a Perl 5 compiler
(perlito5) and a Perl 6 compiler (perlito6).

The following backends are supported:

- Perl5 in Javascript (browser and command-line)

- Perl6 in Javascript (browser and command-line)
- Perl6 in Perl5 (v6.pm, runs in perl 5.8 and up)
- Perl6 in Python (Python 2.6 and 2.7)

There are also many experimental backends.

Web

Run Perlito online, in the browser:

http://perlcabal.org/~fglock/perlito5.html

http://perlcabal.org/~fglock/perlito6.html

From CPAN:

http://search.cpan.org/dist/v6/lib/v6.pm

$ cpan v6

Perlito is a work in progress. For the source code, to-do list, and bugs:

http://github.com/fglock/Perlito

- Flávio S. Glock (fglock)

Perl 6 Advent Calendar: Day 24 — Subs are Always Better in multi-ples

Sat, 24 Dec 2011 02:15:26 +0000

Hey look, it’s Christmas Eve! (Also, the palindrome of 42!) And today, we’re going to learn about multi subs, which are essentially synonyms (like any natural language would have). Let’s get started!

An Informative Introduction

multi subs are simply subroutines (or anything related to it, such as methods, macros, etc.) that start with the multi keyword and are then allowed to have the same name as another sub before, provided that sub starts with a multi (or proto — that’s later) keyword. What has to be different between these subs is their signature, or list of formal parameters.

Sound complicated? It isn’t, just take a look at the example below:

multi sub steve(Str $name) {
    return "Hello, $name";
}

multi sub steve(Int $number) {
    return "You are person number $number to use this sub!";
}

Every sub was started with the multi keyword, and has the same name of “steve”, but its parameters are different. That’s how Perl 6 knows which steve to use. If I were to later type steve("John"), then the first steve gets called. If, however, I were to type steve(35), then I’d get the second steve sub.

Equal Footing with built-ins

When you write a multi sub, and it happens to have the same name as some other built-in, your sub is on equal footing with the compiler’s. There’s no preferring Perl 6′s multi sub over yours, so if you write a multi sub with the same name as a built-in and with the same signature, say

multi sub slurp($filename) {
    say "Yum! $filename was tasty. Got another one?";
}

And then try calling it with something like slurp("/etc/passwd"), I get this:

Ambiguous dispatch to multi 'slurp'. Ambiguous candidates had signatures:
:(Any $filename)
:(Any $filename)

Why? Because Perl 6 found two equally valid choices for slurp("/etc/passwd"), my sub and its own, and was unable to decide. That’s the easiest way I know to demonstrate equal footing.

A Fun Conclusion

Now, since it’s Christmas, let’s try writing another open sub, but unlike the built-in open sub, which opens files, this one open presents! Here’s our Present class for this example:

class Present {
    has $.item;
    has $.iswrapped = True;

    method look() {
        if $.iswrapped {
            say "It's wrapped.";
        }
        else {
            say $.item;
        }
    }

    method unwrap() {
        $!iswrapped = False;
    }
}

Now, our open multi sub looks like this:

multi sub open(Present $present) {
    $present.unwrap;
    say "You unwrap the present and find...!";
}

The signature is vastly different from Perl 6′s own open sub, which is a good thing. And here’s the rest of the code, which makes a Present and uses our new multi sub:

my $gift = Present.new(:item("sock"));
$gift.look;
open($gift);
$gift.look;

But wait!

Running this gets us an error in the latest pull of Rakudo:

$ ./perl6 present.p6
It's wrapped.
This type cannot unbox to a native string
⋮

This means that Perl 6′s original open sub is being used, so perhaps it’s being interpreted as an only sub (only subs are the default — only sub unique() {...} and sub unique() {...} mean the same thing). No matter, let’s try adding a proto sub line before our multi sub:

proto sub open(|$) {*}

A proto sub allows you to specify the commonalities between multi subs of the same name. In this case, |$ means “every possible argument”, and {*} means “any kind of code”. It also turns any sub with that name into a multi sub (unless explicitly defined as something other than multi). This is useful if you’re, say, importing a &color sub from a module that isn’t defined as multi (or explicitly as only) and you want to have your own &color sub as well.

After adding this before our multi sub open, we get this result:

$ ./perl6 present.p6
It's wrapped.
You unwrap the present and find...!
sock

It works! Well, that’s it for multi subs. For all the nitty-gritty details, see the most current S06. Enjoy your multi subs and your Christmas Eve!

Perl 6 Advent Calendar: Day 23 – Idiomatic Perl 6

Fri, 23 Dec 2011 23:16:52 +0000

Perl 6 Idioms, and Idiomatic Perl 6

(Butterflies of the world, alight!)

Perl is a richly expressive language, with a warm and playful community. When someone crafts a succinct way to solve a common problem, the Perl community often adopts that solution's phrasing as a idiom. (Other-times, the community recoils in horror and proposes a less obtuse phrasing.)

Perl 6 adds many elements to the language, and not just keywords; there are metaoperators, clean exceptions, new contexts, better interpolation, frugal OO, and much more. Those additions were shaped by patterns of Perl 5 use (with an eye to future uses), so at least one should scratch some itch you have long ignored. The new bits aren't required, but we like the shiny, so be prepared for a slew of new idioms soon after Christmas.

Most Perl 6 idioms will not just be translated versions of familiar Perl 5 idioms; they will use new features where appropriate, and may seem unfamiliar at first. If you regularly use any of the Perl 5 idioms below, you can expect to grok its new form soon after you embrace Perl 6 itself.

Idiomatic Perl 6 should feel just as Perlish as Perl 5, once you get used to it. After all, it is all Perl.

Conventions

(Details, details)

Definitions (corrupted^W adapted for computer languages):

Idiom: A phrase expected to be used to express an idea.
Idiomatic: Expressed as the language's native users would state it.

In most examples below, the code is shown in four versions:

1. Non-idiomatic Perl 5,
2. then made idiomatic.
#
3. Perl 5 idiom, naively translated into Perl 6,
4. then made idiomatic.

Any versions past #4 are to show TMTOWTDI. Notice how the code gets clearer or more concise as it goes from 1 to 4.

Idiom ==> Word

(Movin' on up)

Some Perl 5 idioms were so useful and conceptually concise, that they became Perl 6 "words" in their own right. These words take the form of new built-in operators, functions, and methods.

 # Pick a random array element
 $z = $array[ int(rand scalar(@array)) ];
 $z = $array[ rand @array ];
 #
 $z = @array[ rand*@array ];
 $z = @array.pick;            

 # Loop over the keys (indexes) of an array
 for ( my $i=0; $i<@array; $i++ ) {...}
 for my $i ( 0 .. $#array ) {...}
 #
 for 0 .. @array.end -> $i {...}
 for @array.keys -> $i {...}

 # Whole number division
 ( ($x - ($x % 3) ) / 3 )
 int( $x / 3 )
 #
 Int( $x / 3 )
 $x div 3                  # Integer division op

 # Print the count of the elements of an array.
 say scalar @array;
 say 0+@array;
 #
 say 0+@array;      # Identical in Perl 6
 say +@array;       # + forces the new "numeric" context
 say @array.elems;  # .elems method is more explicit.

 # Do something every 5th time
 if ( ($x/5) == int($x/5) ) {...}
 if ( !($x % 5) ) {...}
 #
 if !($x % 5) {...}
 if $x %% 5 {...}           # %% means "is evenly divisible by"

 # Do something $n times, counting up to $n-1
 for ( $_=0; $_ < $n; $_++ ) {...}
 for ( 0 .. ($n-1) ) {...}
 #
 for 0 ..^ $n {...}
 for ^$n {...}                      # ^9 means 0 ..^ 9, or 0..8

Bare method calls are *always* methods on $_, eliminating Perl 5's confusion on which functions default to $_.

Other defaults have been tweaked to move from something-else-I-must-remember to obvious.

 # Split on whitespace
 @words = split /\s+/, $_;
 @words = split;          # Default is useful, but not intuitive
 #
 @words = .split(/\s+/);  # split() now has no default pattern
 @words = .words;         # split's old default behavior is now a separate method: .words

 # Split a string into individual characters.
 @chars = map { substr $word, $_, 1 } 0..length($word);
 @chars = split '', $word; # Split on nothingness
 #
 @chars = $word.split('');
 @chars = $word.comb;      # Default is to "keep everything"

 # Infinite loop
 for (;;) {...}    # Spoken with a 'C' accent
 while (1) {...}
 #
 while 1 {...}
 loop {...}        # No limit given, so endless by default

Some idioms that became words were already words in Perl 5, if you used the appropriate module.

 # Return the unique elements from a list, in original order
 my %s, @r; for @a { push @r, $_ if !$s{$_}; $s{$_}++; } return @r;
 my %s; return grep { !$s{$_}++ } @a;    # or List::MoreUtils::uniq
 #
 my %s; return grep { !%s{$_}++ }, @a;
 return @a.uniq;

 # Add up all list elements
 my $sum = 0; for my $num (@a) { $sum += $num }
 my $sum; $sum += $_ for @a;    # or List::Util::sum
 #
 my $sum = @a.reduce(*+*);
 my $sum = [+] @a;              # [op] applies op to entire list

Idiom ==> Idiom

(The song remains the same)

Some idioms remain the same, modulo required syntax changes.

 @alpha = 'A' .. 'Z';

 @a = qw{ able baker charlie };

 %meta = ( foo => 'bar', baz => 'quz' );

 @squares = map { $_ * $_ }, @a;

 @starts_with_number = grep { /^\d/ }, @a;

Didn't those all look familiar? Sure, parenthesis are no longer needed on list assignment, and qw{} now has a an angle-bracket shortcut, but those are optional, and don't really change the gist of the idioms. Add that comma after the map|grep block, and these Perl 5 idioms still stand strong in Perl 6.

Perl 5's "magic diamond" idiom still exists, just spelled more sanely (and less diamond-ly).

 # Process each line from STDIN or from command-line files.
 for my $file (@ARGV) { open FH, $file; while (<FH>) {...} }
 while (<>) {...}               # Null filehandle is magical
 #
 for $*ARGFILES.lines {...}
 for lines() {...}              # lines() defaults to $fh = $*ARGFILES

Idiom 5 ==> Idiom 6

(The more things change, the more they remain the same)

Some idioms have taken new form, when the idea behind the original idiom found better expression through new language elements.

 # Hash initialization to constant
 my %h;   for (@a) { $h{$_} = 1 }
 my %h = map { $_ => 1 } @a;
 #
 my %h = map { $_ => 1 }, @a;
 my %h = @a X=> 1;

 # Hash initialization for enumeration
 my %h;   for (0..$#a) { $h{ $a[$_] } = $_ }
 my $c;   my %h = map { $_ => ++$c } @a;
 #
 my $c;   my %h = map { $_ => ++$c }, @a;
 my %h = @a Z=> 1..*;
 my %h = @a.pairs».invert;  # if zero based

 # Hash initialization from parallel arrays
 my %h;   for (@a) { $h{$_} = shift @b }
 my %h;   @h{@a} = @b;
 #
 my %h;   %h{@a} = @b;
 my %h = @a Z=> @b;

 # Swap two variables
 my $temp = $x; $x = $y; $y = $temp;
 ( $x, $y ) = ( $y, $x );
 #
 ( $x, $y ) =   $y, $x;
 ( $x, $y ) .= reverse;   # .= makes reverse into a "mutating" method
 # Tastes great on array swaps, too!   @a[ $j, $k ] .= reverse;

 # Rotate array left by 1 element
 my $temp = shift @a; push @a, $temp;
 push @a, shift @a;
 #
 @a.push: @a.shift;
 @a .= rotate;

 # Create an object
 my $pet = new Dog;
 my $pet = Dog->new;
 #
 my $pet = Dog.new;
 my Dog $pet .= new;    # $pet *always* isa Dog; Compiler can optimize!

Combining transformation with selection was an advanced idiom in Perl 5. The new return values for if provide a bite-sized idiom.

 # Three copies of elements > 5
 @z = map { ($_) x 3 } grep { $_ > 5 } @y;    # map,grep
 @z = map { $_ > 5 ? ($_) x 3 : () } @y;      # map as grep
 #
 @z = map { $_ > 5 ?? ($_) xx 3 !! Nil }, @y;
 @z = @y.map: { $_ xx 3 if $_ > 5 };          # !if == Empty list
 @z = ($_ xx 3 if $_ > 5 for @y);             # List comprehension

That fifth form is a list comprehension, popular in functional languages, and in our closer cousin.

Sentence|Paragraph ==> Idiom

(The territory becomes the map)

Some larger code blocks can now be expressed so concisely that they become idioms in their own right.

 # Random integer between 3 and 7 inclusive
 do { $z = int rand 8 } until $z >= 3;
 $z = 3 + int rand 5;
 #
 $z = 3 + Int(5.rand);
 $z = (3..7).pick;

 # Count by 3 in an infinite loop
 for ( my $i = 1; ; $i++ ) { my $n = 3 * $i; ... }
 for ( my $n = 3; ; $n += 3 ) {...}
 #
 loop ( my $n = 3; ; $n += 3 ) {...}
 for 3, * + 3 ... * -> $n {...}      # `...` is the "sequence" operator
 for 3, 6, 9 ... * -> $n {...}       # `...` can infer from example list

 # Loop over a range, excluding the start and end points
 for my $i ( $start .. $limit ) { next if $i == $start or $i == $limit; ... }
 for my $i ( ($start+1) .. ($limit-1) ) {...}
 #
 for ($start+1) .. ($limit-1) -> $i {...}
 for $start ^..^ $limit -> $i {...}

Predictions

(Gaze into my crystal ball)

The idioms above are not written in stone. After Christmas, they may be given the boot, eighty-sixed, or traded-in for a newer model. Furthermore, other forces are at work to season this stew. Please indulge these prognostications:

We will see more use of exceptions. They are now cleaner and cheaper and don't need eval(), and users will be introduced to them much sooner due to open()'s new exception-based error handling.
We will see more hash|array slicing. Beginners had to grok another sigil rule to use slices, now they don't! In fact, single-element access now just looks like a degenerate case of multiple-element access.
Lazy lists are cool and efficient. Coding practices will be shaped to allow lists to stay lazy longer.
The new hyper-operators are implicitly parallel. We will see less explicit threading, and more set-at-a-time thinking.
In Perl 5, OO (before Moose) was overly wordy to the point of being grating. Functional programming worked best using the `&` prototype, but we are all told to never use Perl 5's prototypes!
In Perl 6, OO and functional styles are so low-friction that you will hoist code across paradigm boundaries with the same ease that you refactor same-paradigm code today.
`say` will rule the world, with legions of inlined ».methods as its army.

OK, maybe not that last one :)

The crystal ball may be hazy, but the future looks bright!

Perl 6 Advent Calendar: Day 22 – Operator overloading, revisited

Thu, 22 Dec 2011 20:58:11 +0000

Today’s post is a follow-up. Exactly two years ago, Matthew Walton wrote on this blog about overloading operators:

You can exercise further control over the operator’s parsing by adding traits to the definition, such as tighter, equiv and looser, which let you specify the operator’s precedence in relationship to operators which have already been defined. Unfortunately, at the time of writing this is not supported in Rakudo so we will not consider it further today.

Rakudo is still lagging in precedence support (though at this point there are no blockers that I know about to simply going ahead and implementing it). But there’s a new implementation on the block, one that didn’t exist two years ago: Niecza.

Let’s try out operator precedence in Niecza.

$ niecza -e 'sub infix:<mean>($a, $b) { ($a + $b) / 2 }; say 10 mean 4 * 5'
15

Per default, an operator gets the same precedence as infix<+>. This is per spec. (How do we know it got the same precedence as infix<+> above? Well, we know it’s not tighter than multiplication, otherwise we’d have gotten the result 35.)

That’s all well and good, but what if we want to make our mean little operator evaluate tighter than multiplication? Nothing could be simpler:

$ niecza -e 'sub infix:<mean>($a, $b) is tighter(&infix:<*>) { ($a + $b) / 2 }; say 10 mean 4 * 5'
35

See what we did there? is tighter is a trait that we apply to the operator definition. The trait accepts an argument, in this case the language-provided multiplication operator. It all reads quite well, too: “infix mean is tighter [than] infix multiplication”.

Note the explicit use of intuitive naming for the precedence levels. Rather than the inherently confusing terms “higher/lower”, Perl 6 talks about “tighter/looser”, as in “multiplication binds tighter than addition”. Easier to think about precedence that way.

Internally, the precedence levels are stored not as numbers but as strings. Each original precedence level gets a letter of the alphabet and an equals sign (=). Subsequent added precendence levels append either a less-than sign (<) or a greater-than sign (>) to an existing precedence level representation. Using this system, we never “run out” of levels between existing ones (as we could if we were using integers, for example), and tighter levels always come lexigographically before looser ones. Language designers, take heed.

A few last passing notes about operators in Perl 6, while we’re on the subject:

In Perl 6, operators are subroutines. They just happen to have funny names, like prefix:<-> or postfix:<++> or infix:<?? !!>. This actually takes a lot of the hand-wavey magic out of defining them. The traits that we’ve seen applied to operators are really subroutine traits… these just happen to be relevant to operator definitions.
As a consequence, just like subroutines, operators are lexically scoped by default. Lexical scoping is something we like in Perl 6; it keeps popping up in unexpected places as a solid, sound design principle in the language. In practice, this means that if you declare an operator within a given scope, the operator will be visible and usable within that scope. You’re modifying the parser, but you’re doing it locally, within some block or other. (Or within the whole file, of course.)
Likewise, if you want to export your operators, you just use the same exporting mechanism used with subroutines. See how this unification between operators and subroutines keeps making sense? (In Perl 6-land, we say “operators are just funny-looking subroutines”.)
Multiple dispatch in operators works just as with ordinary subroutines. Great if you want to dispatch your operators on different types. As with all other routines in the core library in Perl 6, all operators are declared multi to be able to co-exist peacefully with module extensions to the language.
Operators can be macros, too. This is not an exceptions to the rule that operators are subroutines, because in Perl 6, macros are subroutines. In other words, if you want some syntactic sugar to execute at parse time (which is what a macro does), you can dress it up either as a normal-looking sub, or as an operator.

That’s it for today. Now, go forth and multiply, or even define your own operator that’s either tighter or looser than multiplication.

Perl 6 Advent Calendar: Native libraries, native objects

Wed, 21 Dec 2011 21:42:29 +0000

Last year flussence++ wrote a nice post about writing XMMS bindings for Perl 6 using the Native Call Interface. It has improved a bit since then, (at least NCI, I don’t know about XMMS), so let’s show it off a bit.

To run the examples below you need a NativeCall module installed. Then add use NativeCall; at the top of the file.

Previously, we were carefully writing all the C subs we needed to use and then usually writing some Perl 6 class which wrapped it in a nice, familiar interface. That doesn’t change much, except that now a class is not really an interface for some C-level data structure. Thanks to the new metamodel we can now make our class to actually be a C-level data structure, at least under the hood. Consider a class representing a connection to Music Player Daemon:

    class Connection is repr('CPointer') {
        sub mpd_connection_new(Str $host, Int $port)
            returns Connection
            is native('libmpdclient.so') {}
        sub mpd_connection_free()
            is native('libmpdclient.so') {}
        method new(Str $host, Int $port) {
            self.bless(mpd_connection_new($host, $port))
        }
        method DESTROY {
            mpd_connection_free(self)
        }
    }

The first line does not necesarilly look familiar. The is repr trait tells the compiler that the internal representation of the class Connection is a C pointer. It still is a fully functional Perl 6 type, which we can use in method signatures or wherever (as seen in the lines below).

We then declare some native fuctions we’re going to use. It’s quite convenient to put them inside the class body, so they don’t pollute the namespace and don’t confuse the user. What we are really exposing here is the new method, which uses bless to set the object’s internal representation to what mpd_connection_new has returned. From now on our object is a Perl 6 level object, while under the hood being a mere C pointer. In method DESTROY we just pass self to another native function, mpd_connection_free, without the need to unbox it or whatever. The NativeCall module will just extract its internal representation and pass it around. Ain’t that neat?

Let’s see some bigger example. We’ll use taglib library to extract the metadata about some music files lying around. Let’s see the Tag class first:

    class Tag is repr('CPointer') {
        sub taglib_tag_title(Tag)  returns Str is native('libtag_c.so') {}
        sub taglib_tag_artist(Tag) returns Str is native('libtag_c.so') {}
        sub taglib_tag_album(Tag)  returns Str is native('libtag_c.so') {}
        sub taglib_tag_genre(Tag)  returns Str is native('libtag_c.so') {}
        sub taglib_tag_year(Tag)   returns Int is native('libtag_c.so') {}
        sub taglib_tag_track(Tag)  returns Int is native('libtag_c.so') {}
        sub taglib_tag_free_strings(Tag)       is native('libtag_c.so') {}

        method title  { taglib_tag_title(self)  }
        method artist { taglib_tag_artist(self) }
        method album  { taglib_tag_album(self)  }
        method genre  { taglib_tag_genre(self)  }
        method year   { taglib_tag_year(self)   }
        method track  { taglib_tag_track(self)  }

        method free   { taglib_tag_free_strings(self) }
    }

That one is pretty boring: plenty of native functions, and plenty of methods being exactly the same things. You may have noticed the lack of new: how are we going to get an object and read our precious tags? In taglib, the actual Tag object is obtained from a Tag_File object first. Why didn’t we implement it first? Well, it’s going to have a method returning the Tag object shown above, so it was convenient to declare it first.

    class TagFile is repr('CPointer') {
        sub taglib_file_new(Str) returns TagFile is native('libtag_c.so') {}
        sub taglib_file_free(TagFile)            is native('libtag_c.so') {}
        sub taglib_file_tag(TagFile) returns Tag is native('libtag_c.so') {}
        sub taglib_file_is_valid(TagFile) returns Int
            is native('libtag_c.so') {}

        method new(Str $filename) {
            unless $filename.IO.e {
                die "File '$filename' not found"
            }
            my $self = self.bless(taglib_file_new($filename));
            unless taglib_file_is_valid($self) {
                taglib_file_free(self);
                die "'$filename' is invalid"
            }
            return $self;
        }

        method tag  { taglib_file_tag(self)  }

        method free { taglib_file_free(self) }
    }

Note how we use native functions in new to check for exceptional situations and react in an appropriately Perl 6 way. Now we only have to write a simple MAIN before we can test it on our favourite music files.

    sub MAIN($filename) {
        my $file = TagFile.new($filename);
        my $tag  = $file.tag;
        say 'Artist: ', $tag.artist;
        say 'Title:  ', $tag.title;
        say 'Album:  ', $tag.album;
        say 'Year:   ', $tag.year;

        $tag.free;
        $file.free;
    }

Live demo! Everyone loves live demos.

    $ perl6 taginfo.pl some-track.mp3
    Artist: Diablo Swing Orchestra
    Title:  Balrog Boogie
    Album:  The Butcher's Ballroom
    Year:   2009

Works like a charm. I promise I’ll wrap it up in some nice Audio::Tag module and release it on Github shortly.

Of course there’s more to do with NativeCall than just passing raw pointers around. You could, for example, declare it as a repr('CStruct') and access the struct field directly, as you would in good, old C. This is only partly implemented as for now though, but that shouldn’t stop you from experimenting and seeing what you can do before Christmas. Happy hacking!

Perl 6 Advent Calendar: Paired up Hashes

Tue, 20 Dec 2011 14:31:49 +0000

What is is possible with arrays and lists in Perl 6 is truly remarkable and was demonstrated here several times. But what about hashes?
Superficially not much has changed.
(Following Damian’s rule from PBP to name a hash variable in singular.)

    %song = Panacea => 'found a lover', Photek => 'ni ten ichi ryu';
    say keys %song;   # also %song.keys
    say values %song; # %song.values

Yes, the sigils are now invariant, so you get values with:

    %song{'Panacea'}
    %song{'Panacea', 'Photek'}

That can be shortened, because <> is the new qw():

    %song<Panacea>
    %song<Panacea Photek>

Frankly, almost everything else has changed. Perl 6 can be sometimes hideous, just mimicking to be your good old pal Perl 5 while being a friendly T-X, blasting behind your back your programming problems away. The fat arrow is no longer a fancy comma but an infix operator, creating an object that contains a key-value pair.

    my $song = paniq => 'Godshatter';
    say $song.WHAT; # says: "Pair()"
    $song.key;     # as expected is paniq
    $song.value;  # you guess it

There is another Syntax for that, heavily used in signatures:

    my $song = :paniq('Godshatter');

But what happens if I:

    my @songs = %song; # same as @(%songs)

You maybe predicted it, @songs gets a list of pairs. For the old behaviour, you have to say explicitly: “I want the keys and values as a list.”:

    my @songs = %song.kv; # key 1, value 1, key 2 ....

This new setup of hashes is not only theoretically very pleasing. It also allows iterating over hashes, without the risk of loosing the precious key => value correlation. That’s handy for all kinds of sorting and mashing of data, for which Perl is famous. What else did Randal L. Schwartz once upon a time than creating a list of pairs, sorting them and then picking the needed data bits.

Having pairs as a built-in type helps also subs and methods to handle their parameters. Some of the can be positional, which could be ordered in an array. Some of them are named and could be stored in a hash. But they are actually stored in a “Parcel”, a list that can contain pairs. This way the order of the parameters and the key => value correlations are preserved.

A very similar type is the Capture, which can hold all parameter sent to a routine. Therefore it has to behave more like routine and pass all the named arguments under their names. But if you ask for the positional parameter, you get only them, not the named ones. With a Capture full of values you can ask with a smartmatch if it would pass a certain subroutine and many fine things more. The vaults are going here deeper and deeper, but lets get back to the daylight of everyday hash-usage.

Panacea aka Mathis Mootz had a lot of great tracks. But when I do:

    %song<Panacea> = "state of extacy";

“found a lover” gets overwritten. Nothing new so far, but there are times I just don’t want to loose my data. Then I need to execute some force onto the hash.

    %song.push( Panacea => "state of extacy" );

Whole lists of pairs can be pushed into another hash this way. The result will be (not surprisingly) still a hash but the key ‘Panacea’ now points to an array, containing both song titles. That’s also useful when inverting a hash, that means pulling out a pair-list where key and value are flipped. Pasting that into a hash may lead to collisions, if several keys have the same value. A simple:

    # list with song => artist pairs
    %song = %artist.invert;

might produce losses, but the following does not:

    %song.push( %artist.invert ); # or:
    %song.push: %artist.invert;

While doing some heavy data munging you might regroup the values under a different set of keys. In that case it is likely that several values will end up under the same key. Given you have a sub that recognizes a genre of a song, you might write something like:

    map { %genre.push(genre_of_song($_) => $_) }, %song.values;

But as you already guessed it, there is an easier way to do that:

    %genre = classify { (genre_of_song($_) }, %song.values;

Now you probably say: “That’s unrealistic!”. There are songs for instance from “Magnetic Man” that can be labeled “Drum ‘n Base”, “Dubstep” or even “Pop”. Larry knows that. (This kind of problem of course.) That’s why he pulled out a second hash generating method.

    %genre = categorize { (genre_of_song($_) }, %song.values;

Unlike classify, which expects exactly one value (Called scalar in P5 but item in Perl 6 world), categorize can handle a list of results returned by the closure (block). It also happily accepts a Nil, which means, unlike undef in Perl 5, really nothing. When classify gets a Nil, the song will then not appear in any category, meaning under no hash key.

Imagine a routine of real artificial intelligence that can distinct good from bad songs. (Your definition of good of course!)

    my %quality = classify { good_music($_) ?? 'good' !! 'bad' }, %song.values;

Hence %quality<good> contains all the songs I sent to my music player and %quality<bad> to /dev/null (which is the name of another electronic music artist).

Perl 6 Advent Calendar: Day 19 – Abstraction and why it’s good

Mon, 19 Dec 2011 21:30:09 +0000

Some people are a bit afraid of the word “abstract”, because they’ve heard math teachers say it, and also, abstract art freaks them out. But abstraction is a fine and useful thing, and not so complicated. As programmers, we use it every day in different forms. The term is from Latin and means “to withdraw from” or “to pull away from”, and what we’re pulling away from is the specifics so we can focus on the big picture. That’s often mighty useful.

Here are a few examples:

Variables

If your computer only knew how to handle one specific number at a time, it’d be an abacus. Pretty early on, the programmer guild figured out it made a lot of sense to talk about the memory address of a value, and let that address contain whatever it pleased. They abstracted away from the value, and thus made the program more general.

As time passed, addresses were replaced by names, mostly as a convenience. Some people found it a good idea to give their variables descriptive names, as opposed to things like $grbldf.

Subroutines

Code re-use. We hear so much about it in the OO circles, but it holds equally well for subroutines. You write your code once, and then call it from all over the place. Convenient.

But, as I point out in an announcement pretending to be a computer science professor from an alternate timeline, there’s also the secondary benefit of giving your chunk of code a good mnemonic name, because that act in a sense improves the programming language itself. You’re giving yourself new verbs to program with.

This is especially true in Perl 6, because subroutines are lexically scoped (as opposed to Perl 5) and thus you can easily put a subroutine inside another routine. I use it when writing a Connect 4 game, for example.

Packages and modules

In Perl, packages don’t do much. They pull things together and keep them there. In a sense, what they abstract away is a set of subroutines from the rest of the world.

Perl 5 delivers its whole OO functionality through packages and a bit of dispatch magic on the side. It’s quite a feat, actually, but sometimes a bit too minimal. Moose fixes many of those early issues by providing a full-featured object system. Perl 6 lets packages go back to just being collections of subroutines, but provides a few dedicated abstractions for OO, a kind of built-in Moose. Which brings us to…

Classes

Object-orientation means a lot of different things to different people. To some, it’s the notion of an object, a piece of memory with a given set of operations and a given set of states. In a sense, we’re again in the business of extending the language like with did with subroutines. But this time we’re building new nouns rather than new verbs. One moment the language doesn’t know about a Customer object type; the next, it does.

To others, object-orientation means keeping the operations public and the states private. They refer to this division as encapsulation, because the object is like a little capsule, protecting your data from the big bad world. This is also a kind of abstraction, built on the idea that the rest of the world shouldn’t need to care about the internals of your objects, because some day you may want to refactor them. Don’t talk to the brain, talk to the hand; do your thing through the published operations of the object.

Roles

But class-based OO with inheritance will take you only so far. In the past 10 years or so, people have become increasingly aware of the limitations of inheritance-based class hierarchies. Often there are concerns which cut completely across a conventional inheritance hierarchy.

This is where roles come in; they allow you to apply behaviors in little cute packages here and there, without being tied up by a tree-like structure. In a post about roles I explore how this helps write better programs. But really the best example nowadays is probably the Rakudo compiler and its extensive use of roles; jnthn has been writing about that in an earlier advent post.

If classes abstract away complete sets of behaviors, roles abstract away partial sets of behaviors, or responsibilities.

You can even do so at runtime, using mixins, which are roles that you add to an object as the program executes. Objects changing type during runtime sounds magic almost to the point of recklessness; but it’s all done in a very straightforward manner using anonymous subclasses.

Metaobjects

Sometimes you want extra control over how the object system itself works. The object system in Perl 6, through one of those neat bite-your-own-tail tricks, is written using itself, and is completely modifiable in terms of itself. Basically, a bunch of the complexity has been removed by not having a separate hidden, unreachable system to handle the intricacies of the object system. Instead, there’s a visible API for interacting with the object system.

And, when we feel like it, we can invent new and exotic varieties of object systems. Or just tweak the existing one to our fancy.

Macros

On the way up the abstraction ladder, we’ve abstracted away bigger and bigger chunks of code: values, code, routines, behaviors, responsibilities and object systems. Now we reach the top, and there we find macros. Ah, macros, these magical, inscrutable beasts. What do macros abstract away?

Code.

Well, that’s rather disappointing, isn’t it? Didn’t we already abstract away code with subroutines? Yes, we did. But it turns out there’s so much code in a program that sometimes, it needs to be abstracted away on several levels!

Subroutines abstract away code that can then run in several different ways. You call the routine with other values, and it behaves differently. Macros abstract away code that can then be compiled in several different ways. You write a macro with other values, and it gets compiled into different code, which can then in turn run differently.

Essentially, macros give you a hook into the compiler to help you shape and guide what code it emits during the compilation itself. In a sense, you’re abstracting certain parts of the compilation process, the parsing and the syntax manipulation and the code generation. Again, you’re shaping the language — but this time not inventing new nouns or verbs, but whole ways of expressing yourself.

Macros come in two broad types: textual (a la C) and syntax tree (a la Lisp). The textual ones have a number of known issues stemming from the fact that they’re essentially a big imprecise search-and-replace on your code. The syntax tree ones are hailed as the best thing about Lisp, because it allows Lisp programs to grow and adapt to the needs of the programmer, by inventing new ways of expressing yourself.

Perl 6, being Perl 6, specifies both textual macros and syntax tree macros. I’m currently working on a grant to bring syntax macros to Rakudo Perl 6. There’s a branch where I’m hammering out the syntax and semantics of macros. It’s fun work, and made much more feasible by the past year’s improvements to Rakudo itself.

In conclusion

As an application grows and becomes more complex, it needs more rungs of the abstraction ladder to rest on. It needs more levels of abstraction with which to draw away the specifics and focus on the generalities.

Perl 6 is a new Perl, distinct from Perl 5. Its most distinguishing trait is perhaps that it has more rungs on the abstraction ladder to help you write code that’s more to the point. I like that.

Moritz Lenz (Perl 6): Fourth Grant Report: Structured Error Messages

Mon, 19 Dec 2011 18:28:48 +0000

Progress on my grant for error message is slow but steady. Since my last report, I've done the following things:

Merged the nom-exceptions Rakudo branch, so now you can reliably throw Perl 6 objects as exceptions.
Implemented several error classes and roles in Rakudo
Started to throw typed errors both from runtime libraries and from inside the compiler
Hacked the default exception printer Rakudo to be much more flexible, for example you can now write exception classes that supress backtraces from the standard handler.
Wrote tests for typed run time and compile time errors, and at the same time developed a test function that makes it easy to write such tests.

It's time for a quick review of how far I am along the various deliverables in the original grant proposal.

D1: Specification. I think the hard work here is done already, what remains to do is finding good default and how to manipulate them (for example, how to generally switch on/off printing of backtraces?).
D2: Error catalogue, tests: I've not worked on this one too much. The error classes and roles so far mostly served to exercise the implementation; going through the existing errors from the various compilers and formalizing them will be quite a bit of work, but only moderately complicated.
D3: Implementation, documentation. Like D1, the hard part is mostly done. We can now throw errors from within the compiler actions and from the setting, next up will be the grammar. Then all places where errors are thrown need to be changed to use the new error classes. Again that'll be much work, but easy to do. Documentation is still missing.

All in all I feel I'm well on the way, and most complex decisions have been made.

For a more user oriented view of the new exception system I'd like to point you to my Perl 6 advent calendar post on exceptions.

Perl 6 Advent Calendar: The view from the inside: using meta-programming to implement Rakudo

Sun, 18 Dec 2011 22:49:25 +0000

In my previous article for the Perl 6 advent calendar, I looked at how we can use the meta-programming facilities of Rakudo Perl 6 in order to build a range of tools, tweak the object system to our liking or even add major new features “from the outside”. While it’s nice that you can do these things, the Perl 6 object system that you get by default is already very rich and powerful, supporting a wide range of features. Amongst them are:

Classes
Parametric roles
Attributes
Methods (including private ones)
Delegation
Introspection
Subset (aka. refinement) types
Enums

That’s a lot of stuff to implement, but it’s all done by implementing meta-objects, and therefore we can take advantage of OOP – with both classes and roles – to factor it. The only real difference between the meta-programming we saw in my last article and the meta-programming we do while implementing the core Perl 6 object system in Rakudo is that the meta-objects are mostly written in NQP. NQP is a vastly smaller, much more easily optimizable and portable subset of Perl 6. Being able to use it also helps us to avoid many painful bootstrapping issues. Since it is mostly a small subset of Perl 6, it’s relatively easy to get in to.

In this article, I want to take you inside of Rakudo and, through implementing a missing feature, give you a taste of what it’s like to hack on the core language. So, what are we going to implement? Well, one feature of roles is that they can also serve as interfaces. That is, if you write:

role Describable {
    method describe() { ... }
}
class Page does Describable {
}

Then we are meant to get a compile time error, since the class Page does not implement the method “describe”. At the moment, however, there is no error at compile time; we don’t get any kind of failure until we call the describe method at runtime. So, let’s make this work!

One key thing we’re going to need to know is whether a method is just a stub, with a body containing just “…”, “???” or “!!!”. This is available to us by checking its .yada method. So, we have that bit. Question is, where to check it?

Unlike classes, which have the meta-object ClassHOW by default, there isn’t a single RoleHOW. In fact, roles show up in no less than four different forms. The two most worth knowing about are ParametricRoleHOW and ConcreteRoleHOW. Every role is born parametric. Whether you explicitly give it extra parameters or not, it is always parametric on at least the type of the invocant. Before we can ever use a role, it has to be composed into a class. Along the way, we have to specialize it, taking all the parametric things and replacing them with concrete ones. The outcome of this is a role type with a meta-object of type ConcreteRoleHOW, which is now ready for composition into the class.

So that’s roles themselves, but what about composing them? Well, the actual composition is performed by two classes, RoleToClassApplier and RoleToRoleApplier. RoleToClassApplier is actually only capable of applying a single role to a class. This may seem a little odd: classes can do multiple roles, after all. However, it turns out that a neat way to factor this is to always “sum” multiple roles to a single one, and then apply that to the class. Anyway, it would seem that we need to be doing some kind of check in RoleToClassApplier. Looking through, we see this:

my @methods := $to_compose_meta.methods($to_compose, :local(1));
for @methods {
    my $name;
    try { $name := $_.name }
    unless $name { $name := ~$_ }
    unless has_method($target, $name, 1) {
        $target.HOW.add_method($target, $name, $_);
    }
}

OK, so, it’s having a bit of “fun” with, of all things, looking up the name of the method. Actually it’s trying to cope with NQP and Rakudo methods having slightly different ideas about how the name of a method is looked up. But that aside, it’s really just a loop going over the methods in a role and adding them to the class. Seems like a relatively opportune time to spot the yada case, which indicates we require a method rather than want to compose one into the class. So, we change it do this:

my @methods := $to_compose_meta.methods($to_compose, :local(1));
for @methods {
    my $name;
    my $yada := 0;
    try { $name := $_.name }
    unless $name { $name := ~$_ }
    try { $yada := $_.yada }
    if $yada {
        unless has_method($target, $name, 0) {
            pir::die("Method '$name' must be implemented by " ~
            $target.HOW.name($target) ~
            " because it is required by a role");
        }
    }
    elsif !has_method($target, $name, 1) {
        $target.HOW.add_method($target, $name, $_);
    }
}

A couple of notes. The first is that we’re doing binding, because NQP does not have assignment. Binding is easier to analyze and generate code for. Also, the has_method call is passing an argument of 0 or 1, which indicates whether we want to consider methods in just the target class or any of its parents (note that there’s no True/False in NQP). If the class inherits a method then we’ll consider that as good enough: it has it.

So, now we run our program and we get:

===SORRY!===
Method 'describe' must be implemented by Page because it is required by a role

Which is what we were after. Note that the “SORRY!” indicates it is a compile time error. Success!

So, are we done? Not so fast! First, let’s check the inherited method case works out. Here’s an example.

role Describable {
    method describe() { ... }
}
class SiteItem {
    method describe() { say "It's a thingy" }
}
class Page is SiteItem does Describable {
}

And…oh dear. It gives an error. Fail. So, back to RoleToClassApplier. And…aha.

sub has_method($target, $name, $local) {
    my %mt := $target.HOW.method_table($target);
    return nqp::existskey(%mt, $name)
}

Yup. It’s ignoring the $local argument. Seems it was written with the later need to do a required methods check in mind, but never implemented to handle it. OK, that’s an easy fix – we just need to go walking the MRO (that is, the transitive list of parents in dispatch order).

sub has_method($target, $name, $local) {
    if $local {
        my %mt := $target.HOW.method_table($target);
        return nqp::existskey(%mt, $name);
    }
    else {
        for $target.HOW.mro($target) {
            my %mt := $_.HOW.method_table($_);
            if nqp::existskey(%mt, $name) {
                return 1;
            }
        }
        return 0;
    }
}

With that fixed, we’re in better shape. However, you may be able to imagine another case that we didn’t yet handle. What if another role provides the method? Well, first let’s see what the current failure mode is. Here’s the code.

role Describable {
    method describe() { ... }
}
role DefaultStuff {
    method describe() { say "It's a thingy" }
}
class Page does Describable does DefaultStuff {
}

And here’s the failure.

===SORRY!===
Method 'describe' must be resolved by class Page because it exists
in multiple roles (DefaultStuff, Describable)

So, it’s actually considering this as a collision. So where do collisions actually get added? Happily, that just happens in one place: in RoleToRoleApplier. Here’s the code in question.

if +@add_meths == 1 {
    $target.HOW.add_method($target, $name, @add_meths[0]);
}
else {
    # More than one - add to collisions list.
    $target.HOW.add_collision($target, $name, %meth_providers{$name});
}

We needn’t worry if we just have one method and it’s a requirement rather than an actual implementation – it’ll just do the right thing. So it’s just the second branch that needs consideration. Here’s how we change things.

if +@add_meths == 1 {
    $target.HOW.add_method($target, $name, @add_meths[0]);
}
else {
    # Find if any of the methods are actually requirements, not
    # implementations.
    my @impl_meths;
    for @add_meths {
        my $yada := 0;
        try { $yada := $_.yada; }
        unless $yada {
            @impl_meths.push($_);
        }
    }

    # If there's still more than one possible - add to collisions list.
    # If we got down to just one, add it. If they were all requirements,
    # just choose one.
    if +@impl_meths == 1 {
        $target.HOW.add_method($target, $name, @impl_meths[0]);
    }
    elsif +@impl_meths == 0 {
        $target.HOW.add_method($target, $name, @add_meths[0]);
    }
    else {
        $target.HOW.add_collision($target, $name, %meth_providers{$name});
    }
}

Essentially, we filter out those that are implementations of the method rather than just requirements. If we are left with just a single method, then it’s the only implementation, and it satisfies the requirements, so we add it and we don’t need to do anything further. If we discover they are all requirements, then we don’t want to flag up a collision, but instead we just pick any of the required methods and pass it along. They’ll all give the same error. Otherwise, if we have multiple implementations, then it’s a real collision so we add it just as before. And…it works!

So, we run the test suite, things look good…and commit.

3 files changed, 48 insertions(+), 6 deletions(-)

And there we go – Rakudo now supports a part of the spec that it never has before, and it wasn’t terribly much effort to put in. And that just leaves me to go to the fridge and grab a Christmas ale to relax after a little meta-hacking. Cheers!

Perl 6 Advent Calendar: Day 17: Gtk Mandelbrot

Sat, 17 Dec 2011 20:26:00 +0000

Two years ago today, the Advent post was on making Mandelbrot sets in Perl 6. At the time, they were in black and white, slow to produce, Rakudo was prone to crashing, and the only user interface thing you could control was how big the resulting PPM file was.

As they say, that was then. This is now.

The new gtk-mandelbrot.pl script is 423 lines of Perl 6 code — targeted at Niecza, threaded, and using the GtkSharp library. It allows you to move and resize the windows, zoom in (left mouse button, drag across image to define zoom boundaries), create Julia set images (right click on a Mandelbrot set image), increase the number of iterations (press ‘m’), and output a PPM file for a window (press ‘s’).

The threading doesn’t actually improve performance on my MacBook Pro (still looking into why) but it does make the script much more responsive.

It would be far too long to go through all the code, but lets hit on the highlights. The core is almost unchanged:

        sub julia(Complex $c, Complex $z0) {
            my $z = $z0;
            my $i;
            loop ($i = 0; $i < $max_iterations; $i++) {
                if $z.abs > 2 {
                    return $i + 1;
                }
                $z = $z * $z + $c;
            }
            return 0;
        }

It’s named julia instead of mandel now, because it is more general. If you call it with $z0 set to 0, it calculates the same thing the old mandel did. Allowing $z0 to vary allows you to calculate Julia sets as well.

The code around it is very different, though! Stefan O’Rear wrote the threading code, using Niecza’s Threads library, which is a thin wrapper on C#’s threading libraries, and probably not very close to what Perl 6′s built-in threading will look like when it is ready to go. He establishes a WorkQueue with a list of the work that needs to be done, and then starts N running threads, where N comes from the environment variable THREADS if it is present, and the reported processor count otherwise:

for ^(%*ENV<THREADS> // CLR::System::Environment.ProcessorCount) {
    Thread.new({ WorkQueue.run });
}

WorkQueue.run is pretty simple:

    method run() {
        loop {
            my $item = self.shift;
            next if $item.cancelled;
            $item.run.();
            $item.mark-done;
        }
    }

This is an infinite loop that starts by getting the next WorkItem off the queue, checks to see if it has been cancelled, and if it hasn’t, calls the .run Callable attribute and then the mark-done method.

The WorkItems on the queue look like this:

class WorkItem {
    has Bool $!done = False;
    has Bool $!cancelled = False;

    has Callable &.run;
    has Callable &.done-cb;

    method is-done() { WorkQueue.monitor.lock({ $!done }) }
    method mark-done() {
        &.done-cb.() unless WorkQueue.monitor.lock({ $!done++ })
    }

    method cancelled() { WorkQueue.monitor.lock({ $!cancelled }) }
    method cancel() { WorkQueue.monitor.lock({ $!cancelled = True }) }
}

Each WorkItem has two flags, $!done and $!cancelled, and two Callable attributes, &.run, already mentioned as what is called by WorkQueue.run, and &.done-cb, which is the callback function to be called when the &.run method finishes.

The two methods (for now) we use in our WorkItem are relatively simple:

            sub row() {
                my $row = @rows[$y];
                my $counter = 0;
                my $counter_end = $counter + 3 * $width;
                my $c = $ur - $y * $delta * i;

                while $counter < $counter_end {
                    my $value = $is-julia ?? julia($julia-z0, $c) !! julia($c, 0i);
                    $row.Set($counter++, @red[$value % 72]);
                    $row.Set($counter++, @green[$value % 72]);
                    $row.Set($counter++, @blue[$value % 72]);
                    $c += $delta;
                }
            }

            sub done() {
                Application.Invoke(-> $ , $ {
                    $.draw-area.QueueDrawArea(0, $y, $width, 1);
                });
            }

            my $wi = WorkItem.new(run => &row, done-cb => &done);
            WorkQueue.push($wi);
            push @.line-work-items, $wi;

As you might expect, row calculates one line of the set we are working on. It may look like it is using global variables, but these subs are actually local to the FractalSet.start-work method and the variables are local to it from there. The done invokes a Gtk function noting that a portion of the window needs to be redrawn (namely the portion we just calculated).

The above block of code is called once for each row of the fractal window being generated, which has the effect of queuing up all of the fractal to be handled as there are available threads.

Moving upward in the code's organization, each fractal window we generate is managed by an instance of the FractalSet class.

class FractalSet {
    has Bool $.is-julia;
    has Complex $.upper-right;
    has Real $.delta;
    has Int $.width;
    has Int $.height;
    has Int $.max_iterations;
    has Complex $.c;
    has @.rows;
    has @.line-work-items;
    has $.new-upper-right;

    has $.draw-area;

$.is-julia and $.max_iterations are self-explanatory. $.upper-right is the fixed complex number anchoring the image. $.delta is the amount of change in the previous number per-pixel; we assume the pixels are square. $.width and $.height are the size of the window in pixels. $.c only has meaning for Julia sets, where it is the value $c in the equation $new-z = $z * $z + $c. @.rows the pixel information generated by the row sub above; @.line-work-items saves a reference to all of the WorkItems generating those rows. $.new-upper-right is temporary used during the zoom mouse operation. $.draw-area is the Gtk.DrawingArea for the related window.

Once all that is set up, the rest of the code is pretty straightforward. The Gtk windowing code is set up in FractalSet.build-window:

    method build-window()
    {
        my $index = +@windows;
        @windows.push(self);
        self.start-work;

        my $window = Window.new($.is-julia ?? "julia $index" !! "mandelbrot $index");
        $window.SetData("Id", SystemIntPtr.new($index));
        $window.Resize($.width, $.height);  # TODO: resize at runtime NYI

        my $event-box = GtkEventBox.new;
        $event-box.SetData("Id", SystemIntPtr.new($index));
        $event-box.add_ButtonPressEvent(&ButtonPressEvent);
        $event-box.add_ButtonReleaseEvent(&ButtonReleaseEvent);

        my $drawingarea = $.draw-area = GtkDrawingArea.new;
        $drawingarea.SetData("Id", SystemIntPtr.new($index));
        $drawingarea.add_ExposeEvent(&ExposeEvent);
        $window.add_DeleteEvent(&DeleteEvent);
        $event-box.Add($drawingarea);

        $window.Add($event-box);
        $window.add_KeyReleaseEvent(&KeyReleaseEvent);
        $window.ShowAll;
    }

We store a global array @windows tracking all the FractalSets in play. Each of the different objects here gets the "Id" data set to this set's index into the @windows array so we can easily look up the FractalSet from callback functions. The rest of the method is just plugging the right callback into each component -- simple conceptually but it took this Gtk novice a lot of work figuring it all out.

As an example, consider the KeyReleaseEvent callback, which responds to presses on the keyboard.

sub KeyReleaseEvent($obj, $args) {
    my $index = $obj.GetData("Id").ToInt32();
    my $set = @windows[$index];
    
    given $args.Event.Key {
        when 'm' | 'M' {
            $set.increase-max-iterations;
        }
        when 's' | 'S' {
            $set.write-file;
        }
    }
}

First we lookup the index into @windows, then we get the $set we're looking at. Then we just call the appropriate FractalSet method, for instance

    method increase-max-iterations() {
        self.stop-work;
        $.max_iterations += 32;
        self.start-work;
    }

.stop-work cancels all the pending operations for this FractalSet, then we bump up the number of iterations, and then we .start-work again to queue up a new set of rows with the new values.

The full source code is here. As of this writing it agrees with the code here, but this is an active project, and probably will change again in the not-too-distant future. Right now my biggest goals are figuring out how to get the threading to actually improve performance on my MacBook Pro and cleaning up the code. Both suggestions and questions are welcome.

Perl 6 Advent Calendar: Where Have All The References Gone?

Fri, 16 Dec 2011 14:00:59 +0000

Perl 5 programmers that start to learn Perl 6 often ask me how to take a reference to something, and my answers usually aren’t really helpful. In Perl 6, everything that can be held in a variable is an object, and objects are passed by reference everywhere (though you don’t always notice that, because objects like strings and numbers are immutable, so there’s no difference to passing by value). So, everything is already treated as a reference in some sense, and there’s no point in explicitly taking references.

But people aren’t happy with that answer, because it doesn’t explain how to get stuff done that involved references in Perl 5. So here are a few typical use cases of references, and how Perl 6 handles them.

Creating Objects

In Perl 5, an object is really just a reference to a blessed value (but people usually say "blessed reference", because you virtually never use the blessed value without going through a reference).

So, in Perl 5 you’d write

 package My::Class;
 # constructor
 sub new { bless {}, shift };
 # an accessor
 sub foo {
     my $self = shift;
     # the ->{} dereferences $self as a hash
     $self->{foo} // 5;
 }

 # use the object:
 say My::Class->new->foo;

In Perl 6, you just don’t think about references; classes are much more declarative, and there’s no need for dereferencing anything anywhere:

 class My::Class {
     # attribute with accessor (indicated by the dot)
     # and default value
     has $.foo = 5;
 }

 # use it:
 say My::Class.new.foo

If you don’t like the default constructor, you can still use bless explicitly, but even then you don’t have to think about references:

 method new() {
     # the * specifies the storage, and means "default storage"
     self.bless(*);
 }

So, no explicit reference handling when dealing with OO. Great.

Nested Data Structures

In both Perl 5 and Perl 6, lists flatten automatically by default. So if you write

 my @a = (1, 2);
 my @b = (3, 4);
 push @a, @b

then @a ends up with the four elements 1, 2, 3, 4, not with three elements of which the third is an array.

In Perl 5, nesting the data structure happens by taking a reference to @b:

 push @a, \@b;

In Perl 6, item context replaces this use of references. It is best illustrated by a rather clumsy method to achieve the same thing:

 my $temp = @b;
 push @a, $temp;  # does not flatten the two items in $temp,
                 # because $temp is a scalar

Of course there are shortcuts; the following lines work too:

 push @a, $(@b);
 push @a, item @b;

(As a side note, push @a, $@b is currently not allowed, it tries to catch a p5ism; I will also try to persuade Larry and the other language designers to allow it, and have it mean the same thing as the other two).

On the flip side you need explicit dereferencing to get the values out of item context:

 my @a = 1, 2;
 my $scalar = @a;
 for @a {
     # two iterations
 }
 for $scalar {
     # one iteration only
 }
 for @$scalar {
     # two iterations again
 }

This explicit use of scalar and list context is the closest analogy to Perl 5 references, because it requires explicit context annotations in the same places where referencing and dereferencing is used in Perl 5.

But it’s not really the same, because there are cases where Perl 5 needs references, but Perl 6 can deduce the item context all on its own:

 @a[3] = @b; # automatically puts @b in item context

Mutating Arguments

Another use references in Perl 5 is for passing data to routines that should be modified inside the routine:

 sub set_five; {
     my $x = shift;
     # explicit dereferencing with another $:
     $$x = 5;
 }
 my $var;
 # explicit taking of a reference
 set_five \$var;

In Perl 6, there is a separate mechanism for this use case:

 sub set_five($x is rw) {
     # no dereferencing
     $x = 5;
 }
 my $var;
 # no explicit reference taking
 set_five $var;

So again a use case of Perl 5 references is realized by another mechanism in Perl 6 (signature binding, or binding in general).

Summary

Nearly everything is a reference in Perl 6, but you still don’t see them, unless you take a very close look. The control of list flattening with item and list context is the one area where Perl 5′s referencing and dereferencing shines through the most.

Perl 6 Advent Calendar: Day 15 – Something Exceptional

Thu, 15 Dec 2011 12:39:31 +0000

The Perl 6 exception system is currently in development; here is a small example demonstrating a part of the current state:

 use v6;

 sub might_die(Real $x) {
     die "negative" if $x < 0;
     $x.sqrt;
 }

 for 5, 0, -3, 1+2i -> $n {
     say "The square root of $n is ", might_die($n);

     CATCH {
         # CATCH sets $_ to the error object,
         # and then checks the various cases:
         when 'negative' {
             # note that $n is still in scope,
             # since the CATCH block is *inside* the
             # to-be-handled block
             say "Cannot take square root of $n: negative"
         }
         default {
             say "Other error: $_";
         }
     }
 }

This produces the following output under rakudo:

 The square root of 5 is 2.23606797749979
 The square root of 0 is 0
 Cannot take square root of -3: negative
 Other error: Nominal type check failed for parameter '$x'; expected Real but got Complex instead

A few interesting points: the presence of a CATCH block automatically makes the surrounding block catch exceptions. Inside the CATCH block, all lexical variables from the outside are normally accessible, so all the interesting information is available for error processing.

Inside the CATCH block, the error object is available in the $_ variable, on the outside it is available in $!. If an exception is thrown inside a CATCH block, it is not caught — unless there is a second, inner CATCH that handles it.

The insides of a CATCH block typically consists of when clauses, and sometimes a default clause. If any of those matches the error object, the error is considered to be handled. If no clause matches (and no default block is present), the exception is rethrown.

Comparing the output from rakudo to the one that niecza produces for the same code, one can see that the last line differs:

 Other error: Nominal type check failed in binding Real $x in might_die; got Complex, needed Real

This higlights a problem in the current state: The wording of error messages is not yet specified, and thus differs among implementations.

I am working on rectifying that situation, and also throwing interesting types of error objects. In the past week, I have managed to start throwing specific error objects from within the Rakudo compiler. Here is an example:

 $ ./perl6 -e 'try eval q[ class A { $!x } ]; say "error: $!"; say $!.perl'
 error: Attribute $!x not declared in class A
 X::Attribute::Undeclared.new(
         name => "\$!x",
         package-type => "class",
         package-name => "A", filename => "",
         line => 1,
         column => Any,
         message => "Attribute \$!x not declared in class A"
 )
 # output reformatted for clarity

The string that is passed to eval is not a valid Perl 6 program, because it accesses an attribute that wasn’t declared in class A. The exception thrown is of type X::Attribute::Undeclared, and it contains several details: the name of the attribute, the type of package it was missing in (could be class, module, grammar and maybe others), the name of the package, the actual error message and information about the source of the error (line, cfile name (empty because eval() operates on a string, not on a file), and column, though column isn’t set to a useful value yet).

X::Attribute::Undeclared inherits from type X::Comp, which is the common superclass for all compile time errors. Once all compile time errors in Rakudo are switched to X::Comp objects, one will be able to check if errors were produced at run time or at compile with code like

 eval $some-string;
 CATCH {
     when X::Comp { say 'compile time' }
     default      { say 'run time'     }
 }

The when block smart-matches the error object against the X::Comp type object, which succeeds whenever the error object conforms to that type (so, is of that type or a subclas of X::Comp).

Writing and using new error classes is quite easy:

 class X::PermissionDenied is X::Base {
     has $.reason;
     method message() { "Permission denied: $.reason" };
 }

 # and using it somewhere:
 die X::PermissionDenied.new( reason => "I don't like your nose");

So Perl 6 has a rather flexible error handling mechanism, and libraries and applications can choose to throw error objects with rich information. The plan is to have the Perl 6 compilers throw such easily introspectable error objects too, and at the same time unify their error messages.

Many thanks go to Ian Hague and The Perl Foundation for funding my work on exceptions.

Perl 6 Advent Calendar: Meta-programming: what, why and how

Wed, 14 Dec 2011 21:55:48 +0000

Sometimes, it’s good to take ones understanding of a topic, throw it away and try to build a new mental model of it from scratch. I did that in the last couple of years with object orientation. Some things feel ever so slightly strange to let go of and re-evaluate. For many people, an object really is “an instance of a class” and inheritance really is a core building block of OOP. I suspect many people who read this post will at this point be thinking, “huh, of course they really are” – and if so, that’s totally fair enough. Most people’s view of OOP will, naturally, be based around the languages they’ve applied object orientation in, and most of the mainstream languages really do have objects that are instances of classes and really do have inheritance as a core principle.

Step back and look around, however, and things get a bit more blurry. JavaScript doesn’t have any notion of classes. CLOS (the Common Lisp Object System) does have classes, but they don’t have methods. And even if we do just stick with the languages that have classes with methods, there’s a dizzying array of “extras” playing their part in the language’s OO world view; amongst them are interfaces, mixins and roles.

Roles – more often known as traits in the literature – are a relatively recent arrival on the OO scene, and they serve as an important reminder than object orientation is not finished yet. It’s a living, breathing paradigm, undergoing its own evolution just as our programming languages in general are.

And that brings me nicely on to Perl 6 – a language that from the start has set out to be able to evolve. At a syntax level, that’s done by opening up the grammar to mutation – in a very carefully controlled way, such that you always know what language any given lexical scope is in. Meta-programming plays that same role, but in the object orientation and type system space.

So what is a meta-object? A meta-object is simply an object that describes how a piece of our language works. What sorts of things in Perl 6 have meta-objects? Here’s a partial list.

Classes
Roles
Subsets
Enumerations
Attributes
Subroutines
Methods
Signatures
Parameters

So that’s meta-objects, but what about the protocol? You can read protocol as “API” or “interface”. It’s an agreed set of methods that a meta-object will provide if it wants to expose certain features. Let’s consider the API for anything that can have methods, such as classes and roles. At a minimum, it will provide:

add_method – adds a method to the object
methods – enables introspection of the methods that the object has
method_table – provides a hash of the methods in this type, excluding any that may be inherited

What about something that you can call a method on? It just has to provide one thing:

find_method – takes an object and a name, and returns the method if one exists

By now you may be thinking, “wait a moment, is there something that you can call a method on, but that does not have methods”? And the answer is – yes. For example, an enum has values that you can call a method on – the methods that the underlying type of the enumeration provides. You can’t actually add a method to an enum itself, however.

What’s striking about this is that we are now doing object oriented programming…to implement our object oriented language features. And this in turn means that we can tweak and extend our language – perhaps by subclassing an existing meta-object, or even by writing a new one from scratch. To demonstrate this, we’ll do a simple example, then a trickier one.

Suppose we wanted to forbid multiple inheritance. Here’s the code that we need to write.

my class SingleInheritanceClassHOW
    is Mu is Metamodel::ClassHOW
{
    method add_parent(Mu $obj, Mu $parent) {
        if +self.parents($obj, :local) > 0 {
            die "Multiple inheritance is forbidden!";
        }
        callsame;
    }
}
my module EXPORTHOW { }
EXPORTHOW.WHO.<class> = SingleInheritanceClassHOW;

What are we doing here? First, we inherit from the standard Perl 6 implementation of classes, which is defined by the class Metamodel::ClassHOW. (For now, we also inherit from Mu, since meta-objects currently consider themselves outside of the standard type hierarchy. This may change.) We then override the add_parent method, which is called whenever we want to add a parent to a class. We check the current number of (local) parents that a class has; if it already has one, then we die. Otherwise, we use callsame in order to just call the normal add_parent method, which actually adds the parent.

You may wonder what the $obj parameter that we’re taking is, and why it is needed. It is there because if we were implementing a prototype model of OOP, then adding a method to an object would operate on the individual object, rather than stashing the method away in the meta-object.

Finally, we need to export our new meta-object to anything that uses our module, so that it will be used in place of the “class” package declarator. Do do this, we stick it in the EXPORTHOW module, under the name “class”. The importer pays special attention to this module, if it exists. So, here it is in action, assuming we put our code in a module si.pm. This program works as usual:

use si;
class A { }
class B is A { }

While this one:

class A { }
class B { }
class C is A is B { }

Will die with:

===SORRY!===
Multiple inheritance is forbidden!

At compile time.

Now for the trickier one. Let’s do a really, really simple implementation of aspect oriented programming. We’ll write an aspects module. First, we declare a class that we’ll use to mark aspects.

my class MethodBoundaryAspect is export {
}

Next, when a class is declared with “is SomeAspect”, where SomeAspect inherits from MethodBoundaryAspect, we don’t want to treat it as inheritance. Instead, we’d like to add it to a list of aspects. Here’s an extra trait modifier to do that.

multi trait_mod:(Mu:U $type, MethodBoundaryAspect:U $aspect) is export {
    $aspect === MethodBoundaryAspect ??
        $type.HOW.add_parent($type, $aspect) !!
        $type.HOW.add_aspect($type, $aspect);
}

We take care to make sure that the declaration of aspects themselves – which will directly derive from this class – still works out by continuing to call add_parent for those. Otherwise, we call a method add_aspect, which we’ll define in a moment.

Supposing that our aspects work by optionally implementing entry and exit methods, which get passed the details of the call, here’s our custom meta-class, and the code to export it, just as before.

my class ClassWithAspectsHOW
    is Mu is Metamodel::ClassHOW
{
    has @!aspects;
    method add_aspect(Mu $obj, MethodBoundaryAspect:U $aspect) {
        @!aspects.push($aspect);
    }
    method compose(Mu $obj) {
        for @!aspects -> $a {
        for self.methods($obj, :local) -> $m {
            $m.wrap(-> $obj, |$args {
                $a.?entry($m.name, $obj, $args);
                my $result := callsame;
                $a.?exit($m.name, $obj, $args, $result);
                $result
            });
        }
        }
        callsame;
    }
}
my module EXPORTHOW { }
EXPORTHOW.WHO.<class> = ClassWithAspectsHOW;

Here, we see how add_aspect is implemented – it just pushes the aspect onto a list. The magic all happens at class composition time. The compose method is called after we’ve parsed the closing curly of a class declaration, and is the point at which we finalize things relating to the class declaration. Ahead of that, we loop over any aspects we have, and the wrap each method declared in the class body up so that it will make the call to the entry and exit methods.

Here’s an example of the module in use.

use aspects;
class LoggingAspect is MethodBoundaryAspect {
    method entry($method, $obj, $args) {
        say "Called $method with $args";
    }
    method exit($method, $obj, $args, $result) {
        say "$method returned with $result.perl()";
    }
}
class Example is LoggingAspect {
    method double($x) { $x * 2 }
    method square($x) { $x ** 2 }
}
say Example.double(3);
say Example.square(3);

And the output is:

Called double with 3
double returned with 6
6
Called square with 3
square returned with 9
9

So, a module providing basic aspect orientation support in 30 or so lines. Not so bad.

As you can imagine, we can go a long way with meta-programming, whether we want to create policies, development tools (like Grammar::Debugger) or try to add entirely new concepts to our language. Happy meta-hacking.

Perl 6 Advent Calendar: Bailador — A small Dancer clone

Tue, 13 Dec 2011 18:58:49 +0000

Today we’ll write a simple Dancer clone in Perl 6. Simple also means Very Minimal — it will only recognize basic GET requests. Let’s look at how the simplest Dancer app possible looks like:

    get '/' => sub {
        "Hello World!"
    };
    dance;

So we need something to add routes to our app, and something to run it. Let’s take care of adding routes first. We’ll create an array to store all those, and thus get() will just add them to it.

    my @routes;
    sub get(Pair $x) is export {
        @routes.push: $x;
    }

In case you’re not familiar with the Pair thing, in Perl 6 a fat comma (=>) creates an actual data structure containing a key and a value. In this case, the key is just a string ‘/’, and the value is a subroutine.

Having @routes being a simple array of keys and values we can now write a simple dispatcher:

    sub dispatch($env) {
        for @routes -> $r {
            if $env<REQUEST_URI> ~~ $r.key {
                return $r.value.();
            }
        }
        return "404";
    }

dispatch() takes a hash representing our environment, which contains the REQUEST_URI string, basing on which we’ll try to find an appropriate candidate to dispatch to.

The above example is also cheating a bit: it just returns a ’404′ string instead of creating a proper HTTP response. Making it respond properly is left as an exercise for the reader (not the last one in this short article, I assure you :)).

Since we got that far already, writing our own dance() is a piece of cake. We’re going to call it baile() though. Why do we write all this in Spanish? If you can guess on which classes I was bored and wrote this thing on a piece of paper, then on the next YAPC I’ll show you the fastest possible way to tie a shoe. No kidding! But let’s finish this thing first.

Luckily we don’t need to write our own web server now. We have HTTP::Server::Simple::PSGI in Perl 6, which will do most of the hard work for us. The only thing we have to do is to create a PSGI app. In case you’ve never heard of it, a PSGI app is a mere subroutine, taking the environment as an argument, and returning an array of three things: an HTTP response code, an array of HTTP headers and a response body. Once we have our PSGI app ready, we just feed HTTP::Server::Simple::PSGI with it, and we’re good to go.

    sub baile is export {
        my $app = sub($env) {
            my $res = dispatch($env);
            return ['200', [ 'Content-Type' => 'text/plain' ], $res];
        }

        given HTTP::Server::Simple.PSGI.new {
            .host = 'localhost';
            .app($app);
            .run;
        }
    }

Yep, we’re cheating again and returning 200 no matter what. Remember the part about “an exercise for the reader?” You can think about it while celebrating a working Dancer clone.

But wait, there’s more!

Let’s look at our dispatch() once again:

    sub dispatch($env) {
        for @routes -> $r {
            if $env<REQUEST_URI> ~~ $r.key {
                return $r.value.();
            }
        }
        return "404";
    }

You probably noticed that we’ve used ~~ — a smartmatching operator. Thanks to that, we can match REQUEST_URI against strings, but not only. Junctions will work fine too:

    get any('/h', '/help', '/halp') => sub {
        "A helpful help message"
    }

And regexes:

    get /greet\/(.+)/ => sub ($x) {
        "Welcome $x"
    }

The last one will need a bit of tweaking in dispatch(). Yes, ~~ does the regex matching for us, but we have to take care of passing the match results to the callback function. Let’s modify the if body then:

    sub dispatch($env) {
        for @routes -> $r {
            if $env<REQUEST_URI> ~~ $r.key {
                if $/ {
                    return $r.value.(|$/.list);
                } else {
                    return $r.value.();
                }
            }
        }
        return "404";
    }

The if $/ part checks whether the match resulted in setting the Match object in the $/ variable. If it did, we flatten the Match to a list, and pass it to the callback function. We need a | prefix, so it becomes expanded to a parameter list instead of being passed as a mere array. From now on, the above example with greet will Just Work. Yay!

The Bailador code presented here is available in the Github repository. If you feel challenged by the “exercises for the reader”, or just want to make it a bit more proper Dancer port, you’re welcome to hack on it a bit and contribute. I hope I showed you how simple it is to write a simple, yet useful thing, and going with those simple steps we can hopefully get to something close to a full-blown Dancer port. Happy hacking, and have an appropriate amount of fun!

Perl 6 Advent Calendar: Privacy and OOP

Sun, 11 Dec 2011 21:31:55 +0000

There are a number of ways in which Perl 6 encourages you to restrict the scope of elements of your program. By doing so, you can better understand how they are used and will be able to refactor them more easily later, potentially aiding agility. Lexical scoping is one such mechanism, and subroutines are by default lexically scoped.

Let’s take a look at a class that demonstrates some of the object oriented related privacy mechanisms.

    class Order {
        my class Item {
            has $.name;
            has $.price;
        }

        has Item @!items;

        method add_item($name, $price) {
            @!items.push(Item.new(:$name, :$price))
        }

        method discount() {
            self!compute_discount()
        }

        method total() {
            self!compute_subtotal() - self!compute_discount();
        }

        method !compute_subtotal() {
            [+] @!items>>.price
        }

        method !compute_discount() {
            my $sum = self!compute_subtotal();
            if $sum >= 1000 {
                $sum * 0.15
            }
            elsif $sum >= 100 {
                $sum * 0.1
            }
            else {
                0
            }
        }
    }

Taking a look at this, the first thing we notice is that Item is a lexical class. A class declared with “my” scope can never be referenced outside of the scope it is declared within. In our case, we never leak instances of it outside of our Order class either. This makes our class an example of the aggregate pattern: it prevents outside code from holding direct references to the things inside of it. Should we ever decide to change the way that our class represents its items on the inside, we have complete freedom to do so.

The other example of a privacy mechanism at work in this class is the use of private methods. A private method is declared just like an ordinary method, but with an exclamation mark appearing before its name. This gives it the same visibility as an attribute (which, you’ll note, are also declared with an exclamation mark – a nice bit of consistency). It also means you need to call it differently, using the exclamation mark instead of the dot.

Private methods are non-virtual. This may seem a little odd at first, but is consistent: attributes are also not visible to subclasses. By being non-virtual, we also get some other benefits. The latest Rakudo, with its optimizer cranked up to its highest level, optimizes calls to private methods and complains about missing ones at compile time. Thus a typo:

    self!compite_subtotal() - self!compute_discount();

Will get us a compile time error:

    ===SORRY!===
    CHECK FAILED:
    Undefined private method 'compite_subtotal' called (line 18)

You may worry a little over the fact that we now can’t subclass the discount computation, but that’s likely not a good design anyway; for one, we’d need to also expose the list of items, breaking our aggregate boundary. If we do want pluggable discount mechanisms we’d probably be better implementing the strategy pattern.

Private methods can, of course, not be called from outside of the class, which is also a compile time error. First, if you try:

    say $order!compute_discount;

You’ll be informed:

    ===SORRY!===
    Private method call to 'compute_discount' must be fully qualified
    with the package containing the method

Which isn’t so surprising, given they are non-virtual. But even if we do:

    say $o!Order::compute_discount;

Our encapsulation-busting efforts just get us:

    ===SORRY!===
    Cannot call private method 'compute_discount' on package Order
    because it does not trust GLOBAL

This does, however, hint at the get-out clause for private methods: a class may choose to trust another one (or, indeed, any other package) to be able to call its private methods. Critically, this is the decision of the class itself; if the class declaration didn’t decide to trust you, you’re out of luck. Generally, you won’t need “trusts”, but occasionally you may be in a situation where you have two very closely coupled classes. That’s usually undesirable in itself, though. Don’t trust too readily. :-)

So, lexical classes, private methods and some nice compiler support to help catch mistakes. Have an agile advent. :-)

Perl 6 Advent Calendar: Documenting Perl 6

Sat, 10 Dec 2011 18:41:08 +0000

A wise man once said that programs must be written for people to read, and only incidentally for machines to execute. But aside from being read, your code is also going to be used by people, who don’t really want to dive into it to understand what it does. That’s where the documentation comes in.

In Perl 5 we had POD, which stands for Plain Old Documentation. In Perl 6 we have Pod, which is not really an abbreviation of anything. As its specification says, “Perl 6′s Pod is much more uniform, somewhat more compact, and considerably more expressive”. It has changed slightly compared to Perl 5 Pod, but most of the stuff remains the same, or at least very similar.

There are three main types of Pod blocks in Perl 6. Delimited blocks are probably the most obvious and simple ones:

    =begin pod

    <whatever Pod content we want>

    =end pod

Paragraph blocks are a bit more magical. They begin with =for, and end on the nearest blank line (as the name, Paragraph, suggest):

    my $piece = 'of perl 6 code'

    =for comment
    Here we put whatever we want.
    The compiler will not notice anyway.

    our $perl6 = 'code continues';

Abbreviated blocks are similar to Paragraph blocks. The leading = is followed immediately by a Pod block identifier and the content. Sounds familiar?

    =head1 Shoulders, Knees and Toes

That’s right, =head is nothing magical in Perl 6 Pod. That means you can write it also as a paragraph block

    =for head1
    Longer header
    than we usually write.

Or a full-blown delimited block

    =begin head1

    This header is longer than it should be

    =end head1

Any block can be written as a delimited block, paragraph block, or abbreviated block. No magic. Not all blocks are created equal, of course. =head will be treated differently than plain =pod. By whom? By the Pod renderer, of course, but also by the Perl 6 compiler itself. In Perl 6, Pod is not a second-class citizen, ignored during the program compilation. Pod in Perl 6 is a part of the code; along with parsing and constructing AST of the code to be executed, the compiler also parses and builds AST of all Pod blocks. They are then kept in the special $=POD variable, and can be inspected by the runtime:

    =begin pod

    Some pod content

    =end pod

    say $=POD[0].content[0].content;

The say line may look a little complicated. Content, of a content, of a what? What really happens, is that ‘Some pod content’ is parsed as an ordinary paragraph, and kept in the Pod::Block::Para object. The delimited block started with =begin pod becomes a Pod::Block::Named, and it can contain any number of child blocks. It’s also a first block in our example code, so it ends up in $=POD[0].

You now probably think “geez, how ugly is that. Who’s going to use it in this form”. Don’t worry. Frankly, I don’t expect anyone to use the AST directly. That’s what Pod renderers are useful for. Take for example Pod::To::Text:

    =begin pod

    =head1 A Heading!

    A paragraph! With many lines!

        An implicit code block!
        my $a = 5;

    =item A list!
    =item Of various things!

    =end pod

    DOC INIT {
        use Pod::To::Text;
        pod2text($=POD);
    }

Ran as it is, the code doesn’t produce any output. Why is it so? The DOC INIT block looks a little special. It gets run with every other INIT block, but also only when the --doc flag is passed to the compiler. Let’s take a look:

    $ perl6 --doc foo.pl
    A Heading!

    A paragraph! With many lines!

        An implicit code block!
        my $a = 5;

     * A list!

     * Of various things!

Actually, when no DOC INIT block exists in the code, the compiler generates a default DOC INIT, identical to the one in the example above. So you could really omit it, leaving only the Pod in the file, and perl6 --doc will produce exactly the same result.

But wait, there’s more!

I wrote about 3 types of Pod blocks, but there’s another one I didn’t talk about before. They are Declarator blocks, and they single purpose is to document the actual Perl 6 objects. Take a look.

    #= it's a sheep! really!
    class Sheep {

        #= produces a funny sound
        method bark {
            say "Actually, I don't think sheeps bark"
        }
    }

Every declarator block gets attached to the object which comes after it. It’s available in the .WHY attribute, so we can use it like this:

    say Sheep.WHY.content;                      # it's a sheep! really!
    say Sheep.^find_method('bark').WHY.content; # produces a funny sound

In this case we also don’t need to care about doing a ^find_method and all this for every piece of documentation we want to read. The mighty Pod::To::Text takes care about it too. If we run the Sheep code with --doc flag, we get the following:

    class Sheep: it's a sheep! really!

    method bark: produces a funny sound

The specification says it’s possible to document all the class attributes and all the arguments that methods or subroutines take. Unfortunately no Perl 6 implementation (that I know of) implements those yet.

There are dozens of Pod features that are not covered by this post, for example the formatting codes (<, > and so), or tables. If you’re interested take a look at Synopsis 26 (http://perlcabal.org/syn/S26.html). It’s actually written in Pod 6, and rendered by Pod::To::HTML. Not all features it describes are implemented yet, but most of them are (see the test suite linked below), and some modules are actually documented with it (Term::ANSIColor for example).

Some useful links:

Synopsis 26
Pod::To::Text source code
Term::ANSIColor documentation
Pod test suite (shows what Pod in Rakudo is capable of)

Happy documenting!

Perl 6 Advent Calendar: Day 9: Contributing to Perl 6

Fri, 09 Dec 2011 14:27:40 +0000

This time, instead of sharing some cool feature of Perl 6, I’d like to talk about how easy it is to contribute usefully to the project. So I’m going to walk you through the process of making a change to Niecza. It does require a bit of domain knowledge (which the fine folks on #perl6 will be happy to help you with) but it’s definitely not rocket science. It’s not even particularly deep computer science, for the most part.

A few days ago, Radvendii asked on #perl6 if there was a round function in the core. The correct answer is “There should be one”, and it lead to a couple of bug fixes in Rakudo. But it got me to thinking — is Niecza supporting round (and its relatives ceiling, floor, and truncate) correctly?

Perl 6 has a huge suite of tests to see if an implementation is conforming to the spec, including a file for the round tests, S32-num/rounders.t. My first step then was to check the spectests currently being run by Niecza. Just like in Rakudo, this is stored in a file named t/spectest.data. So

Wynne:niecza colomon$ grep round t/spectest.data 
Wynne:niecza colomon$

Okay, clearly we’re not running the S32-num/rounders.t test file. (Note, in case you’re getting confused — the links in this post are to the latest versions of the files, which include all the changes I made writing this post.) That’s a sign that something is not properly supported yet. So let’s go ahead and run it by hand to see what happens. Both Niecza and Rakudo use a fudging process, allowing you to mark the bits of a test file that don’t work yet in a particular compiler. So we need to use a special fudging tool to run the code:

Wynne:niecza colomon$ t/fudgeandrun t/spec/S32-num/rounders.t
1..108
not ok 1 - floor(NaN) is NaN
# /Users/colomon/tools/niecza/t/spec/S32-num/rounders.t line 16
#    Failed test
#           got: -269653970229347386159395778618353710042696546841345985910145121736599013708251444699062715983611304031680170819807090036488184653221624933739271145959211186566651840137298227914453329401869141179179624428127508653257226023513694322210869665811240855745025766026879447359920868907719574457253034494436336205824

That’s followed by about 15 similar errors, then

Unhandled exception: Unable to resolve method truncate in class Num
  at /Users/colomon/tools/niecza/t/spec/S32-num/rounders.t line 34 (mainline @ 32) 
  at /Users/colomon/tools/niecza/lib/CORE.setting line 2224 (ANON @ 2) 
  at /Users/colomon/tools/niecza/lib/CORE.setting line 2225 (module-CORE @ 58) 
  at /Users/colomon/tools/niecza/lib/CORE.setting line 2225 (mainline @ 1) 
  at <unknown> line 0 (ExitRunloop @ 0)

Okay, so that’s at least two errors that need fixing.

We’ll go in order here, even though it means tackling what is most likely the most complicated error first. (If you do think this part of the problem is too hard to tackle, please skip ahead, because the last few improvements I made really were incredibly easy to do.) Opening src/CORE.setting, we find the following definition for round:

sub round($x, $scale=1) { floor($x / $scale + 0.5) * $scale }

Okay, so the real problem is in floor:

sub floor($x) { Q:CgOp { (floor {$x}) } }

What the heck does Q:CgOp mean? It means floor is actually implemented in C#. So we open up lib/Builtins.cs and search for floor, eventually finding public static Variable floor(Variable a1). I won’t print the full source code here, because it is on the long side, with a case for each of the different number types. We’re only interested in the floating point case here:

        if (r1 == NR_FLOAT) {
            double v1 = PromoteToFloat(r1, n1);
            ulong bits = (ulong)BitConverter.DoubleToInt64Bits(v1);
            BigInteger big = (bits & ((1UL << 52) - 1)) + (1UL << 52);
            int power = ((int)((bits >> 52) & 0x7FF)) - 0x433;
            // note: >>= has flooring semantics for signed values
            if ((bits & (1UL << 63)) != 0) big = -big;
            if (power > 0) big <<= power;
            else big >>= -power;
            return MakeInt(big);
        }

We don’t actually need to understand how all that works to fix this problem. The important bit is the PromoteToFloat line, which sets v1 to the floating point value which is the input to our floor. If we add a trap right after that, it should fix this bug. A quick C# websearch shows me that Double has member functions IsNaN, IsNegativeInfinity, and IsPositiveInfinity. Looking a bit around the Niecza source shows there is a MakeFloat function for returning floating point values. Let’s try:

if (Double.IsNaN(v1) || Double.IsNegativeInfinity(v1) || Double.IsPositiveInfinity(v1)) {
    return MakeFloat(v1);
}

One quick call to make later, I can try the test file again:

Wynne:niecza colomon$ t/fudgeandrun t/spec/S32-num/rounders.t
1..108
ok 1 - floor(NaN) is NaN
ok 2 - round(NaN) is NaN
ok 3 - ceiling(NaN) is NaN
not ok 4 - truncate(NaN) is NaN
# /Users/colomon/tools/niecza/t/spec/S32-num/rounders.t line 19
#    Failed test
#           got: -269653970229347386159395778618353710042696546841345985910145121736599013708251444699062715983611304031680170819807090036488184653221624933739271145959211186566651840137298227914453329401869141179179624428127508653257226023513694322210869665811240855745025766026879447359920868907719574457253034494436336205824

Progress! Apparently truncate uses a separate method, so we’ll have to fix it separately.

sub truncate($x) { $x.Int }
method Int() { Q:CgOp { (coerce_to_int {self}) } }

    public static Variable coerce_to_int(Variable a1) {
        int small; BigInteger big;
        return GetAsInteger(a1, out small, out big) ?
            MakeInt(big) : MakeInt(small);
    }

Oooo, this is perhaps a little bit trickier. Still a basic variant on the previous method, grabbing boilerplate code from a nearby function:

        int r1;
        P6any o1 = a1.Fetch();
        P6any n1 = GetNumber(a1, o1, out r1);

        if (r1 == NR_FLOAT) {
            double v1 = PromoteToFloat(r1, n1);
            if (Double.IsNaN(v1) || Double.IsNegativeInfinity(v1) || Double.IsPositiveInfinity(v1)) {
                return MakeFloat(v1);
            }
        }

I skipped the HandleSpecial2 bit in the boilerplate, because I’m never quite sure how that works. Luckily, we have the spectests to check and see if I have broken something by doing this.

Now the first 15 tests in rounders.t pass, leaving us with the

Unhandled exception: Unable to resolve method truncate in class Num

error. That should be easy to handle! If we go back to lib/CORE.setting and search for ceiling, we see it appears two times: in the catch-all base class Cool and as a stand-alone sub. If we look at the neighboring subs, we see floor, ceiling, round, and truncate are all defined. If we look in Cool, however, only floor, ceiling, and round defined. That’s the source of our trouble!

The method definitions of the others in Cool are really simple; all they do is forward to the sub versions. It’s very easy to add a truncate that does that:

    method truncate() { truncate self }

And poof! This time when we run rounders.t, we pass all 108 tests.

At this point we’ve got three things left to do. First, now that rounders.t passes, we need to add it to t/spectest.data. The list of tests there is ordered, so I just find the S32-num section and add S32-num/rounders.t in alphabetical order.

Next I will commit all the changes to my copy of the git repo. (I won’t explain how to do that, there are lots of git tutorials on the web.) Then I run make spectest to make sure I haven’t broken anything with these changes. (Hmm… actually a few TODO passing, bugs elsewhere that this patch has fixed! Oh, and one test broken, but it’s one which we were only passing by accident before, so I won’t feel bad about fudging it.)

Once that is done, you need to send the patch on to the Niecza developers; I believe the easiest way to do this is via github.

I’ve got one more little change to make that popped into my head while I was working on this. One naive way of implementing, say floor would be to convert the input into a floating point value (a Num in Perl 6) and then do Num.floor. That doesn’t work for all numbers, however, as most of the other number types are capable of storing numbers larger than will fit in a standing floating point double. So we probably need tests in the test suite to check for these cases. Let’s add them.

The tests in rounders.t are weirdly organized for my taste. But hey, we can always add our tests at the bottom.

{
    my $big-int = 1234567890123456789012345678903;
    is $big-int.floor, $big-int, "floor passes bigints unchanged";
    is $big-int.ceiling, $big-int, "ceiling passes bigints unchanged";
    is $big-int.round, $big-int, "round passes bigints unchanged";
    is $big-int.truncate, $big-int, "truncate passes bigints unchanged";
}

That passes okay in Niecza. (Probably out of courtesy we should check it on Rakudo as well and fudge it appropriately to make sure we’re not breaking their spectest!) We need to remember to add the count of new tests to the plan at the top of the test file. And then we can push that fix to github as well.

In conclusion, contributing to Perl 6 is easy. Anyone who tries writing Perl 6 code and reports problems they have to #perl6 is helping in a very real way. If you can write even fairly simple Perl 6 code, then you can write useful spec tests. It’s only marginally harder to write new methods for the setting in Perl 6. And even when you have to get down and dirty and start dealing with the language the compiler is implemented in, it’s still quite possible to do useful work without any deep understanding of how the compiler works.

Perl 6 Advent Calendar: Lexicality and Optimizability

Thu, 08 Dec 2011 13:32:17 +0000

Traditional optimizations in compilers rely on compile-time knowledge about the program. Usually statically typed langauges like Java and C are rather good at that, and dynamic languages like Perl 5, ruby and python are not.

Perl 6 offers the flexibility of dynamic languages, but tries to provide much optimizability nonetheless by gradual typing, that is offering optional static type annotations.

But even in the presence of type annotations, another piece is needed for compile time dispatch decision and inlining: the knowledge about the available routines (and in the case of multi subs, the available candidates).

To provide that knowledge, Perl 6 installs subroutine in lexical scopes (and not packages / symbol tables, as in Perl 5), and lexical scopes are immutable at run time. (Variables inside the lexical scopes are still mutable, you just cannot add or remove entries at run time).

To provide the necessary flexibility, Perl 6 allows code to run at compile time. A typical way to run code at compile time is with the use directive:

 {
    use Test;  # imports routines into the current
               # lexical scope, at compile time
    plan 1;
    ok 1, 'success';
 }
 # plan() and ok() are not available here,
 # outside the scope into which the routines has been imported to.

The upside is that a sufficiently smart compiler can complain before runtime about missing routines and dispatches that are bound to fail. Current Rakudo does that, though there are a certainly cases that rakudo does not detect yet, but which are possible to detect.

 sub f(Int $x) {
          say $x * 2;
           }
 say "got here";
 f('some string');

produces this output with current Rakudo:

 ===SORRY!===
 CHECK FAILED:
 Calling 'f' will never work with argument types (str) (line 5)
     Expected: :(Int $x)

Since built-in routines are provided in an outer scope to the user’s program, all built-in routines are automatically subjected to all the same rules and optimizations as user-provided routines.

Note that this has other implications: require, which loads modules at run time, now needs a list of symbols to stub in at compile time, which are later wired up to the symbols loaded from the module.

The days are past where "a sufficiently smart compiler" was a legend; these days we have a compiler that can provide measurable speed-ups. There is still room for improvements, but we are now seeing the benefits from static knowledge and lexical scoping.

Perl 6 Advent Calendar: Adventures in writing a simple grammar profiler

Wed, 07 Dec 2011 00:00:22 +0000

Inspired by jnthn’s earlier post on Grammar::Debugger, I wondered how hard it would be to implement a simple Perl 6 grammar profiler. Turns out it wasn’t that hard at all.

As far as profiling goes, all I wanted was counts of how many times each rule was executed and the cumulative time each rule took to execute. The interface I had in mind was something simple–a multi-level hash with names of grammars at the first level then, at the second level, names of the individual rules within the grammar, and finally the actual timing information. The timing information would be accessed thusly:

say "MyGrammar::MyRule was called " ~ %timing<MyGrammar><MyRule><calls> ~ "times";
say "and took " ~ %timing<MyGrammar><MyRule><time> ~ " seconds to execute";

But first I had to figure out what jnthn’s code was doing.

From the outside looking in, the basic technique is to replace the normal grammar meta-object with a custom meta-object that inherits most of the behavior of the normal grammar meta-object but replaces the normal method lookup with a custom one that returns a routine that collects the timing information while calling the original method.

Looking at jnthn’s code, I see that if the method name starts with ! or is any one of “parse”, “CREATE”, “Bool”, “defined” or “MATCH”, we just return the original method without modification. This is so that we don’t trace private methods or accidentally trace methods that aren’t directly part of the grammar but are used by it. In my simple profiler, I need to get the name of the grammar, which I do by calling my $grammar = $obj.WHAT.perl. So it looks like I need to add “perl” to that list of methods to pass through unscathed. Otherwise, I get an infinite recursion.

Anyway, for those method names that don’t match the aforementioned criteria, we return a custom built routine that accumulates the execution time and increments a counter for the number of calls. Seems straight-forward enough … below is the code (somewhat untested):

my %timing;

my class ProfiledGrammarHOW is Metamodel::GrammarHOW is Mu {

    method find_method($obj, $name) {
        my $meth := callsame;
        substr($name, 0, 1) eq '!' || $name eq any(<parse CREATE Bool defined MATCH perl>) ??
            $meth !!
            -> $c, |$args {
                my $grammar = $obj.WHAT.perl;
                %timing{$grammar} //= {};                   # Vivify grammar hash
                %timing{$grammar}{$meth.name} //= {};       # Vivify method hash
                my %t := %timing{$grammar}{$meth.name};
                my $start = now;                            # get start time
                my $result := $meth($obj, |$args);          # Call original method
                %t<time> += now - $start;             # accumulate execution time
                %t<calls>++;
                $result
            }
    }

    method publish_method_cache($obj) {
        # no caching, so we always hit find_method
    }
}

sub get-timing is export { %timing }
sub reset-timing is export { %timing = {} }

my module EXPORTHOW { }
EXPORTHOW.WHO.<grammar> = ProfiledGrammarHOW;

Assuming the above code was saved in file called “GrammarProfiler.pm”, you’d use it by adding the line use GrammarProfiler; to the top of any program that makes grammar declarations. Then after you parse your grammar, you can call get-timing() to obtain the hash that has the timing information for the individual rules that were executed during the parse or reset-timing() to clear the timing information.

Of course, a more full-fledged profiler would do much more work and provide many more profiling options, but this isn’t bad for a quick hack and it just might be useful too.

Perl 6 Advent Calendar: Tetris on Niecza

Mon, 05 Dec 2011 23:45:26 +0000

Niecza, the Other Perl 6 Implementation on Mono and .NET, recently gained the ability to call almost any Common Language Runtime library. In Niecza’s examples directory, a simple 30 line script called gtk1.pl shows how to use gtk-sharp, and thus Gtk and Gdk, the graphical basis of Gnome. Here is gtk1′s central working part:

my $btn = Button.new("Hello World");
$btn.add_Clicked: sub ($obj, $args) { #OK
    # runs when the button is clicked.
    say "Hello World";
    Application.Quit;
};

The add_Clicked method defines a callback routine, essential to process user input. Running gtk1.pl makes the following resizeable button in a window, and it closes when clicked:

From gtk1 to Tetris is not far, see the source also in niecza/examples. Two extra ingredients make it possible: a timer tick callback routine to animate the falling pieces, and non blocking keyboard input to give the user the illusion of control. Add some simple physics and Cairo graphics and you have a playable game (modulo scoring and similar low hanging fruit) in under 170 lines of Perl 6.

Animation by timer tick works by causing the whole window to be redrawn by an ExposeEvent at regular intervals. The redraw tries to move the falling piece downwards, and if the physics says no, it adds a new piece at the top instead. (Bug: that should eventually fail with a full pile of pieces.) GLibTimeout sets up the timer callback handler which invokes .QueueDraw. The default interval is 300 milliseconds, and if the game engine wants to speed that up, it can adjust $newInterval which will replace the GLibTimeout on the next tick (sorry about the line wrap):

my $oldInterval = 300;
my $newInterval = 300;
...
GLibTimeout.Add($newInterval, &TimeoutEvent);
...
sub TimeoutEvent()
{
    $drawingarea.QueueDraw;
    my $intervalSame = ($newInterval == $oldInterval);
    unless $intervalSame { GLibTimeout.Add($newInterval, &TimeoutEvent); }
    return $intervalSame; # True means continue calling this timeout handler
}

Thanks to the excellent way Gtk handles typing, the keystroke event handler is fairly self documenting. The Piece subroutines do the physics ($colorindex 4 is the square block that does not rotate):

$drawingarea.add_KeyPressEvent(&KeyPressEvent);
...
sub KeyPressEvent($sender, $eventargs) #OK not used
{
    given $eventargs.Event.Key {
        when 'Up' { if $colorindex != 4 { TryRotatePiece() } }
        when 'Down' { while CanMovePiece(0,1) {++$pieceY;} }
        when 'Left' { if CanMovePiece(-1,0) {--$pieceX;} }
        when 'Right' { if CanMovePiece( 1,0) {++$pieceX;} }
    }
    return True; # means this keypress is now handled
}

With a bit more glue added, here is the result:

This post has glossed over other details such as the drawing of the graphics, because a later Perl 6 Advent post will cover that, even showing off some beautiful fractals, so keep following this blog! The above software was presented at the London Perl Workshop 2011.

Perl 6 Advent Calendar: The Flip-Flop operator

Mon, 05 Dec 2011 07:00:59 +0000

Perl 5 has a binary operator called flip-flop that is false until its first argument evaluates to true and it stays true (flips) until the second argument evaluates to true at which point it becomes false again (flops). This is such a useful operator that Perl 6 also has flip-flop, only it’s spelled ff and has a few variants:

    ff
    ff^
    ^ff
    ^ff^

The circumflex means to skip the end point on that end.

Perhaps some examples are in order …

    for 1..20 { .say if $_ == 9  ff  $_ == 13; }     # 9 10 11 12 13
    for 1..20 { .say if $_ == 9  ff^ $_ == 13; }     # 9 10 11 12
    for 1..20 { .say if $_ == 9 ^ff  $_ == 13; }     #   10 11 12 13
    for 1..20 { .say if $_ == 9 ^ff^ $_ == 13; }     #   10 11 12

In each example we’re iterating over the range of numbers from 1 to 20 and output those numbers where the flip-flop returns true. Both the right hand side of the flip-flop ($_ == 9) and left hand side of the flip-flop ($_ == 13) are evaluated on each iteration of the loop. (I’ve used simple numeric comparison on both sides of the flip-flop operators here but, in general, any boolean expression could be used.)

Each instance of the flip-flop operator maintains it’s own little bit of internal state to decide when to return True or False. All flip-flop operators are born with their internal state set to return False waiting for the moment they can be flipped and start returning True.

In the first and second examples when $_ == 9, the flip-flop operators flips their internal state to True and immediately return True. In the third and fourth examples when $_ == 9 the flip-flop operators set their internal state to True but they return False on that iteration because of the leading circumflex.

Similarly, in the first and third examples above, once the RHS evaluates to True, the flip-flop operators flop their internal state back to False on next evaluation and return True. In the third and fourth examples, the flip-flops operators flop sooner by returning False immediately upon evaluating the RHS True.

To make the flip-flop operator flip, but never flop, use a * on the RHS:

    for 1..20 { .say if $_ == 15 ff *; }     # 15 16 17 18 19 20

Perl 6 has another set of flip-flop operators that function similar to the ones mentioned above, except the RHS isn’t evaluted when the LHS becomes true. This is particularly important when both the RHS and the LHS of the flip-flop could evaluate to True at the same time. These operators are spelled fff, fff^, ^fff, and ^fff^.

Perl 6 Advent Calendar: Traits — Meta Data With Character

Sun, 04 Dec 2011 07:00:15 +0000

Traits are a nice, extensible way to attach meta data to all sorts of objects in Perl 6.

An example is the is cached trait that automatically caches the functions return value, based on the argument(s) passed to it.

Here is a simple implementation of that trait:

 # this gets called when 'is cached' is added
 # to a routine
 multi sub trait_mod:<is>(Routine $r, :$cached!) {
     my %cache;
     #wrap the routine in a block that..
     $r.wrap(-> $arg {
         # looks up the argument in the cache
         %cache.exists($arg)
             ?? %cache{$arg}
             # ... and calls the original, if it
             # is not found in the cache
             !! (%cache{$arg} = callwith($arg))
         }
     );
 }

 # example aplication:
 sub fib($x) is cached {
     say("fib($x)");
     $x <= 1 ?? 1 !! fib($x - 1) + fib($x - 2);
 }

 # only one call for each value from 0 to 10
 say fib(10);

A trait is applied with a verb, here is. That verb appears in the routine name that handles the trait, here trait_mod:<is>. The arguments to that handler are the object on which the trait is applied, and the name of the trait (here cached) as a named argument.

Note that a production grade is cached would need to handle multiple arguments, and maybe things like limiting the cache size.

In this example, the .wrap method is called on the routine, but of course you can do whatever you want. Common applications are mixing roles into the routine or adding them to a dispatch table.

Traits can not only be applied to routines, but also to parameters, attributes and variables. For example writable accessors are realized with the is rw trait:

 class Book {
     has @.pages is rw;
     ...
 }

Traits are also used to attach documentation to classes and attributes (stay tuned for an addvent calendar post on Pod6), marking routine parameters as writable and declaring class inheritance and role application.

This flexibility makes them ideal for writing libraries that make the user code look like a domain-specific language, and supplying meta data in a safe way.

Perl 6 Advent Calendar: Buffers and Binary IO

Sat, 03 Dec 2011 07:30:36 +0000

Perl 5 is known to have very good Unicode support (starting from version 5.8, the later the better), but people still complain that it is hard to use. The most important reason for that is that the programmer needs to keep track of which strings have been decoded, and which are meant to be treated as binary strings. And there is no way to reliably introspect variables to find out if they are binary or text strings.

In Perl 6, this problem has been addressed by introducing separate types. Str holds text strings. String literals in Perl 6 are of type Str. Binary data is stored in Buf objects. There is no way to confuse the two. Converting back and forth is done with the encode and decode methods.

    my $buf = Buf.new(0x6d, 0xc3, 0xb8, 0xc3, 0xbe, 0x0a);
    $*OUT.write($buf);

    my $str = $buf.decode('UTF-8');
    print $str;

Both of those output operations have the same effect, and print møþ to the standard output stream, followed by a newline. Buf.new(...) takes a list of integers between 0 and 255, which are the byte values from which the new byte buffer is constructed. $*OUT.write($buf) writes the $buf buffer to standard output.

$buf.decode('UTF-8') decodes the buffer, and returns a Str object (or dies if the buffer doesn’t consistute valid UTF-8). The reverse operation is $Buf.encode($encoding). A Str can simply be printed with print.

Of course print also needs to convert the string to a binary representation somewhere in the process. There is a default encoding for this and other operations, and it is UTF-8. The Perl 6 specification allows the user to change the default, but no compiler implements that yet.

For reading, you can use the .read($no-of-bytes) methods to read a Buf, and .get for reading a line as a Str.

The read and write methods are also present on sockets, not just on the ordinary file and stream handles.

One of the particularly nasty things you can accidentally do in Perl 5 is
concatenating text and binary strings, or combine them in another way (like with join or string interpolation). The result of such an operation is a string that happens to be broken, but only if the binary string contains any bytes above 127 — which can be a nightmare to debug.

In Perl 6, you get Cannot use a Buf as a string when you try that, avoiding that trap.

The existing Perl 6 compilers do not yet provide the same level of Unicode support as Perl 5 does, but the bits that are there are much harder to misuse.

Perl 6 Advent Calendar: Grammar::Tracer and Grammar::Debugger

Fri, 02 Dec 2011 12:37:00 +0000

Grammars are, for many people, one of the most exciting features of Perl 6. They unify parsing with object orientation, with each production rule in your grammar being represented by a method. These methods are a little special: they are declared using the keywords “regex”, “rule” or “token”, each of which gives you different defaults on backtracking and whitespace handling. In common is that they lead to the body of the method being parsed using the Perl 6 rule syntax. Under the hood, however, they really are just methods, and production rules that refer to others are really just method calls.

Perl 6 grammars also give you a seamless way to combine declarative and imperative parsing. This means efficient mechanisms, such as NFAs and DFAs, may be used to handle the declarative parts – the things that your tokens tend to be made up of – while a more imperative mechanism drives the parsing of larger structures. This in turn means that you don’t need to write a tokenizer; it can be derived from the rules that you write in the grammar.

So what is the result of parsing some text with a grammar? Well, provided it’s able to match your input, you get back a parse tree. This data structure – made up of Match objects – captures the structure of the input. You can treat each Match node a little bit like a hash, indexing in to it to look at the values that its production rules matched. While you can build up your own tree or other data structure while parsing, sometimes the Match tree you get back by default will be convenient enough to extract the information you need.

That’s wonderful, but there was a key clause in all of this: “provided it’s able to match”. In the case that the grammar fails to match your input, then it tells you so – by giving back an empty Match object that, in boolean context, is false. It’s at this point that many people stop feeling the wonder of grammars and start feeling the pain of trying to figure out why on earth their seemingly fine grammar did not accept the input they gave it. Often, it’s something silly – but in a grammar of dozens of production rules – or sometimes even just ten – the place where things go wrong can be elusive.

Thankfully, help is now at hand, in the form of two modules: Grammar::Tracer, which gives you a tree-like trace output of your grammar, and Grammar::Debugger, which gives the same trace output but also enables you to set breakpoints and single step through the grammar.

A picture is worth a thousand words, so here’s how Grammar::Tracer looks in action!

What we’re seeing here is a tree representation of the production rules that were called, starting at “TOP”, next trying to parse a production rule called “country”, which in turn wants to parse a name, two “num”s and an “integer”. The green indicates a successful match, and next to it we see the snippet of text that was captured.

So what happens when things go wrong? In that case, we see something like this:

Here, we see that something happened during the parse that caused a cascade of failures all the way back up to the “TOP” production rule, which meant that the parse failed overall. Happily, though, we now have a really good clue where to look. Here is the text my grammar was trying to match at the time:

Russia
	Ulan Ude : 51.833333,107.600000 : 1
	Moscow : 55.75000,37.616667 : 4

Looking at this, we see that the “name” rule appears to have picked up “Ulan”, but actually the place in question is “Ulan Ude”. This leads us directly to the name production in our grammar:

token name { \w+ }

Just a smattering of regex fu is enough to spot the problem here: we don’t parse names that happen to have spaces in them. Happily, that’s an easy fix.

token name { \w+ [\h+ \w+]* }

So how do we turn on the tracing? Actually, that’s easy: just take the file containing the grammar you wish to trace, and add at the top:

use Grammar::Tracer;

And that’s it; now whenever you use the grammar, it will be traced. Note that this statement has lexical effect, so if you’re using modules that also happen to have grammars – which you likely don’t care about – they will not end up getting the tracing behavior.

You can also do this:

use Grammar::Debugger;

The debugger is the tracer’s big sister, and knows a few more tricks. Here’s an example of it in action.

Instead of getting the full trace, now as soon as we hit the TOP production rule the program execution breaks and we get a prompt. Pressing enter allows you to step rule by rule through the parse. For some people, this may be preferable; others prefer to get the full trace output and analyze it. However, there are a few more tricks. In the example above, I added a breakpoint on the “name” rule. Using “r” informs the debugger to keep running through the production rules until it hits one called “name”, at which point it breaks. It is also possible to add breakpoints in code, for more extended debugging sessions with many runs. There’s one additional feature in code, which is to set a conditional breakpoint.

Sound interesting? You can get modules from GitHub, and if you want to see a live demo of a grammar being debugged using it, then there is a video of my Debugging Perl 6 Grammars talk from YAPC::Europe 2011; slides are also available to make the sample code more clear than it is on the video. Note that the modules need one of the compiler releases from the Rakudo “nom” development branch; we’ll be making a distribution release later this month based on that, though, and these modules will come with it.

You may also be thinking: I bet these are complex modules doing lots of guts stuff! In fact, they are 44 lines (Grammar::Tracer) and 171 lines (Grammar::Debugger), and written in Perl 6. They are built using the meta-programming support we’ve been working on in the Rakudo Perl 6 compiler during the course of the last year – and if you want to know more about that, be sure to check out my meta-programming post coming up later on in this year’s advent calendar.

Carl Masak: Macros progress report: a bit of D1

Thu, 01 Dec 2011 22:17:28 +0000

I've not been making much noise about it, but work on macros is progressing nicely. D1 is about providing a macro declarator to parallel the sub declarator, and a quasi construct which creates ASTs... all of that works already, in a branch of Rakudo. Try it out! Be the first one on your block to run macros in Perl 6!

November has been a busy month for me, $dayjob-wise. I knew that was going to happen, even though it ended up being even more busy than I had imagined. Now I'm taking a well-deserved two-week vacation, and then I'll be back and actually have some time for Perl 6 hacking. Looking forward to that. 哈哈

A bit of the part of D1 that doesn't work yet: it turns out that there are "interesting" things happening with lexical lookup and quasiquotes. It's actually nothing very complicated, but it requires some extra wiring. So it's actually possible to cause Null PMC accesses right now because variable lookups from inside of the quasiquote end up confused.

I know how to solve this, in theory. ASTs have to start carrying around their own lexical environment. But I haven't had time to actually sit down and type out the solution. Will write more when I've done that. Till then, feel free to play around with the parts of macros that don't do too wild lexical lookups.

Beyond that, I'd like to give D1 a bit of test coverage in roast, and then I think we can merge the D1 work into nom. Looking forward to that.

Perl 6 Advent Calendar: Day 1: Catching Up With Perl 6

Thu, 01 Dec 2011 14:40:26 +0000

When we started the Perl 6 Advent Calendar back in 2009, Rakudo was really the only game in town if you wanted to play with Perl 6. But Perl 6 was intended from the start to be a language with multiple implementations, and at the moment there are four different Perl 6 implementations of interest. Because there are so many implementations, I’m not going to give instructions for getting each; instead I’m linking to those instructions.

The most stable and complete implementation is Rakudo Star. This is currently based on the last major revision of Rakudo. It’s been frozen since July, and so lags a bit behind the current Perl 6 spec. It’s slow. But it’s also pretty reliable.

The current Rakudo development version is called “Nom”. It’s full of great improvements over the last Rakudo Star release, notably native types, improved performance, and a much better metamodel. (For example, check out the Grammar::Tracer module, which takes advantage of the new metamodel to add regex tracing in just 44 lines of code.) It’s not quite ready for prime time yet, as it still misses some features that work in Rakudo Star, but progress has been incredible, and it’s quite possible a new Rakudo Star based on Nom will be released during this month.

Stefan O’Rear’s Niecza was just a fledging compiler during last year’s Advent calendar, but it’s a serious contender these days. Built to run on the CLR (.NET and Mono), it is relatively zippy, implements a significant portion of Perl 6, and works easily with existing CLR libraries.

Lastly, ingy and Mäsak have plans afoot to revive Pugs, the original Perl 6 implementation in Haskell. So far they’ve just got it building again on current Haskell compilers, but the long-term goal is to get it running on the spec tests again and bring it closer to the current spec.

Which implementation should you use? If you’re looking for a stable, fairly complete Perl 6, Rakudo Star is it. If you just want to explore the language, try Rakudo Nom — you will probably run into bugs, but it’s significantly more advanced than Rakudo Star, and exposing the bugs is a big help to Rakudo’s development. If you have an idea which would benefit from being able to use CLR libraries, Niecza is fantastic. There’s a handy comparison chart of the different features available.

Personally, I have all three of these installed on my machine, and have different projects underway on each of them.

Finally, please don’t hesitate to ask for help, either in the comments here or on the #perl6 IRC channel on Freenode. The Perl 6 community is very friendly.

Perl 6 Advent Calendar: Perl 6 Advent Calendar 2011

Thu, 01 Dec 2011 00:19:32 +0000

For the third year in a row, we are going to post something about Perl 6 every day until Christmas. This post will serve as a table of contents for the entire month.

Day 1: Catching Up With Perl 6
Day 2: Grammar::Tracer and Grammar::Debugger
Day 3: Buffers and Binary IO
Day 4: Traits — Meta Data with Character
Day 5: The Flip-Flop operator
Day 6: Tetris on Niecza
Day 7: Adventures in writing a simple grammar profiler
Day 8: Lexicality and Optimizability
Day 9: Contributing to Perl 6

perl6.announce: Announce: Niecza Perl 6 v12 by Stefan O'Rear

Mon, 28 Nov 2011 20:56:52 +0000

Announce: Niecza Perl 6 v12

This is the twelth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. I've not had very much time for Niecza
this month. This release marks one year since the first public release,
and about a year and a half since the project began. It's hard to
believe I've been at it this long.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Minor features]

:BASE<DIGITS> literals are now supported. (Solomon Foster)

[Bug fixes]

sqrt(-1) is NaN, complex math is strictly opt-in (#64).

Undefined values can be passed correctly to CLR interop (#66).

-I is now mentioned by --help (#52).

Constants, including 'our constant's, are now truly immutable (#46).

$@ and $^X now properly handled as the last characters in the file (#63).

Left side of xx is now autothunkified so [] xx 4 works correctly (#70).

[Other]

The internals documentation has been largely rewritten to match the
refactor in v11.

Added a copy of Martin Berends' London Perl Workshop slides.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

My current priorities are:
1. Make regexes much more feature-complete, including general Unicode
properties and grapheme mode
2. Prototype the debugger
3. 6model convergence work, including roles/native types
4. Figure out how modules and S11 stuff should work in Niecza. Do it.

[1] https://github.com/downloads/sorear/niecza/niecza-12.zip
[2] https://github.com/sorear/niecza

Jonathan Worthington (6guts): Rakudo: this week’s release, and the next Rakudo Star

Sun, 20 Nov 2011 19:10:18 +0000

On Thursday, tadzik++ cut release #46 of Rakudo. This time, we named it for the London Perl Mongers, organizers of this year’s outstanding and very well attended London Perl Workshop. I’d not been to one for a couple of years, and I’d forgotten what a fun event it is. This one felt better than ever. So, thanks folks! :-)

So, what was in the Thursday release?

Big Integer Support: the Int and Rat type now support very large values without loss of precision/coercing to floating point. This means you can do such things as compute factorial to 1000 (which stringifies to 2568 characters, by the way :-)). You don’t need to get any libraries set up for this – we bundle a slightly extended libtommath with NQP, and it’s exposed by some ops and an extra 6model representation, which the Int type inlines into its body (so Int is still a single garbage collectable object at the VM level). Thanks go to moritz++ for doing much of the work on this.
Protoregexes with LTM: it’s taken us a lot longer to get protoregexes back into the nom development branch than we’d initially hoped. Here’s the story. A while ago, pmichaud++ worked on real Longest Token Matching support – driven by an NFA – in a re-development of the regex engine. We’d only ever had “cheating” protoregexes in Rakudo before – ones that could work on literal prefixes, but nothing more. The improved engine is a really nice piece of work – amazingly little code for what it does, and very elegant. Sadly, pmichaud has not been able to work so much on Rakudo of late, so the work lay not-quite-done and unmerged for a while. Just ahead of the release, I picked it up, and found it was far enough along and sufficiently easy to get in to that I could get a first cut integrated in time for this month’s release. Since the release, along with diakopter++, I’ve been hacking away at regexy bits, so there’ll be many improvements in this area in next month’s release. It was nice to get something in place for this month’s, though.
CATCH improvements: mls++ did a bunch of work that made our handling of CATCH blocks far, far more in line with the Perl 6 specification. Naturally, this makes them much more useful; you now can write when blocks in your CATCH, and any exceptions you don’t match will get re-thrown to the next handler. Write a default block to swallow everything. We also do the stack unwinding at the correct point now. Great work!
Improved MAIN argument parsing: japhb++ has been leading the way on improving our support for the MAIN subroutine, including nice usage message generation. This is a really nice Perl 6 feature for writing command line applications, so it’s good to see work here. :-)
6model REPR API 2: this is very much a behind the scenes thing, and deserves a post of its own for those who follow the blog in order to know more about metamodel design. However, in short: I re-worked the REPR API in 6model somewhat, and updated the various representations to use it. The immediate result was that we could efficiently have Perl 6′s Int type do big integer stuff. However, it’s also the foundation for some forthcoming work on compact structs and packed arrays. Hopefully at a user level, though, you’ll notice nothing at all about this change; it’s all guts. :-)
Various other fixes and improvements: bugs fixed, performance improved, missing stuff implemented…we had quite a few smaller, but very worthwhile things done in this release too.

So, this is great, but the next question on everybody’s mind is: when is a nom-based Rakudo Star release coming out? The answer – epic disasters aside – is “December”. Yes, next month. Two big things have happened in the last week: grammar support in the nom development branch drastically improved, and this unblocked tadzik++ to get our ecosystem tools (for module build and installation) working with it. The focus from here is…

Native call support: get our support for loading and calling into C libraries re-established. The NativeCall module basically worked…apart from it made a load of guts-y assumptions that don’t hold true any more, and was somewhat restricted in what it could handle (for multiple reasons). Now we have 6model and native type support, we can do far better. My task for over the next week is to get our native call support back in shape, all being well with a bunch more features.
More regex work: while we’ve come a really long way in recent days – we have many things working that never worked in the previous development branch – there are still some issues to address, including one nasty backtracking issue.
Fixing Modules: quite a few modules already work just fine on nom. Some block on native call support, others on the remaining regex stuff. Others will point to bugs or will be using outdated semantics or relying on previous bugs. tadzik++ has figured out how to automatically run through the module ecosystem, build modules and run their tests, so we can get a good sense of what needs doing.

And, of course, the usual round of feature addition, bug fixing and performance work.

And, with that, it’s back to the hacking. :-)

Solomon Foster: colomon

Fri, 11 Nov 2011 17:43:40 +0000

chromatic has come up with a lovely new metaphor to replace the idea of technical debt: “technical insurance”. I love this sentence: “Given the premiums for your team, you then have to decide whether or not to buy a technical insurance policy.”

It hits at something that has bugged me about testing enthusiasts — the notion that you should always have tests. Because there is a balance that needs to be found — the cost of the tests versus the cost of not having the tests.

I’ve talking about this with respect to the ABC module before. Properly testing the abc2ly script would require constructing an OCR system for reading sheet music. That would be very expensive technical insurance indeed.

And what would it be protecting against? abc2ly is a open source tool for creating sheet music. It makes no money for me. As far as I know, I am the only active user. The only real risk I can see is if I am printing a long musical document, and a change I’ve made breaks something subtle that only affects, say, one thing on page 14, so I don’t notice the problem in a timely fashion. So basically, there’s a potential for wasting some paper.

In other words, in this case technical insurance would be expensive, and have very limited benefits. I’d be a fool to “buy” it.

On the other hand, the Perl 6 spectests are a fantastic technical insurance policy. For the most part they are low cost (easy to write) AND they are shared across multiple implementations of Perl 6. On the flip side, they help protect against subtle flaws in the implementations, which have the potential of causing problems for every person using Perl 6. We’d have to be crazy to do without this policy.

Jonathan Worthington (6guts): Slides from my Optimizing Rakudo Perl 6 talk

Wed, 09 Nov 2011 22:41:22 +0000

Over the weekend, I visited to Bratislava for a few days, the beautiful city I once called home. It felt oddly familiar, and I found myself noticing all kinds of little changes here and there – where one shop had given way to another, or a statue had appeared or changed. Happily, my favorite eating and watering holes were still there, and my sadly somewhat rusted Slovak language skills were still up to decoding menus and ordering tasty beer, and I did plenty of both. :-)

I was there was to attend the Twin City Perl Workshop. I repeated my Perl 6 grammars talk, and gave a new one about optimizing Rakudo. This included both the optimization work myself and others have been doing, but also some details about the optimizer. I also made a couple of nice diagrams of Rakudo’s overall architecture and what it does with a program.

You can get the slides here, or if you’re heading to the London Perl Workshop this coming Saturday, I’ll be delivering it there too. Enjoy! :-)

Solomon Foster: Fixing Tags

Wed, 09 Nov 2011 03:12:44 +0000

So, in my last post I identified 3906 files in my MP3 collection with missing tags. This time I set out to fix some of them.

So, first I went through the list I generated with the last script and singled out all 2294 files which used a standard pattern of Artist / Album / Track Number - Track Name. Then I wrote this script:

my $for-real = Bool::True;

constant $TAGLIB  = "taglib-sharp,  Version=2.0.4.0, Culture=neutral, PublicKeyToken=db62eba44689b5b0";
constant TagLib-File    = CLR::("TagLib.File,$TAGLIB");
constant String-Array   = CLR::("System.String[]");

for lines() -> $filename {
    my @path-parts = $filename.split('/').map(&Scrub);
    my $number-and-title = @path-parts.pop;
    next unless $number-and-title ~~ m/(\d+) \- (.*) .mp3/;
    my $track-number = ~$0;
    my $title = ~$1;
    my $album = @path-parts.pop;
    my $artist = @path-parts.pop;
    say "$artist: $album: $title (track $track-number)";

    if $for-real {
        my $file;
        try {
            $file = TagLib-File.Create($filename);
            CATCH { say "Error reading $filename" }
        }

        $file.Tag.Track = $track-number.Int;
        $file.Tag.Album = $album;
        $file.Tag.Title = $title;
        $file.Tag.Performers = MakeStringArray($artist);
        
        try {
            $file.Save;
            CATCH { say "Error saving changes to $filename" }
        }
    }
}

sub Scrub($a) {
    $a.subst('_', ' ', :global);
}

sub MakeStringArray(Str $a) {
    my $sa = String-Array.new(1);
    $sa.Set(0, $a);
    $sa;
}

For the main loop, the first half uses standard Perl techniques to extract the artist, album, and track info from the path. The second half sets the tags. Opening the file is the same as last time, and then setting Track, Album, and Title is as simple as could be. The Performers tag is a bit tricky, because it’s a string array (the others are simple strings or integers) and Niecza doesn’t know how to do the coercion automatically. MakeStringArray gets the job done nicely.

So, if you’ve done this sort of thing in Perl 5 using the MP3 CPAN modules, there’s nothing at all revolutionary about this code. But it feels really good to be able to do it with Perl 6!

Carl Masak: Macro grant accepted

Tue, 08 Nov 2011 10:20:23 +0000

A while ago, I applied for a Hague grant to give Rakudo macros. It has now been accepted. My reaction:

Yay! \o/

I haven't blogged much lately, and I suspect it might be because my blogging plans are too ambitious to fit in my fragmented schedule. So I'll stop here. But expect the first status report of the grant soonish.

Solomon Foster: Examining MP3 Tags

Sun, 06 Nov 2011 08:54:29 +0000

I’ve been playing around with Niecza’s ability to handle CLR libraries. It’s actually kind of intoxicating; it’s the closest thing yet to having a CPAN for Perl 6. So I decided to see what I could do with the TagLib# library for dealing with media file tags.

Now, I’ve got an MP3 library with 23564 MP3 files in it, the majority of which were created by older ripping files that didn’t do anything with the ID tags. Most of those have been updated to include tags, but every now and then I add one of the old directories to iTunes and get a bunch of “Unknown Artist” / “Unknown Album” tracks.

So I thought a nice first project would be figuring out which of the tracks was correct. The first thing to do was to get TagLib# properly installed on my MacBook Pro. make install didn’t add the DLL to the main GAC; I ended up installing it there by hand, which was trivially easy once I knew what to do:

sudo gacutil -i taglib-sharp.dll
sudo gacutil -i policy.2.0.taglib-sharp.dll

Once I had that done, I experimented with it for a bit, and ended up with this script:

constant $TAGLIB  = "taglib-sharp,  Version=2.0.4.0, Culture=neutral, PublicKeyToken=db62eba44689b5b0";
constant TagLib-File    = CLR::("TagLib.File,$TAGLIB");

for lines() -> $filename {
    try {
        my $file = TagLib-File.Create($filename);
        unless $file.Tag.JoinedPerformers ~~ m/\S/ && $file.Tag.Title ~~ m/\S/ {
            say $filename;
        }
        CATCH { say "Error reading $filename" }
    }
}

The first line specifies the exact assembly we want to use; you can get the details from gacutil -l. The next line effectively imports the TagLib::File class into Niecza. I get my filenames from stdin, as that allows me to use find to generate the list of MP3 files.

This was my first use of exception handling in Perl 6. I needed it because TagLib-File.Create throws an exception when it isn’t happy with the MP3 file. When it is happy with it, $file is an object of type CLR::TagLib::Mpeg::AudioFile. $file.Tag.JoinedPerformers gives the list of performers (AKA artists) as a single string; $file.Tag.Title gives the title as a string. Unless we find a valid non-space character in both of them, we flag the file by printing it out.

Really, the only way it could be significantly simpler than this would be if the constant TagLib-File line were unnecessary!

End result: I have a list of 3906 files it flagged, 77 of which were read errors.

My next step is to write some code which translates the filenames (which are mostly of the form /Volumes/colomon/Albums/Dervish/Live_in_Palma/04-Slow_Reels.mp3) into artist, album, and track name fields, and then set those tags. Based on my initial experiments, I think it’s is going to be incredibly easy…

perl6.announce: Announce: Niecza Perl 6 v11 by Stefan O'Rear

Tue, 01 Nov 2011 12:04:38 +0000

Announce: Niecza Perl 6 v11

This is the eleventh release of Niecza Perl 6, as usual scheduled on
the last Monday of the month, at least in the US west coast time zone.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Major features / Breaking changes]

The compile time / runtime metamodel unification mentioned in the v10
announce is done now. Real Sub and ClassHOW objects are created at
compile time and stored in a .ser (serialized data) file alongside the
.dll. When using modules the .ser file is loaded; .nam files are gone.
Some operations are faster and others are slower.

All non-dotnet backends no longer work and have been removed, since
niecza now requires much closer integration between the front and back
ends. Restoring them would not be impossible.

BEGIN time code execution is now supported! In particular, a BEGIN in
a module is run *once*; any objects it creates will be serialized, and
the BEGIN is not re-run at runtime. It is erroneous to modify objects
owned by a different module at BEGIN time.

[Minor new features]

The x*+ syntax is now supported as a synonym of x**, doing possessive
quantification like Perl 5.

"make spectest" now respects TEST_JOBS. (Will Coleda)

todo is now supported in tests.

Various improvemends to the p5 interop system. (Paweł Murias)

CLR interop example improvements: make clock resizable, add notepad,
tetris, webbrowser. (Martin Berends)

Fleshed out TextWriter and open to some semblance of usability.
Added close, unlink, mkdir. (Solomon Foster)

Added .pick and .roll. (Solomon Foster)

Added log, log10, exp, cis, polar, gcd, lcm. (Solomon Foster)

Handling of variable types that are constrained to Mu but default to
Any are more consistant.

[Selected bug fixes]

grep /regex/, @list no longer crashes.

" (a source file with an unclosed string) no longer crashes the compiler.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

In the wake of the /serialize branch merge there will be a lot of
bugs to fix and documents to update. I am also looking at designing
a good practical realization of S11 and module packaging.

[1] https://github.com/downloads/sorear/niecza/niecza-11.zip
[2] https://github.com/sorear/niecza

Tadeusz Sośnierz (tadzik): MuEvent: AnyEvent lookalike for Perl 6

Mon, 24 Oct 2011 21:06:12 +0000

Trying to struggle with Select in Parrot, I accidentally discovered that its Socket has a .poll method. What a trivial, yet satisfying way to have some simple non-blocking IO. Thus, MuEvent was born.

Why MuEvent? Well, in Perl 6, Mu can do much less than Any. MuEvent, as expected, can do much less than AnyEvent, but it’s trying to keep the interface similar.

You’re welcome to read the code, and criticise it all the way. Keep in mind that I can no idea how should I properly write an event loop, so bonus points if you tell me what could have been done better. I don’t expect MuEvent to be an ultimate solution for event-driven programming in Perl 6, but I hope it will encourage people to play around. Have an appropriate amount of fun!

perl6.announce: Parrot 3.9.0 "Archaeopteryx" Released by Jonathan "Duke" Leto

Tue, 18 Oct 2011 20:19:18 +0000

On behalf of the Parrot team, I'm proud to announce Parrot 3.9.0
"Archaeopteryx".
Parrot (http://parrot.org/) is a virtual machine aimed at running all
dynamic languages.

Parrot 3.9.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/supported/3.9.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 3.9.0 News:
- Core
+ The whiteknight/kill_threads branch was merged, which
removes the old and broken
thread/concurrency implementation. Better and more flexible
concurrency primitives
are currently being worked on. This also involved removing
some of the last vestiges
of assembly code from Parrot as well as removing the share and
share_ro vtables.
+ random_lib.pir was removed, since better alternatives already exist
+ The freeze and thaw vtables were removed from Default PMC,
because they weren't
useful and caused hard-to-find bugs.
+ A new subroutine profiling runcore was added. It can be
enabled with the command-line
argument of -R subprof . The resulting data can be analyzed
with kcachegrind.
+ Added get_string VTABLE to FixedIntegerArray and FixedFloatArray PMCs
+ The update() method was added to the Hash PMC, which updates
one Hash with the contents
of another. This speeds up rakudo/nqp startup time.
- Languages
+ Winxed
- Updated snapshot to version 1.3.0
- Added the builtin sleep
- Modifier 'multi' allows some more multi functionality
- Community
+ New repo for the Parrot Alternate Compiler Toolkit, a
re-implementation of
PCT in Winxed: https://github.com/parrot/PACT
- Documentation
+ We are in the process to migrating our Trac wiki at
http://trac.parrot.org/ to Github
at https://github.com/parrot/parrot/wiki
+ Packfile PMC documentation was updated
- Tests
+ Select PMC tests improved to pass on non-Linuxy platforms

The SHA256 message digests for the downloadable tarballs are:

923b5ef403c26dd94c04127940659aea94516f79243a80de65fbababff44bfad
parrot-3.9.0.tar.bz2
568bfffad0bc7595164f342cd39c33ac967286423844491e85a8f9767f15871c
parrot-3.9.0.tar.gz

Many thanks to all our contributors for making this possible. This
release comprises
182 commits by 17 authors on the master branch since the previous release:

Michael Schroeder, Whiteknight, soh_cah_toa, Jonathan "Duke" Leto,
Brian Gernhardt, Andy Lester, Christoph Otto, Peter Lobsinger,
jkeenan, NotFound,
Jimmy Zhuo, Stefan Seifert, Andrew Whitworth, Francois Perrad, Moritz Lenz,
Tadeusz Sośnierz, gerd

Our next scheduled release is 15 November 2011.

Enjoy and may the force be with you!

Duke

--
Jonathan "Duke" Leto <jonathan@leto.net>
Leto Labs LLC
209.691.DUKE // http://labs.leto.net
NOTE: Personal email is only checked twice a day at 10am/2pm PST,
please call/text for time-sensitive matters.

Solomon Foster: Ease of FatRat construction

Tue, 18 Oct 2011 17:18:18 +0000

So, on #perl6 today tried using the numeric literal .3333333333333333333333333333333. (Warning: exact number of 3‘s may not match original example.) By the spec (as I understand it), this is a Num, because a Rat isn’t accurate enough to represent it. (Not that a Num is, mind you!)

And that got me to thinking: What if you really wanted a FatRat, so you actually got that exact number? Well, if you’re using Niecza (the only p6 to implement FatRat so far), the answer is FatRat.new(3333333333333333333333333333333, 10000000000000000000000000000000). IMO, that’s ridiculously awkward.

The spec may imply you can do it with ".3333333333333333333333333333333".FatRat. That at least avoids the problem of counting the zeros, but it’s still on the ugly side. Likewise FatRat.new(".3333333333333333333333333333333") is awkward. Still, we should certainly support at least one of these options.

I would like to propose again adding an F suffix to indicate a numeric literal should be a FatRat. I don’t think this is something that can reasonably be done with a postfix operator, because if you treat .3333333333333333333333333333333 like a normal numeric value and then try to FatRat it, you will lose the precision you want.

Just as a quick comparison, here’s a bit of the old endless pi code using the FatRat constructor:

sub unit() { LFT.new(q => FatRat.new(1, 1),
                     r => FatRat.new(0, 1),
                     s => FatRat.new(0, 1),
                     t => FatRat.new(1, 1)); }

I’m proposing we should be able to write that as

sub unit() { LFT.new(q => 1F,
                     r => 0F,
                     s => 0F,
                     t => 1F); }

Much shorter and much clearer. I think that’s a big win.

(Note: I’m in no way particularly attached to the letter “F” for this, that was just the first thing that came to mind.)

Jonathan Worthington (6guts): An optimizer lands, bringing native operators

Sat, 15 Oct 2011 15:56:43 +0000

For some weeks now, I’ve been working on adding an optimizer pass to Rakudo and implementing an initial set of optimizations. The work has been taking place in a branch, which I’m happy to have just merged into our main development branch. This means that the optimizer will be included in the October release! :-) In this post, I want to talk a little about what the optimizer can do so far.

When Optimization Happens

When you feed a Perl 6 program to Rakudo, it munches its way through your code, simultaneously parsing it and building an AST for the executable bits, and a bunch of objects that represent the declarative bits. These are in various kinds of relationship; a code object knows about the bit of as-yet uncompiled AST that corresponds to its body (which it needs to go and compile just in time should it get called at BEGIN time), and the AST has references to declarative objects (types, subs, constants). Normally, the next step is to turn this AST into intermediate code for the target VM (so for Parrot, that’s PIR). The optimizer nudges its way in between the two: it gets to see the fully constructed AST for the compilation unit, as well as all of the declarative objects. It can twiddle with either before we go ahead and finish the compilation process. This means that the optimizer gets to consider anything that took place at BEGIN and CHECK time also.

Using The Optimizer

The optimizer has three levels. The default level is 2. This is “optimizations we’re pretty comfortable with having on by default”. It’s possible to pass –optimize=3, in which case we’ll throw everything we’ve got at your program. If it breaks as a result, please tell us by filing an RT ticket; this is the pool of candidate optimizations to make it into group 2. After an optimization has had a while at level 2, combined with a happy and trouble-free history, we’ll promote it into level 1. Using –optimize=1 at the moment gets you pretty much nothing – the analysis but no transformations. In the long run, it should get you just the optimizations we feel are really safe, so you won’t lose everything if you need to switch down from –optimize=2 for some reason. Our goal is that you should never have to do that, of course. However, it’s good to provide options. My thanks go to pmichaud++ for suggesting this scheme.

Compile Time Type Checking of Sub Arguments

One thing the optimizer can do is consider the arguments that will be passed to a subroutine. If it has sufficient type information about those arguments, it may be able to determine that the call will always be successful. In this case, it can flag to the binder that it need never do the type checks at run time. This one can actually help untyped programs too. Since the default argument type is Any, if you pass a parameter of one subroutine as an argument to another, it can know that this would never be a junction, so it never has to do the junction fail-over checks.

Compile Time Multiple Dispatch Resolution

While the multiple dispatch cache current Rakudo has is by some margin the best it has ever had in terms of lookup performance, it still implies work at run time. Given enough information about the types of the arguments is present, the optimizer is able to resolve some multiple dispatches at compile time, by working out cases where the dispatch must always lead to a certain candidate getting invoked. Of course, how well it can do this depends on the type information it has to hand and the nature of the candidates. This is a double saving: we don’t have to do the multiple dispatch, and we don’t have to do the type checks in the binding of the chosen candidate either.

Basic Inlining

In some (currently very constrained) cases, if we know what code is going to be called at compile time, and we know that the types of arguments being passed are all OK, we can avoid making the call altogether and just inline the body of the subroutine right into the caller. Of course, this is only beneficial in the case where the work the subroutine does is dominated by the overhead of calling it, and there are some cases where inlining is impossible to do without causing semantic differences. For now, the focus has been on doing enough to be able to inline various of the setting built-ins, but it’s in no way restricted to just doing that. With time, the inline analysis will be made much smarter and more capable.

Native Operators

As part of getting the optimizer in place, moritz++ and I have also worked on native operators (that is, operators that operate on native types). This boils down to extra multiple dispatch candidates for various operators, in order to handle the natively typed case. However, something really nice happens here: because you always have to explicitly declare when you are using native types, we always have enough type information to inline them. Put another way, the native operator multis we’ve declared in the setting will always be inlined.

We’ve some way to go on this yet. However, this does already mean that there are some nice performance wins to be had by using native types in your program (int and num) where it makes sense to do so.

As an example, with –optimize=3 (the maximum optimization level, not the default one), we can compare:

my $i = 0; while $i < 10000000 { $i = $i + 1 }; say $i

Against:

my int $i = 0; while $i < 10000000 { $i = $i + 1 }; say $i

On my box, the latter typed version completes in 4.17 seconds, as opposed to the untyped version, which crawls in at 33.13 (so, a factor of 8 performance gain). If you’re curious how this leaves us stacking up against Perl 5, on my box it does:

my $i = 0; while ($i < 10000000) { $i = $i + 1 }; say $i

In 0.746 seconds. This means that, with type information provided and for this one benchmark, Rakudo can get within a factor of six of Perl 5 – and the optimizer still has some way to go yet on this benchmark. (Do not read any more into this. This performance factor is certainly not generally true of Rakudo at the moment.)

We’ll be continuing to work on native operators in the weeks and months ahead.

Immediate Block Inlining

We’ve had this in NQP for a while, but now Rakudo has it too. Where appropriate, we can now flatten simple immediate blocks (such as the bodies of while loops) into the containing block. This happens when they don’t require a new lexical scope (that is, when they don’t declare any lexicals).

That Could Never Work!

There’s another nice fallout of the analysis that the optimizer does: as well as proving dispatches that will always work out at compile time, it can also identify some that could never possibly work. The simplest case is calling an undeclared routine, something that STD has detected for a while. However, Rakudo goes a bit further. For example, suppose you have this program:

sub foo($x) { say $x }
foo()

This will now fail at compile time:

CHECK FAILED:
Calling 'foo' will never work with no arguments (line 2)
    Expected: :(Any $x)

It can also catch some simple cases of type errors. For example:

sub foo(Str $s) { say $s }
foo(42)

Will also fail at compile time:

CHECK FAILED:
Calling 'foo' will never work with argument types (int) (line 2)
    Expected: :(Str $s)

It can handle some basic cases of this with multiple dispatch too.

Propagating Type Information

If we know what routine we’re calling at compile time, we can take the declared return type of it and use it in further analysis. To give an example of how this aids failure analysis, consider the program:

sub foo() returns Int { 42 }
sub bar(Str $s) { say $s }
bar(foo())

This inevitable failure is detected at compile time now:

CHECK FAILED:
Calling 'bar' will never work with argument types (Int) (line 3)
    Expected: :(Str $s)

The real purpose of this is for inlining and compile time multi-dispatch resolution though; otherwise, we could never fully inline complex expressions like $x + $y * $z.

Optimizing The Setting

Since we have loads of tests for the core setting (many of the spectests cover it), we compile it with –optimize=3. This means that a bunch of the built-ins will now perform better. We’ll doubtless be taking advantage of native types and other optimizations to further improve the built-ins.

Gradual Typing

Many of these optimizations are a consequence of Perl 6 being a gradually typed language. You don’t have to use types, but when you do, we make use of them to generate better code and catch more errors for you at compile time. After quite a while just talking about these possible wins, it’s nice to actually have some of them implemented. :-)

The Future

Of course, this is just the start of the work – over the coming weeks and months, we should gain plenty of other optimizations. Some will focus on type-driven optimizations, others will not depend on this. And we’ll probably catch more of thsoe inevitable run time failures at compile time too. In the meantime, enjoy what we have so far. :-)

Carl Masak: Macros — what are they, really?

Sat, 15 Oct 2011 14:13:00 +0000

Apparently, if you schedule all of my talks at YAPC::EU 2011 on the first day, I will spend the remaining time of the conference thinking intently about how macros work. (I did some socializing too, don't worry. I even distinctly remember talking to people about other things than macros on at least one occasion.)

Like most of the rest of you, I'd heard about C preprocessor macros (and how they're both useful and kinda dangerous if you don't know what you're doing), and Lisp macros (and how they're part of what makes Lisp the awesomest programming language in the universe forever). Which one of these types does Perl 6 specify?

Both, duh. 哈哈

But I'm going to talk about the latter kind. I'll call them "AST macros", to differentiate them from "textual macros". ("AST" simply means "Abstract Syntax Tree". Forget the "abstract" part, it's just been put there to scare you into thinking this is tricky.)

Why ASTs matter

When the complexity of a codebase increases, it inevitably becomes a part of the problem it is trying to solve. We need to combat the complexity in the code itself, and we need to start talking about the code in the code. There are three broad ways we can describe code:

Code as strings. Humans can read it, but the computer can't, because it's just strings.
Code as ASTs. It has a lot of structure, which both humans and computers can use.
Code as executable code objects. It's opaque to humans, but the computer run it, because it's just code.

These three forms — string, AST, code block — reflect what a compiler does when it prepares your source code for execution:

Code starts out as text, as source code.
Compiler parses the text and builds an AST. (And some other information, such as a symbol table.)
Compiler generates the AST into executable code.

The reason the compiler takes the detour through ASTs when creating your code is that trees are much easier to reason about and manipulate than the "flat" representations of code. An AST contains a lot of explicit relations that don't stand out in the original or final, "flat" representations of code. ASTs can be manipulated, stitched together, optimized, etc. It's this strength that AST macros make use of.

Since ASTs are the way code looks before code generation, AST macros give you a say in what code will be generated in your program.

Macros are a way to transform code. AST macros transform code by giving you the tools to build your own AST.

How to make an AST

How to construct an AST in Perl 6? Using the quasi keyword:

quasi { say "OH HAI" }

What this evaluates to is a Perl6::AST object holding a tree structure representing the program code say "OH HAI". Exactly how that tree structure looks may or may not be implementation-dependent.

quasi stands for "quasi-quote", a concept invented by Quine, the logician, who liked to think about self-reference, paradox, and words starting with the letter Q. Just as we quote code with a string literal and the result is a Str, so we can quote code with the quasi keyword and the result, in the case of Perl 6, is a Perl6::AST object.

What macros are

Macros work just like subroutines, but AST macros are expected to return a Perl6::AST. How the AST is created is the macro author's business. But we can use quasi to create them:

macro LOG {
    quasi {
        $*ERR.say(DateTime.now, ": some logging information here");
    }
}

# Meanwhile, later in the code:
LOG();

You see, it looks just like a subroutine call. But the call is made by the compiler, not by the runtime as with ordinary subroutines. And the return value is a Perl6::AST object containing the code to print something to $*ERR.

But wait, there's more!

It's a pretty useless LOG macro that doesn't take an argument with a $message. We'll fix that. There's one twist, though: AST macros deal in AST, so the $message that gets passed to the macro won't be a Str. It'll be a Perl6::AST:

macro LOG($message) {
    quasi {
        $*ERR.say(DateTime.now, ": ", {{{$message}}});
    }
}

LOG("Evacuation complete.");

When we call LOG, we do it with a Str, just as with a usual subroutine. The parser sees the string literal and does its thing with turning stuff into ASTs. The compiler then calls the macro with one argument: the resulting Perl6::AST object. In the quasi, we make sure to take this object and stitch it right into the code that says "print a bunch of stuff to $*ERR". It's right there, at the end of that line, enclosed in triple curly braces.

What do the triple curly braces do, exactly? They allow you to say "I want you to incorporate this already-parsed AST into this currently-being-parsed code". Triple curly braces are only recognized inside of quasi-quote blocks. In fact, this is what quasi-quotes specialize in: allowing an escape hatch from code to ASTs, so we can mix them. (This is what Quine used quasi-quoting for too, except in the domain of logic.)

If we didn't write {{{$message}}} there, but just the normal form $message, guess what? The LOG function would stringify the Perl6::AST object, probably to something boring like Perl6::AST()<0x80681e0>, and print that.

Right, so AST macros take ASTs, allow us to manipulate ASTs, and return ASTs. Fine. We get the message. But what makes them so powerful?

The real power comes from the fact that we can steer this process any which way we want. For example, maybe we'd like to turn logging on and off at the switch of a constant:

constant LOGGING_ENABLED = True;

macro LOG($message) {
    if LOGGING_ENABLED {
       quasi {
           $*ERR.say(DateTime.now, ": ", {{{$message}}});
        }
    }
    else {
        quasi {}
    }
}

LOG(crazily-expensive-computation());

Turn LOGGING_ENABLED off, and the crazily-expensive-computation() call will be parsed, but never executed.

This is the essence of AST macros. There's much more to it than that, but we'll get to the other parts in later posts.

Moritz Lenz (Perl 6): The Three-Fold Function of the Smart Match Operator

Wed, 12 Oct 2011 18:37:08 +0000

In Perl 5, if you want to match a regex against a particular string, you write $string =~ $regex.

In the design process of Perl 6, people have realized that you cannot only match against regexes, but lots of other things can act as patterns too: types (checking type conformance), numbers, strings, junctions (composites of values), subroutine signatures and so on. So smart matching was born, and it's now written as $topic ~~ $pattern. Being a general comparison mechanism is the first function of the smart match operator.

But behold, there were problems. One of them was the perceived need for special syntactic forms on the right hand side of the smart match operator to cover some cases. Those were limited and hard to implement. There was also the fact that now we had two different ways to invoke regexes: smart matching, and direct invocation as m/.../, which matches against the topic variable $_. That wasn't really a problem as such, but it was an indicator of design smell.

And that's where the second function of the smart match operator originated: topicalization. Previously, $a ~~ $b mostly turned into a method call, $b.ACCEPTS($a). The new idea was to set the topic variable to $a in a small scope, which allowed many special cases to go away. It also nicely unified with given $topic { when $matcher { ... } }, which was already specified as being a topicalizer.

In the new model, MATCH ~~ PAT becomes something like do { $_ = MATCH; PAT.ACCEPTS($_) } -- which means that if MATCH accesses $_, it automatically does what the user wants.

Awesomeness reigned, and it worked out great.

Until the compiler writers actually started to implement a few more cases of regex matching. The first thing we noticed was that if $str ~~ $regex { ... } behaved quite unexpectedly. What happend was that $_ got set to $str, the match was conducted and returned a Match object. And then called $match.ACCEPTS($str), which failed. A quick hack around that was to modify Match.ACCEPTS to always return the invocant (ie the Match on which it was called), but of course that was only a stop gap solution.

The reason it doesn't work for other, more involved cases of regex invocations is that they don't fit into the "does $a match $b?" schema. Two examples:

# :g for "global", all matches
my @matches = $str ~~ m:g/pattern/; 

if $str ~~ s/pattern/substitution/ { ... }

People expect those to work. But global matching of a regex isn't a simple conformance check, and that is reflected in the return value: a list. So should we special-cases smart-matching against a list, just because we can't get global matching to work in smart-matching otherwise? (People have also proposed to return a kind of aggregate Match object instead of a list; that comes with the problem that Match objects aren't lazy, but lists are. You could "solve" that with a LazyMatch type; watch the pattern of workarounds unfold...)

A substitution is also not a simple matching operation. In Perl 5, a s/// returns the number of successful substitutions. In Perl 6, that wouldn't work with the current setup of the smart match operator, where it would then smart-match the string against the returned number of matches.

So to summarize, the smart match operator has three functions: comparing values to patterns, topicalization, and conducting regex matches.

These three functions are distinct enough to start to interact in weird ways, which limits the flexibility in choice of return values from regex matches and substitutions.

I don't know what the best way forward is. Maybe it is to reintroduce a dedicated operator for regex matching, which seems to be the main feature with which topicalization interacts badly. Maybe there are other good ideas out there. If so, I'd love to hear about them.

perl6.announce: Announce: Niecza Perl 6 v10 by Stefan O'Rear

Tue, 27 Sep 2011 06:19:12 +0000

Announce: Niecza Perl 6 v10

This is the tenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Major features]

CLR interoperation is now fairly well supported! You can create
objects, call methods, get and set fields and properties, create
delegates, etc from Perl 6 code. See examples/ for usage ideas.
(Examples by Martin Berends)

The Mono.Posix dependency has been relaxed from load time to run
time, meaning .NET support is back if you don't use file tests.

[Minor new features]

\qq[] syntax is now implemented.

qp|| now returns a path object.

New Test.pm6 methods succeeds_ok and fails_ok (and eval_ variants) to
catch warnings. (Design by flussence)

@foo? and %foo? in signatures are now correctly supported.

Many more trig functions now implemented. (Solomon Foster)

Standard grammar has been updated, in particular bringing the new
concept of regex separators; x ** y is now spelled x+ % y. Do
not expect other forms of % and %% to work just yet.

[Selected bug fixes]

sqrt now returns the correct value for arguments with a negative
imaginary part. Also sqrt(0) returns Num not Complex now.

[Other]

docs/compiler.pod is more current. (Martin Berends)

Prototyping has begun on Perl 5 interoperation. (Paweł Murias)

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

I have an active branch (started this month) to unify compile-time and
run-time metamodel representations, using serialization to bridge the
gap. It doesn't work yet, but when it does it will enable many
improvements, most importantly real support for BEGIN and roles.

[1] https://github.com/downloads/sorear/niecza/niecza-10.zip
[2] https://github.com/sorear/niecza

perl6.announce: Parrot 3.8.0 "Magrathea" Released by Kevin Polulak

Tue, 20 Sep 2011 21:07:34 +0000

On behalf of the Parrot team, I'm proud to announce Parrot 3.8.0, also known
as "Magrathea". Parrot (http://parrot.org/) is a virtual machine aimed at
running all dynamic languages.

Parrot 3.8.0 is available on Parrot's FTP site (
ftp://ftp.parrot.org/pub/parrot/releases/devel/3.8.0/), or by following the
download instructions at http://parrot.org/download. For those who would
like to develop on Parrot, or help develop Parrot itself, we recommend using
Git to retrieve the source code to get the latest and best Parrot code.

Parrot 3.8.0 News:
- Core
+ New tools/release/auto_release.pl script automates most of release
- Languages
+ Winxed
- Updated snapshot to version 1.2.0
- allowtailcall modifier in try
--debug command-line option, __DEBUG__ predefined constant
and __ASSERT__ builtin
- namespace, class, and ~ (bitwise not) operators
- Implicit nested namespace in namespace and class
declarations
- -X command-line arg
- Documentation
+ Improved release manager guide
- Tests
+ New Makefile target "resubmit_smolder" to resubmit test results
+ New Makefile target "all_hll_test" runs the test suite of all
HLLs and libraries known to work on Parrot
+ New Makefile target "interop_tests" run language
interoperability tests, which runs as part of the normal "make test"
as well

The SHA256 message digests for the downloadable tarballs are:

-

f26d9c1a5d7723b1e778394f87f8bb993e188fb05a719a78eb0204612329cd75

parrot-3.8.0.tar.bz2
-

ae10e52eaf150870949aa51c7588e3a09f8f0588c9e0a7a76c2201672b7c5c7a

parrot-3.8.0.tar.gz

Many thanks to all our contributors for making this possible, and our
sponsors for supporting this project. Our next scheduled release is 18
October 2011.

Enjoy!

Excerpt from *The Hitchhiker's Guide to the Galaxy*, page 634784, section
5a. Entry: *Magrathea*

*Far back in the mists of ancient time, in the great and glorious days of
the former Galactic Empire, life was wild, rich and largely tax free. Mighty
starships plied their way between exotic suns, seeking adventure and reward
among the farthest reaches of Galactic space. In those days spirits were
brave, the stakes were high, men were real men, women were real women and
small furry creatures from Alpha Centauri were real small furry creatures
from Alpha Centauri. And all dared to brave unknown terrors, to do mighty
deeds, to boldly split infinitives that no man had split before – and thus
was the Empire forged.*
*

Many men of course became extremely rich, but this was perfectly natural and
nothing to be ashamed of because no one was really poor – at least no one
worth speaking of. And for all the richest and most successful merchants
life inevitably became rather dull and niggly, and they began to imagine
that this was therefore the fault of the worlds they'd settled on. None of
them was entirely satisfactory: either the climate wasn't quite right in the
later part of the afternoon, or the day was half an hour too long, or the
sea was exactly the wrong shade of pink.

And thus were created the conditions for a staggering new form of specialist
industry; custom-made luxury planet building. The home of this industry was
the planet Magrathea, where hyperspatial engineers sucked matter through
white holes in space to form it into dream planets – gold planets, platinum
planets, soft rubber planets with lots of earthquakes – all lovingly made to
meet the exacting standards that the Galaxy's richest men naturally came to
expect.

But so successful was this venture that Magrathea itself soon became the
richest planet of all time and the rest of the Galaxy was reduced to abject
poverty. And so the system broke down, the Empire collapsed, and a long
sullen silence settled over a billion hungry worlds, disturbed only by the
pen scratchings of scholars as they labored into the night over smug little
treatises on the value of a planned political economy.

Magrathea itself disappeared and its memory soon passed into the obscurity
of legend.
*

*In these enlightened days, of course, no one believes a word of it.*
--
- Kevin Polulak (soh_cah_toa)

Jonathan Worthington (6guts): This is not enough!

Sat, 17 Sep 2011 00:11:51 +0000

The time for some shiny new hardware came around. Sat next to me, purring decidedly more quietly that its predecessor, is my new main development machine: a quad core Intel Core i7, pimped out with 16 GB of RAM and a sufficiently generous SSD that it can hold the OS, compiler toolchain and projects I work most actively on. It’s nice having a $dayjob that likes keeping their hackers…er, consultants…well kitted out. :-)

So, the question I had to ask was: how fast can this thing run the Rakudo spectests? I tried, and with –jobs=8 (the sweet spot, it seems) it chugged its way through them in 220s. That’s vastly better than I’d ever been able to do before, and I could immediately see it was going to be a boon for my Rakudo productivity. 3 minutes 40 seconds. Not so long to wait to know a patch is fine to push. But…what if it was less? It’s fast but…this is not enough!

A while ago, moritz++ showed how the nom branch of Rakudo ran mandelbrot 5 times faster than master. This was a fairly nice indicator. Around the time my new hardware arrived, an update was posted on #perl6: mandelbrot was now down to 2 minutes on the same machine the original tests were done. Again, I was happy to see progress in the right direction but I couldn’t help but feel…this is not enough!

So, I took a few days break from bug fixing and features, and decided to see if things could get faster.

Faster Attribute Access

One of the things I’ve had planned for since the early days of working on 6model is being able to look up attributes by index in the single inheritance case, rather than by name. I finally got around to finishing this up (I’d already put in most of the hooks, just not done the final bits). It’s not an entirely trivial thing to make work; at the point we parse an attribute access we don’t know enough about how the eventual memory layout of the object will be, or whether an indexed lookup will even work. Further, we have to involve the representation in the decision, since we can’t assume all types will use the same one. Mostly, it just involves a later stage of the code generation (PAST => POST in this case) having the type object reachable from the AST and asking it for a slot index, if possible.

Since I implemented it at the code-gen level, it meant the improvement was available to both NQP and Rakudo, so we get compiler and runtime performance improvements from it. Furthermore, I was able to improve various places where the VM interface does attribute lookups (for example, invocation of a code object involves grabbing the underlying VM-level thingy that represents an executable thing, and that “grabbing” is done by an attribute access on the code object). Attribute lookups never really showed up that high in the (C-level) profile, but now they’re way, way down the list.

The P6opaque Diet

P6opaque is by far the most common object representation used in NQP and Rakudo. It’s generally pretty smart; it has a header, and then lays out attributes – including natively typed ones – just like a C structure would be laid out in memory. In fact, it mimics C structures well enough that for a couple of parts of the low-level parts of Rakudo we have C struct definitions that let us pretend that full-blown objects are just plain old C structures. We don’t have to compromise on having first class objects in order to write fast low-level code that works against them any more. Of course, you do commit to a representation – but for a handful of built-in types that’s fine.

So, that’s all rainbows and butterflies, so what was the problem? Back last autumn, I thought I knew how implementing mix-ins and multiple inheritance attribute storage was going to look; it involved some attributes going into a “spill hash” if they were added dynamically, or all of them would go there apart from any in a common SI prefix. Come this spring when I actually did it for real, a slightly smarter me realized I could do much better. It involved a level of indirection – apart from that level already existed, so there was actually no added cost at all. Thing is, I’d already put the spill slot in there, and naughtily used the difference between NULL and PMCNULL as the thing that marked out whether the object was a type object or not.

This week, I shuffled that indicator to be a bit in the PMC object header (Parrot makes several such bits available for us to use for things like that). This meant the spill slot in the P6opaque header could go away. Result: every object using the P6opaque representation got 4 (32-bit) or 8 (64-bit) bytes lighter. This has memory usage benefits, but also some speed ones: we get more in the CPU cache for one, and for another we can pack more objects into fixed sized pools, meaning they have less arenas to manage. Win.

Constant Pain

In Perl 6 we have Str objects. Thanks to 6model’s capability to embed a native Parrot string right into an object, these got about three times cheaper in terms of memory already in nom. Well, hopefully. The thing is, there’s a very painful way to shoot yourself in the foot at the implementation level. 6model differentiates coercion (a high level, language sensitive operation) from unboxing (given this object, give me the native thingy inside of it). Coercion costs somewhat more (a method call or two) than unboxing (mostly just some pointer follows). If you manage to generate code that wants a VM-level string, and it just has an object, it’ll end up doing a coercion (since at that level, it doesn’t know the much cheaper unbox is possible/safe). After reading some of the compiler output, I spotted a bunch of cases where this was happening – worst of all, with constant strings in places we could have just emitted VM-level constant strings! Fixing that, and some other unfortunate cases of coercion instead of unbox, meant I could make the join method a load faster. Mandelbrot uses this method heavily, and it was a surprisingly big win. String concatenation had a variant of this kind of issue, so I fixed that up too.

Optimizing Lexical Lookup

We do a lot of lexical lookups. I’m hopeful that at some point we’ll have an optimizer that can deal with this (the analysis is probably quite tricky for full-blown Perl 6; in NQP it’s much more tractable). In the meantime, it’s nice if they can be faster. After a look over profiler output, I found a way to get a win by caching a low-level hash pointer directly in the lexpad rather than looking it up each time. Profilers. They help. :-)

Optimized MRO Compuation

The easiest optimizations for me to do are…the ones somebody else does. Earlier this week, after looking over the output from a higher level profiler that he’s developing for Parrot, mls++ showed up with a patch that optimized a very common path of C3 MRO computation. Curiously, we were spending quite a bit of time at startup doing that. Of course, once we can serialize stuff fully, we won’t have to do it at all, but this patch will still be a win for compile time, or any time we dynamically construct classes by doing meta-programming. A startup time improvement gets magnified by a factor of 450 times over a spectest run (that’s how many files we have), and it ended up being decidedly noticeable. Again, not where I’d have thought to look…profiling wins again.

Multi-dispatch Cache

We do a lot of multiple dispatch in Perl 6. While I expect an optimizer, with enough type information to hand, will be able to decide a bunch of them at compile time, we’ll always still need to do some at runtime, and they need to be fast. While we’ve cached the sorted candidate list for ages, it still takes a time to walk through it to find the best one. When I was doing the 6model on CLR work, I came up with a design for a multi-dispatch cache that seemed quite reasonable (of note, it does zero heap allocations in order to do a lookup and has decent cache properties). I ported this to C and…it caused loads of test failures. After an hour of frustration, I slept on it, then fixed the issue within 10 minutes the next morning. Guess sleep helps as well as profilers. Naturally, it was a big speed win.

Don’t Do Stuff Twice

Somehow, in the switch over to the nom branch, I’d managed to miss setting the flag that causes us not to do type checks in the binder if the multi-dispatcher already calculated they’d succeed. Since the multi-dispatch cache, when it gets a hit, can tell us that much faster than actually doing the checks, not re-doing them is a fairly notable win.

Results

After all of this, I now have a spectest run in just short of 170 seconds (for running 14267 tests). That’s solidly under the three minute mark, down 50s on earlier on this week. And if it’s that much of a win for me on this hardware, I expect it’s going to amount to an improvement measured in some minutes for some of our other contributors.

And what of mandelbrot? Earlier on today, moritz reported a time of 51 seconds. The best we ever got it to do in the previous generation of Rakudo was 16 minutes 14 seconds, making for a 19 times performance improvement for this benchmark.

This is not enough!

Of course, these are welcome improvements, and will make the upcoming first release of Rakudo from this new “nom” development branch nicer for our users. But it’s just one step on the way. These changes make Rakudo faster – but there’s still plenty to be done yet. And note that this work doesn’t deliver any of the “big ticket” items I mentioned in my previous post, which should also give us some good wins. Plus there’s parsing performance improvements in the pipeline – but I’ll leave those for pmichaud++ to tell you about as they land. :-)

Jonathan Worthington (6guts): What’s coming up in September/October

Mon, 12 Sep 2011 23:41:05 +0000

So, YAPC has come, been great, gone and been recovered from, I’m done with my summer visiting, the $dayjob speaking trip has taken place, I’ve shaken off the obligatory start of autumn cold and it’s time to get back to hacking on stuff. Actually, I’ve no more workshops or other trips until November and I’ve got Perl 6 time marked in my schedule, so all being well there should be plenty of time to Get Stuff Done. :-) So what have I got planned for the next couple of months?

Ship a ”nom”-based Release

This is the current priority. Day by day, we’re fixing up test files we lost in the refactor. This weekend I got most of the missing bits of parametric role support back in place (and I’m overall happy with the resulting factoring; it’s a massive amount better than what we had before). Our biggest remaining holes are in the regex and grammar handling, which pmichaud++ is on with (I’m quite excited about what’s coming here). Other than that, it’s little bits here and there. We’re getting there. :-)

A Basic Rakudo Optimizer

I’ve started playing with this a bit, in a branch. Nothing interesting to see yet, other than an optimizer that only knows one optimization, makes a few things a little faster and regresses a couple of spectests (so, some weird bug in the analysis or transform somewhere). The good news is that it does successfully make it through applying that optimization to CORE.setting. This is an important part of developing the optimizer: if we’re going to write our built-ins in Perl 6, we really want an optimizer to go over them too. My aim is to teach this a couple more things and have it in the October release.

A Basic NQP Optimizer

NQP is the subset of Perl 6 that we write most of the compiler in. We also implement the various built-in meta-objects in it (so we want it to be fast here, and of course we want faster compiles!) It currently has no optimizer, a situation I plan to change. NQP has many restrictions that full-blown Perl 6 does not have, and as a result we’ll be able to do some more aggressive optimizations, or be able to apply them with far simpler analysis. My goal is to have some form of basic optimizer in NQP by the October release. Of course, since NQP is bootstrapped, NQP’s optimizer can be used to optimize NQP itself (yes, it can optimize the optimizer…) “So we just keep running NQP on itself until it runs crazy fast?” Er, no, sorry, it doesn’t work like that. :-)

Bounded Serialization

Currently, as we compile programs, we build up a complete “model” of the runtime environment (for example, we build Signature/Parameter objects, meta-objects to represent classes, and so forth). If we’re going to just run the program, we carry these objects over to runtime and use them. If we’re in compilation mode, like we are with the setting, then we generate a bunch of “deserialization code”. This gets run first of all when we load the setting/module in question. This introduces a couple of problems.

It’s a lot of code to generate. In fact, it accounts for a whopping 60% of the lines of PIR in CORE.setting! This probably means that about 60% of the PAST nodes we generate and then have to process into POST and then PIR are also related to this. Assuming we could replace this process with something much faster, we’re looking at cutting compilation time in half for the setting. Note that this only really affects pre-compilation time, not “run immediately” mode.
We have to run it all at load time. Now, having it all in one huge blob is decidedly more efficient than the thousands of little loadinit subs we had everywhere in previous generations of Rakudo. But we still have to do it, and it means that loading the setting – something we have to do every time we start Rakudo to compile or run code – pays that cost. We need better startup time.
We have to make sure that every single change that gets made gets recorded. We cheat a bit here. This gets in the way of many things, including various optimizations and proper BEGIN time support in pre-compiled modules (note that we don’t re-run BEGIN, but we should maintain effects of it).

The solution is to find a way to efficiently serialize everything we create, and then be able to deserialize it efficiently at load time and do the few needed fixups. While in theory it’s “easy” (keeping iterating over a worklist until you’ve serialized everything, basically), there’s a bunch of really tricky things that also come up (especially any closures that got taken at compile time). I hope to dig into this before the end of the month, and my target is to have it for the November release (October one is a bit ambitious, unless it goes crazily well; in reality, I’d like to land it late October, so we have a couple of weeks before the November release to get it in shape).

Revive the CLR Backend

Running NQP on the CLR got a long, long way. At the time I last touched it, the majority of the non-regex tests in the NQP test suite were passing, and diakopter++ was making progress on the regex ones too. It’s been dormant for a while, but it’s time to get back to work on it. Getting NQP, and then Rakudo, to run on the CLR is now a vastly more tractable task to back then, since:

Back then, I was in the process of designing 6model and doing an NQP port and working out what NQP on 6model would look like. Now I only have to worry about one of those things, not all three at the same time.
NQP itself is 6model based now. Thus there won’t be a need for the “JnthnNQP” fork (NQP itself ended up looking much more like JnthnNQP anyway, which was the idea). If I do need tweaks that I can’t immediately reconcile with NQP on Parrot, they can be done just by subclassing NQP::Actions or so and overriding the things in question. Either way, it’ll be the same – or a slightly twiddled – version of NQP, not a vastly different one.
Rakudo itself is now mostly written in NQP and Perl 6. We used to have a bunch of PIR builtins, but they’re long gone (along with various ugly issues they caused, like not having proper Perl 6 signatures and so forth). I’d guesstimate that about 5% – at most 10% – of the Rakudo code is backend specific. Some of it will be tricky, but a lot of it is – at least if you’re familiar with it – pretty mundane.

Of course, some problems I chose to ignore will need to be dealt with (like, how to best harness the way to CLR thinks of types in order to implement 6model representations more efficiently, and how to support gather/take). It’ll need generating IL rather than C#. And…plenty more. I’m not going to set any targets for this just yet; I’ll just dig in, have fun, and we’ll see where things land up in a month or two. I found it great fun to work on this last time – the CLR is a nice VM and well tuned – so it shouldn’t be hard to get a round tuit. :-)

Other Bits

Of course, there’s still bits of the Perl 6 spec that needs implementing, and things in module space. Amongst things I’d like to hack on soon are big integer support (so we can do Int right), natively typed operators and teaching NativeCall to handle structures.

Other 6model Bits

I want to vastly improve the state of 6model’s documentation. Taking a moment to look further ahead than the next couple of months, once the updated CLR implementation of it comes together, and when some other language’s object systems have been shown to be buildable on 6model, I also want to think about declaring a “6model API v1″ or so. This is so that if/when Parrot integrates 6model, or others implementations show up that I won’t have a close hand in, there’s a clear idea of what it’s expected to look like from the outside, and what are implementation details (and thus can be done in whatever way is appropriate). I also expect further extensions, refinements, and so forth, and I think it’d be best to give folks who implement 6model – myself included! – some coarser grained way of saying “we support this set of things” than just listing off implemented features. This is some way off, though I do already have a slowly forming picture of what I’d like to tackle in the area of meta-model design in the future.

So, that’s what I’ve got in mind. Oh, and I should be sure to blog as I work on this stuff! Feel free to prod me if I forget. :-)

Carl Masak: -n and -p, part three

Fri, 09 Sep 2011 22:20:34 +0000

(This blog post is part three of a series; there's also a part one and a part two.)

Shortly after I wrote the last post on -n and -p, and how I didn't really understand what a setting was in Perl 6, sorear++ and TimToady++ filled me in on all of the details. So here I am, a third time, to pass the knowledge on.

I wrote last time that I found the term "setting" confusing and overloaded. That's because I thought it was a single, defined thing. In effect, there's no the setting in Perl 6; there can be many at the same time.

A setting is simply something that surrounds your code on the outside. (Haskell has a Prelude, but a prelude only comes before your code. A setting envelopes your code both before and after.) Your code simply finds itself lexically inside some setting or other. In technical parlance, whatever is the OUTER:: of your code is a setting.

So, you can have several settings, just like I wished for. They stack, you see. Or rather peel, like onion layers. One man's setting is another man's code, all the way outwards into the final OUTER:: nothingness of empty space.

And — the final piece of the puzzle — the big default "here, friend, are all of your builtins" setting is called CORE. I'd always wondered why we keep saying both "setting" and CORE. (And why Rakudo calls the directory src/core, not src/setting.) That's why; CORE is just a setting among others.

The discussion — which consists mostly of sorear and TimToady telling how things really are — can be found here.

And now I think I'm finally done writing on how -n and -p work. 哈哈

rakudo.org: ‘nom’ branch is now default, release status

Fri, 09 Sep 2011 04:59:53 +0000

I’ve just now switched the Rakudo repository on GitHub such that the default branch is now “nom” instead of “master”. The nom branch is where nearly all significant Rakudo development has taken place over the last several months — over 1,800 commits since being forked from the ‘master’ branch back in April.

In order to avoid confusion with git’s conventional use of the “master” branch name, I’ve now renamed “master” to “ng” (the branch’s original name from early 2010). However, most Rakudo developers will likely refer to the version of Rakudo in this older branch as either “2011.07″ or “Beijing”, these being the tagnames of the final compiler release from that particular branch.

We have not as yet created a 2011.08 release based on nom. After much useful discussion and contributions from people on #perl6, I feel that there are still too many feature regressions and some performance regressions that really need to be addressed before making a release. Ultimately, it comes down to something like “It’s far better to explain and justify delaying a release than to issue a release that we know will cause problems for our clients.”

I don’t want to specify an exact date for the next release; certainly it will occur within the next three weeks. As in previous announcements, we continue to recommend using the Rakudo Star 2011.07 release until the releases from the new development line are ready.

We did discuss the possibility of simply creating a 2011.08 release from the old branch, but since it would be essentially identical to the 2011.07 that doesn’t seem to serve much purpose other than to fill a slot, so we’ve decided against that.

Today I’m planning to begin updating the perl6 RT ticket queue to reflect things as they stand after the branch rename. I
expect many tickets to be closed due to the many fixes that nom brings. New incoming tickets will now be evaluated against the ‘nom’ branch, and it’s okay to file tickets for regressions from 2011.07. (Perhaps cut us a little slack for a week or two on this, though.) More details on the RT ticket queue will be described in an upcoming message.

Comments and questions can be sent to the perl6-users@perl.org and perl6-compiler@perl.org mailing lists, or join us on IRC
freenode/#perl6 and ask there.

Carl Masak: -n and -p, part two

Mon, 05 Sep 2011 19:51:06 +0000

(This blog post is part two of a series; there's also a part one and a part three.)

I wrote last time about how -n and -p were implemented in a text-oriented way in Perl 5, and in an AST-oriented way in Rakudo.

Afterwards, TimToady said he thought I was going to write about settings and {YOU_ARE_HERE}. You see, the spec doesn't talk about toying around with ASTs, it talks about -n being equivalent to having a setting that looks something like this:

for lines() {
    {YOU_ARE_HERE}
}

(This is completely equivalent to the current AST approach, but with code instead; the {YOU_ARE_HERE} gets replaced by your program. In a way, it's a nice full circle back to a text-based way of doing things, but correctly this time. Kissing eskimos still need not apply.)

The whole notion confused me, because by "setting", I generally mean the set of builtins provided by Perl 6. Here it seemed to mean "a layer of code immediately surrounding your program". I asked on the channel. Turns out no-one else knew, either.

There was plenty of good discussion, though. Didn't make me any wiser, but it was at least interesting.

Finally jnthn suggested that maybe Rakudo actually does this right already (with the ASTs) and the spec is wrong. I can agree with that, at least to the extent that I don't see how the current setting/{YOU_ARE_HERE} spec is s'posed to work, but I feel pretty comfortable about those AST transformations.

So, um, yeah.

Solomon Foster: Complex Issues

Wed, 31 Aug 2011 15:43:58 +0000

Sorry for the long silence here, it’s been a busy summer with far too little Perl 6. But I did squeeze in some work on trig, both on nom and niecza. And I ran into a very interesting issue.

My local copy of niecza has S32-trig/sin.t almost working. A few needed skips, but all the core numeric types work. Except…

> is_approx(asin(0.785398163404734 + 2i), 0.341338918259482 + 1.49709293866352i
# got:      2.80025373533031-1.49709293866352i
# expected: 0.341338918259482+1.49709293866352i

niecza> asin(0.785398163404734 + 2i)
2.80025373533031-1.49709293866352i

rakudo> asin(0.785398163404734 + 2i)
0.341338918259481 + 1.49709293866352i

Woah, what’s up with that? Well, it turns out both answers are right in some sense:

niecza> sin(asin(0.785398163404734 + 2i))
0.785398163404734+2i

rakudo> sin(asin(0.785398163404734 + 2i))
0.785398163404734 + 2i

The thing here is that sin is periodic; there are an infinite number of complex numbers it maps to the same result value. That means when you call asin, there are an infinite number of possible results for each input value, and you must somehow choose one of them.

But let’s take a step back from that and look at why I got different results, because I used the exact same formula for asin in both Rakudo and Niecza. That formula is -1i * log(($x)i + sqrt(1 - $x * $x)). Let’s look at the sqrt first:

niecza> my $x = 0.785398163404734 + 2i; sqrt(1 - $x * $x)
-2.21086930051619+0.710488099157523i

rakudo> my $x = 0.785398163404734 + 2i; sqrt(1 - $x * $x)
2.21086930051619 - 0.710488099157523i

As you can see, one answer is the negative of the other. Of course, when you square the results, that additional factor of -1 just goes away, so these are both valid results.

So this leads me to two questions:
1) Should we define one of these two answers as being correct, as far as Perl 6 is concerned? (Or should they both be considered valid results?)

2) If so, which one? And how do we coherently specify that branch?

I thought at first it might be as simple as saying “The branch where the complex result of sqrt for complex numbers with an imaginary value of 0 agrees with the real sqrt result.” But in fact both Rakudo and Niecza already seem to agree for the sqrts of real-valued Complex numbers.

Anyone else out there have a notion?

perl6.announce: Announce: Niecza Perl 6 v9 by Stefan O'Rear

Tue, 30 Aug 2011 07:57:40 +0000

Announce: Niecza Perl 6 v9

This is the ninth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Mono implementaton of the Common Language Runtime.

List of changes

[Breaking changes]

Niecza v9 no longer supports .NET. Niecza v9 has Mono-specific code
for performing file access tests. Implementation of runtime probing
has been deferred until demand arises.

[Major features]

Niecza now supports the $::("foo") indirect name access syntax,
including indirect access to pseudopackages such as CALLER.

Trailing double colons on package names are now parsed, allowing
for working Foo::<$bar> syntax.

Pseudo-packages (CALLER, OUTER, etc) are now supported.

The new gist method is used to render say output one argument at a time,
and no longer has the effect of joining lists without spaces.

Nil is now supported according to spec. It is an undefined value, and
is treated as the lack of an argument by the binder.

Perl 6 subroutine-scope phasers are now implemented: LEAVE, ENTER, UNDO,
KEEP, TEMP, PRE, POST. Class-level design-by-contract is still NYI.

CATCH and CONTROL are now implemented.

[Minor new features]

Name handling is more uniform and corner cases such as class :: is Str
now work.

"is dynamic" is now parsed and ignored on variables.

Exception handling through nested runloops is now more transparent;
control exceptions in particular now work properly.

File access tests (rwxoRWXO) are now supported. (Mathew Wilson)

Regex modifiers :nth :x :continue :pos are now supported.

Added &dir, Pair::ACCEPTS.

Added sprintf (Mathew Wilson and Martin Berends).

General cases of the temp and let pseudo-declarators are now
implemented.

Warnings are now implemented as control exceptions and can be caught
using CONTROL.

[Selected bug fixes]

% now handles signs correctly. (Larry Wall)

[Other]

The binary distribution has been slightly simplified to reduce potential
confusion.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

The main shiny thing I want to chase next month is asynchronous I/O
and networking. Niecza's poor startup speed makes long-running
servers an attractive niche to target.

[1] https://github.com/downloads/sorear/niecza/niecza-9.zip
[2] https://github.com/sorear/niecza

Moritz Lenz (Perl 6): Third Grant Report: Structured Error Messages

Mon, 29 Aug 2011 17:56:41 +0000

Progress on my grant for error message is slower than expected, as expected :-). Yes, you've read that sentence before.

In the past months, general hacking on the nom branch of Rakudo was just too much fun -- and partially a prerequisite for the exceptions work.

I did manage to redo the backtraces that are generated from error messages.

Backtraces are now generated mostly in Perl 6 code, making them much more hackable. There's a Backtrace class, which is a list of Backtrace::Frame objects, each knowing the code object associated with it, as well as line number and file. (This is both specced and works in Rakudo)

Routines can have the is hidden_from_backtrace trait, which makes them not show up in the default backtrace stringification (one can still request a .full string representation). This is useful for routines which are internally used to generate exceptions, like die().

Rakudo also has a --ll-exceptions command line option which provides PIR-level backtraces, in the rare case the Perl 6 level backtraces hide too much information.

I've also started the nom-exceptions branch in Rakudo, which aims at lifting current limitations in Rakudo's exception handling. Currently die() and friends generate a parrot exception, and then there's a routine that fills the error variable $!. This routine generates a new Exception object, and sticks the parrot exception into it.

This practice means that if you create a subclass of Exception, instantiate it and throw it, you still only get an Exception in the error handler, not an object of the subclass. Since the actual exception type is very important for the ongoing work, that has to change. The branch mentioned earlier allows one to generate a Perl 6 exception, and pass that on as the payload of the parrot exception, which is then unwrapped when filling $!.

As a proof of concept this works, but it suffers from not being robust enough -- as it is, we could accidentally unwrap the payload of a CONTROL exception, placing meaningless junk into $!. So this needs a bit more work, which I plan to do this week (or next, if it proves to be more difficult than anticipated).

As always, your feedback is very welcome.

Carl Masak: -n and -p

Sun, 28 Aug 2011 21:54:43 +0000

(This blog post is part one of a series; there's also a part two and a part three.)

With -n on the command line in Perl 5, you can create an implicit loop for lines of input:

$ cat > input
1 big cup of tea
2 cookies
5 pages of a good book
^D

$ perl -nE '$sum += $_; END { say $sum }' input
8

Pretty neat.

Oh, and it works in Perl 6, too.

$ perl6 -n -e 'our $sum += $_; END { say $sum }' input
8

Yay!

There's a -p flag that also does the loop thing, but it prints $_ at the end of each loop iteration:

$ perl -pE '$_ = uc' input 
1 BIG CUP OF TEA
2 COOKIES
5 PAGES OF A GOOD BOOK

$ perl6 -p -e '.=uc' input
1 BIG CUP OF TEA
2 COOKIES
5 PAGES OF A GOOD BOOK

Now, let's look at the implementations of these flags in Perl 5 and in Rakudo.

Perl 5, has a file perl.c, let's look there:

PL_minus_n      = FALSE;
PL_minus_p      = FALSE;

/* meanwhile, much later */

case 'n':
    PL_minus_n = TRUE;
    s++;
    return s;
case 'p':
    PL_minus_p = TRUE;
    s++;
    return s;

Ok, um. So that clearly wasn't it. That's just the code to prepare for applying the flags. Let's keep looking.

Oh, here's a file, toke.c. But that's the lexer for Perl 5, clearly the code for handling the flags can't be in there, can it?

if (!PL_in_eval && PL_minus_p) {
    sv_catpvs(linestr,
        /*{*/";}continue{print or die qq(-p destination: $!\\n);}");
    PL_minus_n = PL_minus_p = 0;
} else if (!PL_in_eval && PL_minus_n) {
    sv_catpvs(linestr, /*{*/";}");
    PL_minus_n = 0;
} else
    sv_catpvs(linestr, ";");

/* much, much later */

if (PL_minus_n || PL_minus_p) {
    sv_catpvs(PL_linestr, "LINE: while (<>) {"/*}*/);
    /* handling of -l, -a, and -F */

Oh wow. sv_catpvs. That's some kind of string concatenation. So when perldoc perlrun says that the -n and -p flags cause Perl 5 to "assume a loop around your program", it actually means something more like "stick a loop right into your program".

I won't toke — sorry, poke — too much fun of the Perl 5 solution. After all, I've used it many times, and I really like it. I bet it's fast to do it with strings like that. And elegant. No wait, the other thing.

Let's look at Rakudo's implementation of the same flags. In Rakudo, we find the code in src/Perl6/Actions.pm, a code-oriented companion to src/Perl6/Grammar.pm:

if %*COMPILING<%?OPTIONS><p> { # also covers the -np case, like Perl 5
    $mainline := wrap_option_p_code($mainline);
}
elsif %*COMPILING<%?OPTIONS><n> {
    $mainline := wrap_option_n_code($mainline);
}

# meanwhile, earlier

# Turn $code into "for lines() { $code }"
sub wrap_option_n_code($code) {
    return PAST::Op.new(:name<&eager>,
        PAST::Op.new(:pasttype<callmethod>, :name<map>,
            PAST::Op.new( :name<&flat>,
                PAST::Op.new(:name<&flat>,
                    PAST::Op.new(
                        :name<&lines>,
                        :pasttype<call>
                    )
                )
            ),
            make_block_from(
                Perl6::Compiler::Signature.new(
                    Perl6::Compiler::Parameter.new(
                        :var_name('$_'), :is_copy(1)
                    )
                ),
                $code
            )
        )
    );
}

# Turn $code into "for lines() { $code; say $_ }"
# &wrap_option_n_code already does the C<for> loop, so we just add the
# C<say> call here
sub wrap_option_p_code($code) {
    return wrap_option_n_code(
        PAST::Stmts.new(
            $code,
            PAST::Op.new(:name<&say>, :pasttype<call>,
                PAST::Var.new(:name<$_>)
            )
        )
    );
}

Don't get bogged down by detail. There's a bit more code, but the big difference is that Rakudo operates on the syntax tree of the code, whereas Perl 5 operates on the text of the code.

In particular, this means that Rakudo parses the program code first, and then adds the -n and -p code.

Which means that the eskimo operator doesn't work in Rakudo:

$ perl -nE '$sum += $_ }{ say $sum' input
8

$ perl6 -n -e 'our $sum += $_ }{ say $sum' input
===SORRY!===
Confused at line 1, near "}{ say $su"

Call me conservative, but I think this is a good thing.

Carl Masak: Why tests will change the way you code (if they haven't already)

Fri, 26 Aug 2011 22:45:43 +0000

A dear colleague of mine keeps saying things like "I'll just do a few tests", or "I'll need to carry out these tests". It throws me off practically every time.

You see, I've been so completely swallowed up by the Unit Testers' Underground Movement, that to me these perfectly reasonable phrases sound slightly... wrong. You don't do tests, you write them. You don't carry out tests yourself, you have the computer run them for you.

Clearly my indoctrination has been effective, when conversation that sounds completely correct to my colleagues sets off silent alarm bells in my mind. I'm all like, "they're using the word "tests", to mean what? Manual testing?" (Nothing wrong with manual testing per se. It's a good tool in your toolbox. And ad-hoc tests often produce some of the best unit tests.)

There's a reason my mind ended up in this twisted state. We'll get to that.

For a number of years now — I don't actually remember how many — I've been quitting IRC with the message "tests rock!". I always type this message out, no automation involved. (Yes, I do see the sweet irony there.) Consider it my green wristband, if you like.

There's a reason I keep writing that, too. You see...

Tests rock. They truly and utterly rock.

I'll explain, and you might not believe me. I am after all, dear reader, trying to indoctrinate you, just as I was once the victim of the Unit Tester's Underground Movement's corrosive propaganda. You'll resist, you'll fight, but eventually we'll get you too. Resistance is futile, unless there's a unit test for it somewhere.

If you let down your guard even a little, you'll start discovering the three benefits, one by one. And then there's really no turning back — you'll be test-oriented before you can say "but I don't have time to write tests".

Mwhahaha.

First benefit

It all begins so innocently. You're thinking "hey, I keep pressing the <up> arrow in my shell a lot, just to run the same test code all over again". So you decide to exercise some laziness, and put all your testing snippets in a file somewhere. Or a group of files, whatever.

This is the first benefit: by treating tests as program code, as something more permanent than just a shell command somewhere, you're unwittingly building up a list of things that are true about your program. That's all that unit tests really are: an executable list of assertions in a file somewhere. And then — oops — you checked it into your repository, too! Now other people can run those same tests. Fancy that.

In order to completely obviate the <up> arrow, I've come up with a small harness that runs all of your tests as soon as you hit Save in your editor. It has the same kind of addictive effect as does a good game: it sucks you in and keeps you there. I've written about it here and here; a small prototype of it can be found here.

Second benefit

Of course, while this sounds trivial on paper, you actually don't want your tests to go wreak havoc with your production database, or send emails to customers, or whatever. Basically, you want to exercise all the important program logic while still having the program be side-effect-free. The Unit Testers' Underground Movement have had great successes in this department, deviously convincing people that testing is so important that it actually makes sense to "de-couple" the pure business logic from the different kinds of side effects.

Think of your normal application as a giant octopus. It has a brain, and it has arms. The tests want to interact with the brain without getting tangled up in the arms. With the correct decoupling, the brain can be lifted out and placed in a vat in a laboratory, and a set of virtual octopus arms can be plugged in. So the octopus brain still thinks it has all of its arms, and when it sends signals to flex them, the virtual arms happily send back signals of movement as if they were real, physical arms.

Coders do this kind of thing under various slightly overlapping names: dependency injection, mocks, fakes, stubs... They do this, and as an annoying little side effect of doing this, they just have to factor their code to be receptible to this kind of octopus plug-and-play behavior, and thus also more modular. How insidious.

You might think that this modularity is the second benefit. It's not. Good modularity is just common sense for a big system. It is a nice side effect of tests, though.

No, the second benefit is this: it's been said that your API won't be really stable/usable until you've made it work with three different clients. Usually, the application itself, or some component of it, or some component near it, makes up the first client. With the test, you immediately have a second client. That's well on your way to having three clients.

This is more important than it might at first seem. As a second client (or a first client, depending how you see it), tests will put you in the habit of thinking about the interface up front. Yes, that thing that your OO teacher went on about: the outwards-facing layer that the external world gets to interact with to get to query or modify the internal, heavily protected gooey stuff in the middle. It's true for the objects of your application, but it's also true for your whole application, even a non-OO application. You just need to momentarily view the application as a single coherent object. To me, that's what object-orientation really means nowadays: coming up with a decent interface behind which to hide the internal gooey stuff.

Put differently, the tests make you think about and design your interfaces. The face of design changes from one of invention to one of discovery.

The tests push the buttons and twist the dials on the outside of the machine without opening it up. So your first order of business is to make sure that all the right buttons and dials are there. When you feel like it, you could switch out the out-moded transistor innards for contemporary circuitry. The tests don't care, they just care about the buttons and dials on the outside.

And it's because the tests are internals-agnostic that you dare make such a crazy leap from transistors to circuits. Would you have made such a daring move before you had tests? Heavens no, something important might break! (A revision system helps greatly here, too.)

At this point, you're hooked on two of three benefits. You're already getting that glassy-eyed look, as if there's a chance you might just tell a random stranger on the street about test-driven development. You're starting to suspect that the Unit Testers' Underground Movement actually is a real movement.

Third benefit

This is when the third and final benefit triggers, if it hadn't already. And I'm really sorry to report this, but when it does, you're a goner. There's just no going back after that.

You see... oh, how shall I put this? These test files, these dumb, inert lists of assertions about your program, they find bugs for you. Since they run your program as often as you want, with the simple-minded patience of a computer, testing every little thing you ever thought of during your brightest moments, they find your bugs before you do! At this point you might have accrued hundreds or thousands of individual tests in dozens or hundreds of files. They all just tirelessly iterate through everything that could conceivably go wrong with your program, and try it out without your having to lift a finger. The command line can go <up> itself.

And, boy, do the tests find bugs. Your reaction the first few times this happens will range from "oh right, forgot about that other bit" to "oh dear... that would've been embarrasing, wouldn't it?". Then you get used to the tests being a bit ahead of the curve.

When the third benefit started hitting me hard, when I had begun to use tests so much that they actually began reporting bugs for me in this way, I got to thinking that maybe I was becoming soft and careless due to having all those tests. Maybe the safety net that the tests provide had made me think less about the correctness of the code I was writing.

Then I realized that the bugs the tests were finding for me corresponded to bugs that I never found in my older programs, or bugs that came back later and bit me hard when I least needed it.

It's been said that an ounce of prevention is worth a pound of cure. This is where it all pays off. If you thought that all that explicit test writing and dependency injection and mocking and all that stuff was a bit, you know, tedious — this is where it all pays off in reduced debugging time. Just to see why, let's run a few common scenarios with our test-colored glasses on:

Does the program work? Run tests. Green? Yes, it works. Red? No, it doesn't work.
Will this change break something? Run tests. Green? No, the change is fine. Red? Yes, it broke something.
If I add this new feature, do the old features still work? Write a test for your new feature. Code until it turns from red to green. Run all the tests. Green? Yes, the old features still work. Red? Well, something broke, and you're really happy to realize this now rather than later, during a customer demo or something.

Making changes/additions without unit tests? Well, you would either have to proceed very slowly, possibly doing a lot of tiresome debugging anyway — or you might just never consider them as something that you would dare do to the application. Sans tests, it's far too easy to have technical debt pile up into such tall, insane piles of Jenga blocks that pulling out another block just isn't considered. There's something immensely refreshing about how care-free you can be with tests backing you up.

Conclusion

This is why I write tests. They're to a programmer as swimfins are to an underwater diver — sure, you can jump into the fray without them, buy you won't move as fast or as accurately. Or as far.

To summarize, these are the three benefits of tests, as I see them:

Tests are simply static versions of various checks that you would've (or, as they grow, should've) run anyway.
Tests provide you with a "second client" for free, and a very diligent client at that.
Tests find bugs in your code before you do. It's awesome.

May your unit tests be fruitful and multiply. You can reach me by email to learn the secret handshake of the Movement. ;-)

Moritz Lenz (Perl 6): Why Rakudo needs NQP

Wed, 24 Aug 2011 20:43:55 +0000

Rakudo, a popular Perl 6 compiler, is built on top of a smaller compiler called "NQP", short for Not Quite Perl.

Reading through a recent ramble by chromatic, I felt like he said "Rakudo needs NQP to be able to ditch Parrot, once NQP runs on a different platform" (NQP is the "another layer", which sits between Rakudo and Parrot, mentioned in the next-to-final paragraph).

I'm sure chromatic knows that VM independence is the least important reason for having NQP at all, but the casual reader might not, so let me explain the real importance of NQP for Rakudo here.

The short version is just a single word: bootstrapping.

The longer version is that large parts of Rakudo are written in Perl 6 itself (or a subset thereof), and something is needed to break the circularity.

In particular the base of the compiler is written in a subset of Perl 6, and NQP compiles those parts to bytecode, which can then compile the rest of the compiler.

This is not just because we have a fancy for Perl 6, and thus want to write as much of the code in Perl 6, but there are solid technical reasons for writing the compiler in Perl 6.

In Perl 6, the boundary between run time and compile time is blurred, as well as the boundary between the compiler, the run time library and user-space code. For example you alter the grammar with which your source code is parsed, by injecting your own grammar rules.

"Your own grammar rules" above refers to user-space code, while the grammar that is being altered is part of the compiler. If we had written the compiler in something else than Perl 6 (for example Java), it would be horribly difficult to inject user-space Perl 6 code into compiled code from a different language.

And the code not only needs to be injected, but the data passed back and forth between the compiler and the user space need to be Perl 6 objects, so all important data structures in the compiler need to be Perl 6 based anyway.

And it's not just for grammar modifications: At its heart, Perl 6 is an object oriented language. When the compiler sees a class definition, it translates them to a series of method calls on the metaobject, which again needs to be a Perl 6 object, otherwise it wouldn't be easily usable and extensible from the user space.

Now you might think that grammar modifications and changes to the Metaobject are pretty obscure features, and you could get along just fine with an incomplete Perl 6 compiler that neglected those two areas. But even then you'd have lots of interactions between run time and compile time. For example consider a numeric literal like 42. Obviously that needs to be constructed of type Int. What's less obvious is that it needs to be constructed to be of type Int at compile time already, because Perl 6 code can run interleaved with the compilation. So the compiler needs to be able to handle Perl 6 objects in all their generality, which is a huge pain if the compiler is not written in Perl 6.

Rakudo has cheated on that front in the past, and consequently has had lots of bugs and limitations due to non-Perl 6 objects leaking out at unexpected ends. If you ever got a "Null PMC Access" from Rakudo, you know what I mean.

The lesson we learned was that you need a Perl 6 compiler to implement a Perl 6 compiler, even if that first Perl 6 compiler can handle only a rather limited subset of Perl 6.

And there are also quite some benefits to this approach. For example NQP's new regex engine is implemented as a role in NQP. It is mixed into an NQP class which allows us to build Rakudo, but it is also mixed in a Perl 6 class, which allows the generation of Perl 6-level Match objects without any need to create NQP-level match objects first, and then wrap them in Perl 6 Match objects.

That's what NQP does for us. It allows us to actually write a Perl 6 compiler.

Tadeusz Sośnierz (tadzik): What is Production Ready?

Wed, 24 Aug 2011 14:39:05 +0000

“When will Perl 6 be production ready?” – they ask from time to time. I know the feeling, there was a time I wanted to know too, and after a year working on Rakudo, I can truly say,

I have no freaking idea!

I’d really like to tell you, seriously. If you ask #perl6, they will start tricking you into thinking that it’s ready enough and they’re actually using it, right? Tricky bastards. But, what do you actually ask for? What is this mighty Production Ready?

I dedicated some thinking to this today. What makes something Production Ready? I can think of two possibilities

The creators declare it Production Ready
People start using it in Production Environment

The first one is a bit tricky to achieve when it comes to Perl 6. As we know, Perl 6 is a language. How can language be Production Ready? Think, think. Is there another example of something which is rather a spec than an end-user product, and is either not declared as finished, or the spec freeze date is ridiculously far in the future? Right, it’s HTML5. Spec is a draft, it’s nowhere near finished, and neither of the implementation implement all of it. So what makes HTML5 production-ready? I don’t think it’s declared ready by its creators. It’s that people didn’t bother with official opinions and started actually solving problems with it. Took the existing implementations and made use of it. Therefore, we can safely assume that by “Production Ready Perl 6” we really mean “A Perl 6 Compiler I can use to get the job done”. So what are the current compilers lacking for the majority of people?

Yes, I’m asking you. You don’t really know, do you? You didn’t even try them? It’s just that people don’t use them too often, so they’re probably crap, right? Ok, there’s some logic in that.

There is a possibility that Perl 6 is already capable of solving your problems. You should try it. But! Enough of the advertising business, I’m wondering here.

“So what is your Production Ready?”, you may ask. What do I expect from Perl 6 before it will be Production Ready for me? It’s not, I’m not gonna lie. It’s solving my problems, it pays my bills, but it lacks this Something that will make it Purely Awesome. In my opinion, there are two major things we’re missing:

Speed. Not all things I write need to be blazingly fast, but what is the point of amazingly expressive language, if the bottleneck of the development process is program runtime?
Access to Perl 5 modules from CPAN. Yes, I know of modules.perl6.org fairly well, believe me. Still, it will take ages, if not infinity to make it as awesome as CPAN is. Blizkost is a bridge between Perl 5 and Parrot and it’s capable of running simple stuff already.

That’s it. I can live without most of the things. But what I’m really looking for, is a better Perl 5. It needs CPAN, and it needs to be less slow that it is. I’m not looking for a C performance. I could live with Perl 5 performance here probably.

That’s what I’m missing. And what is Your Production Ready?

Jonathan Worthington (6guts): My YAPC::EU talks

Tue, 23 Aug 2011 21:40:14 +0000

I had a wonderful time at YAPC::EU in Riga this year. Riga is a very beautiful city – there’s many nice things to look at, not to mention the excellent food and a wide range of beer at sensible prices. Once place I went had over 30 types of beer on tap! That’s the city, but then there was the YAPC itself: smooth running, well organized, nice venue, all breaks and lunches on site, and a great conference dinner. I’ve had the fortune to be at many events where Andrew Shitov has led the organization (I just checked and I think this must have been the 9th one!), and I knew he could do something excellent for YAPC::EU. He did, and was a very deserving recipient of a White Camel award this year. So, congrats to Andrew, and many thanks for the steady stream of excuses to travel to East Europe and talk about Perl. :-)

I gave a couple of talks.

Rakudo Evolved: Speed, Feedback and Hackability covered my work on 6model and Rakudo over the last year, and laid out some of the plans for the coming year. It had some code, but was more of a wordy talk: some theory, some lessons learned, some plans.
Debugging Perl 6 Grammars was my more hands-on talk this year. During the talk I unveiled Grammar::Debugger, a module for debugging grammars.

So, another year, another great YAPC. And now, it’s back to work on Rakudo!

Carl Masak: YAPC::EU without the adjectives and adverbs

Mon, 22 Aug 2011 21:03:00 +0000

YAPC::EU 2011 deserves adjectives and adverbs. A lot of them. But I hear constraint liberates, so let's try a post without them.

The venue. I liked it.
Andrew Shitov. I would heap adjectives on him. That guy can arrange conferences. How does he do it? It impresses me. Andrew plus plus.
Giving talks. Years pass, and the energy and nervousness I expend before and during talks keeps decreasing. I relaxed this time.
Listening to people's talks. I liked doing that. I might have bias, but I preferred the ones about Perl 6. 哈哈
Hallway track. It alone brought value to this conference. Discussion about the spec excels when the conversation takes place face to face.
Riga. I must come to Riga in the future. It is a city I like. Its buildings have age and the food has tastes that I appreciate.
Esperanto guide. She came from nowhere and she showed us town. Thank you, Esperanto guide.
Hotel Konventa Sēta. Rooms, personel, location, everything. Full of wonder.

This blog lacks a commenting system; had it had one, I would have asked you to comment without adjectives and adverbs.

rakudo.org: Rakudo 2011.08 release status

Fri, 19 Aug 2011 01:09:03 +0000

As noted in a previous post, we had planned to issue a new release of Rakudo sometime around YAPC::EU, based on the new nom branch. The nom branch is a substantial redevelopment of the Rakudo compiler, and generally shows a 3x speed improvement over previous versions of Rakudo.

However, in the past few days at YAPC::EU, some extended testing exposed some significant regressions both in performance and features. So, rather than rush out a buggy release we’ve decided to spend a few more days cleaning things up a bit. Because of the magnitude of changes in the switchover to nom we had already planned to allow ourselves a fair bit of wiggle room in terms of exact release dates.

I expect releases to start as early as this weekend, or perhaps a few days after that. We definitely will have some August 2011 releases, they just won’t be “exactly two days after the Parrot release” as has been typical in previous months. In the meantime, we continue to encourage existing users of Rakudo Perl to use the 2011.07 release of Rakudo Star.

More details as we make progress.

Patrick R. Michaud: Some thoughts on YAPC::EU 2011

Thu, 18 Aug 2011 22:03:04 +0000

YAPC::EU 2011 in Riga has just about finished, and it has been great seeing long-time friends again and making new ones. I’ve heard many people remark that we really wish there could be more weeks like these.

There are two items that stand out in my mind about this year’s conference:

1. Andrew Shitov and his crew are absolutely amazing at organizing and running a conference. This was the most flawlessly executed conference or event I think I’ve ever been to. Not only that, but Andrew and the other organizers made it look effortless, which to me is a mark of true greatness. I’m certain that in fact there was a lot of planning and effort behind it, but the entire team just looked relaxed and at ease throughout the event. I’d definitely encourage folks to attend any event that Andrew and this group organizes.

2. Riga is a stunningly beautiful place. I definitely want to return here again some day, and I’m grateful that the organizers chose this location.

perl6.announce: Parrot 3.7.0 "Wanda" released by Will Coleda

Tue, 16 Aug 2011 20:09:04 +0000

OTTO. Apes don't read philosophy.

WANDA. Yes they do, Otto. They just don't understand it. Now let me correct you
on a couple of things, OK? Aristotle was not Belgian. The central message of
Buddhism is not 'Every Man For Himself'.

OTTO. You read...

WANDA. And... the London Underground is not a political movement. Those are all
mistakes, Otto. I looked them up.

Now. You have just assaulted the one man who can keep you out of jail and
make you rich. So, what are you going to do about it, huh? What would an
intellectual do? What would Plato do?

-- A Fish Called Wanda, by John Cleese.

With that, I'd like to apologize for Parrot 3.7.0, also known
as "Wanda". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 3.7.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/3.7.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend getting
the latest and best Parrot code from github (https://github.com/parrot/parrot).

Parrot 3.7.0 News:
- Core
+ Added mem_sys_strndup function.
+ Added new load_bytecode_p_s opcode as an eventual replacement for
load_bytecode_s
+ Added new :tag() syntax to IMCC for PIR
+ Improved configuration support for msys
+ known-buggy parrot_debugger is no longer installed
- Languages
+ Winxed
- Updated snapshot to version 1.1.0
- multi functions and methods
- cast to var
- load_packfile builtin
- __NAMESPACE__ and __CLASS__ predefined constants
- Documentation
+ The Archive::Tar and Archive::Zip libraries now have POD docs

The SHA256 message digests for the downloadable tarballs are:
939d10eaeec7fdbe689a7e1cc6cd613e343b32eda0f5394b79f98347b68e37f8
parrot-3.7.0.tar.bz2
494eb7749fbec101acaca890e1272c63192f274796a76463ffa42b99642b3e9b
parrot-3.7.0.tar.gz

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. The following people (in no particular
order) contributed to this release. Thanks!

Aaron Faanes, Allison Randal, Andrew Whitworth, Andy Dougherty, Andy
Lester, Bob Kuo, Christoph Otto, Will "Coke" Coleda, Dan Bolser,
Felipe Pena, Francois Perrad, Gerhard R, Jimmy Zhuo, Jonathan "Duke"
Leto, Jonathan Worthington, Martin von Gagern, Michael H. Hind, Moritz
Lenz, Peter Lobsinger, Jim Keenan, Kevin Polulak, Julian Albo

Our next scheduled release is 20 September, 2011.

Enjoy!

--
Will "Coke" Coleda

Tadeusz Sośnierz (tadzik): S26 in HTML

Wed, 10 Aug 2011 08:56:12 +0000

On http://perlcabal.org/syn/, Synopsis 26 is the only one without the HTML page. That’s of course due to the lack of Pod (Pod6) to HTML converter. Today there has been a breakthrough in this embarrasing situation :)
My Pod parser integrated into Rakudo is capable of parsing S26 completely, so this morning I wrote a Pod::To::HTML module for it. Parsing and generating HTML output from S26 takes about 4 minutes on my machine, but the outcoming document is not that bad. You can see some still NYI features, like the lack of formatting codes and correctly interpreting semantic blocks. The first one is a part of my GSoC work, the second one is not but I’ll probably do it anyway, just for the sake of having prettier S26 :)

My HTML-fu is a bit Less Than Awesome, so if anyone knows how to make it any better (and it won’t be cheating as Pod::To::HTML is not a part of my GSoC work), I’m willing to hand out commit bits to anyone interested.

Maybe it’s finally time to read this one. Have the appropriate amount of fun.

Jonathan Worthington (6guts): A hint of meta-programming

Mon, 01 Aug 2011 22:59:34 +0000

After a few patches to deal with some meta-circularity issues, I’ve got to the point where I have at least a very basic example of a custom meta-class working in the nom branch of Rakudo. All this does is print out the name of each method that gets looked up, before deferring to the normal handling of method dispatch. However, I’m sure that now that I’m at this point, much more interesting things are already very possible, though no doubt there’s some rough edges I still need to deal with.

Anyways, to the code! Here’s a module, LoggedDispatch.pm.

my class LoggedDispatch is Metamodel::ClassHOW is Mu {
    method find_method($obj, $name) {
        note "Looking up method $name";
        nextsame;
    }
    
    method publish_method_cache($obj) {
        # Suppress this, so we always hit find_method.
    }
}

# Export this as the meta-class for the "class" package declarator.
my module EXPORTHOW { }
EXPORTHOW.WHO.<class> = LoggedDispatch;

Taking it from the top, we inherit from Metamodel::ClassHOW, which is the default implementation of Perl 6 classes. There’s actually no requirement to do this – in some future posts I’ll show you how you can easily piece together meta-classes from the ground up – but in this case all we really want to do is a small tweak to the existing behavior. For now, we also inherit from Mu. (Aside: at the moment, since ClassHOW is what defines classes, it falls “outside” of the Perl 6 type hierarchy. That’s in some ways understandable but also in some ways very awkward. I’ve more than one approach to dealing with this, but need to think more on the problem.)

Now, on to the meat of the problem. Conceptually, every method dispatch on an object gets that object’s meta-object and calls “find_method” on it, passing in the method name to find. Therefore, in order to intercept this, we need to override “find_method” – which is what we do. We simply print the name of the method to STDERR using “note”, then use “nextsame” to defer to the actual method lookup algorithm. Pretty easy. However, there’s a twist.

As you might imagine, dispatching methods is a hot-path. If we have to call a method in order to work out what method to call – and if that had to do the same in turn – things would get really very slow indeed. Thus, it’s possible to publish a method cache, which is dead fast to look in. However, it’s also going to mean that find_method never gets called – which is where we’re going to put our logging stuff. While for an optimized version of this we could compute a bunch of closures to do the logging and populate a cache with those, for now we just suppress generation of the cache.

That gets us down to the last bit. There’s a (lexical) EXPORT package for exporting symbols, but we want to do something a bit different here: we want to tell the compiler that when this module is used, it should use our meta-class whenever the user writes a class (that is, uses the package declarator “class”). For now (note, API liable to change) that’s done by installing an entry in an EXPORTHOW module.

And with that, we’re done. Here’s an example program.

use LoggedDispatch;

class A {
    method m1() { say 42 }
    method m2() { say 99 }
}

for 1..2 {
    A.m1;
    A.m2;
}

And here’s the output:

Looking up method BUILD
Looking up method m1
42
Looking up method m2
99
Looking up method m1
42
Looking up method m2
99

The first line of this may be a little surprising – something goes looking for a BUILD method at some point. But it’s maybe not such a surprise: it’s a method with special meaning to object construction, and we can imagine that at some stage during the class composition, something got interested in whether the class had one (in this case, it’s the bit of code that creates a build plan, in order to optimize object instantiation).

At the start of the post, I used the term meta-circularity. If you weren’t sure what it meant, you’re now in a position to understand it. Here, we wrote a Perl 6 class to extend the meaning of Perl 6 classes. Meta-circularity is very simply the property of being able to use a language to extend itself, rather than having to do so by going and writing some other special or lower-level language to do so. And I’m really happy that I’ve been able to tie up the last few loose ends so that we can start to do that in the nom branch now.

Tadeusz Sośnierz (tadzik): GSoC – The podparser branch has landed

Sun, 31 Jul 2011 19:14:11 +0000

Just a few minutes ago, with a total count of 28 files changed, 1627 insertions and 41 deletions, the podparser branch in which I’ve been doing my GSoC work has been merged into nom, the current development branch of Rakudo and the soon-to-be master. So, what do we get?

Pod is now parsed, not ignored. The whole parse tree is available in runtime in the $=POD variable
A basic support for documenting classes and subroutines with #= comments, and the .WHY method on each object returning the attached documentation
A –doc switch, a not-really-replacement for perldoc
A simple Pod::To::Text output generator
and more, of course :)

So, how does it look like in action?

$=POD

All the Pod we have written in our code is now available in the $=POD variable. Example?

=begin pod
Some documentation
=end pod

say $=POD[0].content[0].content;

Results in

Some documentation

That’s not very useful per se, so how about this one:

=begin pod

=head1 NAME

A basic pod document

=head2 Running Perl programs

To run a Perl program from the Unix command line:

    perl progname.pl

=head2 Things on my desk

There are the following things on my desk right now:

=item A cup of tea
=item A couple of pens
=item A stereo
=item A couple of books
=item A laptop, obviously

=end pod

DOC INIT {
    use Pod::To::Text;
    say pod2text($=POD);
    exit;
}

say "I'm just a simple program";

Now what’s the DOC INIT thing? Let’s see. How about we run the above program:

$ perl6 foo.pl
I'm just a simple program

No suprises. Let’s introduce the –doc switch then:

$ perl6 --doc foo.pl
NAME

A basic pod document

  Running Perl programs

To run a Perl program from the Unix command line:

    perl progname.pl

  Things on my desk

There are the following things on my desk right now:

 * A cup of tea
 * A couple of pens
 * A stereo
 * A couple of books
 * A laptop, obviously

That’s right. The DOC blocks are executed only when the –doc command line option has been given. At the moment you have to write them yourself, but maybe even in the nearest few days there will be a default DOC INIT block doing What You mean all by itself. There we go, a perldoc-alike built-in :)

WHY

That’s probably the biggest killer feature in the whole project. Although it’s not yet fully implemented in Rakudo (suprise segfaults here and there, don’t worry, they’re not permanent :)), it looks pretty much like this:

#= our zebra class
class Zebra {
...
}

say Zebra.WHY # prints: "our zebra class"

Yes yes, a documentation inspection in runtime. See the potential?

# what was that sub again?
&sort.WHY.say # get the documentation for the sort() builtin

That opens a way for lots of awesome userspace tools too.

So, what’s still not quite there?

.WHY (some cases)
Formatting Codes (C<code>, B<bold> etc)
DOC use (that can be written as DOC INIT { use … } for now)
Default DOC INIT (as mentioned above)
A Pod::Parser as a standalone module

With that done, my GSoC project will be finished, and I’ll hopefully have time to hit some things I was planning to poke in some spare time, for example:

HTML output generator
Extending the parser so Synopse 26 can be fully parsed
Other fancy userspace tools; I don’t want to spoil the suprise :)

What can I say? Pull it, compile it, play with it, report bugs and have the appopriate amount of fun :)

Jonathan Worthington (6guts): July: Beijing Perl Workshop, vacation and hacking

Sun, 31 Jul 2011 00:19:53 +0000

It’s been an interesting month. I started it over in Beijing, for the Beijing Perl Workshop. It’s the first Perl event that I’d been to in China, and it was a great experience. People were very friendly and welcoming, the venue was nice and cool (good job, since I had a hard time coping with the Beijing summer heat), I had a good audience and good response to my talks, and the meals before and after it were great too. I gave two full-length talks, plus a lightning talk:

I got to see various things in Beijing. I avoided a couple of the most famous attractions – I saw them last time I was in Beijing, and they’re incredibly crowded mid-summer – and instead enjoyed some of the still wonderful, but maybe slightly lesser known ones. I managed to get a decent bit of hacking in too, mostly because I simply can’t handle being outside for too long in high temperature and humidity. Happily, I had a hotel that provided both wired and wireless internet with the room, and a couple of free beers a day too!

Anyway, that was my fun conference trip of the month. This was also the month where I took my one totally laptop-free vacation of the year. That was just the mental and physical break I needed from the hacking, though of course I couldn’t totally break away from things and ended up reading The Art of the Metaobject Protocol on the journey there and back. I’d read a couple of extracts from it before, but this was the first time through the whole book. It was decidedly worthwhile, though it shows its age a little at times too! ;-)

Amongst the trips, I’ve got some decent work in on the “nom” branch. It’s going well – a look at the excellent new compiler features page shows that there’s not too many areas left where we lack things (at least on that list) that master has. On some features, we’ve decidedly pulled ahead of master too. I’ll write up some more detailed news soon, but overall, we’re making good progress.

Excitingly, YAPC::Europe is now not far off too! Last year I had one talk, so it was relatively relaxing. This year, I’ve two talks, but also a full day teaching class on OOP in Perl 6. So, I’m busily preparing for all of that too, and of course very much looking forward to seeing lots of familiar – and new – faces there. :-)

Anyways, my patches for today are pushed, and my beer glass is empty, so I guess it’s time to take a little rest.

rakudo.org: Rakudo Star 2011.07 released

Fri, 29 Jul 2011 18:18:23 +0000

Announce: Rakudo Star – a useful, usable, “early adopter” distribution of Perl 6

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the July 2011 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the July 2011 release is available from http://github.com/rakudo/star/downloads.

IMPORTANT NOTE ABOUT UPCOMING RELEASES: This is the last Rakudo Star release to use a compiler from the current “master” branch of Rakudo. For the past several months, Rakudo compiler development has primarily occurred in a separate “nom” (“New Object Model”) branch of its repository. Shortly after this release, the “nom” branch will become the new “master” branch and will be the source for future compiler and Star releases.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. The July 2011 Star release includes release #43 of the Rakudo Perl 6 compiler [1], version 3.6.0 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

The Rakudo team plans to issue another distribution release in August 2011 based on the new “nom” compiler. The new compiler contains many improvements over the current line of development (including vastly improved performance), but also will not be 100% backwards-compatible due to changes in the Perl 6 language specification and elimination of some workarounds. Existing Rakudo Star users looking for stability should continue to use the July 2011 release as the new distribution releases stabilize over the next several months.

Details about deprecations and migration guidelines will be provided in future releases and on rakudo.org .

This release of Rakudo Star adds the following features over the previous Star release:
* 10%-30% improvement in compile and execution speed
* New gcd (greatest common divisor) and lcm (largest common multiple) operators
* Int.base
* Call counter for Perl 6-level subroutines
* Better handling of &infix:<=> right associativity
* Fix bug in exponentiation of negative numbers
* Fix building on systems with smaller RAM

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:
* nested package definitions
* binary objects, native types, pack and unpack
* typed arrays
* macros
* state variables
* threads and concurrency
* Unicode strings at levels other than codepoints
* pre and post constraints, and some other phasers
* interactive readline that understands Unicode
* backslash escapes in regex <[...]> character classes
* non-blocking I/O
* most of Synopsis 9
* perl6doc or pod manipulation tools

There is a new online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo Star 2011.07 and other Perl 6 implementations.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see http://rakudo.org/how-to-help, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

[1] http://github.com/rakudo/rakudo

[2] http://parrot.org/

Moritz Lenz (Perl 6): Perl 6 Compiler Feature Matrix

Tue, 26 Jul 2011 19:07:37 +0000

We now have a nice table that tells you which Perl 6 compiler implements what..

Such a thing was long overdue. When the topic came up in the past, people have suggested mostly automated solutions that compared test coverage of compiles to generate such a table. Nothing came out of it, it would have been a rather large endeavor. Now Eevee blogged about the lack of some easy overview that tells you what is implemented in Rakudo., and I thought it was time to tackle the problem.

Instead of some advanced automated system, we now have a simple text file, and a short perl script that converts it to a HTML page.

I'd like to thanks Will Coleda, Patrick Michaud and Stefan O'Rear for their contributions, and encourage everybody to keep the data up to date.

perl6.announce: Announce: Niecza Perl 6 v8 by Stefan O'Rear

Tue, 26 Jul 2011 07:51:34 +0000

Announce: Niecza Perl 6 v8

This is the eighth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. No major general updates; having
courted alternatives I feel satisfied with the name "Niecza" again.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Major features]

$_ $/ $! are more in line with the spec. In particular they are
automatically declared in all Routines, and normal references are
lexical, not dynamic.

Rakudo-style fallback of postcircumfixes to
{at,delete,bind,exists}_{key,pos} is now implemented.

[Minor new features]

[+] is now parsed as a listop, rather than an ad-hoc
half-PREfix half-term.

$$var now functions for captures created using \$var.

substr has been extended to support calling with 4 arguments and with
WhateverCode closures.

<-> now generates rw parameters.

Parameters in blocks now default to Mu.

term:<proceed> is now available.

Added names: term:<proceed>, Complex.Complex,

(Solomon Foster) ln, log, exp, sin

$*foo is a simple lexical fetch if there is $*foo in the current scope.

Improvements to variables: The name can now be omitted; state-variables
are processed correctly when parenthesized or at the top level; will
start is implemented.

Flipflop operators are now supported. (Many thanks to Fitz Elliott for
taking the time to work out the dark corners in the spec and write
comprehensive tests.)

[Selected bug fixes]

do { } is now properly handled as a loop, and responds to labels and
control operators.

given/when no longer eats implicit return values
(Example: "say given 5 { when Int { 42 } }")

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

(Yes, this is the same as last month...)

Next month will see a major focus on spectests. Solomon Foster thinks
I can quadruple passing tests in a week; I'm not sure about that but
hope to at least do so in a month. Other features that are planned for
next month include transparent use of CLR libraries, Buf, roles, and
pseudo packages like OUTER::.

[1] https://github.com/downloads/sorear/niecza/niecza-8.zip
[2] https://github.com/sorear/niecza

Carl Masak: What I learned from the June blogging

Mon, 25 Jul 2011 20:47:37 +0000

So, I confess: I had fun. Pushing out a blog post a day for a month sounds like a lot of work, but it was mostly enjoyable.

I've been wanting to try this kind of thing for a year or so. Now that it's behind me, it feels like the material produced begs for being put into a form other than just blog posts. Therefore, I've created a repository for the chapters. From now on, changes/improvements will go into those, and the blog posts are essentially frozen. I expect to have a PDF of all the chapters soonish.

Having done three November blogging months, I guess I didn't expect to learn much new. But this kind of setup was sufficiently different, and I ended up with a few take-aways.

What I learned from blogging once a day

It's easy to fall off. More exactly, I have a full-time job now, thanks to Perl 6. I thought that, if anything, would be what would prevent me from doing this. But no, it was essentially going to a conference and do the usual last-minute dance with slide preparations. D'oh! Having fallen off, it was a bit hard to get back on again.
It's fun. My regular hobby hacking more or less got put on hold, but every day I feel like I've accomplished something. It's great. There was never any real lack of things to write about; making up the topics ahead of time took more thinking than actually writing the posts.
I should have had a blog commenting system when doing a blogging month. Been wanting to build one, but ENOTUITS. Disqus looks great, and people have been pointing me to it, but I think I'd prefer in the long run to build something myself. Yeah, that means it's going to take a bit longer, I know. As it was, I got a lot of good feedback from #perl6 (as usual). But not everyone hangs out on #perl6.

What I learned from trying to bootstrap Perl 6 knowledge

Things can be bootstrapped fairly easily. This surprised me. There's at least one fairly linear order for concepts to be introduced.
I need to go back and re-do everything. For most games, but especially the last one, I found myself using concepts that I should have introduced in some earlier chapter, but didn't. Of course I should've mentioned bareword keys in the post about hashes. It's evident in retrospect that I should've spent more time on public accessors and constructors in the chapter about classes. And probably .can and the .?method syntax too. While the themes for each day seem to have been mostly well-chosen, there's just so much that I failed to mention.
It's easier to explain a concept than to explain why you wrote a game the way you did. Or rather, it got more difficult to adequately explain the games the bigger they got. It's like small games contain all these interesting little decisions, and that's fine to explain. But then as they grow bigger they also start to contain all these overarching, architectural decisions. And there just isn't enough blog space and reader attention to focus both on the big and the small issues, so I increasingly focused on the big ones. But the little decisions are still there, even in the bigger games.
There's just so much to tell. I'm thinking of adding sections to each chapter named "The full story" or something, just to bring up the slightly more advanced topics that aren't necessarily in the path of someone who just wants to learn the basics and implement a game.

What I learned from writing a text adventure game

It was just as enjoyable as I hoped it would be. I had been looking forward to this.
It is a bigger project and takes more time than one might think. How hard can it be, right? Just four, no five, no six rooms. Be done in a jiffy. Looking at the commit logs for the game, it seems it took closer to 14 days. Keep in mind, that's for a game I expected to have ready in two days. Oh, the optimism.
I wish I had written tests for it. Not sure how it would have affected the code, but the game definitely is complex enough to merit tests. Henceforth, my answer to the question "Shall I write tests?" will be "Is it more than a screenful of code?". (The sad thing is, I know this, and have for a long time. I should listen more to myself.)
The game isn't finished. I had to push it out into the world, but feedback keeps coming in, and it'll keep settling and evolving for quite some time yet. It's a big little program now, and won't just sit around doing nothing. All I can do as a proud parent is look on from a distance and hope for the best.

Fun as it was, I can't think of a followup for this project for next June. Let's just see where we are at that point and decide then. 哈哈

Carl Masak: June 30 2011: Text Adventure II

Sun, 24 Jul 2011 21:55:28 +0000

Welcome back to the putting-together of a real, actual, old-school text adventure game. Today we'll see the second and final part of building the game. This also marks the finale of this blogging "month". Here we go.

We set the stage yesterday with the rooms of the game, and the ability to walk between them. Today we'll "just" add the things that occupy those rooms — and this, somewhat typically, turns out to constitute around 80% of the game.

In running through this, we'll see a number of useful techniques mirroring what we've been looking at earlier this month. Everything enters into the mix at this point: input/output, variables, if statements, various kinds of loops, string operators, arithmetic, given/when, arrays, boolean operators, subroutines, hashes, junctions, files, nested data, format strings, map and grep and first, subtypes, signatures, and definitely classes/objects/attributes/methods and roles. Yep, that's the whole month. We need it all to build this game.

Here's an excerpt from the final game:

> s
Chamber

This is a cramped space just inside the hidden opening in the hill. The
sun gets in enough to illuminate the place. There are some scribblings on
the wall.
There is a sign here.
There is a basket here.
You can go north.

> help   
Here are some (made-up) examples of commands you can use:

look (l)                             | take banana
examine banana (x banana)            | drop banana
[go] north/south/east/west (n/s/e/w) | put banana in bag
open bag                             | close bag

> read walls
"This sentence no verb."

> read walls
"Help, I'm stuck!"

> inventory
You are carrying:
  A flashlight.
  A rope.

> read sign
It says "LEAVE " with big, scrawly lettering. Maybe a warning or a threat.
On closer inspection, though, it looks like there might once have been one
more letter at the end, but it has since been worn away.

> put rope in basket
You put the rope in the basket.

>

Here we see the span of different commands that can be issued. A movement command (s) causes a new room to be described, along with the things in the room and the visible exits. There are some "special" commands, such as help and inventory. Apart from that we can do several things (take, drop, open, close) with single objects, as well as move objects in more advanced ways (put rope in basket).

What happens when we write read sign on the command line of the game? The command is parsed, which means that the program assigns some structure to the text. In this case, it is recognized that read is a verb and sign is a noun standing for a thing known to the game. A bit of validation and some magic later, and the user-provided command read sign has been translated to the computer-understandable method call %things<sign>.read. %things is a hash that maps all the nouns to their corresponding objects. In this case, it maps to an instance of the class Sign which just happens to have a .read method, provided through a role Readable.

Similarly, via a slightly different code path, the human command put rope in basket gets translated to the computer command %things<rope>.put( %things<basket> ). The Rope object thus found just happens to have a .put method by virtue of being Takable. (And this has been validated before the method is called.) Not only that, but the Basket object allows things to be put inside of it because it is a Container (that is, doing the Container role) and therefore being provided with an .add method.

From a sufficiently high-level perspective, all the common actions on things in the game are enabled by some class in the source code doing some role. "Program to interfaces", says the experienced OO programmer. This is what that's about. Several things in the game can be read, and that's because their object does the role Readable, giving them a .read method.

I bet you want to have a look at the source code. Here it is. If you're concerned that peeking might be considered cheating, you hereby have the author's assurance that learning the principles of programming takes precedence over not gaining inside information in the game. (Or you could just finish the game first, then read the source. ☺)

I'd recommend keeping the source around in a tab while reading the following sections. The program is subdivided in eight parts, and the corresponding sections pull the noteworthy chunky bits out of the source code cookie. The explanations don't contain any particular spoilers, but the source code itself does.

Predeclarations

Perl 6 programs are read through once by the compiler, from top to bottom. If you refer to a class or role before it's been defined, the compiler becomes confused and gives you an error. (In the future, Rakudo will give a compile-time error. Unfortunately, as of this writing it still gives a run-time error, which is worse.)

In order to avoid references to types that haven't been declared yet, we could be very meticulous about the ordering of our types, so that they always refer backwards to already defined types. For small programs, that's what we tend to do, and it works fine. (Sometimes, though, referral cycles between types are inevitable, and we can't use this solution.)

Or we do as in this script: we predeclare everything. A predeclaration looks like this: class Sign { ... }. The three dots there are to be pronounced "yadda yadda yadda", and here they mean that the class isn't defined here. But the class name is registered, and that's exactly what we wanted from the predeclaration. Further down in the code, we get to actually defining the insides of the class (as well as the roles it composes, etc).

In a way, the predeclarations form a "table of contents" of classes and roles. You'll note that they also are grouped suggestively: first comes a list of various roles that give objects various capabilities; then come the things themselves, then a few rooms/locations. As it happens, this is the order that these types are later introduced in the program.

Global variables

It is generally agreed-upon that global variables are Bad with a capital "B". Especially in a program that otherwise strives to be object-oriented, each global variable is like a defeat, a strike against all that we love and hold dear. Global variables can make people see red, and make them want to rip out all the code and rewrite it.

Why? How come globals are perceived so negatively? Well, it's easy: a global variable has a tendency to thread through the whole program, weaving in and out of classes/roles, forming unholy dependencies everywhere, and making the classes/roles far less re-usable. A class uses a global in a few places, and suddenly that class is dependent on that global. Want to move the class to a different file? That will be difficult; it's coupled to the global variable. Tough luck.

If they're so bad, why are they here, in a program supposed to teach good programmer values? Simply because the cure in this case is deemed to be worse than the desease. We're using these globals in various places — for example the %things hash — and encapsulating it properly would merely put it further away from everything else, resulting in more unnecessary code to get to it. Our smug sense of having implemented things right would be complemented by heaps of efficious code "doing the right thing" but amounting to nothing.

Strive to do things right, and by all means keep to principles. But be practical too, and know when to deviate from common rules of thumb. (And be prepared to take the ire from people who don't.)

Utility subroutines

Just as classes are concrete representations of things in our program that we talk about a lot, so subroutines often "condense" out of relations or circumstances that we find ourselves mentioning in a lot of places around the program. For example, whether there is any light in the current room is central to whether we can examine things, read things, or get any location descriptions at all. So we define there_is_light in one place, and then we can use and re-use the concept everywhere.

sub there_is_light() {
    my $there_is_sun = $room !~~ Darkness;
    return True if $there_is_sun;

    my $flashlight = %things<flashlight>;
    my $flashlight_is_here = player_can_see($flashlight);
    return True if $flashlight_is_here && $flashlight.is_on;

    my $fire = %things<fire>;
    my $fire_is_here = player_can_see($fire);
    return True if $fire_is_here;

    return False;
}

The sub has four parts.

The sun is considered to "be there" if the current Room is one that does not do Darkness. (That's locations that are outside and thus technically not rooms at all, or rooms that are sufficiently close to outside that the sun is shining in.) If the sun is shining, there is light.
If there's a flashlight here and it's on, there is light. Note that the player doesn't have to hold the flashlight or anything like that; we only require that player_can_see the flashlight. (This use of "see" has nothing to do with available light sources, so there's no risk of recusion here; can_see simply means that the thing is in the same room and not hidden in a box or something.)
If there's a fire in the room, that also counts as a light source.
Having exhausted those three options, we give up and return a False value, indicating that the room is without light.

A concept such as there_is_light builds on other concepts, such as player_can_see, which in turn build on other concepts. Unlike classes and roles, subroutines don't mind referring to each other forwards or backwards, so we don't have to think about predeclaring those. (When the Perl 6 compiler sees a name of something it hasn't seen before, it assumes that it's a subroutine call. Therefore, referring to subroutines that haven't been defined Just Works.)

Roles for things and rooms

The meat of the game; also, the most fun part to write. If something is meant to contain things, it's a Container. If something is meant to be picked up and carried around, it's Takable. All things in the game, directly or indirectly, do the general role Thing, which provides them with a name and a description, and an .examine method.

A couple of insights surprised me along the way. Starting out, I didn't suspect that Inventory could be a Container, although it's quite evident in retrospect. Shortly thereafter, it became clear that a Room is also a kind of Container. Then putting things in different places mostly becomes a matter of moving them between various Containers.

The roles sometimes interact in fanciful ways. A thing can be Openable but not a Container (like a Door), or a Container but not Openable (like the Inventory). But if it is both Openable and a Container, the .open method in Openable makes sure to call self.?on_open, a method that — voilà — Container just happens to have. (This method shows the contents of the just-opened container, if any.)

Things

Most of the work is done and represented in the roles above, which means that in this section when we construct classes for the things in the game, they can be as simple as class Sign does Readable {}. Empty class — all we needed to say was that the Sign is Readable.

Sometimes, though, we add attributes and methods "at the last minute"; notable among these are Doom, the only thing in the game that can kill the player. It has a ticking $!time_left attribute, and it can cause the whole cave to collapse. Talk about a powerful object.

At other times, a class will override a method from a role. The class Walls would've got a .read method from Thing, but instead it provides its own, which allows random quips to be read from the walls in the various rooms.

Directions

This is a pleasant case of several language elements cooperating. We define a couple of global variables for the various "standard directions" and their abbreviations; then we define a subtype Direction that restricts Str to only the standard directions. Finally we draw up a subroutine that knows how to get the reverse Direction from a given Direction.

Observe the trick in that subroutine: we only specify one half of the mappings of opposites, and then we "mirror" the hash with %opposites.invert, automatically creating the other half.

Rooms

A Room is a Container with a few extra methods to handle connecting rooms together, examining the room, and entering the room. A few rooms have a few event methods to trigger various events.

The biggest class in the game is the Hall, a room with its own little subgame. The room redefines how it presents its contents, and encourages the player to move disks between rods. This goes to show that even when we've built a set of roles for our classes to compose, there's still plenty of wiggle-room outside of that framework. At any point, we can let a class bloom out into something quite different. The roles are just there for the standard behaviors.

The game itself

The rest of the game consists almost entirely of the command loop with when statements to direct the player's commands to the right method on the right object.

Worth noting is how, in the first few when statements, we use proceed a lot to "massage" any movement commands into a Direction that we can then just use to look up the exit directly. This is slightly easier and more maintainable than one big expression for trying to match and then translate all the possible variants at once.

Sometimes we match commands with string values, but most of the time we match them with regexes. Here the regex syntax comes very much in handy, being able to both match the command string and pull out the pertinent information from it. That's what things like $<verb> are about in this piece of the code: when a regex matches, the special capture variable $<verb> will contain a verb, and we can process it based on that knowledge.

In that vein, it might be instructive to see how we are often working on "two levels", as often happens when interacting with a user.

my $verb = $<verb>;
if %verb_synonyms{$verb} -> $synonym {
    $verb = $synonym;
}

[...]

unless $thing.can($verb) {
    say "You can't $<verb> the $<noun>.";
    succeed;
}

On the game level, we prefer to handle $verb which has been de-synonymized and therefor is more regular. (pick up, pick and get all become take, for example.) But on the user interface level, we still refer to it as $<verb>, the thing we got originally. That way, the user who writes get basket will see You can't get the basket. and not You can't take the basket. A small thing, but usability is king.

Finally, the real magic resides in the method call .can above. This method checks ahead of time whether a given method exists in an object. So if the program wants to read a $thing, it first checks whether $thing.can("read") — if the answer is yes, it proceeds to call the .read method on $thing.

How can we call a method with a name that varies depending on what $verb we provided? We do it like this:

$thing."$verb"();

So, if $verb is read, we call $thing.read, etc. This is the core, the treasure, the butterfly of the game code. Most of the verbs are routed through that line, which maps a player's verb into a method.

Have you finished the game yet? If not, get to it! Good luck!

rakudo.org: Upcoming Rakudo releases

Tue, 19 Jul 2011 19:38:08 +0000

Over the past 6+ months the Rakudo developers have been working on a new version of the Rakudo compiler (code named “nom”) that uses a new object model implementation, has lots of bugfixes, and contains many performance enhancements over the existing version.

We plan to switch over all mainline development for Rakudo to this new branch within the next couple of weeks. As a result, the upcoming July 2011 releases of the compiler and Rakudo Star will be the last releases to use the older compiler codebase.

In early August, sometime prior to or concurrent with YAPC::EU, we plan to issue another “distribution release” based on the new compiler implementation, and will encourage newcomers to Rakudo Perl 6 to start with that distribution release. We haven’t yet decided if the new distribution will continue the “Rakudo Star” series of distribution releases or will constitute a new phase of releases.

Existing users of Rakudo Star can continue to use the July 2011 releases and have some comfort of stability with previous Star releases, but should be aware that we expect future releases of Rakudo to come from the new compiler. We will be developing a list of deprecations and guidelines for migrating programs from the older compiler to the new one.

We also plan to hold an online hackathon on July 30th on IRC #perl6 to focus on updating Perl 6 modules in the Star distribution to work with the new compiler implementation. In many cases the modules will not require any modifications to work, but there are some cases where changes to the Perl 6 language and/or the fixing of bugs in Rakudo may require a change to the module. Our goal and expectation is to have a module-compatible update for Rakudo Star in either August 2011 or September 2011.

In short, we expect there to be a flurry of “releases” of various Rakudo components over the next couple of months. If you’re looking for stability in that time, we recommend sticking with the July release of Rakudo Star while we iron out the new features. If you’re looking to learn Perl 6 or start a new project, we recommend starting with the new releases that will be coming out in August.

perl6.announce: Parrot 3.6.0 "Pájaros del Caribe" Released by James E Keenan

Tue, 19 Jul 2011 00:34:46 +0000

We are proud to announce Parrot 3.6.0, also known as "Pájaros del
Caribe". Parrot is a virtual machine aimed at running all dynamic languages.

Parrot 3.6.0 is available on Parrot's FTP site, or by following the
download instructions. For those who want to hack on Parrot or languages
that run on top of Parrot, we recommend our organization page on GitHub,
or you can go directly to the official Parrot Git repo on Github.

To clone the the Parrot Git repo into a directory called 'parrot', use
the following:

git clone git://github.com/parrot/parrot.git

If you want it to be in a directory other than 'parrot', then just give
that as a second
argument to clone:

git clone git://github.com/parrot/parrot.git parrot_foo

Parrot 3.6.0 News:

- Core
+ Class.add_vtable_override now allows you to overwrite an override
+ Integers, floats and strings can now be stored in lexical variables.
Previously, only PMCs could be stored in lexicals.
+ VTABLE_substr has been removed. VTABLE_substr_str has been renamed to
VTABLE_substr
+ Added a new PackfileView PMC type, an intended replacement for the
deprecated
Eval PMC.
+ The is_integer vtable on the String PMC now works for all string
encodings.
+ Unicode error messages on Win32 now work correctly.
+ A memory leak in IMCC was fixed.
- Languages
+ The snaphost of Winxed included with Parrot was updated to version
1.0.0
+ Winxed has migrated to Github: https://github.com/NotFound/winxed

The SHA256 message digests for the downloadable tarballs are:

a6ae5c4a1af3602043d1139a12ae9d4dfe2dd000250b1a76fc339bf4a004f8c7
parrot-3.6.0.tar.bz2
8280930a12a66c384b397b31482df2efc1fcc65efd723b279c71b1e283c2638f
parrot-3.6.0.tar.gz

Thanks to all our contributors for making this possible, and our
sponsors for supporting this project. The following people (in random
order) contributed to this release. Thanks!

Julian Albo, Matt Boyle, Vasily Chekalkin, chromatic, Will Coleda, Bruce
Gray, Brian Gernhardt, Michael H. Hind, James E Keenan, Bob Kuo, Andy
Lester, Jonathan "Duke" Leto, Peter Lobsinger, Patrick Michaud,
Christoph Otto, Felipe Pena, Francois Perrad, Gerd Pokorra, Kevin
Polulak, Tadeusz Sośnierz, Andrew Whitworth, Jonathan Worthington.

Our next release is 16 August 2011.

Did you think we were going to leave without a quotation? Wrong!

"El concierto de tantos pájaros distinto llegó a ser tan aturdidor,
que Úrsula se tapó los oídos con cera de abejas para no perder el
sentido de la realidad. La primera vez que llegó la tribu de Melquíades
vendiendo bolas de vidrio para el dolor de cabeza, todo el mundo se
sorprendió de que hubieran podido encontrar aquella aldea perdida en el
sopor de la ciénaga, y los gitanos confesaron que se habían orientado
por en canto de los pájaros."

The Release Manager will offer a public commendation and a suitable
prize to the first person who correctly identifies the source of this
quotation WITHOUT USING THE INTERNET. Send your determination of the
source to jkeenan at cpan dot org and testify that you did not use the
http(s) protocol, directly or indirectly, in researching the source.

Enjoy!

James E Keenan
jkeenan at cpan dot org; IRC #parrot kid51
Release Manager, Parrot 3.6.0

Carl Masak: June 29 2011: Text Adventure I

Mon, 18 Jul 2011 21:26:00 +0000

The adventure game is large and we will approach it in two stages. We introduce a kind of skeleton today, consisting of only the six rooms of the game.

Have a look at the code. Also try to run it (with the accompanying room-descriptions file).

Most of the code is built up of things we've seen before, but a few things merit special focus.

Directions

Note how we use the < > quoting environment to create both arrays and hashes without too much repetetive writing. It works in the hash case as well, because assigning a list to a hash container makes it interpret the list as a sequence of keys and values.

The subset declaration allows us to typecheck strings to make sure they belong to a predefined set.

Note the use of .invert to extrapolate the second half of the opposite directions from the first half.

Rooms

There are two roles, Thing and Room. That will make more sense tomorrow, when we actuall introduce things that aren't rooms.

In the game, the rooms are connected as shown on this map:

There are no coordinates placing the rooms anywhere. One could view them as sitting in a 2x3x2 grid, with coordinates to denote position, and neighbourhood relations being indicated by nearness, or something. It would be a bit fragile, but could be made to work.

What we're really interested in, however, is how the rooms are connected. During the course of the game, we'll also want to connect and disconnect rooms as new exits are presented and old ones are taken away. That's what the .connect and .disconnect methods do.

There are two methods whose names begin with on_. Per convention, these are callback methods, which are normal methods called under certain circumstances. For example, on_examine is called on anything we examine, allowing some special thing to happen when we examine something. Similarly, on_enter is called when we enter a room.

All the callback methods are called with a self.?on_something syntax — note the question mark — allowing the method to not be there without an error occurring. Most rooms and things will not have a given callback method.

The on_try_exit callback method is called before an exit is taken, and its return value (a true or a false value) is used to determine whether the player is actually allowed to take that exit. This is what happens in the Cave: a fire is preventing the player from exiting to the northwest, and it's on_try_exit which enforces that.

Walking around

The game loop takes care of all the ways a player might type a move command. All of the following ones work:

east
go east
e
go e

This works by massaging all of the above into a "standard form" (east, incidentally) and then acting on that. Liberal use of the proceed keyword allows the given statement to fall through several when cases if necessary.

Some massaging is also done with the directions in and out, which aren't considered primary directions, but more like aliases for some actual direction.

...and that's it for today. Tomorrow we'll add objects into the mix, filling the game with puzzles and interactions.

Patrick R. Michaud: New regex engine for nqp and nom, now passing 7K spectests

Thu, 14 Jul 2011 06:45:59 +0000

Nom and nqp now have a new regular expression engine (currently known as “QRegex”) that I’ve implemented over the past week.

As progress continued on the new “nom” branch of Rakudo since my last posting, it was becoming increasingly evident that regular expression support would end up being the next major blocker. I think we were all expecting that nom would initially use the same regular expression engine that nqp (and nqp-rx) have traditionally used. However, as I starting working on this, it began to look as though the amount of effort and frustration involved would end up being almost as large as what would be needed to make a cleaner implementation up front, and would leave a quite messy result.

So, last week I started on designing and implementing a new engine. Today I’m happy to report that nom is now using the new QRegex engine for its pattern matching, and that making a new engine was undoubtedly a far better choice than trying to patch in the old one in an ugly manner.

So far only nom’s runtime is using the new regex engine; the nqp and rakudo parsers are still using the older (slow) one, so I don’t have a good estimate of the speed improvement yet. The new engine still needs protoregexes and a couple of other features before it can be used in the compilers, and I hope to complete that work in the next couple of days. Then we’ll have a good idea about the relative speed of the new engine.

I’m expecting QRegex to be substantially faster than the old one, for a variety of reasons. First, it should make far fewer method calls than the old version, and method calls in Parrot can definitely be slow. As an example I did some profiling of the old engine a couple of weeks ago, and the “!mark_fail” method accounted for something like 60% or more of the overall method calls needed to perform the parse.

Qregex does its backtracking and other core operations more directly, without any method calls for backtracking. So I expect that this one change will reduce the number of method calls involved in parsing by almost a factor of 3. Other common operations have also eliminated the method call overhead of the previous engine.

The new engine also uses a fixed-width encoding format internally, which means that we no longer pay a performance penalty for matching on unicode utf-8 strings. This will also enable us to eventually use the engine to do matching on bytes and graphemes as well as codepoints.

I also found quite a few places where I could drastically reduce the number of GCables being created. In some cases the old engine would end up creating multiple GCables for static constants, the new engine avoids this. A couple of new opcodes will enable QRegex to do substring comparisons without having to create new STRING gcables, which should also be a dramatic improvement.

I’ve already prototyped some code (not yet committed) that will integrate a parallel-NFA and longest-token-matching (LTM) into QRegex, so we’ll see even more speed improvement.

And did I mention the new engine is implemented in NQP instead of PIR? (Although it definitely has a lot of PIR influence in the code generation, simply by virtue of what it currently has to do to generate running code.)

Ultimately I’m expecting the improvements already put into QRegex to make it at least two to three times faster than its predecessor, and once the NFA an LTM improvements are in it ought to be even faster than that. And I’ve already noted new places ripe for optimizations… but I’m going to wait for some new profiles before doing too much there.

Another key feature of the new engine is that the core component is now a NQP role instead of a class. This means that it’s fairly trivial for any HLL to make use of the engine and have it produce match objects that are “native” to the HLL’s type system, instead of having to be wrapped. The wrapping of match objects in the old version of Rakudo was always a source of bugs and problems, that we can now avoid. Credit goes to Jonathan Worthington for 6model, which enables QRegex to do this, and indeed the ability to implement the engine using roles was what ultimately convinced me to go this route.

While I’ve been working on regexes, Moritz Lenz, Will Coleda, Tadeusz Sośnierz, Solomon Foster, and others have continued to add features to enable nom to pass more of the spectest suite. As of this writing nom is at 244 test files and 7,047 tests… and that’s before we re-enable those tests that needed regex support. The addition of regexes to nom should unblock even more tests and features.

Some of the features added to nom since my previous post on July 2:
* Regexes
* Smart matching of lists, and other list/hash methods and functions
* Fixes to BEGIN handling and lexicals
* Implementation of nextsame, callsame, nextwith, callwith
* More introspection features
* Methods for object creation (.new, .bless, .BUILD, etc.)
* ‘is rw’ and return value type checking traits on routines
* Auto-generation of proto subs
* Junctions
* Backtraces

We’ve also done some detailed planning for releases that will transition Rakudo and Rakudo Star from the old compiler to the new one; I’ll be writing those plans up in another post in the next day or two.