Planet Perl Six

t4: Hex puzzle

2012-05-16T16:05:28+00:00

Despite a rather long absence from such matters, we haven't forgotten that we're still in the midst of reviewing Perl 6 Coding Contest 2011 code submissions. The t4 task was of the puzzle kind. See the post about counting t4 configurations for some overview of the static parts of the problem.

In this post, it's time to get dynamic and look at how to solve actual hex puzzles. The rest of the problem went like this:

A valid playing move on this board consists of sliding a piece along
its groove, either forwards or backwards. There are a few things which
are *not* allowed:

* Two pieces may never overlap and occupy the same location. (Note that
  the above three representations of the board actually denote the same
  board; the three sets of grooves intersect each other.)

* A piece may not "push" another piece as it slides; it is simply locked
  in by that other piece.

* A piece may not "jump" over another piece as it slides; it is restricted
  in its movement by the current positions of the other pieces.

* A piece may not rotate, move sideways, or otherwise leave its groove.

It's perfectly valid for a groove to contain more than one piece (as
long as they don't overlap).

For this problem, we will restrict ourselves to initial board configurations
with a piece at l1 and l2 (written as "l12"). We call this piece the "bullet".
The goal is to slide the bullet to l56, through a valid sequence of moves.
Thus, other pieces may have to be moved in order to get the bullet to l56.

                ..  ..  ..  ..  ..
              ..  ..  ..  ..  ..  ..
                ..  ..  ..  ..  ..
    start --> l1  l2  ..  ..  l5  l6 <-- goal
                ..  ..  ..  ..  ..
              ..  ..  ..  ..  ..  ..
                ..  ..  ..  ..  ..

Some initial configurations won't have a solution at all. (For example, the
bullet will never get through if there are other pieces in its groove.)

Write a program that accepts an initial board configuration on standard input.
The format looks as follows:

    d67
    i12
    l12
    u345
    v34

The program should reject any initial board configuration that has illegal
piece specifications, contains overlapping pieces, or lacks the bullet at
l12.

If there is possible solution, the program should output

    No solution.

Otherwise it should output one solution as a sequence of valid moves on
this format:

    u[345 -> 456]
    d[67 -> 23]
    u[456 -> 123]
    v[34 -> 23]
    l[12 -> 56]

A solution doesn't have to be minimal in the number of moves, but it may
count in your favor if it is. Even more so if it's minimal in the total
distance the pieces were moved. Arriving at a solution quickly is an
even more important success metric than minimal solutions.

I strongly encourage you to try a few problems; they're often quite exquisite. Each puzzle instance requires you to move the bullet from one side of the board to the other, but in order to do so, you must move aside other pieces, and in order to do that, you must move yet other pieces. Everyone who has ever tried their hand at Sokoban knows that this quickly grows non-trivial. (The problem of solving Sokoban puzzles has been proven to be NP-hard. I know of no such result for this hex puzzle, but let's just say it wouldn't surprise me.)

The necessity of moving pieces aside forms an implicit dependency tree. If that were all, these puzzles would be merely mechanical and boring. But what often happens is that the dependencies are not cleanly separated, and you have the additional problem of not tripping over your own pieces. Here's an example:

  ..  q4  ..  ..  t7
..  q3  j3  j4  t6  ..
  q2  e3  k3  k4  ..
l1  l2  e4  ..  c4  ..
  m1  m2  ..  d5  c5
..  ..  n3  n4  d6  c6
  ..  o2  o3  o4  ..

We trivially notice that in order to get the bullet (the l piece) across, we need to move aside the e piece and the c piece. But in the starting configuration, both of these pieces are blocked by other pieces. So we must move them first. And so on.

Reasoning about this kind of board takes place in a backwards manner. We figure out which pieces we have to move out of the way to be able to move the pieces we are really interested in. We follow the dependencies backwards until we bottom out.

In this particular problem, it's a bit worse than that: the two subproblems of "move aside the e piece" and "move aside the c piece" interact. Why? Because the c piece is blocked by the k piece, which is blocked by the e piece. So we can't just solve the subproblems in any order we like, we have to find a way to solve them that works. The whole thing has a feel of people attempting to execute a ballet number in a crowded elevator. It's exquisite. It's frustrating.

So, people solve this problem by reasoning backwards. This is the only working approach I've seen, and I've talked to quite a few people about this problem. How should a machine solve it?

Well, there's always the brute force approach. Try all possible moves from the starting configuration, and all moves from the configurations that result, and so on until you either (a) run out of new configurations to try, or (b) solve the problem. If you do this in a breadth-first way, i.e. you examine the new configurations in a first-in-first-out manner, you're also guaranteed to find a shortest possible solution first. (Shortest in terms of moves required.)

This is fine. It's slow, but it's fine. What some of our contestants ended up doing was to improve on this by using A* search to guide the search. A perfect fit, it seems, for this kind of problem. The extra complexity from upgrading from BFS to A* is paid back in spades by the problems being solved faster. A success story.

The other possible approach, which none of the submitted entries attempted, is to do the reasoning about blocking dependencies much in the way humans do. Though such a solution is certainly possible, I have a creeping suspicion that it would be far more complex than the A* approach. It's unclear how many special cases it would need to contain in order to work out all the dependencies. One gets a bit of extra respect at the pattern-matching and hypithetical-future algorithms in one's own brain when trying to code up the same things as a program.

Be that as it may. Have a look at people's solutions. Admire people's ingenuity on this one. Even though virtually everyone solves the problem in the same way, there's no end to how differently they factor their solutions.

Parrot 4.4.0 "Banana Fanna Fo Ferret" Released! by Andrew Whitworth

2012-05-15T20:25:03+00:00

Its existence guarantees nothing in itself, and the catalytic or
Promethean moment only occurs when one individual is prepared to cease being
the passive listener to such a voice and to become instead is spokesman, or
representative.

But it's important to remember the many dreary years when the prospect of
victory appeared quite unattainable. On every day of those years,
the "as if"
pose had to be kept up, until its cumulative effect could be felt.
-- Christopher Hitchens, Letters to a Young Contrarian

On behalf of the Parrot team, I'm proud to announce the 4.4.0 release of
Parrot "Banana Fanna Fo Ferret". Parrot (http://parrot.org/) is a virtual
machine aimed at running all dynamic languages.

Parrot 4.4.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/stable/4.4.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 4.4.0 News:
- Core
+ Most internal calls to libc exit(x) have been replaced with
Parrot_x_* API calls or PARROT_FORCE_EXIT
- Documentation
+ 'pdd31_hll.pod' made stable in 'docs/pdds/'.
+ Updated main 'README' to 'README.pod'
+ Updated various dependencies, e.g., 'lib/Parrot/Distribution.pm'.
+ Updated all 'README' files to 'README.pod' files.
+ Added 'README.pod' files to top-level directories.
- Tests
+ Update various tests to pull from new 'README.pod'
+ Updated 't/tools/install/02-install_files.t' to pull from new
'README.pod'
- Community
- Platforms
- Tools
+ pbc_merge has been fixed to deduplicate constant strings and
merge annotations segments

The SHA256 message digests for the downloadable tarballs are:

348ce13fc136afc74a7b50b094f64d8cb00f83f0cd3d59acc6fa4e63c824fa4d
parrot-4.4.0.tar.bz2
02495c0d11d3977a615bb76d3219f12bc6543b8cf12c596dfd5c35e98d95218a
parrot-4.4.0.tar.gz

Alvis Yardley (or a delegate) will release Parrot 4.5.0, the next scheduled
monthly release, on June 16th 2012. Subsequent release managers are to be
announced. A special thanks to our donors, contributors and volunteers for
making this release possible.

Enjoy!

--Andrew Whitworth

No such pipe, or this pipe has been deleted

2012-05-15T19:42:30+00:00

This data comes from pipes.yahoo.com but the Pipe does not exist or has been deleted.

Since the Hackathon…

2012-05-08T22:32:23+00:00

Last time I got around to writing here was while I was at the Oslo Hackathon. It was a truly great event: hugely productive, a great deal of fun and a real motivation booster too. I’d like to again thank Oslo.pm, and especially Salve, Rune and Jan, for thinking to organize this, and then making a superb job of doing so. Everything ran smoothly, there was lots of undistracted hacking time, and a couple of evening dinner and beer outings provided chance to enjoy time together as a team in real life, without the restricted bandwidth IRC normally enforces on us.

After the hackathon, it was right back to $dayjob, though in a fun way: teaching Git for a couple of days. Less fun was the couple of days after, which involved a lot of pain followed by a lot of noisy drilling at my local dental clinic. It took some days beyond that before I stopped feeling unusually tired… Anyways, things are back to normal again, or as normal as things ever tend to look for me… :-)

So, what’s been going on in Rakudo land since the hackathon? Lots and lots, as it turns out. Most public-facing, the April 2012 Rakudo Star release landed; moritz++ is to thank for getting the release out of the door this month. It’s the first distribution release to incorporate the bounded serialization work, which delivers the much-improved startup time. While I’ve talked about many of the compiler side improvements in previous posts, Rakudo Star also includes modules, and we had some exciting additions in this release too: Bailador (a Perl 6 port of Dancer), LWP::Simple and JSON::RPC.

Of course, one nice release out the door means time to start working on things to make the next one interesting. Here’s some of what to expect.

Following on from removing “lib” from the default @*INC, the current working directory (“.”) is also gone now. While they may be nice for development, they are not good defaults from a security perspective when you actually run stuff in a production environment. Making this a more comfortable removal, as well as supplying search paths through the PERL6LIB environment (as has been supported for a long while), the -I command line option and “use lib” have now been implemented, and will be in the next release. moritz++ is primarily to thank for this work.
I realized that now we have LEAVE and UNDO phasers, I could implement “temp” and “let”. “temp” saves away the current value of a variable, and arranges for that value to be restored when the block is left, so any changes made to it will be undone. “let” is a way of making hypothetical changes to a variable; if the block is left due to an exception or if it evaluates to an undefined value, the change will be rolled back just like “temp”, but a successful completion of the block will leave the value in place. Both are now in place.
Several days back, moritz++ asked me for some inspiration of what to hack on. Amongst my suggestions was working on tagged imports/exports. And sure enough, with me throwing the odd commit in now and then, we have ‘em working. Actually the main thing I did was improve the trait handling to support passing the tags to export; some work had been left undone awaiting real serialization, which we now have, so happily it was fairly straightforward to sort things out there.
pmichaud++ has been working on making things better if you build Parrot and Rakudo without ICU. Previously, any case-insensitive string match – even if it didn’t use anything beyond ASCII – would immediately die over a missing ICU. Now you can get away with that.
A while back I did a first cut implementation of “ff”, the flip-flop operator. It sorta worked, but we missed the “fff” form and, true to form, masak++ managed to find a bug to submit too. :-) Today I tossed the previous approach and re-did it, using a state variable for the underlying storage (since I did the first cut at it, the spec was updated to explicitly say that it should be done with a state variable). I also got the “fff” form in place. We now pass the vast majority of spectests for this feature.
A few days back, tadzik++ showed up with a snippet of code that ran insanely slowly. It was doing regex interpolation: / <$search_term> / or the like. Here, if $search_term is a string, it will be compiled to a regex and evaluated. Turns out that we weren’t caching that compilation with the string, though. So, if it had to scan or if you backtracked over the <$search_term>, then tried it again, you’d end up compiling it every time. A little caching later and things went rather faster. I was happy to have a profiler to hand; the call counts into the compiler very quickly showed what was up.
Our END phasers didn’t always run when you’d hope they did. In the last few days, moritz++ ensured they run on exceptional exits, and I made sure they run when execution is terminated using the “exit” function.
This evening, kboga++ showed up with a patch to turn Real, which ended up as a class during the early days of the “nom” development branch, into a role as it should be. This enabled custom numeric types. It also enabled us to run real-bridge.t, which brought and extra 200 spectests, pushing us nicely over the 22,000 passing spectests mark.
In various other fixes: //=, ||= and &&= now short-circuit properly, reduction meta-operators on list-associative ops do the right thing (pmichaud++), %*ENV propagates changes properly (tadzik++), ms// now works (moritz++) and enumeration types can now turn themselves into roles and be composed or mixed in to things.

There’s still over a week before the next compiler release, and I’m sure there will be some more things beyond this. It’s already looking like a nice bunch of improvements, though.

In other news, the work on QAST has also been chugging along. QAST is a replacement for PAST, which is a set of AST nodes together with something that maps them down to POST (Parrot Opcode Syntax Tree), which then becomes PIR code (which Parrot then turns into bytecode and runs). In essence, it’s part of the compiler guts. The not-quite-gone-yet old regex engine aside, PAST is the only significant part of our codebase that is still written in PIR (an assembly-ish language for Parrot). It also predates 6model, bounded serialization and many years of learned lessons. All that said, it gets a lot right, so much will stay the same.

QAST really is a simultaneous port to NQP, leaving out things we came to realize were bad or unrequired, adding much better 6model integration, unifying the notion of “ops” somewhat, improving native type support and taking advantage of native types during its implementation to get the AST nodes more compact, so we can save on memory during the compilation. Additionally, it will unblock masak++’s work on quasi splicing in macros, and is a step towards Rakudo targeting other backends being a reality rather than a nice idea. So, lots of win lies ahead…after the hard work of getting it landed. I’m hoping to make some significant steps forward on it during May.

Last but not least, masak++ and I will be heading over to the Bristol IT Megameet the weekend after next, giving a 30 minute joint talk on Perl 6, followed by a couple of hour tutorial. Looking forward to it – and hopefully I’ll be able to sneak a British ale in while I’m over there too. :-)

SQLite support for DBIish

2012-05-03T18:02:47+00:00

DBIish, the new database interface for Rakudo Perl 6, now has a working SQLite backend. It uses prepared statements and placeholders, and supports standard CRUD operations.

Previously the SQLite driver would randomly report "Malformed UTF-8 string" or segfault, but usually worked pretty well when run under valgrind. The problem turned out to be a mismatch between the caller's and the callee's ideas about memory management.

In particular, parrot's garbage collector would deallocate strings passed to sqlite3_bind_text after the call was done, but sqlite wants such values to stay around until the next call to sqlite3_step in the very least.

Fixing this mismatch was enabled by this patch, which lets you mark strings as explicitly managed. Such strings keep their marshalled C string equivalent around until they are garbage-collected themselves. So now the sqlite driver keeps a copy of the strings as long as necessary, and the SQLite tests pass reliably.

Currently it still needs the cstr branches in the nqp and zavolaj repositories, but they will be merged soon -- certainly before the May release of Rakudo.

Announce: Niecza Perl 6 v17 by Carl Mäsak

2012-04-30T14:22:40+00:00

Announce: Niecza Perl 6 v17

This is the seventeenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you still need a
binary for bootstrapping.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Minor changes]

The .Str methods of Set, KeySet, Bag, and KeyBag now just list the elements
(Solomon Foster).

Out-of-range digits in a radix conversion now warn appropriately (Solomon
Foster).

The [Z] operator now doesn't hang on empty inputs (Stefan O'Rear).

Cool.eval added (Solomon Foster).

[Major changes]

List comprehension syntax like ($_*$_ for 0..9) works now (Stefan O'Rear).

'no strict' (Stefan O'Rear).

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the stefanor@cox.net.
Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

[1] https://github.com/downloads/sorear/niecza/niecza-17.zip
[2] https://github.com/sorear/niecza

Meet DBIish, a Perl 6 Database Interface

2012-04-28T19:03:10+00:00

In the aftermath of the Oslo Perl 6 hackathon 2012, I have decided to fork and rename MiniDBI. MiniDBI is intended as a compatible port of Perl 5's excellent DBI module to Perl 6. While working on the MiniDBI backends, I noticed that I became more and more unhappy with that. Perl 6 is sufficiently different from Perl 5 to warrant different design decisions in the database interface layer.

Meet DBIish. It started with MiniDBI's code base, but has some substantial deviations from MiniDBI:

Connection information is passed by named arguments to the driver (instead of a single DSN string)
Different naming of several methods. There's not much point in having both fetchrow_array and fetchrow_arrayref in Rakudo. fetchrow simply returns an array or a list, and the caller decides what to do with it.
Backends only need to implement fetchrow and column_names, and get all the other fetching methods (like fetchrow-hash, fetchall-hash) for free.
Error handling from DB connection and statement handle are unified into a single row

The latter two changes brought quite a reduction in backend code size.

My plans for the future include experimenting with different names and maybe totally different APIs. When a language has lazy lists, one can simply return all rows lazily, instead of encouraging the user to fetch the rows one by one.

Currently the Postgresql and mysql backends support basic CRUD operations, Postgresql with proper prepared statements and placeholders. An SQLite backend is under way, but still needs better support from our native call interface.

Announce: Rakudo Star 2012.04 – a useful, usable, “early adopter” distribution of Perl 6

2012-04-26T14:19:13+00:00

On behalf of the Rakudo and Perl 6 development teams, I’m happy to
announce the April 2012 release of “Rakudo Star”, a useful and
usable distribution of Perl 6. The tarball for the April 2012
release is available from github.

In the Perl 6 world, we make a distinction between the language
(“Perl 6″) and specific implementations of the language such as
“Rakudo Perl”. This Star release includes release 2012.04.1 [0] of the
Rakudo Perl 6 compiler [1], version 4.3 of the Parrot Virtual
Machine [2], and various modules, documentation, and other
resources collected from the Perl 6 community.

Here are some of the major improvements in this release over the
previous distribution release.

* much improved startup time

* much more robust module precompilation

* autovivification for arrays and hashes is implemented again

* many phasers like PRE, POST and REDO are now implemented

* improved support for calling C functions and modelling structs and arrays
via NativeCall.pm6

* now includes modules URI, LWP::Simple, jsonrpc and Bailador (a Perl 6 port
of Dancer)

This release also contains a range of bug fixes, improvements to error
reporting and better failure modes. Many more exceptions are thrown
as typed exceptions.

Some notable incompatible changes from the previous release include

* the ‘lib’ directory is not included in the default module search path
anymore. You can manipulate the search path with the PERL6LIB environment
variable

* ‘defined’ used to be a prefix operator, and is now a regular subroutine.
This means you must updated code that relies ‘defined’ taking only one
argument. For example ‘defined $x ?? $a !! $b’ should be written as
‘$x.defined ?? $a !! $b’ or ‘defined($x) ?? $a !! $b’.

There are some key features of Perl 6 that Rakudo Star does not
yet handle appropriately, although they will appear in upcoming
releases. Some of the not-quite-there features include:
* pack and unpack
* macros
* threads and concurrency
* Unicode strings at levels other than codepoints
* interactive readline that understands Unicode
* non-blocking I/O
* much of Synopsis 9

There is a new online resource at http://perl6.org/compilers/features
that lists the known implemented and missing features of Rakudo Star
2012.04 and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the
programmer that a given feature isn’t implemented, but there are
many that we’ve missed. Bug reports about missing and broken
features are welcomed at.

See http://perl6.org/ for links to much more information about
Perl 6, including documentation, example code, tutorials, reference
materials, specification documents, and other supporting resources.
An updated draft of a Perl 6 book is available as
in the release tarball.

The development team thanks all of the contributors and sponsors
for making Rakudo Star possible. If you would like to contribute,
see , ask on the perl6-compiler@perl.org
mailing list, or join us on IRC #perl6 on freenode.

[0] https://github.com/rakudo/rakudo/blob/nom/docs/announce/2012.04.1
[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Announce: Rakudo Star 2012.04 - a useful, usable, "early adopter"distribution of Perl 6 by Moritz Lenz

2012-04-26T09:59:03+00:00

On behalf of the Rakudo and Perl 6 development teams, I'm happy to
announce the February 2012 release of "Rakudo Star", a useful and
usable distribution of Perl 6. The tarball for the February 2012
release is available from <http://github.com/rakudo/star/downloads>.

In the Perl 6 world, we make a distinction between the language
("Perl 6") and specific implementations of the language such as
"Rakudo Perl". This Star release includes release 2012.04.1 [0] of the
Rakudo Perl 6 compiler [1], version 4.3 of the Parrot Virtual
Machine [2], and various modules, documentation, and other
resources collected from the Perl 6 community.

Here are some of the major improvements in this release over the
previous distribution release.

* much improved startup time

* much more robust module precompilation

* autovivification for arrays and hashes is implemented again

* many phasers like PRE, POST and REDO are now implemented

* improved support for calling C functions and modelling structs and
arraysvia NativeCall.pm6

* now includes modules URI, LWP::Simple, jsonrpc and Bailador (a Perl 6
port of Dancer)

This release also contains a range of bug fixes, improvements to error
reporting and better failure modes. Many more exceptions are thrown
as typed exceptions.

Some notable incompatible changes from the previous release include

* the 'lib' directory is not included in the default module search
path anymore. You can manipulate the search path with the PERL6LIB
environment variable

* 'defined' used to be a prefix operator, and is now a regular
subroutine. This means you must updated code that relies 'defined'
taking only one argument. For example 'defined $x ?? $a !! $b' should be
written as '$x.defined ?? $a !! $b' or 'defined($x) ?? $a !! $b'.

There are some key features of Perl 6 that Rakudo Star does not
yet handle appropriately, although they will appear in upcoming
releases. Some of the not-quite-there features include:
* pack and unpack
* macros
* threads and concurrency
* Unicode strings at levels other than codepoints
* interactive readline that understands Unicode
* non-blocking I/O
* much of Synopsis 9

There is a new online resource at http://perl6.org/compilers/features
that lists the known implemented and missing features of Rakudo Star
2012.04 and other Perl 6 implementations.

In many places we've tried to make Rakudo smart enough to inform the
programmer that a given feature isn't implemented, but there are
many that we've missed. Bug reports about missing and broken
features are welcomed at <rakudobug@perl.org>.

See http://perl6.org/ for links to much more information about
Perl 6, including documentation, example code, tutorials, reference
materials, specification documents, and other supporting resources.
An updated draft of a Perl 6 book is available as
<docs/UsingPerl6-draft.pdf> in the release tarball.

The development team thanks all of the contributors and sponsors
for making Rakudo Star possible. If you would like to contribute,
see <http://rakudo.org/how-to-help>, ask on the perl6-compiler@perl.org
mailing list, or join us on IRC #perl6 on freenode.

[0] https://github.com/rakudo/rakudo/blob/nom/docs/announce/2012.04.1
[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Post-hackathon thoughts

2012-04-25T11:15:00+00:00

It has been a few days, and we have (sort of) landed.

I shall call this year's Oslo.pm Perl 6 Patterns hackathon a success, judging by how the participants seemed to enjoy themselves during the hackathon, and the inspired blogging afterwards.

Salve (sjn) and Rune (krunen) have, on behalf of Oslo.pm, delivered to standards that I am a bit worried that guests now will come to expect ;), and although I did not have capacity to help with much, I am happy with how everything turned out, except for one thing:

Damian caught a cold, and was essentially out of the loop for most of the hackathon.

Darnit.

But he delivered a kick-ass talk on Thursday, and was very helpful and a very good resource before he somewhat reluctantly accepted the thrown towel.

I did not do much hacking this year myself, but I managed to revive some of my Perl 6 skills. I even fiddled a bit with the cookbook (dormant for three years or so, and therefore slightly out-of-date in some respects). I hope to contribute more to that piece of documentation in the coming months.

The bonus if I manage to work my way through it, is that I will be pretty much up to speed with the language. Win!

My thanks go to all the participants of the Hackathon, without you, Perl 6 would be much the poorer!

開源之道

2012-04-24T12:40:05+00:00

(這是 Allison Randal 在 OSDC.tw 的演講中譯本。請參見原文及錄影。)

這幾年來，我慢慢覺得，我們參與開源社群，就像是在一條道路上並肩而行：這不僅讓我們成為更好的程式設計者，也讓我們通過與人合作，而成為更好的人。

您可以將它想成一條修行之道，讓身而為人的我們能夠不斷成長。接下來，我想談談我對開源世界的個人觀點，希望能與您分享。

首先，人是一切開源專案的核心。程式碼是很重要，但最核心的永遠是人。

人們透過各種不同的方式來參與專案：有人寫程式，有人寫文件，有人寫測試。而使用軟體的人，同樣也是專案裡不可或缺的一部分。

您的專案也許會用到別人開發的軟體，而因此接觸到上游的專案，或許偶爾也會向他們提出建議和修正。

又或許您開發的是一套程式庫或模組，提供給其他專案的人使用。此時，您就是他們的上游專案，他們也會用相同的方式來與您溝通。

所以，人們到底為什麼要做開源軟體呢？如果您想理解開源模式如何運作，這是一個很關鍵的問題。

許多人在日常工作中，可能已經常常和軟體打交道了。我們為什麼要花額外的心力，來參與開源專案呢？一部分的原因，是因為這能夠讓人迅速接觸到刺激、有趣的新鮮技術。

能夠與人分享，也是一個主因：透過與人分享，我們可以認識開源專案裡的同好，來提升彼此的樂趣。

投入開源專案的人，往往也帶著分享奉獻的精神。能夠伸出雙手幫助別人，是身而為人很重要的一部份。

除了這些內在因素，參與開源專案工作，也可以得到許多回報。其中一項，是獲得別人的敬重：當我們創造新的事物與人分享，進而吸引人們一同合作時，人們自然會認識我們的人品與才能，從而為我們自己帶來成就感。

換個角度來看，這也意味著：我們應當對於加入專案的人表示尊重，這樣人們才會願意繼續參與專案的活動。

欣賞別人的作品也很重要。當人們發表自己的作品，而您有機會與他們交流時，即使是一封簡單的電子郵件感謝函，說「您的專案對我很重要」，也足以營造出一種正向的文化，讓大家都能保有繼續創造的動力。

懂得讚美也很重要。當您介紹專案時，不要忘了讚賞您身邊的人，讓大家認識這些人是誰、做了多麼棒的貢獻，以建立社群的認同感。

之所以有那麼多人持續對開源專案保持興趣，其中一個原因是這樣的：在合力工作時，我們的能力會愈來愈強，能做的事也愈來愈多。

光用簡單的算數來想：如果我們有兩倍的人，至少就可以寫兩倍多的程式，有三倍的人就可以寫出三倍的程式。不過，我們的能力遠遠不止這些。

在一起合作時，我們可以透過彼此鼓勵，讓彼此變得更好更強大。當您看到其他人正在解決艱難的問題時，您不妨鼓勵他們，跟他們說：「你做得很好，而且我看得出來，你在未來會做得更棒。」

僅僅是透過談話和分享，您就可以為他人培力，讓對方變得更好。

還有一點就是，當許多人聚在一起的時候，每個人都有不同的能力。一起工作時，可能您知道專案需要的五樣東西，而其他人知道另外五樣東西，您們互補長短，就有了一整套技能足以完成專案，而這是單打獨鬥時做不到的事情。

所以在多人合作時，不只是生產力倍增，還可以達到互相加乘的效果。

另一件很重要的事，是鼓勵彼此放眼未來、看得更遠。我們可以給其他人靈感，幫助他們解決有意思的問題。有時，只要說「我有這個想法...」，別人就可以將它化為現實。

有些時候，您只要看看別人在做些什麼，然後告訴他們您想到的關鍵之處，不必自己跳下去實作，也可以幫助他們走得更好更遠。

在做開源工作時，我們得時常提醒自己，我們並不是孤身一人。由於需要和許多人合作，我們最需要注意的，就是不斷改進自己的溝通技巧。

我們經常會彼此溝通對未來的規劃，例如軟體專案的發展藍圖，以及我們的個人計劃，像是接下來想要實作哪些功能等等。

在開源社群中，我注意到一件事情：人們對如何做軟體往往有很好的規劃，可是卻由於缺乏良好的溝通，而讓彼此的計劃互相衝突。如果您朝向某個規劃埋頭開發，而沒有與人溝通的話，很可能會傷害到其他朝向不同方向開發的人。

我們就像一窩在蜂巢裡的蜜蜂，要經常發出嗡嗡聲，才能讓彼此持續發揮功能。

此外，我們還會不時討論技術問題，嘗試找出最好的解決方案。在面對技術問題的時候，人們可能會互相爭論、甚至大動肝火，讓事情難以獲得實質的進展。

所以，我們在工作過程裡，要逐漸學會接受各種各樣的可能性。對於您自己想到的解法，您當然應該持續努力，但也不妨對別人所提出的其他可能性，抱持開放的態度。

而在您自己的工作有所進展時，也可以透過各種通訊管道，讓大家知道您做了些什麼。發電郵、寫推特… 有很多方法能讓人們知道您的進度。

有時候我們可能會覺得害羞，或是不想被別人認為自己在吹噓。但其實事情完全不是這樣！多溝通對專案有好處，對專案裡的人也是好事，因為他們可以從您所作的事情裡學到東西。

溝通的另一個重點是問問題。有社群的好處，就是可能有人已經解決過您正在面對的問題。透過論壇或聊天室主動發問，可以為您省去很多時間。

同樣的道理，當別人想要學習時，您也可以認真回應，而不是對簡單的問題拋下一句「RTFM（去看該死的說明書）」就算了。

如果您回答「RTFM」，的確可以為自己省些時間，但是您一旦這麼做，同時也是在告訴別人說，他們一開始就不應該問問題。而這絕對不是您想要的效果，您要的是培養對方溝通的意願。

學著如何去給別人有幫助的答案，幫助他們一同走上這條開源之道，日後他們才能把這條路走得更長、更遠。

有些時候，批評別人是必要的。雖然我們對各種可能性抱持開放的態度，但針對特定的技術問題，確實可能有某種解法比其他的都要正確。即使如此，當您想要讓別人改變他們的看法，最好的方式是用友善的態度提出回應，對方才會用開放的胸懷來向您學習。

即使對方態度惡劣，也請保持優雅。難免有些人會對您很不客氣，但這也是參與開源的必經之路。有時候，臉皮厚一點也有好處。雖然有些人的溝通方式有待加強，但他們說的內容或許也有可取之處，您還是可以從中學到東西。

從這個角度來看，就算人們說話的時候不禮貌，您還是可以禮貌地回應他們。

溝通的另一部分不是說話，而是傾聽。有時我們須要做的，不是告訴別人我們的想法，而是靜靜地坐好，讓別人暢所欲言。

光是聆聽是不夠的，我們還需要有同理心。英文有句俗話說：「如果您真想瞭解某人的話，請穿上他的鞋走一哩路。」 — 或許只有這樣，您才能明白別人所經過的煎熬。

有些人以為，能夠從事開源軟體工作的人，個個都得是天才。事實絕非如此。的確有 Larry、Guido、Linus 這樣的人物，但其實任何一個專案，都需要各方面具有不同才能的人加入。

重要的是，無論您有多聰明，都要保持謙虛。因為只有謙虛的人，才能以開放的態度面對其他人，學會用新方法來做事。謙遜的心態，讓您能歡迎其他人加入您的專案。相反的，抱持驕傲自大的態度，就等於是在跟其他人說：「我不需要你們，我用自己的方法做事就夠了。」

也是因為謙遜，我們才能歡迎各種性別、各種文化的人加入社群，為開源軟體帶來多元而豐富的人才。

就像各個國家有不同的語言和文化一樣，相同的多元性，也體現在各式各樣的開源專案裡。舉例來說，Linux 社群、Perl 社群、Ruby 社群和 Python 社群，都各自用獨特的方式來交流合作。

只要我們懷著一顆謙卑的心，就可以看到自己專案所屬的社群並不是唯一的途徑，也才能夠欣賞其他社群裡的合作方式。

另外，做開源專案並不只是享受樂趣而已。樂趣當然是有，但同時也有責任。當您承諾參與一個專案時，您是讓雙肩扛上了重量。這是件好事，因為責任能讓我們進步，變成更好的人。

但是人生中還有其他的事情，像是您的伴侶、父母、孩子、職業等等。對於開源專案，我們可能會承擔一段時間的責任，但到了某天，我們可能會發現，自己不能再負起那麼多的責任了。

我們要意識到這是一個循環。一開始我們加入社群，然後逐漸負起越來越多的責任。但當人生到達某個階段之後，您總會逐漸減少所負的責任。這個過程完全是自然的，而且在專案的生命週期裡一定會發生。

所以我們不妨想想：「哪天我無法再付出那麼多心力的時候，誰來繼續我的工作呢？」

為了確保其他人能繼續我們的工作，我們可以創造出某種持續前進的過程：盡力教導與分享我們所學到的一切，同時也向其他人學習更多的事物。這是一個不斷吸收與分享知識的過程。

最後，當您在為開源工作的時候，請保持快樂吧，讓您的臉上帶著笑容，讓其他人分享您的喜悅！因為正是這種樂趣給予我們力量，讓我們能創造出偉大的事物。

您現在更快樂了嗎？:-)

Back from Oslo

2012-04-23T18:30:08+00:00

And so the hackathon is over. I haven’t experienced anything this awesome since I discovered how well bacon tastes with maasdamer.

I seem to be much more productive during hackathons if I have no talk to prepare. I’ve had lots of plans about it (see previous post), and while I haven’t started even half of them I think, I’m really satisfied with what was achieved.

On friday I looked a little bit into how Rakudo loads precompiled modules and why doesn’t it always work as expected. The easiest one to notice was Test.pm, which for some unknown reason was installed only as a source file, not a precompiled module. A simple fix in the Makefile, and suddenly every test file you ever run runs about 3.5 seconds faster. Suddenly it makes module development a lot less painful.

The other thing about module loading was that a lib/ directory was always at the very beginning of the @*INC array – that means that precompiling your modules (which usually puts them in blib/) before running your tests gave you… right. Absolutely nothing. How significant is that? Well, tests of Panda take just under 7 seconds when you have the modules precompiled, and almost 23 seconds when they aren’t. So that’s about 3 times faster test runs for your modules. How cool is that?

That solved pretty much all the issues and wtf’s I’ve experienced with module precompilation. It took me another commit on saturday to teach panda to precompile modules again. Then my mind wandered again around the idea of emmentaler, a service for automatic smoke testing of Perl 6 modules. A quick and dirty 100-lines long script is now able to download all the modules, test them and present the results in a bit silly, but definetely useful way. There’s still lots of things to be done about it, but the fact that you can test all the modules in about 20 minutes is something that wasn’t quite possible before. Huge success; I hope I can turn it into something that the whole community will benefit.

Saturday also brought a couple of fixes to Pod handling and the –doc command line switch. After a quick discussion with Damian Conway, sort of a “what does the spec really expect me to do”, I taught –doc to Do The Right Thing whatever argument you pass to it. So –doc=HTML will use Pod::To::HTML for Pod formatting, and whatever module you write is available to be used this way without changes neither in Rakudo nor in the module code itself. A small thing, but something that waited much too long to be done after GSoC. It should now be possible to extract documentation from modules in a simple, automatic way, possibly for inclusion on modules.perl6.org or somewhere? We’ll see about that.

The same day Marcus Ramberg worked on Pod::To::Text, in particular the .WHY handling part. He also contributed some patches to Rakudo itself, and thanks to him what –doc prints for documented subs and methods is actually Something Useful and not Something That’s Good That It Exists. Much appreciated!

Sunday brought some new fixes to Bailador::Test, which now does almost everything that Dancer::Test does, so some brave soul could actually start porting tests from Dancer to Bailador to see what blows up. I’ve also spent plenty of time (mostly compilation time) chasing small things like a few-months-old panda bug, which actually turned out to be a Rakudobug, making perl6 –doc behave a bit more like perldoc, and less like “what the hell does this beast do”, and some other small things.

Besides hacking there was also plenty of opportunities to walk around Oslo, try some delicious food and beer and meet people mostly known from IRC before. Social aspect of the Hackathon was probably the best part of it, even given the fact that the coding aspect was probably the best one I’ve ever experienced. This wouldn’t have been possible without the work that the organizers have put into the event. It was absolutely perfect. Thank you, Oslo.pm! And thank you, sjn for giving me the opportunity to be there. It was a great hackathon and a time well spent. I hope to see you guys sooner than later. And if later than sooner, let it be the next hackathon in Oslo, for there’s no way that I’ll let this one happen without me.

Perl 6 Hackathon in Oslo: Report From The Second Day

2012-04-23T16:24:30+00:00

Second day of the Perl 6 Patterns Hackathon. My plans to get the rest of placeholders and prepared statements working in the Postgresql backend for MiniDBI succeed about 10 minutes after midnight. I just wanted to give them a very quick try before going to bed, and was successful. Then I went to sleep.

It was night, and it was morning. Second day.

Next I wrote an SQLite backend for MiniDBI. It blocked on missing features in our native call infrastructure, on which arnsholt worked in parallel. So I haven't had a chance to try the SQLite backend yet. It probably requires some substantial amount of work before it will run, but at least it compiles.

I also investigated prepared statements and placeholders for the mysql backend. This is much less straight forward, because it requires filling in members of structs, not just function calls. This by itself wouldn't be much a problem, our native call infrastructure supports that. The problem is that it's a struct of mixed "private" and "public" members, so modelling the structure in Perl 6 requires modeling private data of the mysql client library. While possible, I don't find it desirable, because it is rather fragile.

Another notable event was the hacking dojo, where about 8 of us collaborated to write a roman numeral conversion, using pair programming, and fixed cycles of first writing a failing test, then getting it to run in the simplest possible way, and finally refactoring it. It was quite an interesting and fun experience.

I spent much of the rest of the hackathon discussing things. For example Patrick Michaud gave a quick walk through of how lists and related types are implemented and iterated in Rakudo.

In the evening we had very tasty Vietnamese food, and generally a good time.

Again it was a very productive and enjoyable day, and I'm very grateful for being invited to the Hackathon.

Speed up by a factor of 6 million

2012-04-22T21:39:00+00:00

By the end of March, I received an email saying this:

$ time perl t4.pl
total: 4783154184978

real    0m0.185s
user    0m0.176s
sys     0m0.004s

(requires a perl with 64bit integers)

There was a t4.pl file attached.

You may recognize the total that the program prints out is the total number of t4 configurations, the same number that it took my C program two weeks to calculate on a decent box. So somehow, Salvador Fandino, perl.org blogger and occasional reader of my blog, managed to find way to arrive at the answer 6 million times as fast.

Well, that's interesting. To say the least.

Maybe I should be super-embarrassed. Maybe my cheeks should cycle through previously un-attained shades of crimson as I ponder the fact that my program was 6 million times as slow as someone else's. Ouch! But, I dunno. I don't really see it that way. I got to write about something I care about. Salvador++ cared enough to improve on my methods. The world is a better place. Blogging is cool — I learn stuff. Prestige doesn't much enter into it — the next time I'll have a better tool in my toolbox.

So, let's investigate this new tool, and how it's better.

First off, to get a factor-6e6 speedup, you don't apply some simple optimization somewhere; you use a different method. Salvador's code doesn't try to enumerate all the configurations, it just gets at the number. Which makes a lot of sense in retrospect, since we're not using the individual configurations for anything. My program arrives at each individual configuration, but then just throws it away immediately. Wasteful.

Salvador's blog post is as brief as his email. But let's copy the code over here and talk about it a bit:

#!/usr/bin/perl

use strict;
use warnings;

my $tab = <<EOT;
-----xxx
------xx
x-----xx
x------x
xx-----x
xx------
xxx-----
EOT

my $vertical = index $tab, "\n";
my $diagonal = $vertical + 1;

my $acu = { $tab => 1 };

for my $ix (0 .. length($tab) - 1) {
    my %next;
    while (my ($k, $c) = each %$acu) {
        my $s = substr($k, 0, 1, '');
        $next{$k} += $c;
        if ($s eq '-') {
            my $k1 = $k;
            if ($k1 =~ s/^-/x/) { # horizontal xx
                $next{$k1} += $c;
                if ($k1 =~ s/^x-/xx/) { # horizontal xxx
                    $next{$k1} += $c;
                }
            }
            $k1 = $k;
            if ($k1 =~ s/^(.{$vertical})-/${1}x/os) { # vertical xx
                $next{$k1} += $c;
                if ($k1 =~ s/^(.{$vertical}x.{$vertical})-/${1}x/os) {  # vertical xxx
                    $next{$k1} += $c;
                }
            }
            $k1 = $k;
            if ($k1 =~  s/^(.{$diagonal})-/${1}x/os) { # diagonal xx
                $next{$k1} += $c;
                if ($k1 =~ s/^(.{$diagonal}x.{$diagonal})-/${1}x/os) {  # diagonal xxx
                    $next{$k1} += $c;
                }
            }
        }
    }
    $acu = \%next;
}

my ($k, $c) = each %$acu;
print "total: $c\n";

The code is wonderfully idiomatic and to-the-point. Here are a few highlights, as I see them:

The board is a string. It's one-dimensional, but it plays a 2D array on TV. Some cute regexes then do matches on it according to this 2D representation.
We've "compressed" the hexagonal aspect of the board into a rectangular view. You know brick walls? On every other level the bricks are "between" those on the levels above/below. It's like they have half-valued x coordinates. This board representation removes the halves and just puts the bricks right on top of each other. It's bad for building walls, but useful for memory layout. It does mean that one of the diagonals on the hex layout becomes a vertical in the rectangular layout.
The script "munches" through the board, eating it one character at a time. In a very real way, this program solves the problem by eating it.
At each point it finds an empty location, it tries to put all kinds of 2-pieces and 3-pieces at that location. It diverges into all alternatives, keeping track for each alternative what locations it's used up.
The alternatives will then converge naturally as the same half-munched board shows up in various alternative paths. The script just needs to keep track of multiplicity of each alternative.
By the time we've muched the whole board down to an empty string, everything will have converged, so the multiplicity of the empty board will magically equal all possible ways to munch up the original one.

The program does far too much destructive updating for my tastes. I realize when I look at it that I no longer "think" in terms of these destructive updates. But it does it so successfully and idiomatically, that I find it difficult to list it as a disadvantage. Maybe it's a Perl 5 thing. Constructs like s/// are terribly convenient, and their default is to mutate things. (Even though Perl 5.14 adds /r for non-destructive substitution).

I was curious how this script would look (and perform) in Perl 6, so I wrote a straight port of it, trying to stick to the original as closely as possible:

my $tab = join "\n", <
    -----xxx
    ------xx
    x-----xx
    x------x
    xx-----x
    xx------
    xxx-----
>;

my $vertical = index $tab, "\n";
my $diagonal = $vertical + 1;

my %acu = $tab => 1;

my $vertical_xx = eval("/^ (. ** $vertical) '-'/");
my $vertical_xxx = eval("/^ (. ** $vertical 'x' . ** $vertical) '-'/");
my $diagonal_xx = eval("/^ (. ** $diagonal) '-'/");
my $diagonal_xxx = eval("/^ (. ** $diagonal 'x' . ** $diagonal) '-'/");

for ^$tab.chars {
    my %next;
    for %acu.kv -> $k, $c {
        my $s = $k.substr(0, 1);
        my $k0 = $k.substr(1);
        %next{$k0} += $c;
        next unless $s eq '-';
        my $k1 = $k0;
        if $k1.=subst(/^ '-'/, 'x') ne $k0 { # horizontal xx
            %next{$k1} += $c;
            my $k2 = $k1;
            if $k2.=subst(/^ 'x-'/, 'xx') ne $k1 { # horizontal xxx
                %next{$k2} += $c;
            }
        }
        $k1 = $k0;
        if $k1.=subst($vertical_xx,
                      -> $/ { $0 ~ 'x' }) ne $k0 { # vertical xx
            %next{$k1} += $c;
            my $k2 = $k1;
            if $k2.=subst($vertical_xxx,
                          -> $/ { $0 ~ 'x' }) ne $k1 { # vertical xxx
                %next{$k2} += $c;
            }
        }
        $k1 = $k0;
        if $k1.=subst($diagonal_xx,
                      -> $/ { $0 ~ 'x' }) ne $k0 { # diagonal xx
            %next{$k1} += $c;
            my $k2 = $k1;
            if $k2.=subst($diagonal_xxx,
                          -> $/ { $0 ~ 'x' }) ne $k1 { # diagonal xxx
                %next{$k2} += $c;
            }
        }
    }
    %acu := %next;
}

say "total: %acu.values()";

Ugh! This script is longer than the Perl 5 version, and it looks messier, too. A few factors contribute to that. First, you can't just do s/// in Rakudo in an if statement. (You can in Niecza, though.) Second, there are problems with <atom> ** $repeats, and I got to submit two tickets about that, and then do a workaround with the evals you see above. (Aah. Feels like the old days.)

Furthermore, jnthn++ could put this program into the profiler, and get two optimizations out of it. It went from 40s on my machine, to 37s.

But in the end, I felt that my straight-port version suffers from not playing off Perl 6's strengths. So I wrote a version that leans more towards immutability and closures.

my $tab = join "\n", <
    -----xxx
    ------xx
    x-----xx
    x------x
    xx-----x
    xx------
    xxx-----
>;

my $vertical = index $tab, "\n";
my $diagonal = $vertical + 1;

my %acu = $tab => 1;

sub make_substituter($rx) {
    return sub ($tab) {
        my $newtab = $tab;
        return $newtab
            if $newtab.=subst($rx, -> $/ { $0 ~ 'x' }) ne $tab;
    };
}

sub make_2x_substituter($rx) {
    return sub ($tab) {
        my $newtab = $tab;
        return $newtab
            if $newtab.=subst($rx, -> $/ { [~] $0, 'x', $1, 'x' }) ne $tab;
    };
}

my @pieces = 
    make_substituter(rx/^ ('') '-'/),
    make_substituter(eval("/^ ({'.' x $vertical}) '-'/")),
    make_substituter(eval("/^ ({'.' x $diagonal}) '-'/")),
    make_2x_substituter(rx/^ ('') '-' ('') '-'/),
    make_2x_substituter(eval("/^ ({'.' x $vertical}) '-' ({'.' x $vertical}) '-'/")),
    make_2x_substituter(eval("/^ ({'.' x $diagonal}) '-' ({'.' x $diagonal}) '-'/"));

for ^$tab.chars {
    my %next;
    for %acu.kv -> $k, $c {
        my $s = $k.substr(0, 1);
        my $k0 = $k.substr(1);
        %next{$k0} += $c;
        next unless $s eq '-';
        for @pieces -> &piece {
            if &piece($k0) -> $newtab {
                %next{$newtab} += $c;
            }
        }
    }
    %acu := %next;
}

say "total: %acu.values()";

Hmm. The loop is shorter now, but at the cost of some abstractions in other places. It's an improvement on my first version, but I don't really feel I got close to the succinctness of Salvador's Perl 5 version here either. (And this version runs slower, predictably. Something like 52s on my machine.)

I'm pretty sure it's possible to make even more idiomatic versions. This is a large enough problem to make things interesting. I encourage others to try.

Revenge of the Oslo hackathon

2012-04-22T21:05:00+00:00

A month of blog silence. Ouch. Looking back, the three reasons I can see for my absence from blogging are work, work, and work.

So I saw moritz, jnthn, and pmichaud blog about the weekend, and I must've been too shell-shocked to think to do the same. sjn++ woke me up from my reverie by asking me outright.

So, here goes.

Oslo. We had a hackathon there.

This is the second time. The first time was in 2009, and was quite possible the best hackathon ever, in the history of Perl 6 hackathons. (Or, let's say, certainly among the top 5.)

I haven't fully processed this one, but it's not too early to say this: this one beat the last one.

My weekend, in brief:

jnthn and I arrived Oslo on the Thursday, and watched Damian Conway giving and awesome presentation in a bar. No-one combines geek jokes, almost-accurate physics, and Perl programming like Damian.
On the Saturday, I spent about an hour introducing a group of newcomers to Perl 6, language and culture. Fun!
infosophy++ (Geir Amdal) liked my tote script so much that he did something I never got around to doing: he published it as a module. As part of this, he added back a bunch of IO methods that got lost in the ng→nom transition, and also added spectests for them. As part of this, he became the 100th developer in the perl6 organization.
sergot++ (Filip Sergot) and I built a presentation framework. It's called Sambal, and it turns a DSL into a PDF. I'm happy and proud over how long we managed to get with only two days of work.
sergot++ and I also spit out a Text::Markdown module. It's in early stages yet, but it already services Sambal in its slide generation. It's an easy addition to have its objects model serialize to HTML, too.
I asked whether BEGIN should trigger immediately inside a quasi, or whether it should trigger only after macro application. People around the hackathon table suggested that we should have a QBEGIN that did the latter. I felt it was a singularly bad idea, so I asked TimToady. He suggested the same. I exploded. Then I decided not to listen to anyone, and just implement it in the way that turned out to be natural and convenient. pmichaud joked that he should have adopted that approach long ago with respect to implementing Perl 6.
(But seriously. If you want to execute a BEGIN block at macro-parse time, put it outside of the quasi. If you want to execute it at macro-apply time, put it inside of it. We don't need a Q*bert BEGIN.)
We discussed what s/// should evaluate to. No real consensus. :-(
On the Sunday, we tried a coding dojo (hosted by infosophy++), implementing a roman numerals Int -> Str converter in Perl 6. It led to interesting discussions, and many of us had useful insights in collaborative coding and small-step iterative development.
frettled++ took some nice pictures.
jnthn++ and pmichaud++ evolved a plan for the new QAST redesign, which will enable the next step in the macros grant. jnthn++ invited me to write some tests on this for great success. It looks doable; I'll dig into this during the next week. As I do this, I can also write tests for my new QAST::Unquasi node type.

If Oslo.pm ever hosts a third hackathon, my expectations will found be in geostationary orbit. No chance in the world I'd miss it.

Oslo.pm Patterns Hackathon pictures

2012-04-22T16:59:00+00:00

Hacking: sjn, moritz (back), pmichaud, jnthn, masak, tadzik, sergot, bjarneh

Hacking: imarcusthis, infosophy

Posing, back to front, left to right:
infosophy, moritz, arnsholt, sjn, krunen, masak
tadzik, jnthn, frettled, pmichaud

Perl 6 Hackathon in Oslo: Report From The First Day

2012-04-22T05:10:48+00:00

Yesterday I arrived in the beautiful city of Oslo to attend the Perl 6 Patterns Hackathon. Yesterday we visited a pub, had great discussions, food and beverages, and generally a very good time.

Today we met at 10 am, and got straight to hacking. We are located in an office in the 6th floor of a big building, with a nice view over the center of town, harbor, and even the Holmenkollen.

I worked on the backtrace printer, which in alarmingly many cases reported Error while creating error string: Method 'message' not found for invocant of class 'Any', which wasn't too helpful.

It turns out there were actually two causes. One was a subtle error in the backtrace printer that was triggered by stricter implementation of the specification, which was easy to find. The second bug was harder to find, considering that you don't get easily get backtraces from errors within the backtrace printer. In the end it was the usage of a code object in boolean context, which turned out to be harmful. Because regexes are also code objects, and in boolean context they search for the outer $_ variable and try to match the regex against it. Which failed. Hard to find, but easy to fix.

My second big project today was database connectivity. Part of it was pestering Jonathan to fix the issues that arose from module precompilation mixed with calling C modules, and testing all the iterations he produced. I'm happy to report that it now works fine, which speeds up development quite a bit.

I also fixed the postgres driver. The root cause for the failing tests turned out to be rather simple too (a missing initialization), so simple that it's embarrassing how long it took me to find out. On the plus side I improved the code quite a bit in passing.

So now all tests in MiniDBI pass, which is a nice milestone, and an indication that we need more tests.

Tomorrow I plan to change the postgres driver to use proper prepared statments.

But the real value of such hackathon comes from interacting with the other hackers. I'm very happy about lots of discussions with other core hackers, as well as feedback and patches from new users and hackers.

At this occasion I'd also like to thank the organizers, Salve J. Nilsen, Karl Rune Nilsen and Jan Ingvoldstad. It has been a great event so far, both fun and productive. You are doing a great service to the Perl 6 community, and to the hackers you have invited.

Hackathoning in Oslo

2012-04-21T21:39:38+00:00

I’m in Oslo with a bunch of Perl 6 folks. It’s great to see old friends and meet some new ones – and we’re having a highly productive time. After a wonderful evening of tasty food and lovely beer (amongst others, a delicious imperial stout) yesterday, today has been solidly focused on Getting Stuff Done.

I’m a little tired now, so here’s just a quick rundown of some of what I’ve been up to.

Since bounded serialization landed, we’ve had a few issues with pre-compilation of MiniDBI, the database access module. I’ve now tracked down all of the remaining issues there, fixed them. and happily moritz++ has been hacking lots on improving the module in other ways too. The MySQL and Postgres drivers now pass all the tests we have for them, which is some nice progress.
I’ve been answering a few questions for arnsholt++, who has picked up the Zavolaj (native calling) module where I left off, adding support for passing/returning arrays of structs, structs pointing to arrays and various other permutations. This will greatly improve the range of C libraries that can be used with it.
I had a design session with pmichaud++ on QAST, the successor to our current AST. The new nodes will integrate far better with 6model and bounded serialization, give us better native type handling and be much more memory efficient due to being able to use natively typed attributes in them. This is also a key part of our work towards getting Rakudo up and running on an extra back end.
After that, I got the nodes fleshed out somewhat, and have started a little work on QAST::Compiler too. It’s underway!
I also spent some time in the ticket queue and fixed a bunch of Rakudo bugs: constant initializers containing blocks now work out, state declarations together with list assignment work, $.foo/@.foo/%.foo now contextualize as they should, and :i now also applies to interpolated variables.

So, lots of stuff – and that’s just the things I’ve been directly involved with. It’s nice to be a part of this hive of activity…and tomorrow there’s another day of this! Catch you then. :-)

ABC module now works on Rakudo and Niecza

2012-04-19T18:29:15+00:00

I haven’t had many tuits to work on Perl 6 lately — except this week I needed to have three pages of transposed single-line sheet music ready to go for tonight’s rehearsal. The ABC module (particularly the abc2ly.pl script to convert ABC to Lilypond) was the obvious choice to use. However, to make it work well, I needed to add ties, slurs, multi-measure rests, fermatas, and text messages, as well as fixing the key change and meter change code. While I was at it, I went ahead and got all the tests passing again under Rakudo (version 2012.02-173-gb13c517, the latest won’t build on my Mac) and for the first time, Niecza (latest source from github only, I actually added a small patch in the process).

It feels so good to have this working under both compilers!

t3: Addition chains

2012-04-14T21:24:00+00:00

<arnsholt> Heh. NP-complete problems in a
competition. That's just mean ^_^

Ok, we're in the midst of reviewing Perl 6 Coding Contest 2011 code submissions, and the turn has come to the third task: addition chains.

For a positive integer N, an addition chain for N is a sequence
starting with 1, each subsequent element being the sum of two
earlier elements (possibly the sum of the same element twice),
and ending with N.

For example for N = 9, this is a possible addition chain:

    (1, 2, 4, 5, 8, 9)

because 2 = 1 + 1, 4 = 2 + 2, 5 = 1 + 4, etc.

But a minimal solution would be:

    (1, 2, 3, 6, 9)

Write a program that reads numbers N from standard input, one per line,
and outputs a minimal addition chain like the one above.

Sometimes there will be several possible solutions of minimal length.
That's fine; just pick one of them.

Addition chains are interesting from a computing standpoint, because of a multiplication technique called addition-chain exponentiation, by which you can use an addition chain for a certain N to do a minimum number of multiplications; the addition chain implicitly encodes a sequence of multiplications to perform. So there's a genuine interest in finding shortest addition chains.

This is a hard problem. Finding addition chains is easy, but finding a minimal addition chain is not. Depsite arnsholt's quote above, it hasn't been proven NP-complete. Slightly more general problems have, but not this exact one. We know it's tricky, though.

Wikipedia has this to say about the problem: "There is no known algorithm which can calculate a minimal addition chain for a given number with any guarantees of reasonable timing or small memory usage." That's what we're looking for in this contest: problems that are easy to state, and that look quite straightforward, but that have hidden depth.

Someone may look at the problem and think "aha! dynamic programming!" — but, alas, as Wikipedia patiently explains:

Note that the problem of finding the shortest addition chain cannot be solved by dynamic programming, because it does not satisfy the assumption of optimal substructure. That is, it is not sufficient to decompose the power into smaller powers, each of which is computed minimally, since the addition chains for the smaller powers may be related (to share computations). For example, in the shortest addition chain for a¹⁵ [...] the subproblem for a⁶ must be computed as (a³)² since a³ is re-used (as opposed to, say, a⁶ = a²(a²)², which also requires three multiplies).

This is probably why the problem looks approachable, because it sort of feels like a dynamic-programming problem. But it ain't.

People sent in solutions. Go check them out.

I was a bit concerned that people might find Knuth's solution and just transcribe it into Perl 6. (Which would've been OK, if a bit boring if everyone did it.) But no-one did that; instead, people started from well-known algorithms, either breadth-first or depth-first search.

Perhaps the most remarkable things that can be recounted about the solutions are the cases where they deviate from correctness in various ways. One solution produced the right results for the first 76 chain lengths, but with N = 77, it went awry due to internal optimizations which turned out to be less than innocent. (The first rule of optimaztion? "Make sure you don't get caught.")

Then there were two submitted algorithms that generated Brauer chains. "What's a Brauer chain?", I hear you asking. Hold tight and I'll tell you. A Brauer chain is an addition chain where each new element is formed as the sum of the previous element and some element (possibly the same). Thus, of the two examples from the description,

(1, 2, 4, 5, 8, 9)

and

(1, 2, 3, 6, 9)

The latter is a Brauer chain, but the former isn't, because you can't get 8 by summing 5 and some element in the chain.

The task is to generate minimal addition chains. If some algorithm looks only among the Brauer chains, will it ever omit some shorter chain from its search? The answer, it turns out, highlights exactly why I like mathematics.

A Brauer-based algorithm will fail the first time at N = 12509. (See this reference, provided by hakank++).

Now, you might of course argue that failing at N = 77 is more wrong than failing at N = 12509.

Sheldon: More wrong? "Wrong" is an absolute state and not subject to gradation. Stuart: Of course it is! It's a little wrong to say a tomato is a vegetable. It's very wrong to say it's a suspension bridge.

More precisely, there are infinitely many N for which no Brauer chain is minimal. 12509 just happens to be the smallest one.

This task, understandably, is a tricky one to judge. We've tried to go easy on the contestants (and non-contestants) in the reviews. After all, the problem is hard.

Now, who wants to translate Knuth's solution to Perl 6? 哈哈

Oslo.pm Perl 6 Patterns Hackathon 2012

2012-04-13T10:41:00+00:00

In one week (2012-04-20 – 2012-04-22), a bunch of bright people will attend the Perl 6 Patterns Hackathon in Oslo.

Perl 6 – both its specification and implementations – will become one of the Great Ones. I admit that we are not quite there yet, but to me, Perl 6 is a language for the long term.

In one of the meetings krunen, sjn and I had this winter, we discussed this, and how to get there. I hope the Perl 6 Patterns Hackathon will contribute significantly, and I also hope that in 30-60 years, we will look back and be happy about most of the choices made around this time! I truly believe that we can get there!

Call for sponsor matching

In this spirit, Ingvoldstad IT decided to sponsor this hackathon with NOK 5,000 (≈ EUR 650), and I hereby call for other companies who develop software* to match this amount. Oslo.pm will put those money to good use!

I hope to see you in Oslo in a week!

* I did not write "Perl" here, because I think that is largely irrelevant. Perl 6 is an amalgam of many of the most interesting programming language features, and offers a very compelling path to those who want a next-generation language after the current versions of e.g. Perl, Python, and Ruby.

Rakudo Hack: Dynamic Export Lists

2012-04-13T03:59:43+00:00

Rakudo's meta programming capabilities are very good when it comes to objects, classes and methods. But sometimes people want to generate subroutines on the fly and use them, and can't seem to find a way to do it.

The problem is that subroutines are usually stored (and looked up from) in the lexical pad (ie the same as my-variables), and those lexpads are immutable at run time.

Today I found a solution that lets you dynamically install subroutines with a computed name into a module, and you can then use that module from elsewhere, and have all the generated subroutines available.

module A {
    BEGIN {
        my $name = 'foo';
        my $x = sub { say 'OH HAI from &foo' }
                but role { method name { $name } };
        trait_mod:<is>(:export, $x);
    }
}

Inside the module first we need a BEGIN block, so that the is export trait will run while the module is being compiled, and thus knows which module to associate the subroutine to.

Next comes the actual code object that is to be installed. Since the export trait inspects the name of the subroutine, we need to give it one. Doing that dynamically can be done by overriding the name method, here by mixing in a role with such a method into the code object.

Finally comes the part where the export trait is applied. The code here uses knowledge of the calling conventions that hide behind a trait.

A different script can then write

use A;
foo();

And access the dynamically exported sub just like any other.

In future there will hopefully be much nicer APIs for this kind of fiddling, but for now I'm glad that a workaround has been found.

Announce: Perlito version 9; Perl5 and Perl6 compilers by Flavio S. Glock

2012-04-11T05:55:46+00:00

Perlito is a compiler collection that contains both a Perl 5 compiler
(perlito5) and a Perl 6 compiler (perlito6):

- Perl5 to Javascript (browser and node.js)

- Perl6 to Javascript (browser and v8 command-line)
- Perl6 to Perl5 (v6.pm, runs in perl 5.8 and up)
- Perl6 to Python (Python 2.6 and 2.7)

Run Perlito online, in the browser:

http://perlcabal.org/~fglock/perlito5.html
http://perlcabal.org/~fglock/perlito6.html

From CPAN (perlito6 only):

http://search.cpan.org/dist/v6/lib/v6.pm

$ cpan v6

Perlito is a work in progress. For the source code, to-do list, and bugs:

http://github.com/fglock/Perlito

- Flavio S. Glock (fglock)

Perl 6 Hackathon in Oslo: Be Prepared!

2012-04-11T05:52:08+00:00

The Oslo Perl Mongers invite to the Perl 6 Patterns Hackathon in Oslo. I have previously suggested that we hack on database connectivity, and so far only got positive feedback. If you want to help, here is what you can do to be prepared:

Get a github account
Build and install Rakudo
Build and install zavolaj/NativeCall
download MiniDBI
install and prepare databases to talk to

To hack efficiently on those projects, and to benefit from last-minute fixes, you should obtain Rakudo, NativeCall and MiniDBI from their git source repositories -- that last release is already outdated.

Here are the instructions in detail. If at any point you run into problems, feel free to ask on the #perl6 IRC channel or the perl6-users@perl.org mailing list.

Get a Github account

All the interesting Perl 6 code lives in git repositories on github. If you don't have an account already, sign up -- it's free.

Build and install Rakudo

This step is described well on the Rakudo homepage. Please follow the instruction in section "Building the compiler from source".

For the following steps it is important that you have a fresh perl6 executable file in your $PATH. If you have downloaded rakudo to /home/you/p6/rakudo/, you can run the command

PATH=$PATH:/home/you/p6/rakudo/install/bin

(and put it in your ~/.bashrc file if you want it permanently available, not just in this shell).

Build and install zavolaj/NativeCall

NativeCall.pm is the high-level interface for calling C functions from Perl 6 code. Install it:

$ git clone git://github.com/jnthn/zavolaj.git
$ cd zavolaj
$ cp lib/NativeCall.pm6 ~/.perl6/lib/

If you download and install ufo, you can use it create a Makefile for zavolaj. Then you can also run make test. On Linux it might not find the test libraries (which is mostly harmless, because you usually call libraries that are installed into your operating system, like those from mysql or postgres). In this case you should run LD_LIBRARY_PATH=. make test instead.

Download MiniDBI

That's not hard at all:

$ git clone git://github.com/mberends/MiniDBI.git

Install and Prepare Databases

So far, MiniDBI has (somewhat limited) support for mysql and postgres. Since it is always easiest to start from (at least somewhat) working code, I strongly recommend that you install at least one of those database engines.

Most modern Linux systems allow an easy installation via the package manager, and there are installers available for other operating systems. Be sure to also install the headers or development files if they come as extra packages.

Mysql

As mysql root user, run these statements:

CREATE DATABASE zavolaj;
CREATE USER 'testuser'@'localhost' IDENTIFIED BY 'testpass';
GRANT SELECT         ON   mysql.* TO 'testuser'@'localhost';
GRANT CREATE         ON zavolaj.* TO 'testuser'@'localhost';
GRANT DROP           ON zavolaj.* TO 'testuser'@'localhost';
GRANT INSERT         ON zavolaj.* TO 'testuser'@'localhost';
GRANT DELETE         ON zavolaj.* TO 'testuser'@'localhost';
GRANT LOCK TABLES    ON zavolaj.* TO 'testuser'@'localhost';
GRANT SELECT         ON zavolaj.* TO 'testuser'@'localhost';

Postgres

Launch psql as the postgres user and run these statements:

CREATE DATABASE zavolaj;
CREATE ROLE testuser LOGIN PASSWORD 'testpass';
GRANT ALL PRIVILEGES ON DATABASE zavolaj TO testuser;

You should now be able to connect with:

psql --host=localhost --dbname=zavolaj --username=testuser --password

(psql will ask you for the password. Enter testpass).

Other Databases

If you want to work on a backend for another database, it helps to have that database installed. Sqlite is an obvious choice (easy to install, zero setup), but of course there are other free database too, like firebird.

Project ideas

There is a lot of stuff to do. What follows is only a loose, incomplete collection of ideas.

Fix the postgres backend to actually pass its tests
Both mysql and postgres backends don't implement placeholders properly; change them (or one of them) to pass the placeholder values out of band.
Write an sqlite backend
Currently the user builds a DSN ("data source name") string out of the driver name, database name, db host name and so on, and then the driver parses it again. One could change that to pass all the information as named parameters instead.
Improve test coverage. For example test that numbers round-trip with the correct types.
Write a small application that uses a database. That's the best way to see if MiniDBI and the backends work.

Plans for the Perl 6 Hackathon in Oslo

2012-04-10T20:20:10+00:00

The Perl 6 Patterns hackthon in Oslo is happening next week, gathering most of the Rakudo developers. Awesome!

I’m still looking for the thing I’d like to work on during the event. The first obvious thought is Bailador, the Perl 6 port of Dancer web framework. It’s getting more and more in shape, and I’ve recently started looking through the code of Dancer 2 to see how it all should be done. I really don’t want Bailador to share a fate of neutro (ie reaching a state of being a horrible, unmaintainable mess), and I feel that it’s time for it to be refactored a bit. It came to life as a 10-line module written on a piece of paper during my Spanish classes, and simply adding more and more features to it is not going to end well.

Moritz Lenz is planning to hack on database stuff, which combined with a useful and usable web framework could create a nice platform for web development in Perl 6. I’ve already heard people saying “well, I could write Perl 6 apps if it had a web framework and some database access”, so I guess it’s time to show off something suitable for web development, even simple, but usable and covering most usage scenarios.

While we’re at the Web-y stuff, one thing that we really miss is a good (or at least good-enough) templating engine. Bailador has been using Ratel for some time, yet for one it stopped working on Rakudo recently (it was probably relying on some buggy or incorrect behaviour), and it suffers the problem of having no maintainer (no one really claims ownership of it). Hitomi was to be a templating engine for the new age, so maybe it’s about time to bring it back to life in all its glory? One could definitely find something to hack on.

From the new things, but still related to web development, zby has proposed porting WebNano to Perl 6 as one of the hackathon projects. It’s not something very big, and it should be totally possible to rewrite it in Perl 6 to run on Rakudo.

Or maybe there is something you particulary want to see in Perl 6? I’m sure we’ll not lack people with tuits; what is there that could possibly make Perl 6 a useful tool for you, something that you were missing when you last looked at it?

Back from vacation, hackathon coming up!

2012-04-05T15:56:48+00:00

So, I’m back from vacation. Turns out Argentina is a pretty awesome place to vacation in, too. As well as wonderful food and delicious imperial stout (amongst other good beers), there was walking like this…

…and other cool stuff, like glaciers…

…and so even though the laptop came with me, it was just too much fun to be outside, especially when the weather was good so much of the time. I did sneak in a few patches, though, most notably implementing PRE and POST phasers.

Anyway, I’m safely back, after an 8 hour flight delay from Buenos Aires and a small bus accident at Frankfurt airport. Yes, this airport fails SO hard they managed to screw up the 2 minute bus trip from the plane to the terminal…anyway, I got off with just a few small cuts. Suggest taking the train to YAPC::Europe this summer… :-)

So, what’s coming up? Well, this month brings a Perl 6 hackathon in Oslo, where I look forward to being together with a bunch of other Rakudo and Perl 6 folks. I’m sure we’ll get some nice stuff done, and some future directions worked out. I’m happy that one of the most industrious Perl Mongers groups I know when it comes to organizing such events is also set in a very pleasant city situated in a beautiful country. :-) By the way, it’s still very possible (and very encouraged) to sign up if you want to come along.

As moritz++ noted on rakudo.org, we’re skipping doing a distribution (Star) release based on the March compiler release since an unfortunate bug slipped in that busted precompilation of modules that used NativeCall. We hold ourselves to higher standards of stability in the distribution releases (which are user focused) than the compiler ones (which just ship at the same time each month), and this would have been too big a regression. The good news is that I’ve patched the bug today, so we’re now all clear for doing an April one – and what a nice release it should be.

Well, time for dinner here – which I’ll be having with masak++. No doubt macros will come up, and what’s needed to get us along to the next level with those. Stay tuned; the next month should be interesting in Rakudo land. :-)

No Rakudo Star release for March 2012, stay tuned for April

2012-04-05T08:13:10+00:00

Some of our users have asked what happened to the Rakudo Star release that was planned for March 2012. The short answer is that it didn’t happen.

The long answer is that compiler release for March 2012 include “bounded serialization”, a technique that drastically reduces startup time. Unfortunately it also causes a regression that stops the calling of C functions under cetain conditions.

We have noticed that too late, and decided not to base a Star distribution release on the new compiler release. We hope to fix the regression in time for the April release.

So if everything goes as planned, the April release of Rakudo star will have greatly reduced startup time, many new compiler features and the newly fixed modules URI and LWP::Simple.

We also hope to get some more cool stuff done at the Perl 6 Patterns Hackathon in Oslo, maybe improved database access.

Asynchronous HTTP requests in Perl 6

2012-04-02T15:38:43+00:00

Inspired by http://oylenshpeegul.typepad.com/blog/2012/03/asynchronicity.html, I decided to tackle asynchronous HTTP requests in Perl 6 on Rakudo. Allowing this required a bit of hacking on MuEvent, and the changes are far too ugly and hacky to be released, but here’s how it looks:

    use MuEvent;

    my @urls = <
        cpan.org
        kosciol-spaghetti.pl
        perlcabal.org
        duckduckgo.com
        perl6.org
    >;
    my $count = @urls.elems;
    my $starttime;

    sub handler ($what, $content) {
        say sprintf "%-25s has loaded in %s", $what, now - $starttime;
        unless --$count {
            say "Finished in {now - $starttime} seconds";
            exit 0;
        }
    }

    for @urls -> $url {
        http_get(url => $url, cb => sub { handler($url, $^content) })
    }

    $starttime = now;

    MuEvent::run;

The code results in something like this:

    cpan.org                  has loaded in 0.047303409195493
    perlcabal.org             has loaded in 0.0985883954326499
    duckduckgo.com            has loaded in 0.137316369015216
    perl6.org                 has loaded in 0.175586713955562
    kosciol-spaghetti.pl      has loaded in 0.413971411974607
    Finished in 0.465130828044337 seconds

You may notice that the urls are pretty much ordered – that’s because my internet connection is probably way faster than MuEvent’s event loop :)

For now http_get() is using bare sockets to send HTTP requests, which is, well, Less Than Awesome at least. Once I figure out how to properly tie LWP::Simple to MuEvent I’ll hopefully hack out some proper solution to be released for everyone’s joy. Stay tuned.

Announce: Niecza Perl 6 v16 by Stefan O'Rear

2012-03-26T23:21:55+00:00

Announce: Niecza Perl 6 v16

This is the sixteenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Minor changes]

Improved multi-dispatch failure error messages (Larry Wall).

Added @foo[1;2;3] multi-indexing syntax (but no real multidimensional
arrays yet), and Zen slices @foo[].

Added take-rw and return-rw functions for returning a read-write value.
(This will allow you to futureproof your code against take and return
becoming readonly by default.)

Containers bound into aggregates like %foo<k> := 1..3 now retain list
nature for iteration.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

[1] https://github.com/downloads/sorear/niecza/niecza-16.zip
[2] https://github.com/sorear/niecza

Counting t4 configurations

2012-03-25T13:37:54+00:00

I discovered the t4 task in a beach paradise on the island Ko Lanta in Thailand. I was there with friends, having a long-awaited December vacation, away from dreary Swedish cold and dark and snow. Instead, we were enjoying sun and shade, cool drinks on the beach or the quiet of our air-conditioned condo, getting caught up with reading, talking amongst ourselves, and exploring our surroundings. We seemed to have gotten there just a few weeks before the normal wave of tourism, so we felt uniquely at peace and at ease.

One day I was playing around with my Android phone, looking for some game to distract me in this oddly tranquil place. Since I'm ever fascinated with games on hexagonal grids, I searched for "hex". And I found this hex-grid-based one-player puzzle game that had me instantly hooked.

The easiest way to understand the t4 task is probably to play the game, and actually drag some pieces around.

The description of t4 merely captures in formal writing what the puzzle game shows plainly:

This problem makes use of a board of hexagonal cells that looks like this:

      i1  i2  i3  i4  i5
    j1  j2  j3  j4  j5  j6
      k1  k2  k3  k4  k5
    l1  l2  l3  l4  l5  l6
      m1  m2  m3  m4  m5
    n1  n2  n3  n4  n5  n6
      o1  o2  o3  o4  o5

Each cell has up to six neighbors. l3 has neighbors (clockwise) k3, l4, m3,
m2, l2, and k2.

The board is populated with some number of straight pieces of length 2 and 3.
Pieces never overlap, but they can move in the direction of their length,
which we will refer to as a "groove". So for example, a piece that starts out
on locations l1 and l2 can "slide" over to rest on locations l5 and l6.
No valid move can make a piece leave the groove in which it was first found.

We write "groove" because "row" doesn't quite capture how the pieces can be
situated. Besides the above seven rows of the table, a groove (and thus the
pieces in it) can also run along a diagonal direction, like this:

      e1  d1  c1  b1  a1
    f1  e2  d2  c2  b2  a2
      f2  e3  d3  c3  b3
    g1  f3  e4  d4  c4  b4
      g2  f4  e5  d5  c5
    h1  g3  f5  e6  d6  c6
      h2  g4  f6  e7  d7

Or a groove and its pieces could run along the other diagonal direction:

      p2  q4  r6  s7  t7
    p1  q3  r5  s6  t6  u6
      q2  r4  s5  t5  u5
    q1  r3  s4  t4  u4  v4
      r2  s3  t3  u3  v3
    r1  s2  t2  u2  v2  w2
      s1  t1  u1  v1  w1

There are a total of 23 grooves on the board, 7 horizontal, and 8 for
each diagonal. Grooves vary in length from 2 (out in the corners) to
7 (around the main diagonals).

The t4 description goes on for some time, but I'll stop there for now, because this post doesn't consider the dynamic aspects of the puzzle, only the board configurations.

Parenthetically, figuring out a decent coordinate system for the board, which manages to encode the fact that the grooves constrain the pieces, and the way each location on the board is the intersection of (exactly) three grooves, took me somewhere between hours and days of thinking. I was a bit disappointed to have to make that part public (in order to rein in the solutions enough and to be able to write base tests for the task). Looking at the solutions sent in, I don't think many people appreciated how much one can actually get for free with this particular encoding.

The t4 task is to write an algorithm that gets one special piece from one side of the board to another. Sometimes other pieces will block its way, and will have to be moved away first. And so on, recursively. The "and so on" bit is what makes this an interesting problem.

My attention quickly focused on more basic matters, though.

The app on my Android phone sports a thousand distinct puzzles. Presumably the good people at Tiny Bite Games generated these algorithmically somehow. Some of these are on slightly different boards, but I've chosen to ignore those for the purposes of t4. There's also a non-free version with ten thousand puzzles.

How many puzzles are there?

Or, to be precise, how many board configurations are there? By "board configuration", I mean every legal way to strew pieces on the board, everything from leaving the board empty to completely filling it up with length-2 and length-3 pieces. All of them.

As we headed out from Ko Lanta, I tried to upper-bound this, with just pen and paper, just multiplying out all the ways to place length-2 and length-3 pieces in all the grooves but without accounting for collisions. I arrived at a ridiculous number of combinations: about 3e57. I looked out the window of the taxi-jeep and saw my first real-life elephant. It looked small in comparison.

Now, "all possible board configurations" isn't the only number that might interest us here. It's just the "universe" of objects that we're dealing with in t4. As days and weeks went by (and I eventually got back home from vacation), finding the exact number of configurations became a worthy challenge, a Mount Everest of sorts.

But there are other, smaller numbers, which are also interesting:

The number of configurations with a piece l12. ("start-configurations")
The number of configurations with a piece l12 and no other l-groove piece.
The number of solvable start-configurations.
The number of trivially solvable start-configurations (where the solution is simply l[12 -> 56]).

But I didn't want to attack these problems until I had felled the beast. Finding the number of all possible configurations, the size of the problem space, the extent of the universe. Which was hopefully a lot smaller than 3e57.

Here's one way to solve it, in pseudocode:

Set confs <- 0
For all combinations of piece placements
    If there are no forbidden overlaps
        Set confs <- confs + 1
Output confs

It's a wonderfully simple program to think up. And it only has one loop in it. Let's say we happen to have an unbelievably fast computer at hand, which runs the loop in an optimized way so that each iteration takes a nanosecond on average. That's faster than today's computers, for sure.

Yeah, so. You run that on your fast computer, and I'll go brew some coffee. See you in nine and a half million million million million million million years.

So, that clearly wouldn't work. The program was simple to implement, but it just wouldn't terminate before our physical universe did. Basically, if you have to loop over ~3e57 things, you're screwed.

I needed something that could intelligently weed out impossible branches as it went along, that didn't make the stack or the RAM blow up. Come to think of it, something a little like the Dancing Links algorithm, which computes solutions to "exact cover"-type problems quickly (considering), and in constant space.

Conveniently, I had written a DLX solver in 2011. In C, no less (for speed). The idea with that project, which was only partly realized, is to make it very easy to specify problems in some "human" representation, and then frontends and backends to the solver would take care of the translation.

Of course, the board configuration enumeration problem wasn't an exact cover problem, I knew that. It would be if I was only interested in all the possible ways to fill up the board completely with length-2 and length-3 pieces. (There are 11,071,306 such configurations. Because I knew you were wondering.) But I didn't want that, I wanted board configurations with "gaps" in them too. (Especially since these are the only ones that can actually be solved!) But I figured I could extract the "essence" of the DLX algorithm, which is to be clever about possible alternatives at each choice point.

After a week of trying and failing to write an algorithm that borrowed the "essence" of the DLX algorithm, I came to an embarrassing realization.

The problem I wanted to solve is an exact cover problem. It can be solved directly using the DLX code I already had.

It was the word "gaps" that had me confused. Exact cover problems are characterized by the fact that their solutions have no overlaps, and no gaps. (That's what the "exact" in "exact cover" means.) But, fine, let's replace the word "gap" by the word "marshmallow". Now, what I was looking for was all the possible ways to fill the board with length-2 pieces, length-3 pieces, and marshmallows. Voilà, exact cover problem.

Some problems are made solvable simply by restating them. Also, don't underestimate the power of reifying nothingness.

So, I fed a representation of the problem into my DLX solver. It ran for two weeks on a decent computer, enumerating millions of configurations a second. It came up with this answer:

There are 4,783,154,184,978 board configurations. A bit short of five million million. Graah, I felled the beast!

Before this long calculation was even finished, it had occurred to me that (1), (2) and (4) on my list were also describable as exact cover problems. Only (3) is tricky and requires actually making moves on the board. But the other three questions can be formulated as exact cover problems by changing the shape of the board, essentially "forbidding" either one groove to occupy a location, or forbidding the location outright.

I started generating such exact cover models with a 100-line Perl 6 script. The numbers that fell out were these:

There are 573,538,221,334 configurations with a piece l12.
There are 375,873,151,406 configurations with a piece l12 and no other l-groove pieces.
The number of solvable start configurations is still unknown as of this writing.
There are 7,767,954,496 trivially solvable configurations.

At least now it doesn't feel unlikely at all that some game company manages to generate ten thousand hex puzzles.

Parrot 4.2.0 "Ornithopter" Released! by Jonathan "Duke" Leto

2012-03-24T00:11:25+00:00

"Technology, in common with many other activities, tends toward avoidance of
risks by investors. Uncertainty is ruled out if possible. Capital investment
follows this rule, since people generally prefer the predictable. Few recognize
how destructive this can be, how it imposes severe limits on variability and
thus makes whole populations fatally vulnerable to the shocking ways our
universe can throw the dice."
-- Assessment of Ix, Bene Gesserit Archives (Heretics of Dune)

On behalf of the Parrot team, I'm proud to announce Parrot 4.2.0, also known
as "Ornithopter". Parrot (http://parrot.org) is a virtual machine designed
to run all dynamic languages.

Parrot 4.2.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/4.2.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 4.2.0 News:
- API Changes
+ The signature of getprop was changed from (PMC,String,PMC) to
(PMC, PMC,String) for consistency
- Core
+ Parrot Calling Conventions (pcc) now reuses Continuation PMCs
internally, which reduces GC work by 25% and improves
the fib.pir benchmark by 6%
+ Winxed snapshot updated to 1.6.devel 44a04cfa7b
+ Improved the detection of Clang-ish compilers during configuration
+ Fixed a possible segfault bug when reading packfiles with no
constants or main_sub
+ By default, Parrot has now elevated these GCC warnings to errors
during compile time:
implicit-function-declaration, undef, missing-braces,
nested externs, old-style-definition, strict-prototypes,
+ The OS Dynamic PMC now has separate functions to unlink a file
and remove an empty directory (rmdir)
+ Fix building on Cygwin due to an improperly named DLL file
+ Various small bug fixes pointed out by static and dynamic analysis
tools
- Branches
+ Work on M0 continues now in the m0 branch, which contains both
implementations (currently C and Perl) and specification.
+ Good progress has been made on the threads branch which builds
on the green_threads branch. This gets Parrot much closer to
being able to utilize multiple CPU cores seemlessly. More details
at http://niner.name/Hybrid_Threads_for_the_Parrot_VM.pdf
- Documentation
+ New release manager documentation for parrot.github.com :
http://git.io/parrot-github-guide
- Community
+ Parrot was accepted to Google Summer of Code 2012!
Ideas Page: http://git.io/parrot-gsoc-2012

The SHA256 message digests for the downloadable tarballs are:

a960c89f94099c9643c5623263bdb7fa9e97effb889c975382bbf9c0251186b4
parrot-4.2.0.tar.bz2
69ee93d9f81babcff67b747cc614358958a32f274e407b65771ca5a056d4c3d4
parrot-4.2.0.tar.gz

Many thanks to all Parrot contributors for making this possible, and our
sponsors for supporting this project. This release is comprised of 136 commits
on the master branch since 4.1.0, with contributions from : Andy Lester, Vasily
Chekalkin, Jonathan "Duke" Leto, jkeenan, Alvis Yardley Whiteknight, Brian
Gernhardt, Gerhard R, NotFound, Francois Perrad, Moritz Lenz and luben.

Our next scheduled release is 17 April 2012.

Enjoy!

--
Jonathan "Duke" Leto <jonathan@leto.net>
Leto Labs LLC
209.691.DUKE // http://labs.leto.net
NOTE: Personal email is only checked twice a day at 10am/2pm PST,
please call/text for time-sensitive matters.

Upcoming Perl 6 Hackathon in Oslo, Norway

2012-03-21T01:15:52+00:00

The Oslo Perl Mongers are inviting to the Perl 6 Patterns Hackathon in Oslo in one month, and I very much look forward to being there.

Hackathons can be quite fun and productive if many programmers focus on the same goal. So to make the hackathon a success, I'm willing to work on whatever we decide to set as our goal(s).

One topic that is dear to me, and that is approachable by a horde of programmers (and guided by one or two Rakudo core hackers) is bringing database access into a usable state.

With muchly improved support for calling C functions and NativeCall.pm we should have enough infrastructure for access mysql, postgres and SQLite databases. MiniDBI aims to provide some basic convenience, but currently only the mysql driver partially works.

I believe that with concentrated effort, MiniDBI and the rest of the infrastructure can be improved to the point that it is usable, and other modules can start to rely on it. Databases usable in Perl 6, doesn't that sound good?

I'll see what kind of feedback I get on this idea, and if it's positive, I'll follow up with instructions on how to install the prerequisites for hacking on MiniDBI and its drivers.

t3: Addition chains

2012-03-14T21:24:00+00:00

<arnsholt> Heh. NP-complete problems in a
competition. That's just mean ^_^

Ok, we're in the midst of reviewing Perl 6 Coding Contest 2011 code submissions, and the turn has come to the third task: addition chains.

For a positive integer N, an addition chain for N is a sequence
starting with 1, each subsequent element being the sum of two
earlier elements (possibly the sum of the same element twice),
and ending with N.

For example for N = 9, this is a possible addition chain:

    (1, 2, 4, 5, 8, 9)

because 2 = 1 + 1, 4 = 2 + 2, 5 = 1 + 4, etc.

But a minimal solution would be:

    (1, 2, 3, 6, 9)

Write a program that reads numbers N from standard input, one per line,
and outputs a minimal addition chain like the one above.

Sometimes there will be several possible solutions of minimal length.
That's fine; just pick one of them.

Someone may look at the problem and think "aha! dynamic programming!" — but, alas, as Wikipedia patiently explains:

This is probably why the problem looks approachable, because it sort of feels like a dynamic-programming problem. But it ain't.

People sent in solutions. Go check them out.

Then there were two submitted algorithms that generated Brauer chains. "What's a Brauer chain?", I hear you asking. Hold on tight and I'll tell you. A Brauer chain is an addition chain where each new element is formed as the sum of the previous element and some element (possibly the same). Thus, of the two examples from the description,

(1, 2, 4, 5, 8, 9)

and

(1, 2, 3, 6, 9)

The latter is a Brauer chain, but the former isn't, because you can't get 8 by summing 5 and some element in the chain.

A Brauer-based algorithm will fail the first time at N = 12509. (See this reference, provided by hakank++).

Now, you might of course argue that failing at N = 77 is more wrong than failing at N = 12509.

Sheldon: More wrong? "Wrong" is an absolute state and not subject to gradation.

Stuart: Of course it is! It's a little wrong to say a tomato is a vegetable. It's very wrong to say it's a suspension bridge.

More precisely, there are infinitely many N for which no Brauer chain is minimal. 12509 just happens to be the smallest one.

This task, understandably, is a tricky one to judge. We've tried to go easy on the contestants (and non-contestants) in the reviews. After all, the problem is hard.

Now, who wants to translate Knuth's solution to Perl 6? 哈哈

mjd’s magic z

2012-03-13T15:23:04+00:00

So, I plunged ahead and implemented mjd’s z function in Perl 6. This is the first simple step to full continued fraction arithmetic, allowing addition with a rational, multiplication by a rational, and taking the reciprocal of a continued fraction. (And oh! I just accidentally discovered that I’d missed a bunch of slides at the end of mjd’s talk which discuss implementing the full z function from HAKMEM!)

The implementation turned out to be pretty straightforward, with only the edge cases causing trouble. z maintains its state using four variables, $a, $b, $c, and $d. There are two basic operations which modify this state: input, which takes the next value from the continued fraction provided, and output, which outputs the next value of the continued fraction result. How do you know which to do? Easy! If $a div $c == $b div $d, then we’ve determined the next value which can be output, and should output it. If not, then we need to input another value.

The tricky bits: first, it’s completely legal for $c and/or $d to be 0. That causes a division by zero error in Niecza. So I added a check for each, setting the result of the division to Inf when it needs to be. The second thing is what happens when the given continued fraction runs out of values. In that case, the next value is implicitly Inf. That’s well and good, but mjd somehow performs magic with it. I don’t understand what the heck he was doing there, but I can imitate the result. Here it all is in one function:

sub z($a is copy, $b is copy, $c is copy, $d is copy, @x) {
    gather loop {
        my $a-div-c = $c ?? $a div $c !! Inf;
        my $b-div-d = $d ?? $b div $d !! Inf;
        last if $a-div-c == Inf && $b-div-d == Inf;
        if $a-div-c == $b-div-d {
            my $n = $a-div-c;
            ($a, $b, $c, $d) = ($c, $d, $a - $c * $n, $b - $d * $n);
            take $n;
        } else {
            if @x {
                my $p = @x.shift;
                ($a, $b, $c, $d) = ($b, $a + $b * $p, $d, $c + $d * $p);
            } else {
                ($a, $b, $c, $d) = ($b, $b, $d, $d); # WHY????
            }
        }
    }
}

And there you have it! I’m still a bit worried about the edge cases (I haven’t tested negative numbers yet, for instance), but I think it clearly points the way onto the full HAKMEM z function.

Continued Fractions

2012-03-13T00:23:42+00:00

Four months ago I created Math::ContinuedFractions on Github. But I didn’t actually post any code to it. But I’ve finally started, pushing t/00-experiments.t. It’s very simple so far, but that’s because I’m slowly feeling my way through this unfamiliar subject.

I tried to link the Wikipedia equation graphic for a continued fraction, but that just lost me the first draft of this post, so instead I’ll link to the Wikipedia page. It is the source of our first algorithm, for converting a number to a continued fraction:

sub make-continued-fraction (Real $x is copy) {
    gather loop {
        my $a = $x.floor;
        take $a;
        $x = $x - $a;
        last if $x == 0;
        $x = 1 / $x;
    }
}

This returns the components of the continued fraction as a lazy list. It passes the first few simple tests I’ve thrown at it, but I’m a bit worried about $x = 1 / $x. Should this maybe be FatRat math instead of normal math? I’m not clear on the implications one way or the other.

Unfortunately, the Wikipedia page doesn’t shed any light on how to do basic arithmetic on continued fractions. HAKMEM gives a complete overview on basic continued fraction arithmetic, but I find it quite hard going. MJD has a nifty talk on the subject, including some algorithms I hope to implement soon. Unfortunately, he doesn’t get into the tricky stuff.

Luther Tychonievich has a nice blog post on the subject which brings up a problem I have not seen mentioned elsewhere. Some pretty basic operations may not work because they need every (infinite) input term to determine the first output term. His example is squaring the square root of 2. As I understand it, the problem is that the first term (the integer part) flits back and forth between 1 and 2 as each addition term is fed to the code. It can never settle down on one or the other.

Incidentally, my code fails miserably in Rakudo, but works in Niecza. Unfortunately for this project, Niecza cannot currently handle overloading the arithmetic operators for a new type, which means I’ll have to figure out how to get things working on Rakudo at some point.

Macros progress report: D1 merged

2012-03-10T16:14:00+00:00

I've merged my first chunk of work, known to the grant application as "D1", into the nom branch of Rakudo. Concretely, this means three things:

The next release of Rakudo (release #50, due 2012-03-22) will be the first release of any Perl 6 implementation to feature AST macros.
People who have a new-enough revision of the nom branch checked out can play around with basic macros already.
There's just three chunks of work left to do. :-)

The work has progressed nicely so far. It's been a little trickier than I expected, but not insurmountable yet, and I've received plenty of support from jnthn++. The time estimates were way too optimistic, it turned out — partly because the grant took a while to get accepted, partly because $dayjob keeps wanting tuits — but at least I'm not stuck.

One interesting aspect of the grant that I didn't expect is that much of my thinking happens in gists. Increasingly, I'm beginning to see gists as a sort of "semi-private blog posts". They're notes mainly left for myself and my creative process, and they don't clutter up the blog feed, but they're there if people are interested enough to see how sausage is made. I don't go back and "fix" gists when I learn new things that contradict what's in them — I just write new ones.

Here, just for reference, I collect all the gists I've made so far philosophizing on macros:

Musings on macros
Macros and short-circuiting
More thoughts on macros
(at this point, I wrote the grant application)
D4, deconstructed
I'm thinking about macros again
Cool use for a macro: meta for
The lifetime of a macro invocation

I have one more gist coming up that I haven't written yet, about how macro-argument AST thingies are thunkish and rebind to their outer context just like closures do.

The other "artifacts" that have fallen out of the grant work so far, are:

A spectest file. Too few tests in it, even though I've tested everything I can think of. There will be more tests for D2, I'm sure.
A monolithic commit to Rakudo, representing my D1 work. I did a bunch of rebase operations on the way to keep current with the nom branch, and each time I folded a number of commits into a single one, because I considered most commits after the original one from September to be "oh right"-type afterthoughts. Not sure how this rhymes with jnthn's post about atomic commits... I'm a bit divided on the issue.
A talk, at Deutscher Perl Workshop in Erlangen. You can see the slides here.

Looking ahead a bit, what's up next is D2. Informally, D2 consists of "getting those little {{{$stuff}}} thingies to work". We need a way to represent "there's a placeholder here" in the AST, because that's how the ASTs look during the static part of their lifetime (before macro application). It will require a new AST type. As it happens, PAST in nqp/Rakudo is just about to be replaced by the next generation of AST technology, and jnthn++ is just about to start working on that. So it makes sense for me to assist him, since I'm now blocking the new AST technology landing in Rakudo for my macros work. Aside from blocking, I expect D2 to be quite easy to implement.

See you again at the next milestone. And possibly at part-way resting stops in between.

Meta-programming slides, and some Rakudo news

2012-03-09T00:19:59+00:00

I’m just back from a trip to the German Perl Workshop. Of course, the food was great and the beer was greater. :-) I was also very happy to see various Rakudo hackers in person; moritz++ was one of the organizers, and tadzik++ attended and gave a talk also. Much fun was had, and some hacking happened too. :-)

You may be interested to check out the slides from my talk on meta-programming in Perl 6. It covers some of the things we’ve been able to do for a while, as well as having a new example that shows off an interesting use of bounded serialization in combination with meta-programming. Also, I have now submitted this talk to the French Perl Workshop, which will take place in June. :-)

As for Rakudo progress, here’s some of the things that have been going on.

The bounded serialization branch has landed (and thus will be in the next release)! This has, as hoped, greatly decreased Rakudo’s startup time – an improvement that has been welcomed by a bunch of folks on #perl6. Module and setting pre-compilation is now also faster, along with loading of pre-compiled modules. Win all around.
Thanks to the serialization, various things have been unblocked. Amongst them, the constant declarator now takes non-literals on its RHS, and the BEGIN and CHECK phasers now work in r-value context. In both cases, the values are serialized if these things are done in a pre-compiled module.
masak++ merged his initial work on macros. While there’s much more to do in this area yet, it’s great that we now have the basics in place, enabling declarations of macros and quasis. I’m sure masak will blog about this soon – keep a lookout for it.
tadzik++ has got us some support for the Set, Bag, KeySet and KeyBag types.
moritz++ cleaned up match handling, and also implemented the :exhaustive and :nth match adverbs. So, that’s two out of three branches mentioned in my last post landed. :-)
Stunning progress on phasers! We now support ENTER, LEAVE, KEEP, UNDO and START. The LEAVE, KEEP and UNDO drew very heavily on work by mls++. Additionally, FIRST, NEXT and LAST are supported inside for loops (though not yet available in other loops). I’m fairly fired up to do PRE and POST too, though need some spec clarifications first.

moritz++ has also been continuing his excellent typed exceptions work; we discussed how to handle typed exceptions in the metamodel during the workshop, which should extend their reach to cover even more of the errors that can crop up.

So, all in all, lots of progress, and we’ve still just under two weeks to go until the March release. Having landed the bounded serialization work, I’ve been enjoying doing some feature development; right now, I have a branch that is muchly cleaning up name handling, improving name interpolation and getting various of the missing pseudo-packages in place. I expect to have this work completed in time to go in to the release also.

Beyond that, it’ll be time for me to dig back into some of the fundamentals again. Most immediately, that will be getting QRegex to be the engine we use to parse NQP and Perl 6 code (currently, we use QRegex for compiling the regexes and grammars you write in your Perl 6 programs, but still use the previous generation engine for parsing Perl 6 source). This will allow us to eliminate a vast swathe of Parrot PIR code that we depend on (the new regex compiler is written in NQP), aiding portability. I aim to land this in time for the April release; I’ve already done significant pieces of the work towards it.

Probably somewhat in parallel with that (though set to land after it), I’ll also be taking on our AST and AST compiler. This work will see it being ported from PIR to NQP, switched over to use 6model objects (which means we can use natively typed attributes to make the nodes more memory efficient, and also serialize the nodes as macros require) and given much better handling of native types. Generally I’m hoping for compilation performance improvements and better code generation. This will also eliminate our last large PIR dependency, thus clearing the final hurdle for making a real stab at targeting an extra backend.

One slight slowdown will be that I’m taking a vacation during the second half of March. I’m going to a land far away, and plan to spend most of my time seeing it, and checking out their food and drink. The laptop is coming, though, so it won’t be a complete stop; my stress levels are happily in fairly good shape right now, so I don’t need my vacation to be a complete break with normality, just a refreshing change of scene. :-)

Rakudo Star 2012.02 out – and the focus for 2012.03

2012-02-29T00:21:00+00:00

I’ve just cut the February 2012 release of Rakudo Star. The good news is that we got a bunch of nice improvements into the release.

More typed exceptions, and better backtraces. This work has been done by moritz++; I’ll simply point you to his excellent blog post on that topic today. I think everyone working with Rakudo will appreciate the new backtraces, which cut out a lot of the uninformative details and are thus much more focused.
FatRat is now implemented. This type gives you rational number support with big integer semantics for both the numerator and denominator. The Rat type now correctly degrades to a Num if the denominator gets too large, as per spec (this avoids performance issues in certain cases). Again, we’ve moritz++ to thank for this.
We have object hashes! You can now make a declaration like:
my %hash{Any};
Which will have keys of type Any. You can constrain this as you wish. You can also still constrain the value in the usual way:
my Int %hash{Any};
This latter example gives you a hash with Int values and object keys, meaning that keys are not coerced to strings, as is the default. Thus, using %hash.keys will give you back the original objects. Identify is done based on .WHICH and thus ObjAt.
The coercion syntax, Int($foo), is now implemented. This gives you another – perhaps neater – way to write $foo.Int, but types may also implemented the method postcircumfix:<( )> if the target type is the one that knows how to do the coercion.
The reduction meta-operator used to have crazy bad performance, and parsed dodgy in some cases. Both of these issues are now resolved; it’s an order of magnitude faster and parses more correctly.
Various regex improvements and more problems caught at compile time, which I discussed in my previous post.

As always, there’s a bunch of bug fixes and other tweaks – see the Rakudo ChangeLog for more details.

Naturally, one release done simply means another one to work towards. :-) If you’ve been following along with my posts, you’ll have noticed that the bounded serialization work I’ve been doing didn’t make it in to the 2012.02 release; it just wasn’t done in time. The good news is that it’s well on course to land in very good time for the 2012.03 release.

Today I got to the milestone of having a Rakudo binary, with a serialized CORE.setting, that will compile Test.pm and attempt the spectests. There’s a bunch of failures (which I expected), but a lot more passes than fails. Anyway, that means things are essentially working and I’m into triage mode. I’ve also got some very preliminary data on what kind of improvements it brings.

Just from trying it out, I could feel that startup was faster, as hoped. tadzik++ built the bounded serialization branch and the current main development branch and compared the time they took for a simple “hello world” (such a simple program that startup cost will dominate). The result: the bounded serialization branch ran it in around 25% of the time. I think we can squeeze a bit more out yet, but achieving startup in a quarter of the time we do today is a big improvement. Loading time for pre-compiled modules will see a similar improvement.

I also checked memory consumption when compiling CORE.setting. The bounded serialization branch uses 60% of the memory that the main development branch uses. I’d been hoping that this work would cut memory usage by around a third, and it’s done at least that. I didn’t measure the speedup for this task yet; while the other bits are fair comparisons, this one would not be yet even if I had a figure, since the optimizer is currently disabled (one of the transformations causes issues; I’m not expecting it to be a huge pain to resolve, but was more focused on the critical path to running spectests again so I could understand the fallout better).

Once the busted spectests are running again, I’ll get this merged, then dig in to exploiting some of the opportunities it makes available. I’m hoping we can get some more nice features in place for the next release also; moritz++ has already got branches for sink context and the :exhaustive modifier in regexes in progress, and I’ve got some ideas of things to hack on… :-)

Rakudo Star 2012.02 released

2012-02-28T23:26:39+00:00

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the February 2012 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the February 2012 release is available from <http://github.com/rakudo/star/downloads>.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release #49 of the Rakudo Perl 6 compiler [1], version 4.1 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

Here are some of the major improvements in this release over the previous distribution release.

The FatRat type is implemented, and Rat arithmetic now properly defaults to Num if the denominator is too big
Object hashes are implemented, and can be declared with the syntax my %h{Any} (for a hash with keys of type Any)
The <Some::Grammar::rulename> syntax is now implemented in regexes; <foo> can also be used to call predeclared lexical rules
The Int($x) coercion syntax is implemented
&rename and &copy are now implemented
Improvements to the reduction meta-operator (order of magnitude faster, some parsing issues fixed)
The <prior> regex built-in is now available, and matches whatever the last successful match matched
A $match.make(…) method is available to set the AST for a match object not stored in the $/ variable
Improved backtraces

This release also contains a range of bug fixes, improvements to error reporting and better failure modes. Many more exceptions are thrown as typed exceptions.

Due to continued evolution of and convergence with the Perl 6 spec, there are some changes to existing functionality that may affect your code:

You may no longer use the $.x form in submethods and attribute initializers, as per spec; use the non-virtual $!x instead.
The LHS of the xx operator is now thunked
.conjugate is now called .conj
Enumeration values .gist to just the key, not the full name
An empty Buf is now False in boolean context

Currently, we have maintained backwards compatibility with some changed pieces of syntax, but will drop them in an upcoming release:

“<…>” in proto regex bodies; now this should be written “*”
The use of “**” with a separator in regexes; this is now done by using “%” or “%%” on another quantifier

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:

pack and unpack
macros
threads and concurrency
Unicode strings at levels other than codepoints
pre and post constraints, and some other phasers
interactive readline that understands Unicode
non-blocking I/O
much of Synopsis 9

There is a new online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo Star 2012.02 and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at <rakudobug@perl.org>.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. An updated draft of a Perl 6 book is available as
<docs/UsingPerl6-draft.pdf> in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see <http://rakudo.org/how-to-help>, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Current State of Exceptions in Rakudo and Perl 6

2012-02-28T19:54:25+00:00

It's been a while since my last update on my grant work on exceptions for Perl 6 and Rakudo, and I can report lots of progress.

The work on Rakudo's exception system made me realize that we conflated two concepts in the base exception type: on the one hand the infrastructure for reporting errors and backtraces, and on the other hand holding some sort of error message as an attribute.

As a result, we now have a base class called Exception from which all exception types must inherit. When a non-Exception object is passed to die(), it is wrapped in an object of class X::AdHoc. Other error classes can decide to generate the error message without having an attribute for it (for example hard-coded in a method).

Typed exceptions are now thrown not only from the setting, but also from the compiler itself, namely the grammar and the action methods. In fact the majority of errors from these two parts of the compiler are now handled with dedicated exception types.

The most user-visible change is a new and improved backtrace printer, which produces usually much shorter and more readable backtraces. The old one is still available on demand. Consider the program

sub f {
    g() for 1..10;
}
sub g { die 'OH NOEZ' }
f;

The old backtrace printer produced:

OH NOEZ
  in sub g at /home/moritz/p6/rakudo/ex.pl:4
  in block <anon> at /home/moritz/p6/rakudo/ex.pl:2
  in method reify at src/gen/CORE.setting:4471
  in method reify at src/gen/CORE.setting:4376
  in method reify at src/gen/CORE.setting:4376
  in method gimme at src/gen/CORE.setting:4740
  in method eager at src/gen/CORE.setting:4715
  in method eager at src/gen/CORE.setting:1028
  in sub eager at src/gen/CORE.setting:5000
  in sub f at /home/moritz/p6/rakudo/ex.pl:2
  in block <anon> at /home/moritz/p6/rakudo/ex.pl:5
  in <anon> at /home/moritz/p6/rakudo/ex.pl:1

Where the eager, gimme and reify methods come from the 'for' lop, which is compiled to the equivalent of eager (1..10).map: { g() }.

The new backtrace printer produces

OH NOEZ
  in sub g at ex.pl:4
  in sub f at ex.pl:2
  in block <anon> at ex.pl:5

It is also a special pleasure to report that after a walk through a change to throw a typed exception, we've received a pull request by a new developer which also changes an exception from X::AdHoc to a dedicated type.

Announce: Niecza Perl 6 v15 by Stefan O'Rear

2012-02-27T20:58:33+00:00

Announce: Niecza Perl 6 v15

This is the fifteenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. No, it's not dead.

Niecza now passes more spectests than Rakudo, but that is not an entirely
fair comparison, and we have the feature matrix now which is much more
useful for such comparisons. [3]

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Major changes]

The Unicode character database bundled with Niecza has been updated to
version 6.1.0.

[Minor changes]

Constant folding which fails at compile time, since it cannot succeed, now
generates a compile-time warning. (idea from Darren Duncan)

Perl 5 interoperability has made progress. (Paweł Murias)

Proxy has been added. (No real new functionality, just putting a specced
API on an existing feature)

Unused-variable warnings are now suppressed in the REPL.

Added KeySet, KeyBag; substantially improved Set and Bag. (Solomon Foster)

Set operators (both Texas and Unicode) are now available; it is now possible
to define operators in the setting beyond those defined in the grammar.

Added ability to define custom iffy and diffy operators.

Added $.foo(42) syntax meaning $(self.foo(42)).

@*INC internals have changed again.

Added Range.pick, Range.roll, &rmdir, &rungather. (Solomon Foster)

Added &run (uses GLib if available for more robust argument passing).
(Solomon Foster)

Compiler memory usage has been substantially optimized, approximately 40%
less used for CORE, with a small accompanying size improvement.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

[1] https://github.com/downloads/sorear/niecza/niecza-15.zip
[2] https://github.com/sorear/niecza
[3] http://perl6.org/compilers/features

Parrot 4.1.0 "Black-headed Parrot" Released by Alvis Yardley

2012-02-21T21:29:07+00:00

On behalf of the Parrot team, I'm proud to announce Parrot 4.1.0, also known
as "Black-headed Parrot". Parrot (http://parrot.org/) is a virtual machine aimed
at running all dynamic languages.

Parrot 4.1.0 is available on Parrot's FTP site
(ftp://ftp.parrot.org/pub/parrot/releases/devel/4.1.0/), or by following the
download instructions at http://parrot.org/download. For those who would like
to develop on Parrot, or help develop Parrot itself, we recommend using Git to
retrieve the source code to get the latest and best Parrot code.

Parrot 4.1.0 News:
- Core
+ Shared libraries and installable binaries are now stripped if
built with --optimize on Cygwin, which greatly reduces their
size on disk
+ New experimental PCC-related ops added to core.
- Documentation
+ Revised 'docs/project/release_manager_guide.pod'
- Tests
+ Parrot now uses Travis CI http://travis-ci.org
+ Parrot Continuous Integration (CI) with Travis CI means
every commit of Parrot is now compiled and tested on gcc,
g++ and clang with various Configure.pl options.
+ CI Notifications are sent to parrot-dev, the #parrot
IRC channel and Smolder
+ Cardinal and Rakudo spec tests also on Travis CI

The SHA256 message digests for the downloadable tarballs are:
6d277862cfddc3e13f53684315f1ed3c76a053c01200652436b548be0964cf1d parrot-4.1.0.tar.gz
826465f3b7045cf81768029b1f18a4fa05259b7268c98c9c0436bd77c74785ec parrot-4.1.0.tar.bz2

Many thanks to all our contributors for making this possible, and our sponsors
for supporting this project. Our next scheduled release is 20 March 2012.

Enjoy!

--
Alvis

Results from the Prisoner's Dilemma Challenge

2012-02-18T20:00:00+00:00

The Iterated Prisoner's Dilemma Challenge is now closed; several interesting solutions have been submitted.

Of the basic strategies, tit-for-tat (doing what the opponent did the last time, starting off with cooperating) is usually the strongest. Since the random strategy is, well, random, the results fluctuate a bit.

Most submitted strategies are a variation on tit-for-tat, modified in some way or another to make it stronger. All submissions contained a strategy that is stronger than tit-for-tat when tested against the basic strategies only, though the interaction with other new strategies made some of them come out weaker than tit-for-tat.

Submitted Strategies

Without any further ado, here are the strategies and a few comments on them.

Turn the Other Cheek

## Dean Serenevy; received on 2012-02-07
%strategies<turn-other-cheek-no-deal-with-devil-once-bit-twice-shy-variety-is-the-spice-o-life> = sub (:@mine, :@theirs, *%) {
    my ($bitten, $shy, $they-coop) = (0, 0, False);

    for @mine Z @theirs -> $me, $them {
        if $them          { $they-coop = True; }
        if $me and !$them { $bitten++; $shy = 0; }
        if !$me           { $shy++ }
    }

    return True  if 0 == $bitten;               # Cooperate if we have never been bitten
    return True  if 1 == $bitten and 0 == $shy; # Turn the other cheek once
    return False unless $they-coop;             # Screw you too!
    return $shy >= (2 ** ($bitten-1)).rand      # Once-bitten rand() shy
};

Inevitable Betrayal

## Andrew Egeler, received 2012-02-09

%strategies<inevitable-betrayal> = &inevitable-betrayal;
sub inevitable-betrayal (:@theirs, :$total, *%) { +@theirs <
($total-1) ?? @theirs[*-1] // True !! False }

%strategies<evil-inevitable-betrayal> = &evil-inevitable-betrayal;
sub evil-inevitable-betrayal (:@theirs, :$total, *%) { +@theirs <
($total-1) ?? @theirs[*-1] // False !! False }

These are variations on tit-for-tat and evil-tit-for-tat which always defect in the last round, because then the opponent can't retaliate anymore.

In a typical Iterated Prisoner's Dilemma contest, strategies don't know how many rounds are being played, just to avoid this behavior.

Tit for D'oh and Watch for Random

## Solomon Foster, receievd 2012-02-10

%strategies<tit-for-doh> = -> :@theirs, :$total, *% {
    @theirs < $total - 1 ??  (@theirs[*-1] // True) !! False
}

%strategies<watch-for-random> = -> :@theirs, *% {
    @theirs > 10 && @theirs.grep(* == False) > 5 ?? False !! (@theirs[*-1] // True)
};

tit-for-doh is the same as inevitable-betrayal. watch-for-random defects forever once the opponent has defected too often.

Me

## Audrey Tang, received 2012-02-17
%strategies<me> = -> :@theirs, *% {
    my role Me {};
    (@theirs[*-1] // Me).does(Me) but Me
};

This strategy uses a mixin in its returned boolean values to find out when it plays against itself, or against a strategy that copies its values from @theirs (ie tit-for-tat derivatives), in which case it cooperates. This games the system, though doesn't explicitly violates the stated rules.

Audrey also deserves two dishonorable mentions for two solutions that game the test harness or the other strategies by exploiting the technically imperfect sandboxing:

   au => -> :@theirs, *% {
       use MONKEY_TYPING;
       my role TRUE {};
       augment class Bool {
           method Stringy(Bool:D:) {
               self.^does(TRUE) ?? 'True' !! 'False'
           }
       }
       False but TRUE;
   }, 

   amnesia => -> :@mine, :@theirs, *% {
       my role Uh {};
       my $rv = (@theirs[*-1] // Uh).does(Uh) but Uh;
       @mine = @theirs = ();
       $rv;
   },

Those two strategies did not compete in the tournament

Lenient in the Beginning, Then Strict

I've written my own two strategies before the tournament started. Here is the original, I've only changed the signatures to run under current Niecza:

# a tit for tat, but a bit more friendly at the beginning
# to avoid hacking on forever on evil-tit-for-tat,
# but be very stringent when the other one defects too often
sub moritz-ctft(:@theirs, :$total,  *%) {
    return True if @theirs < 3;
    return False if @theirs.grep(*.not).elems > ($total / 10);

    @theirs[*-1];
};
%strategies<moritz-ctft> = &moritz-ctft;

# the evil clone...
sub moritz-ectft(:@theirs, :$total,  *%) {
    return True if @theirs < 3;
    return False if @theirs.grep(*.not).elems > ($total / 10);
    # did you believe in happy ends?
    return False if @theirs + 1 == $total;

    @theirs[*-1];
};
%strategies<moritz-ectft> = &moritz-ectft;

Results

The results vary quite a bit between runs, mostly because of the random strategy.

Here is the output from a sample run. Please don't use this for determining the "winner", because it is just a random sample with no statistical significance.

SUMMARY
2588    moritz-ectft
2577    me
2560    moritz-ctft
2491    inevitable-betrayal
2483    tit-for-tat
2480    tit-for-doh
2399    turn-other-cheek-no-deal-with-devil-once-bit-twice-shy-variety-is-the-spice-o-life
2319    watch-for-random
2272    good
1876    evil-inevitable-betrayal
1862    evil-tit-for-tat
1538    random
1145    bad

You see, inevitable-betrayal and tit-for-doh are exactly the same strategy, but the random fluctuations place them on different sides of tit-for-tat. Which is why I won't declare a winner at all, there is simply no fair way to determine one.

At first I was surprised how well the me strategy performed. But then I noticed that with the given game harness, a strategy fighting against itself counts double (once for the first copy, once for the second copy). With only 13 strategies participating, and such close results, harmonizing perfectly with yourself gives you a critical advantage.

Visualizations

For each strategy you can find an image that shows how it worked with or against another strategy. Green means cooperate, red means defect, and the height of the bar is proportional to the resulting score.

Trying to Be Fair

In an attempt to reduce the impact of the random strategy, I've changed it to use the same random sequence against each player (and of course against itself, which totally skews that particular result).

Again the rankings vary between different runs of the same program, but now at least same strategies produce mostly the same result (turn-the-other-cheek also has a random component). An example output from such a run is

SUMMARY
2558    moritz-ectft
2543    moritz-ctft
2532    me
2457    inevitable-betrayal
2457    tit-for-doh
2445    tit-for-tat
2387    turn-other-cheek-no-deal-with-devil-once-bit-twice-shy-variety-is-the-spice-o-life
2314    watch-for-random
2248    good
1856    evil-inevitable-betrayal
1844    evil-tit-for-tat
1359    random
1100    bad

TL;DR

It was a lot of fun! Thanks to everybody who submitted a strategy.

t2: Sums of cubes

2012-02-12T16:35:49+00:00

(Guest post by Moritz Lenz.)

The second task from the Perl 6 Coding Contest 2011 needs data structures not available in core Perl 6 to be solved efficiently.

But I'm getting ahead of myself here. First, let's recall the task description:

Some natural numbers can be as the sum of two positive cubes of natural
numbers. For example, 1729 is 12 cubed plus 1 cubed. Some natural numbers
can even be expressed as more than one such sum. For example, 1729 can
also be expressed as 10 cubed plus 9 cubed.

Just for clarity's sake, the sum with the two terms reversed is the
same sum, not a new one. Thus, 91 is 3 cubed plus 4 cubed, but
finding "4 cubed plus 3 cubed" does not count as a distinct sum.

Write a program that accepts an integer N on the command line, and
prints out the first N of these numbers in increasing order. For each
number found, print it out, as well as the sums of cubes that yield that
number.

For example:

    1729 = 12 ** 3 + 1 ** 3 = 10 ** 3 + 9 ** 3

As always, the devil is in the details. Let me add some emphasis:

"For each number found, print it out, as well as the sums of cubes that yield that number." The sums, not both sums.

The 455th of such sums of cubes can be written in three different ways, not just two:

87539319 = 423 ** 3 + 228 ** 3 = 414 ** 3 + 255 ** 3 = 436 ** 3 + 167 ** 3

If advanced number theory offers a direct way to come up with such numbers, we are not aware of it. That leaves the search through the pairs of positive integers as a viable solution.

Now there are basically two approaches to searching that vast space of number pairs. The first is to queue up all such pairs in the order that they produce the cubes, and then iterate them all, looking for double (or triple) consecutive elements.

The second is to iterate the pairs in some simpler way, and keep all sums of cubes in a hash, looking for solutions. Since this can produce the searched numbers out of order, a queue is needed here as well, but on the output end this time. It also helps to have some condition for when one is sure that no numbers smaller than some limit will be found, so that one can safely print out the values already found.

The first one sounds more elegant a priori, and implictly avoids having non-collected garbage in the data structures, but in practice the second one is much faster, because it keeps orders of magnitudes less data in its queue.

All solutions we have seen or written so far check or enqueue the numbers by grazing half of a quadrant of the number plane in a triangular fashion, that is they have a slowly growing first index x, and a fast second index y that runs from 1 to x for each value of x.

Here you'll find people's solutions.

The following chart illustrates the run time of the solutions we have received, as a function of how many target numbers they should find. A drop to zero indicates that the solution threw an error before printing out all requested numbers (limited to 1 GB virtual memory)

One can see that the solutions all scale roughly exponentially (that is, a straight line in the logarithmic plot), and that there are two clusterings: one with steep slopes of slow solutions that enqueue all sums of cubes, and one of milder slopes of fast solutions that filter by hash first.

It might be possible to come up with a cleverer strategy for producing these index pairs, one that is more closely aligned to the contour lines of x * 3 + y * 3. Such a scheme would reduce memory usage, and maybe even run time. If you have a nice idea for such an algorithm, please contact us. :-)

It is possible to walk each possible integer contour line l, and for each smaller integer cube i = x ** 3 check if the third root of l - i is integral. Such an algorithm has the advantage of using up nearly no memory, but it is ridiculously slow in comparison to the other approaches we have discussed so far. But maybe there is a way to combine the advantages of all these approaches?

Bounded serialization, better regexes and better errors

2012-02-10T00:55:18+00:00

Here’s a quick roundup of some of the things that have been going on in Rakudo since the January release.

Bounded Serialization

This is where I’ve been focusing the majority of my efforts. At the moment, when we pre-compile Perl 6 code – such as the CORE setting with all the built-ins – we also build up a bunch of code that reconstructs the various objects that are made as a result of compile time declarations. This is done by recording an “event” for each thing that happens at compile time, then replaying them when the module/setting is loaded.

We’ve done things this way throughout every generation of Rakudo to date, but as I worked on the “nom” development branch and 6model, I built things so that we could later switch over to a different model; one where all of the objects created as a result of compile-time declarations are serialized. Then, when we need to load the module, we deserialize this blob of data back into the required objects, rather than doing all of the method calls and data shuffling to build them up again.

I’ve been working on this since we got the last release out. So far I’ve got the serialization and deserialization engine to a reasonably capable state; it can happily handle references between compilation units, has no problems with circular references between objects, and today I got basic support for serializing closures in place also. At this point, I’ve just got it being exercised by a bunch of tests; the next step will be to integrate it into NQP’s compilation process. I’m hoping I can get through that at the weekend, and after that it’ll be time to try and get Rakudo using it. Whether I get this landed for the February release or if we have to wait until the March one, I’m not yet sure. I know for sure I don’t want a half-baked version of it going out, so I’ll hold it for the March release if I can’t get it working as reliably as I want for the February one.

Why am I working on this? Here’s some of the things that I’m aiming at as a result of this work.

Improved startup time (the deserialization should be less work than rebuilding everything up from scratch)
Reduced memory during pre-compilation of modules. Most notably, I’m hoping to make a notable reduction in the memory (and time) required to build CORE.setting, which will most certainly be welcomed by anyone trying to build in a lower memory environment. The faster build will also be helpful for Rakudo developers, and enable us to be more productive.
The restrictions on the “constant” declarator can be lifted, enabling use of non-literal values.
Phasers as r-values can be implemented.
We can implement constant folding much, much more easily, as well as build other immutables at compile time.
Other nice things, no doubt. :-)

Completing this will also be one prerequisite down for much better inlining optimization support.

More Regex Bits

We now support the use of <x> in a regex to also call any predeclared lexical regex “x”. The <Foo::Bar::baz> syntax for calling a rule from another grammar is also now supported. A nasty bug that caused <!> not to work has been fixed. Finally, <prior> – which matches the same string that the last successful match did – is now implemented.

Better Errors and Exceptions

moritz++ has continued this typed exception work. We’ve also been improving various bits of error reporting to be more informative, and catching a few more things at compile time. moritz++ has also ported STD’s $*HAS_SELF handling, which gives us better handling of knowing when operations needing an invocant can take place. We currently have this work in a branch (it depends on some other work I was doing to align us with changes to how STD parses initializers, and there’s one last bug that prevents us merging it just yet; it’ll be sorted out in the coming days, I’m sure).

Pod Bits

tadzik++ dropped by with a couple of patches, one a bug fix, the other pretty cute: it makes the auto-generated usage message for MAIN subs include the declarator documentation.

Other

Thanks to moritz++, we now support copy and rename functions. There’s also been the usual range of bug fixes that we get into every release, steadily chipping away at making Rakudo better.

Mini-Challenge: Write Your Prisoner's Dilemma Strategy

2012-02-07T16:41:42+00:00

Here is a small task we considered for the Perl 6 Coding Contest, but not chose to not pursue. But it's a nice little challenge for your leisure time.

In the Prisoner's Dilemma, two suspected criminals can choose to not betray each other (which we call "cooperate"), or betraying the other ("defecting"). If only one suspect betrays the other, the traitor gets released and the betrayed one gets a long sentence; if both betray each other, both get a rather long sentence. If both cooperate, both get rather short sentences.

It becomes more interesting when the dilemma is repeated multiple times. Now instead of prison sentences the contestants are assigned scores, which add up over multiple rounds.

I challenge you to write one or two strategies for the iterated prisoner's dilemma, and send them to moritz@faui2k3.org no later than Friday February 17.

You'll find some basic strategies and a harness here. It runs on both newest Rakudo and Niecza.

The scoring is as follows, where True means cooperate and False means defect:

my %scoring =
    'True True' => [4, 4],
    'True False' => [0, 6],
    'False True' => [6, 0],
    'False False' => [1, 1],

Your strategy should be a subroutine or block that accepts the named parameters mine and theirs, which are lists of previous decisions of your own algorithm and of its opponents, and total, which is the number of laps to be played. It should return True if it wishes to cooperate, and False to defect.

Here is an example strategy that starts off with cooperating, and then randomly chooses a previous reaction of the current opponent:

sub example-strategy(:@theirs, *%) {
    @theirs.roll // True;
}

Your strategy or strategies will play against each other and against the example strategies in the gist above. It is not allowed to submit strategies that commit suicide to actively support another strategy.

I too have written two strategies that will take participate in the contest. Here is the checksum to convince you I won't alter the strategies in response to the submissions:

6d4ba99b66e4963a658c8dcfc72922dd0f74e0ad  prisoner-moritz.pl

A four-quarter plan for psyde

2012-02-05T15:15:49+00:00

(If this blog had had a tagging system, this post would have been tagged "meta". You have been warned.)

My blogging engine, psyde, is written in Perl 6. It's been serving me well in the past year-and-a-quarter since use.perl went read-only and forced my hand to move out. It's not too powerful, and it's kinda idiosyncratic, but it has definitely made me happy.

Howver, I've increasingly been picturing where I want to take the engine. It could do more, and I know what bits I want it to do. But I never seem to get to the point where I sit down and tinker with the engine to make it better. You know, 'cus of tuits and yaks and worse-is-better and all that.

So, here goes. Now I'm changing that. Here's a four-quarter plan for 2012:

Q1: A way to create/edit posts online. All my posts sit in a repository, and theoretically I should be able to post from anywhere, just as one would expect from a blogging system. But in practice, I always forget to push the latest changes, which makes my local copy on my laptop at home the athoritative copy, and so I've no choice but to post from home. Better then to author all the posts online, to store the post sources on the server, and to have those be the authoritative copy.

Also, while writing posts in Emacs is hard to beat from a text-editing perspective, one advantage of authoring posts on the web is that I would be able to see the final HTML in real time. Better than having to wait through a compilation process to see how things will turn out.
Q2: A commenting system. Finally. This includes converting the old comments from my use.perl blog and retroactively adding them, too. I'd like to use something like openID as an authentication system, or more generally, something ready-made that makes it easy to just post a comment.

I've had Disqus recommended to me as a way to add a commenting system. While it would certainly work, it's slightly against the spirit of psyde, which is more or less NIH. I believe it's sometimes healthy to reinvent wheels, as much to learn how to make them round as to figure out how they can be made to work better. More importantly, I wouldn't be as happy with someone else's wheel, even a well-oiled one, as with my own.
Q3: A text-oriented graphic library. I often desire to inject an image/graph/diagram or other in my blog posts, explaining the relationship between some things. But there are too many steps involved:
1. open up Inkscape
2. painstakingly draw and position a bunch of rectangles and arrows and text fields
3. export to .png
4. upload the .png
5. insert an <img> tag linking to where I uploaded the .png
...and besides, I always feel like I'm doing work that the computer should be doing for me, drawing the SVG. I've drawn graphs like that a hundred times before, and they all look the same.

Imagine if I could just write a few symbols in my post saying "oh boy, here comes le graph!", a bit like \[ takes me into the displaymath environment in LaTeX. And, and! Because the groundwork I will have done in Q1 enables me to see things as I type in real time, and because the graphic library that I haven't written yet will be mostly implemented in JavaScript, those graphs will appear as I type them, too! Instant graphic!

I haven't seen any other blog system do this. Let me know if there's prior art.

The graphic library will support not just graphs, but any number of pre-set picture drawing modes: graphs, class diagrams, chess boards, hex boards, tables, Nim positions, pentomino pieces, sudoku boards, sparklines, basically anything. Adding another mode should be as simple (for a programmer) as adding a syntax parser and a generator of iamge elements. There will have to be an API for all of this; details pending.
Q4: Productize psyde. After all of this, I actually think I'll have something worth sharing with people. Not everyone will prefer to leave the cozy confines of Wordpress or Movable Type or whatever it is you kids use these days, but some people will like the combination of features in psyde, and will want to set up their own blog with it.

(This already happened once, by the way. I'm not at all prepared for this, so I just sent them a .tar.gz file with the minimal environment needed to get set up, and they succeeded. It was a slightly bumpy ride for them, of course, but they got it working.)

Now psyde isn't really a blogging system at its core, but a static web page generator. Making a product out of it probably means turning a bunch of things into modules, and have them play nicely with the Perl 6 ecosystem. And to make setup a one-click operation, or as close to it as possible. Probably psyde the static web generator and the yet-unnamed blogging software will separate into a module-client relation as a part of this.

When I say "product", by they way, I don't imagine I'll charge money for it. The source will remain free in all senses. I just want to make it easier to approach the software and get started with it. Removing various hard-coded things, as well as adding documentation and installation instructions, will be necessary.

Early on, I told myself that this would be a ten-year blog. We're now a few months into the second year. It's time to introduce some furniture and make this blog engine fit to live in. Fit to relax in after a hard day's work.

I want to be able to blog from my iPad. Or from a 40-inch TV screen. Or, in a pinch, from my Android phone. I want people to be able to comment. I want those graphs, oh nicely laid out, skinnable, auto-generated instant-gratification graphs! And I want to push psyde out into the wild, make it stand on its own legs.

I want to use Perl 6 more in production, because the time is here. Even though I know it's prudent to underpromise and overdeliver, I choose to publish my plans like this because so far I've only been scheming and dreaming — this somehow makes it all more concrete.

Onwards!

t1: Expressing integers using four nines

2012-02-03T22:43:28+00:00

(This is a guest post by Moritz Lenz. If you're wondering what this is all about, it's the aftermath of The 2011 Perl 6 Coding Contest. If you're not wondering, it's still about that.)

Let's consider the first task from the Perl 6 Coding Contest 2011.

Here is the description of the task once more:

What non-negative integers can you write as expressions containing exactly
four occurrences the number 9, and any of the binary operators *, /, %, +,
-, prefix negations, and any number of matching pairs of parentheses you
care to use?

The program should accept an upper limit N as a command-line argument.
It should then print all integers 0..N in increasing order, along with
an expression with four nines, if any such was found.

It was probably the easiest of all tasks, and the one we got the most submissions for. Yet there were still some things that could go wrong, and some submissions got some of them wrong:

non-negative implies that we start at 0 = 9 + 9 - 9 - 9, not at 1
integers means that the result must be an integer; it does not mean that intermediate results are automatically rounded or truncated
the expressions must consist of four times the number 9, not the digit nine. Thus 99 is not a valid expression of two nines.

All contestants solved this task with the following strategy: first find all possible expressions involving four nines (and possibly filter out those out of range), then iterate the numbers from 0 to N and check for each if an expression was found.

One possible approach to generate all expresions is to hard-code all expressions with one nine, namely 9 and -9. Then apply all operators to all combinations of those, and thus generate all expressions with two nines. This is a good time for filtering out duplicates, for example 18 == 9 + 9 == 9 - (-9). Then expressions of greater length can be generated from the shorter ones in the same manner.

There are just two small pitfalls: the first is that one has to remember that not all expressions can be generated by combining expressions of length (N-1) and 1. For example 324 == (9 + 9) * (9 + 9) can only be written as the combination of two expressions with two nines each. The second small pitfall is that three of the operators (%, / and -) are not commutative, so one has to explicitly try both combinations of $a op $b and $b op $a, or take care of that in some other way.

The task description allowed the prefix minus operator, but including it into the production does not produce any new numbers; it is sufficent to seed the expressions of length one with (9, -9).

As usual, we've published each contestant's code along with a review; feel free to have a look. This year, we're also publish solutions from people who sent in solutions but didn't sign up for the contest.

Isn't it noteworthy how, even with this narrow a task, people's solutions are all over the board in terms of the constructs used, code size, and style? We think so.

Stand by for the review on t2: "Sums of cubes".

Announce: Niecza Perl 6 v14 by Stefan O'Rear

2012-01-30T21:00:51+00:00

Announce: Niecza Perl 6 v14

This is the fourteenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. I'm on winter break now, which means lots
of time for improvements. Muahahaha. This month hasn't seen much of a
focus.

With this release, Niecza now passes a similar number of spectests as
Rakudo. (Note that they are not the same tests, and as such the test counts
are not completely comparable.) Solomon Foster has begun efforts to port
panda to niecza, with the goal of designing a multi-implementation ecosystem.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Breaking changes]

Multiple dispatch has been changed to reject all cases of ambiguity using
an explicit list of conflictors; some ambiguous cases were formerly accepted.

List iteration has been modified to throw an exception when the list generator
tries to access the unreified part of the list. The previous code exhibited
undefined behavior that some code actually relied on.

Parameters like @foo and %foo now insist that their arguments be Positional and
Associative respectively.

Rat and Complex stringification has been substantially changed.

Niecza now enforces "trusts".

&nextwith and CallFrame.args now conspire to hide the invocant parameter. If
you are calling nextwith directly, you no longer need to - and must no longer -
pass a self argument.

|$foo capture parameters now capture the logical "current" capture, rather
than the "initial" capture. In particular, 'method (|$foo)' no longer results
in $foo containing self.

[Major features]

END blocks are now suppported.

The subroutine entry process has been significantly modified to allow
signatures to reference variables and anonymous code blocks.

Roles have been overhauled to much more closely match the Rakudo nom behavior.
Importantly, roles now have a $?CLASS parameter, and role blocks are not run
until that is available. Role composition at compile time is now supported,
as is role summation, attributes in roles, role conflict detection, type
checking against roles, etc. &infix:<does> and &infix:<but> now support
much more of specced behaviors.

Accompanying that, the old tag and mixin classes CommonEnum, IntBasedEnum,
StrBasedEnum, Callable, Positional, Associative, Numeric, and Real have been
converted into roles.

Major signature improvements: Added support for sub-signatures, where blocks,
proper MMD with subtypes, values used as types. Parameter and Signature
objects are now reified into Perl 6 space and support a subset of the Rakudo
nom introspection API.

Niecza now supports constant folding! If you mark a sub 'is pure' and call
it with sufficiently constant arguments, it will be replaced at compile time
with the result of the call, provided said call does not throw an exception.
What constitutes "sufficiently constant" is not documented, poorly defined,
and subject to change.

Niecza now keeps attributes from different classes in different namespaces,
so you can have $!x in both a parent and a child class without issues.
Additionally, the sigil is part of the name, so you can now have both $!x
and @!x.

val() is now supported, and is used automatically on <> lists.

MAIN is now supported.

STD-imported syntax changes: \foo and |foo parameters no longer require a
sigil. my \foo = ... works to declare a "raw" variable. Initializer
assignment now binds tightly to the declarator, so that e.g. (5 + my $x = 3)
does something more useful.

.WHICH has been added, and === does the correct thing with value types now.

Perl 5 interop improvements: can now be used from any directory and builds
much more robustly. Also supports more pass and return cases. (Paweł Murias)

[Minor features]

No more pseudo-evaluators - all constructs which contain code that logically
is run at BEGIN time to produce a value, is now actually run rather than
attempting a static evaluation.

Binding to attributes is now supported.

Class attribute forms such as my $.foo, and the aliasing form has $foo, are
now supported.

Within a named 'anon sub', the name is visible, allowing for nicer recursion.

* now properly ignores assignments.

$obj.Foo::bar, $obj.::("Foo::bar"), and /<::("Foo::bar")>/ are now accepted.

You can now set 'is iffy' and 'is diffy' for fully custom operators.

Type adverbs :_ :U :D :T are now accepted.

Signatures like (Int, Int) are now allowed.

Phaser handling has been improved, and support added for CHECK.

Aliasing forms like <foo=$bar> are now allowed.

Version information is now embedded into the build, allowing for --version
and $?PERL support.

Printing of unhandled exceptions uses .gist.

Multiple dispatch now handles junctions.

Exporting multisubs from modules now approximately works.

Added ".Bridge" support. (Solomon Foster)

Attribute-binding parameters (:$!foo) now implemented.

:16() syntax is now supported.

Defaults and type constraints that are constants are now saved as such,
avoiding an unneeded block.

Str.perl now escapes special characters.

CLR interop now supports calls to shadowed and hidden methods, like
$obj.CLR::System::IDisposable.Dispose(). Note that this can NOT be used to
call overriden methods (callvirt semantics are used).

LTM processing ignores arguments and dispatchers forward the arguments to
multi regexes.

infix:<cmp> supports pairs, ±Inf (Solomon Foster)

Hash.perl sorts the output to be slightly more useful.

New setting things:
Cool.polar, Cool.roots, &roots, Array.delete, &infix:<minmax>, &rotate,
CommonEnum.pick, CommonEnum.roll, Any.min, Any.max, Any.minmax, Str.trans,
&elems, Any.reduce, &reduce, &shell, &categorize, &cwd, &chdir, $*CWD,
&printf, IO.copy, IO.chmod
(Solomon Foster)

Hash.push (Moritz Lenz)

Int.base, Array.splice, &splice (Will Coleda)

Set, Bag (Larry Wall, Solomon Foster)

&prefix:<sleep>, Mu.clone, &undefine, $*OS, Order, $*OUT, $*ERR

split and comb now support limits of Whatever (Solomon Foster)

Range coerces values to numbers according to spec (Solomon Foster)

&sort accepts a Callable first argument (Solomon Foster)

Added limit to &lines, :r and :rw to &open (Solomon Foster)

min and max support arbitrarily many arguments (Solomon Foster)

Changed sleep to return a value (Will Coleda)

[Other]

Daniel Ruoso is attempting an implementation of concurrent feeds.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

Next month is likely to see fewer tuits in general. The only concrete plan
I have is to continue with 6model convergence, hopefully reaching a point
where user-defined metaclasses are possible.

[1] https://github.com/downloads/sorear/niecza/niecza-14.zip
[2] https://github.com/sorear/niecza

Macros progress report: interesting things

2012-01-29T18:26:35+00:00

I'm here to report that the macros grant is coming along nicely. I've been busier with $dayjob than anticipated, and so the schedule in the grant application is slipping a bit. But I have a fairly good view of the obstacles and to-do items ahead.

When we last saw each other, I was saying that "variable lookups from inside of the quasiquote end up confused". That's still true, and it's the big thing I want to fix before merging my work so far into the master branch, er, into the nom branch.

What I've done since last time:

I wrote a thorough explanation of why ASTs need to carry around their original lexical environment. (Short explanation: they need to act like closures, but they aren't really so they need to fake it.)
I re-based the macros branch on the latest nom, and fixed a bunch of regressions in my code that fell out of that.
I started a test file with macro tests that are more adapted to the work that I'm doing than the tests that are already there. I hope to be able to absorb the ones that are there as we go along; right now they're not much use to me.

Now I'm well poised to go in and actually implement the lexical fixup I need for the ASTs to behave as if they're normal, honest-to-goblin closures. I just thought I'd blog this report in case I end up walking into the source code, never to return. :-)

The trick is this: .SET_BLOCK_OUTER_CTX. It lets you say "block, your OUTER is now this thingy". It's the kind of internal fixup that makes the cat walk by twice inside the Matrix.

Ok, I'm going in. See you on the other side.

The Perl 6 Coding Contest (2011 edition) is now closed

2012-01-29T12:04:28+00:00

"Thanks for organizing the contest. Staying awake until 3am trying to make my solutions work (and trying to write them in a perl6ish way) was big fun! :)"
        — one of the contestants

Ding! Pencils down.

So, here we are, five weeks later. Let's sum up:

35 people signed up as contestants this year.
6 contestants submitted solutions.
27 submissions passed a base-test run and were accepted.

Last year's figures for the same things are (18, 5, 26), so somehow we ended up with many more contestants and only one more submission this year. That's fine; we know that the step from signing up to actually sending something in is quite a big one to take. Very interested to get feedback on how to make that step easier, though.

Just as last year, we'll synchronously publish people's solutions for each task in sequence, along with a blog post about the task and its solutions.

Unlike last year, I won't promise that these posts will be forthcoming in "the next few days/weeks". Last year it took over two months to process everything and get to the winner. Let's aim for one month this time — we're as expectant as you are to see solutions, blog post explanations, and an eventual winner!

Let's do this! First up: "Four 9s."

Modules and Perl 6s

2012-01-29T03:09:35+00:00

I’ve got my fork of Panda working on Niecza. There are two major issues which remain to be resolved, and they are both related. Panda assumes that the Perl 6 executable is named perl6, and modules should be installed to ~/.perl6. That’s great if you have just one Perl 6 on your system.

But in this crazy modern world, I really want both Rakudo and Niecza on my system. And so assuming that everything can be named “perl6″ is a big problem.

I’m hoping the community can put their heads together and figure out a solution. I guess the obvious one (which would only need sorear to buy in) would be installing Niecza modules to ~/.niecza, and creating a “real” niecza executable. (It might well be as simple as a one-line script “mono path-to-Niecza.exe”.)

Good idea or bad idea?

This month’s Rakudo Star release – and what’s coming next

2012-01-29T00:41:57+00:00

So, we made it – a Rakudo Star release based on the “nom” development branch has landed. It’s based on the compiler release moritz++ cut earlier this week, and pmichaud++, masak++ and myself have been involved in getting the Star-specific build and installation scripts in shape.

Getting to this release has been a lot of work; in many sense this is a revolution in Rakudo’s development rather than a small evolutionary step over what we had before. It’s taken a while (indeed, longer than I’d first hoped) and has been a lot of work – but it was worth it. Not just because of the many improvements that are in this release, but because of the enormous future potential that we now have.

Here’s some of the things I’m happiest about in the release.

The performance improvements in many areas. Yes, we’ve plenty of work to do here – but this is a solid step forward for a wide range of scripts, and in some cases an order of magnitude improvement.
That 6model – something I started designing a year and a half ago – has not only cleanly supported all of the things we needed to do in Rakudo, but also opened up so many other doors. For example, the new NativeCall module uses its representation polymorphism support to great effect.
Protoregexes doing real NFA-driven Longest Token Matching rather than the cheating version we had before that only operated on literals.
The optimizer, along with the various extra compile time error reporting it gives. This will be an important future area for Rakudo.
Initial native type support, and bigint semantics for the Int type.
The POD6 support, thanks to tadzik++’s Google Summer of Code grant in summer.

So, what’s next? Currently I’m working hard on getting true bounded serialization support in place. This should further improve BEGIN time support (including constructs that depend on BEGIN time), greatly cut down resource consumption during CORE.setting compilation (both time and memory) and give us faster startup. It’s hard to guess at figures for the improvement, but I’m expecting it to be a noticeable improvement in all of these areas. I’m aiming at getting this landed for the next Rakudo compiler release (which I expect us to do a Star release based on too), though largely it depends on whether I can get it working and stable enough in time; while some parts are a simple matter or programming, other parts are tricky.

That aside, we’ve already got various other new features in the pipeline; even since last weekend’s compiler release, there are multiple new regex-related things in place, moritz++ has continued with his typed exceptions work, we’re catching a couple more errors at compile time rather than letting them slip through until runtime, and there’s various other miscellaneous bug fixes. Also, masak++ is working on macros in a branch, and I’m optimistic that we’ll have some initial macro support in place by the next release also. Busy times! :-)

Rakudo Star 2012.01 released

2012-01-28T14:34:19+00:00

On behalf of the Rakudo and Perl 6 development teams, I’m happy to announce the January 2012 release of “Rakudo Star”, a useful and usable distribution of Perl 6. The tarball for the January 2012 release is available from http://github.com/rakudo/star/downloads.

In the Perl 6 world, we make a distinction between the language (“Perl 6″) and specific implementations of the language such as “Rakudo Perl”. This Star release includes release #48 of the Rakudo Perl 6 compiler [1], version 3.11 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

Significantly, this is the first distribution release based on the “nom” (New Object Model) development branch of Rakudo. This work has been carried out with the aim of increasing performance and correctness, as well as providing a better base for taking on a
range of missing features. Here are some of the major improvements in this release over the previous distribution release.

Greatly improved performance in many areas. For some scripts, this release offers an order of magnitude performance improvement.
POD6 support, including the $=POD variable to make the POD available at runtime and a –doc option to get at the POD
The Int type now has big integer semantics
Initial work on native types, which can be used to write much more efficient code
LTM-driven protoregexes
Meta-programming support, including custom meta-classes, overriding method dispatch and much more
Exception handling is much closer to the specification, and thus much more useful
Better package handling, including true separate compilation, lexical packages and better nested package handling
An optimizer, which improves generated code as well as catching a range of issues at compile-time that previously made it to runtime
Backslash sequences in character classes
Stubbed methods from roles are now required, providing interface style functionality
Typed arrays and hashes, as well as supporting for binding to array and hash elements

Due to improvements in the Perl 6 language specification, and changes to Rakudo to track them, some existing code will need changes. Here are some of the major differences to be aware of.

Attributes can no longer be initialized using “new” unless they are declared as having an accessor; either give them one or write a BUILD submethod
The proto keyword is no longer used to declare a multi-dispatch fallback
You may no longer do ‘filename’.lines; use ‘filename’.IO.lines

We have maintained backwards compatibility with some changed pieces of syntax, but will drop them in an upcoming release:

“<…>” in proto regex bodies; now this should be written “*”
The use of “**” with a separator in regexes; this is now done by using “%” or “%%” on another quantifier

While this release does contain a great number of improvements, unfortunately we have regressed in a few places. Of note:

Some cases of auto-vivification do not work
The binding of a capture against a signature literal is broken

We will be working to restore this functionality for future Rakudo Star releases; if you depend heavily on it, you may wish to stick with the previous Rakudo Star release for
another month.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:

pack and unpack
macros
threads and concurrency
Unicode strings at levels other than codepoints
pre and post constraints, and some other phasers
interactive readline that understands Unicode
non-blocking I/O
much of Synopsis 9

There is a new online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo Star 2012.01 and other Perl 6 implementations.

In many places we’ve tried to make Rakudo smart enough to inform the programmer that a given feature isn’t implemented, but there are many that we’ve missed. Bug reports about missing and broken features are welcomed at <rakudobug@perl.org>.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. An updated draft of a Perl 6 book is available as <docs/UsingPerl6-draft.pdf> in the release tarball.

[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

Looking back, looking forward

2012-01-15T22:28:47+00:00

So, 2012 is here, and here’s my first Perl 6 post of the year. Welcome! :-)

Looking Back

2011 brought us a faster Rakudo with vastly improved meta-programming capabilities, the first work on exploring native types in Perl 6, the start of a powerful type-driven optimizer and many other bits. It also took me to various conferences and workshops, which I greatly enjoyed. I’d like to take a moment to thank everyone involved in all of this!

This was all great, but slightly tainted by not managing to get a Rakudo Star distribution release out based on a compiler release with all of these improvements. I’d really hoped to get one out in December. So what happened? Simply, there were a certain set of things I wanted to get in place, and while many of them got done, they didn’t all happen. While the compiler releases are time based – we do one every month – the distribution releases are more about stability and continuity. By the time I headed back to the UK to spend Christmas with family, we were still missing a some things I really wanted before a Star release was done. Given the first thing that happened when I started relaxing a little was that I immediately got unwell, I figured I should actually use my break as, well, a break – and come back recharged. So, I did that.

So, let’s try again

So, the new goal is this month. I’m happy to report that in the week since I’ve got back to things, one of the things I really wanted to sort out is now done: Zavolaj, the native calling library, now does everything the pre-6model version of it did. In fact, it does a heck of a lot more. It’s even documented now! It’s also far cleaner; the original implementation was built in the space of a couple of days with mberends++ while I was moving apartment, and was decidedly hacky in places. The missing bits of the NativeCall library were important because they are depended on by MiniDBI, and I really didn’t want to ship a Rakudo Star that can’t connect to a database. So, next up is to make sure that is in working order. I’m not expecting that to be difficult.

That aside, there were some things to worry about in Rakudo itself. I’ve dealt with some of those things in the last week, and perhaps the one important remaining thing I want to deal with before Star is a nasty regex engine backtracking related bug (I’ve been hoping pmichaud++, the regex engine guru, might appear and magic it away, but it seems it’s going to fall on my plate). But overall, we’re well on track to cut the Star release this month.

What’s the direction for the year ahead?

During 2011, Rakudo underwent a significant overhaul. It was somewhat painful, at times decidedly not much fun, but ultimately has been very much worth it: many long standing issues have been put to rest, performance has been improved and many things that were once hard to do are now relatively easy or at least accessible.

I think it goes without saying that we won’t be doing any such wide-ranging overhaul in 2012. :-) The work in 2011 has opened many doors that we have yet to walk through, and 2012 will see us doing that.

At a high level, here’s what I want:

Less bugs, more stability: mostly this is about continuing to work through the ticket queue and fix things, adding tests as bugs are fixed to ensure they stay fixed.
Better error reporting: there are things in STD, the Perl 6 standard grammar, that allow it to give much more informative error reports on syntax errors than we often can in Rakudo today. I want us to bring these things into Rakudo. Additionally, there’s plenty of improvements to be made in runtime errors. Furthermore, I want to expand the various bits of static analysis that I have started doing in the optimizer to catch a much wider range of errors at compile time.
Run programs faster: this process is helped by having decent profiling support these days. There’s a lot more to be done here; the optimizer will help, as will code generation improvements.
Compile programs faster: this will come from more efficient parsing and greatly improving the quality of the code NQP generates (NQP is the language we write much of the compiler in)
Shorter startup time: this mostly involves finishing the bounded serialization work up. I think the best way to describe this stuff is “fiddly”.
Use less memory: Rakudo has got faster in no small part thanks to being able to understand its performance through profiling. Our understanding of its memory consumption is much more limited, which makes it harder to target improvements. That said, I’ve some good guesses, and some ideas for analyzing the situation.
More features: while being able to do the things Rakudo can do today faster and with less bugs would give a very usable language for quite a lot of tasks, there’s still various features to come. In particular, Rakudo’s support for S05 and S09 needs work.
VM Portability: we’ve made good progress towards being able to make a serious stab at this over the last year, while at the same time also managing to perform vastly better on Parrot too. With help from kshannon++, I’m currently working on completing the switch to QRegex (a regex engine with NFA-powered LTM, and written in NQP rather than PIR), which should carry on a pattern of simultaneously increasing performance and portability. Beyond that will be an overhaul of our AST and code generation, with the same goal in mind.

So, lot’s of exciting things coming up, and I look forward to blogging about it here. :-)

A way to help

There are many ways to get involved, depending on what you’re interested in. One way is to take a look at our fixed bugs that need tests writing. At the time of writing, just short of 100 tickets are in this state. No guts-y knowledge needed for this – you just need to understand enough Perl 6 (or be willing to learn enough) to know what the ticket is about and how to write a test for it. Drop by #perl6 for hints. :-)

Last Piece of Pi

2012-01-15T17:43:08+00:00

So, last spring I did a series of posts on making a Pi spigot. Unfortunately, the project foundered a bit, as it turned out that only Rakudo had the subsignatures needed to make the code pretty, but only Niecza had the big integers needed to make the code work.

Fast forward to today. Now both Rakudo and Niecza can handle the best version of the code with no problem!

Let me emphasize that again. A year ago, neither implementation had those two features. Eight months ago, each had one. Today both have both. What’s more, Rakudo seems to have made some pretty impressive speed improvements.

I think the awkward situation with Rakudo Star has helped obscure the fantastic good news in Perl 6. Right now we have two distinct Perl 6 implementations, each of which is markedly better than anything available a year ago. While there are still some rough edges, both implementations are making visible progress every single day.

Okay, enough cheerleading. Time to finish the Pi story. It turned out there was one last complication. I had never actually tried to use the code to generate an infinite stream of Pi. I always stopped at 40 digits. Confident that the algorithm must work, I tweaked the output to just continue generating digits until you stopped the program with control-C. And I ran it using Niecza, and it printed

3.14159265358979323846264338327950288419716
9399375105820974944592307816406286208998628
0348253421170679821480865

And then it just sat there, calculating. After a few minutes I stopped it and tried it in Rakudo, and got the exact same result.

Well, after adding a few debugging says here and there, I found the problem. The extr sub was returning NaN. It turns out that both compilers have issues with dividing huge numbers. Here’s the division operation that was causing the problem:

826855066209083067690330954944954674053
707782399091459328155002954168455127712
564546723209828068849110223672691692080
858850302237001093531862737473606364113
314687502675869281622802970765988449203
963736097729699655628829895255493809983
868753943269929165690008254816168624365
041070395716948346309925280258763697273
816643106559428329680316113883598846477
019844021876290510680558354153412094804
165563855909020631086890050609449881578
622437959410200560054513816596644762131
226627968813825552929967132893776980417
525678140579476414867767644626389410380
794467097761379794479928269796859019439
705966555011741254554959832606241504043
482378842096776403191455346497512084739
323724281071973237937801014210278895804
940475966938880398182275335278425442994
287812050560074302564177393567873480740
249095636709741437469651121924884638352
523975466249955052660310789169884060356
70777782797813415527343750 / 7640045443
915776552858245682965495041201868477244
923723289802372673948570989301189643881
881673616027696317656701576457227272117
067947294675092324411286583110995372015
785893970194452345100095389207557064515
618905243737091067059065039684867000766
465399984513882758095027633673968549038
659642193965599458094646356792444696562
299054844575814305259223023977803302735
242307789179027935107449661143998428584
590618630170775872761731454567203230484
311106708425828778192240791257477924515
937573923664112071637127786446287936043
637833776529791999568414593746973068229
015816020732598109749879566833692821582
816119454436978125677364775318707235283
939392855143663977524974469973442065855
128922452382372338686111634084367257050
255499210918328304009454606504169283500
652033488755998721320134300149134205253
095344802815192912405314659633341062491
389270940370884860337596933277382049709
5584869384765625

Turns out the bottom number is bigger than can be presented by a Num, so the division operation ends up becoming N / Inf, which is NaN. This is a bit obscured in Rakudo because the division operation actually returns a Rat (which should be illegal according to the spec!), but then .floor is called on the result, which tries to convert to Num before doing the floor operation.

This opens several questions, like: Should Perl 6′s Int / Int operator be modified to try to cope with this sort of case?

But for the Pi problem, the good news is this: every time the extr sub is called, its result is fed to .floor. That means we simply needed to replace / with div and get rid of the .floors.

And with that, the code easily produces 2,000+ digits of Pi using either Rakudo or Niecza!

State of Dancer on Perl 6

2012-01-06T00:24:34+00:00

Bailador is growing better and bigger and starts to resemble a real tool more and more. Let’s see what new features it has gained recently.

Remember the first example in perldoc Dancer? It’s not really different in Bailador:

use Bailador;
get '/hello/:name' => sub ($name) {
    return "Why, hello there $name"
}
baile;

Aside of being a little Spanished, what did it give us? We have subroutine signatures in Perl 6 so we can pass the :name parameter to the sub; there’s no need to use param() now: it’s gone.

You don’t need to pass everything using GET of course. post keyword is also supported.

post '/' => sub {
    return request.params.perl
}

The above will print something like ("foo" => "bar").hash, if fed with appropriate request.

any() is a reserved keyword in Perl 6, and while you can use it, it means a completely different thing. Instead of any('get', 'post') you can just do it like this:

get post '/' => sub {
    if request.is_get {
        return "I am GET"
    } else {
        return request.params.perl
    }
}

post, as well as get return their arguments, so you can chain them like in the example above. It also shows the joy of request object, which you can use to inspect the request being processed. It’s not as cool as Dancer::Request, but it does the job, being quite small and simple.

What else do we have? Let’s show off a bit and write a simple-simple pastebin webapp.

use Bailador;

unless 'data'.IO ~~ :d {
    mkdir 'data'
}

get '/' => sub {
    template 'index.tt'
}

post '/new_paste' => sub {
    my $t  = time;
    my $c = request.params<content>;
    unless $c {
        return "No empty pastes please";
    }
    my $fh = open "data/$t", :w;
    $fh.print: $c;
    $fh.close;
    return "New paste available at paste/$t";
}

get /paste\/(.+)/ => sub ($tag) {
    content_type 'text/plain';
    if "data/$tag".IO.f {
        return slurp "data/$tag"
    }
    status 404;
    return "Paste does not exist";
}

baile;

Holy cow, what’s that! Let’s go there piece by piece. First, we’ll create a data directory if it doesn’t already exist. No black magic here, let’s proceed. What’s next? Templates! Here we just load index.tt, not passing any parameters, but that works too and some example apps use that in their example templates.

The handler of new_paste uses our well-known request object again, and creates a new file for a paste, identified by the current time.

The last get block uses some nifty features, so let’s take a look. It uses regexes, and you can see that they also cooperate with subroutine parameters without black magic. We then set a content_type as we’ll do in Dancer, and send status 404 if no paste have been found. Easy peasy? I suppose so. That’s it, it works like a charm.

Thus we’ve covered all the features in Bailador as for now. I don’t think it’s that poor, as for about 100 lines of code.

What’s next? What’s missing? You tell me. Or you contribute; the code is dead simple and implementing stuff like before(), after(), before_template() etc should be a matter of 3-5 lines, I think. Feel encouraged to look into the code and hack on it. If you have any questions, suggestions or criticism, don’t hesitate to tell, or poke me on #perl @ Freenode. Have fun!

Perl 6 in 2011 - A Retrospection

2011-12-31T18:00:00+00:00

The change of year is a good occasion to look back. Here I want to reflect on the development of Perl 6, its compilers and ecosystem.

At the start of the year, masak's Perl 6 Coding Contest continued from 2010, concluding in the announcement of the winner. I must admit that I still haven't read all the books I won :-)

Specification

2011 was a rather quiet year in terms of spec changes; they were a mixture of responses to compiler writer and user feedback, and some simplifications and cleanups.

Positional parameters used to be allowed to be called by name; this feature is now gone. That both makes the signature binder simpler, and removes accidental dependencies on names that weren't meant to be public. Read the full justification for more background.

A small change that illustrates the cleanup of old, p5-inherited features was the change that made &eval stop catching exceptions. There is really no good reason for it to catch them, except Perl 5 legacy.

say now uses a different stringification than print. The reasoning is that print is aimed at computer-readable output, whereas say is often used for debugging. As an example, undefined values stringify to the empty string (and produce a warning), whereas say calls the .gist method on the object to be said, which produces the type name on undefined values.

An area that has been greatly solidified due to implementation progress is Plain Old Documentation or Pod. Tadeusz Sośnierz' Google Summer of Code project ironed out many wrinkles and inconsistencies, and changed my perception of this part of the spec from "speculative" to "under development".

Rakudo

Rakudo underwent a huge refactoring this year; it is now bootstrapped by a new compiler called "nqp", and uses a new object model (nom).

It allows us to gain speed and memory advantages from gradual typing; for example the mandelbrot fractral generator used to take 18 minutes to run on a machine of mine, and now takes less than 40 seconds. Speedups in other areas are not as big, but there is still much room for improvement in the optimizer.

With the nom branch came support for different object representations. It makes it possible to store object attributes in simple C-like structs, which in turn makes it much easier and more convenient to interoperate with C libraries.

Tadeusz' work on Pod gave Rakudo support for converting Pod to plain text and HTML, and attach documentation objects to routines and other objects.

Rakudo now also has lazy lists, much better role handling, typed exceptions for a few errors, the -n and -p command line options, support for big integers, NFA-based support for proto regexes and improvements to many built-in functions, methods and operators.

Niecza

It is hard to accurately summarize the development of Niecza in a few sentences; instead of listing the many, many new features I should give an impression on how it feels and felt for the user.

At the start of 2011, programming in niecza was a real adventure. Running some random piece of Perl 6 code that worked with Rakudo rarely worked, most of the time it hit a missing built-in, feature or bug.

Now it often just works, and usually much faster than in Rakudo. There are still some missing features, but Stefan O'Rear and his fellow contributors work tirelessly on catching up to Rakudo, and it some areas Niecza is clearly ahead (for example Unicode support in regexes, and longest-token matching).

Since Niecza is implemented on top of the Common Language Runtime (CLR) (which means .NET or mono), it makes it easy to use existing CLR-based libraries. Examples include an interactive fractal generator and a small Tetris game in Perl 6.

Perlito

Perlito aims to be a minimal compiler with multiple backends, which can be used for embedding and experimenting with Perl 6. It had several releases in 2011, and has interesting features like a Javascript backend.

Ecosystem

The presence of two usable compilers (and in the case of Rakudo, two viable but very different branches) has led to many questions about the different compilers. The new Perl 6 Compiler Feature matrix tries to answer the questions about the state of the implemented features in the compilers.

With Panda we now have a module installer that actually works with Rakudo. It still has some lengths to go in terms of stability and feature completeness, but it is fun to work with.

The new Perl 6 Modules page gives an overview of existing Perl 6 modules; we hope to evolve it into a real CPAN equivalent.

Community

This year we had another Perl 6 Advent Calendar, with much positive feedback both from the Perl 6 community and the wider programming community.

We were also happy to welcome several new prolific contributors to the Perl 6 compilers and modules. The atmosphere in the community still feels relaxed, friendly and productive -- I quite enjoy it.

The year ends like it started: with a Perl 6 Coding Contest. This is a good opportunity to dive into Perl 6, provide feedback to compiler writers, and most of all have fun.

The -c flag

2011-12-28T20:49:17+00:00

The p6cc contest is underway. Yay.

[Coke]++ discovered on the channel that Rakudo nom didn't have a -c flag. The five base-test files all syntax-check the corresponding code files using the -c flag. Which made Rakudo nom and the base-test files incompatible. Oh noes.

<moritz> moritz-- # not reviewing the test harness properly

The fault is even more mine, of course, since I wrote the test harness. And I may be an "early adopter" with Perl 6, but I'm always very late at switching over to a new Rakudo branch.

I was late at switching over to ng, back when it was still called ng. And I'm late this time in switching over from b (the new name for ng, since the n stands for "new" and ng isn't new anymore) to nom (the new "new").

I'll take the leap any day now, I promise.

moritz++ was quick in patching up the -c omission.

<dalek> rakudo/nom: a9bead6 | moritz++ | src/ (2 files):
<dalek> rakudo/nom: reimplement -c command line option
<dalek> rakudo/nom: review: https://github.com/rakudo/rakudo/commit/a9bead6d48
<moritz> masak: there you go
<masak> moritz++

What this means in practice is: you can't use the latest Rakudo nom compiler release to solve the p6cc problems. Not without modifying the base-test files anyway. But you can use the bleeding-edge git checkout of the nom branch.

If you're on either Niecza or Rakudo b, things should be fine: those have a working -c flag already.

Those are the breaks. Perl 6 is evolving, and the mat is constantly being pulled out from under us. To keep up, one has to do a jig now and then.

We added a NOTES file to keep track of information of this kind that we didn't manage to get into the contest instructions.

On the channel, we also had some nice concluding discussion about the nature of the -c flag.

<moritz> to me it felt a bit like a cheat
<moritz> because there is already some mechanism for specifying the target stage
<moritz> but it's too tightly coupled to the output from the existing stages to
         be easily usable
<moritz> so I feel like hijacking an existing mechanism
<masak> I guess.
<masak> in some sense, "checking syntax" isn't so much of a compiler stage as...
        a decision not to go past a certain compiler stage.
<TimToady> in a sense, -c adds the final CHECK, that just exits with status
<masak> right.
<TimToady> it can even be implemented that way, since CHECKS do lifo order
<masak> "everything turns out to be yet another setting" :)
<TimToady> yes, it could also be done with a variant setting, but that seems
           a bit heavyweight
<TimToady> otoh, it would be possible to sneak CHECK pushes in before the -c,
           so maybe a setting is the cleaner way

Announce: Niecza Perl 6 v13 by Stefan O'Rear

2011-12-26T15:55:30+00:00

Announce: Niecza Perl 6 v13

This is the thirteenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. I'm on winter break now, which means lots
of time for improvements. Muahahaha. A large portion of the improvements
have been in regular expression support.

Will Coleda and Solomon Foster are working on fudging spectests so they run on
Niecza; between that and actual improvements, we've gained 5000+ working
spectests since the last release. See [4] for a dramatic visual.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Breaking changes]

/ <{ foo }> / is no longer accepted as a synonym for / <?{ foo }> /.

$0 is no longer allowed to mean $/ when there is no capture zero.

$/.ast no longer defaults to ~$/. Use $() if you want that.

[Major features]

New-style character class expressions like <:Uppercase & :Greek> are
now supported.

Unicode property access is now supported! In addition to the use in
regexes, there is also a minimal Niecza::UCD module which allows querying
the properties of characters. All non-Unihan properties defined in
Unicode 6.0.0 are available.

Runtime number parsing has been radically extended and now supports the
full gamut of Perl 6 number syntaxes.

[Minor features]

Supplementary characters are now generally supported, though StrPos-type
counting for chars, substr, etc is still in UTF-16 code units.

codes is now available, for when you need to actually count code points
(be aware that it is O(n)).

The regex infix operators & and && are now supported (currently treated
as synonyms, but don't rely on this).

&pow is a little bit smarter and needlessly returns NaN in fewer cases
(Solomon Foster, Stefan O'Rear).

Imported a few tweaks from STD, including a better message for say$_.

\h \v \s \w and \d now use the recommended UTS18[3] definitions.

\n, $$, and ^^ now match any vertical whitespace, including CRLF as
a single unit.

% is now supported for all quantifiers and %% is available too.

$() @() %() are now supported.

/$var/ now allows $var to be a Regex. Likewise /<$var>/, /@var/,
and /<@var>/ have been added. / <{...}> / now does the right thing.

The implementation of $<foo>=[...] has been considerably simplified
and depessimized.

/ <.$foo> / assertion syntax is now supported.

/ <foo> / will now call a lexical "my regex foo" if possible. To minimize
potential accidents, this applies ONLY to regexes, tokens, and rules;
despite using the same namespace, a "my sub foo" will not be called.

/ <&foo(...)> / now allows arguments.

@<foo> now correctly contextualizes.

Nontrivial regex protos, like "proto regex foo { "bar" {*} }", are now
implemented.

<( and )> are now supported.

Added Match methods: kv, keys, values, caps, chunks, prematch, postmatch.

Added \c[] syntax in strings and regexes.

Rat and FatRat now stringify as fractions.

Inf now stringifies as "Inf" rather than leaking C# Infinity.

Added predefined quasi-property rules like <alnum>.

Added Niecza::Benchmark, providing the barest minimum of functionality like
Perl 5's "Benchmark".

Inf, NaN correctly handled in rounding, Cool.truncate, &kv, Pair.invert,
&srand, allow :x(*) in subst, .chr, .ord, .chrs, .ords, &chrs, &ords,
trim methods, &roll, .roll(*), end, min, max, minmax (Solomon Foster)

Added .pick(*), corrected Str and Numeric for Range (Will Coleda)

classify (Moritz Lenz)

[Selected bug fixes]

Threads.pm6 is working again, now with exception-safe locking.

[Other]

There is now documentation on how to prepare releases.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

My current priorities are:
1. Make regexes much more feature-complete, including general Unicode
properties and grapheme mode
2. Prototype the debugger
3. 6model convergence work, including roles/native types
4. Figure out how modules and S11 stuff should work in Niecza. Do it.

[1] https://github.com/downloads/sorear/niecza/niecza-13.zip
[2] https://github.com/sorear/niecza
[3] http://www.unicode.org/reports/tr18/
[4] https://github.com/flussence/specgraphs/raw/master/impls.png

Day 25 – Merry Christmas!

2011-12-25T10:52:41+00:00

The kind elves who spend the rest of the year working in Santa’s shop to bring you more of Perl 6 each year would like to wish you a very warm and fuzzy Christmas vacation. December is always a special time for us, because we get to interact with you all through the interface of the advent calendar. We think that’s wonderful.

Be sure to check out this year’s Perl 6 coding contest, where you can win €100 worth of books!

Merry Christmas!

The 2011 Perl 6 Coding Contest

2011-12-25T10:20:17+00:00

With slightly less gratuitous buildup of expectations this year, but with no less grandeur and excitement, we're proud to announce:

The 2011 Perl 6 Coding Contest

Today there's two of us arranging the contest: the esteemed Moritz Lenz, and me. Hello.

This is a contest for people who are aware that Perl 6 has been "officially released", and who want the perfect excuse to start playing around with the language. As the Perl 6 community steadily works its way towards production-readiness, we can already enjoy ourselves royally by solving interesting Computer Science problems!

Here's the deal:

do five tasks, and you might win Amazon books worth €100!

Addendum: Thanks to an anonymous donor, we're very happy to announce that there is now a second prize too: Amazon books worth 100 USD!

The contest starts now, today on 2011-12-25. It ends five weeks later, on 2012-01-29. Registration is open for two weeks, starting now. Just send an email to cmasak@gmail.com — saying "sign me up!" or even sending your Amazon wishlist so we'll know which books to buy you if you win.

You might be curious about what you need to do to win. Here's an extract from the file RULES:

Since "code quality" is a slightly subjective measure, let us provide a few
hints of what we'll be looking for:

* Correctness.
* Readability.
* Consistency.
* Clarity of intent.
* Algorithmic efficiency.
* Idiomatic use of Perl 6.
* Brevity.

In short, what we're looking for is top-quality code. That's how you win.

Here are the five tasks. Write Perl 6 programs to...

Find a way to express an integer as an expression containing four 9s. (more)
List the numbers which are sums of cubes in more ways than one. (more)
Calculate addition chains. (more)
Slide a hex-shaped piece across a board. (more)
List all possible trees of a certain type. (more)

We've chosen the problems so that they're easy to explain, but allow contestants quite a bit of freedom to play around with various solutions.

If you're curious about the details of the contest, I recommend the file RULES. For easy downloading, here's a .zip file of everything you need for the contest.

Addendum: For useful afterthoughts, we've decided to keep a file NOTES around. Think of it as useful clarifications to the existing files in the repo. We won't pass or fail people based on things found in NOTES, but the items may help you with the tasks.

The rest is up to you. Hope hear from you — we want you to win those books!

Announce: Perlito version 8; Perl5 and Perl6 compilers by Flavio S. Glock

2011-12-24T11:26:26+00:00

Perlito is a compiler collection that contains both a Perl 5 compiler
(perlito5) and a Perl 6 compiler (perlito6).

The following backends are supported:

- Perl5 in Javascript (browser and command-line)

- Perl6 in Javascript (browser and command-line)
- Perl6 in Perl5 (v6.pm, runs in perl 5.8 and up)
- Perl6 in Python (Python 2.6 and 2.7)

There are also many experimental backends.

Web

Run Perlito online, in the browser:

http://perlcabal.org/~fglock/perlito5.html

http://perlcabal.org/~fglock/perlito6.html

From CPAN:

http://search.cpan.org/dist/v6/lib/v6.pm

$ cpan v6

Perlito is a work in progress. For the source code, to-do list, and bugs:

http://github.com/fglock/Perlito

- Flávio S. Glock (fglock)

Day 24 — Subs are Always Better in multi-ples

2011-12-24T02:15:26+00:00

Hey look, it’s Christmas Eve! (Also, the palindrome of 42!) And today, we’re going to learn about multi subs, which are essentially synonyms (like any natural language would have). Let’s get started!

An Informative Introduction

multi subs are simply subroutines (or anything related to it, such as methods, macros, etc.) that start with the multi keyword and are then allowed to have the same name as another sub before, provided that sub starts with a multi (or proto — that’s later) keyword. What has to be different between these subs is their signature, or list of formal parameters.

Sound complicated? It isn’t, just take a look at the example below:

multi sub steve(Str $name) {
    return "Hello, $name";
}

multi sub steve(Int $number) {
    return "You are person number $number to use this sub!";
}

Every sub was started with the multi keyword, and has the same name of “steve”, but its parameters are different. That’s how Perl 6 knows which steve to use. If I were to later type steve("John"), then the first steve gets called. If, however, I were to type steve(35), then I’d get the second steve sub.

Equal Footing with built-ins

When you write a multi sub, and it happens to have the same name as some other built-in, your sub is on equal footing with the compiler’s. There’s no preferring Perl 6′s multi sub over yours, so if you write a multi sub with the same name as a built-in and with the same signature, say

multi sub slurp($filename) {
    say "Yum! $filename was tasty. Got another one?";
}

And then try calling it with something like slurp("/etc/passwd"), I get this:

Ambiguous dispatch to multi 'slurp'. Ambiguous candidates had signatures:
:(Any $filename)
:(Any $filename)

Why? Because Perl 6 found two equally valid choices for slurp("/etc/passwd"), my sub and its own, and was unable to decide. That’s the easiest way I know to demonstrate equal footing.

A Fun Conclusion

Now, since it’s Christmas, let’s try writing another open sub, but unlike the built-in open sub, which opens files, this one open presents! Here’s our Present class for this example:

class Present {
    has $.item;
    has $.iswrapped = True;

    method look() {
        if $.iswrapped {
            say "It's wrapped.";
        }
        else {
            say $.item;
        }
    }

    method unwrap() {
        $!iswrapped = False;
    }
}

Now, our open multi sub looks like this:

multi sub open(Present $present) {
    $present.unwrap;
    say "You unwrap the present and find...!";
}

The signature is vastly different from Perl 6′s own open sub, which is a good thing. And here’s the rest of the code, which makes a Present and uses our new multi sub:

my $gift = Present.new(:item("sock"));
$gift.look;
open($gift);
$gift.look;

But wait!

Running this gets us an error in the latest pull of Rakudo:

$ ./perl6 present.p6
It's wrapped.
This type cannot unbox to a native string
⋮

This means that Perl 6′s original open sub is being used, so perhaps it’s being interpreted as an only sub (only subs are the default — only sub unique() {...} and sub unique() {...} mean the same thing). No matter, let’s try adding a proto sub line before our multi sub:

proto sub open(|$) {*}

A proto sub allows you to specify the commonalities between multi subs of the same name. In this case, |$ means “every possible argument”, and {*} means “any kind of code”. It also turns any sub with that name into a multi sub (unless explicitly defined as something other than multi). This is useful if you’re, say, importing a &color sub from a module that isn’t defined as multi (or explicitly as only) and you want to have your own &color sub as well.

After adding this before our multi sub open, we get this result:

$ ./perl6 present.p6
It's wrapped.
You unwrap the present and find...!
sock

It works! Well, that’s it for multi subs. For all the nitty-gritty details, see the most current S06. Enjoy your multi subs and your Christmas Eve!

Day 23 – Idiomatic Perl 6

2011-12-23T23:16:52+00:00

Perl 6 Idioms, and Idiomatic Perl 6

(Butterflies of the world, alight!)

Perl is a richly expressive language, with a warm and playful community. When someone crafts a succinct way to solve a common problem, the Perl community often adopts that solution's phrasing as a idiom. (Other-times, the community recoils in horror and proposes a less obtuse phrasing.)

Perl 6 adds many elements to the language, and not just keywords; there are metaoperators, clean exceptions, new contexts, better interpolation, frugal OO, and much more. Those additions were shaped by patterns of Perl 5 use (with an eye to future uses), so at least one should scratch some itch you have long ignored. The new bits aren't required, but we like the shiny, so be prepared for a slew of new idioms soon after Christmas.

Most Perl 6 idioms will not just be translated versions of familiar Perl 5 idioms; they will use new features where appropriate, and may seem unfamiliar at first. If you regularly use any of the Perl 5 idioms below, you can expect to grok its new form soon after you embrace Perl 6 itself.

Idiomatic Perl 6 should feel just as Perlish as Perl 5, once you get used to it. After all, it is all Perl.

Conventions

(Details, details)

Definitions (corrupted^W adapted for computer languages):

Idiom: A phrase expected to be used to express an idea.
Idiomatic: Expressed as the language's native users would state it.

In most examples below, the code is shown in four versions:

1. Non-idiomatic Perl 5,
2. then made idiomatic.
#
3. Perl 5 idiom, naively translated into Perl 6,
4. then made idiomatic.

Any versions past #4 are to show TMTOWTDI. Notice how the code gets clearer or more concise as it goes from 1 to 4.

Idiom ==> Word

(Movin' on up)

Some Perl 5 idioms were so useful and conceptually concise, that they became Perl 6 "words" in their own right. These words take the form of new built-in operators, functions, and methods.

 # Pick a random array element
 $z = $array[ int(rand scalar(@array)) ];
 $z = $array[ rand @array ];
 #
 $z = @array[ rand*@array ];
 $z = @array.pick;            

 # Loop over the keys (indexes) of an array
 for ( my $i=0; $i<@array; $i++ ) {...}
 for my $i ( 0 .. $#array ) {...}
 #
 for 0 .. @array.end -> $i {...}
 for @array.keys -> $i {...}

 # Whole number division
 ( ($x - ($x % 3) ) / 3 )
 int( $x / 3 )
 #
 Int( $x / 3 )
 $x div 3                  # Integer division op

 # Print the count of the elements of an array.
 say scalar @array;
 say 0+@array;
 #
 say 0+@array;      # Identical in Perl 6
 say +@array;       # + forces the new "numeric" context
 say @array.elems;  # .elems method is more explicit.

 # Do something every 5th time
 if ( ($x/5) == int($x/5) ) {...}
 if ( !($x % 5) ) {...}
 #
 if !($x % 5) {...}
 if $x %% 5 {...}           # %% means "is evenly divisible by"

 # Do something $n times, counting up to $n-1
 for ( $_=0; $_ < $n; $_++ ) {...}
 for ( 0 .. ($n-1) ) {...}
 #
 for 0 ..^ $n {...}
 for ^$n {...}                      # ^9 means 0 ..^ 9, or 0..8

Bare method calls are *always* methods on $_, eliminating Perl 5's confusion on which functions default to $_.

Other defaults have been tweaked to move from something-else-I-must-remember to obvious.

 # Split on whitespace
 @words = split /\s+/, $_;
 @words = split;          # Default is useful, but not intuitive
 #
 @words = .split(/\s+/);  # split() now has no default pattern
 @words = .words;         # split's old default behavior is now a separate method: .words

 # Split a string into individual characters.
 @chars = map { substr $word, $_, 1 } 0..length($word);
 @chars = split '', $word; # Split on nothingness
 #
 @chars = $word.split('');
 @chars = $word.comb;      # Default is to "keep everything"

 # Infinite loop
 for (;;) {...}    # Spoken with a 'C' accent
 while (1) {...}
 #
 while 1 {...}
 loop {...}        # No limit given, so endless by default

Some idioms that became words were already words in Perl 5, if you used the appropriate module.

 # Return the unique elements from a list, in original order
 my %s, @r; for @a { push @r, $_ if !$s{$_}; $s{$_}++; } return @r;
 my %s; return grep { !$s{$_}++ } @a;    # or List::MoreUtils::uniq
 #
 my %s; return grep { !%s{$_}++ }, @a;
 return @a.uniq;

 # Add up all list elements
 my $sum = 0; for my $num (@a) { $sum += $num }
 my $sum; $sum += $_ for @a;    # or List::Util::sum
 #
 my $sum = @a.reduce(*+*);
 my $sum = [+] @a;              # [op] applies op to entire list

Idiom ==> Idiom

(The song remains the same)

Some idioms remain the same, modulo required syntax changes.

 @alpha = 'A' .. 'Z';

 @a = qw{ able baker charlie };

 %meta = ( foo => 'bar', baz => 'quz' );

 @squares = map { $_ * $_ }, @a;

 @starts_with_number = grep { /^\d/ }, @a;

Didn't those all look familiar? Sure, parenthesis are no longer needed on list assignment, and qw{} now has a an angle-bracket shortcut, but those are optional, and don't really change the gist of the idioms. Add that comma after the map|grep block, and these Perl 5 idioms still stand strong in Perl 6.

Perl 5's "magic diamond" idiom still exists, just spelled more sanely (and less diamond-ly).

 # Process each line from STDIN or from command-line files.
 for my $file (@ARGV) { open FH, $file; while (<FH>) {...} }
 while (<>) {...}               # Null filehandle is magical
 #
 for $*ARGFILES.lines {...}
 for lines() {...}              # lines() defaults to $fh = $*ARGFILES

Idiom 5 ==> Idiom 6

(The more things change, the more they remain the same)

Some idioms have taken new form, when the idea behind the original idiom found better expression through new language elements.

 # Hash initialization to constant
 my %h;   for (@a) { $h{$_} = 1 }
 my %h = map { $_ => 1 } @a;
 #
 my %h = map { $_ => 1 }, @a;
 my %h = @a X=> 1;

 # Hash initialization for enumeration
 my %h;   for (0..$#a) { $h{ $a[$_] } = $_ }
 my $c;   my %h = map { $_ => ++$c } @a;
 #
 my $c;   my %h = map { $_ => ++$c }, @a;
 my %h = @a Z=> 1..*;
 my %h = @a.pairs».invert;  # if zero based

 # Hash initialization from parallel arrays
 my %h;   for (@a) { $h{$_} = shift @b }
 my %h;   @h{@a} = @b;
 #
 my %h;   %h{@a} = @b;
 my %h = @a Z=> @b;

 # Swap two variables
 my $temp = $x; $x = $y; $y = $temp;
 ( $x, $y ) = ( $y, $x );
 #
 ( $x, $y ) =   $y, $x;
 ( $x, $y ) .= reverse;   # .= makes reverse into a "mutating" method
 # Tastes great on array swaps, too!   @a[ $j, $k ] .= reverse;

 # Rotate array left by 1 element
 my $temp = shift @a; push @a, $temp;
 push @a, shift @a;
 #
 @a.push: @a.shift;
 @a .= rotate;

 # Create an object
 my $pet = new Dog;
 my $pet = Dog->new;
 #
 my $pet = Dog.new;
 my Dog $pet .= new;    # $pet *always* isa Dog; Compiler can optimize!

Combining transformation with selection was an advanced idiom in Perl 5. The new return values for if provide a bite-sized idiom.

 # Three copies of elements > 5
 @z = map { ($_) x 3 } grep { $_ > 5 } @y;    # map,grep
 @z = map { $_ > 5 ? ($_) x 3 : () } @y;      # map as grep
 #
 @z = map { $_ > 5 ?? ($_) xx 3 !! Nil }, @y;
 @z = @y.map: { $_ xx 3 if $_ > 5 };          # !if == Empty list
 @z = ($_ xx 3 if $_ > 5 for @y);             # List comprehension

That fifth form is a list comprehension, popular in functional languages, and in our closer cousin.

Sentence|Paragraph ==> Idiom

(The territory becomes the map)

Some larger code blocks can now be expressed so concisely that they become idioms in their own right.

 # Random integer between 3 and 7 inclusive
 do { $z = int rand 8 } until $z >= 3;
 $z = 3 + int rand 5;
 #
 $z = 3 + Int(5.rand);
 $z = (3..7).pick;

 # Count by 3 in an infinite loop
 for ( my $i = 1; ; $i++ ) { my $n = 3 * $i; ... }
 for ( my $n = 3; ; $n += 3 ) {...}
 #
 loop ( my $n = 3; ; $n += 3 ) {...}
 for 3, * + 3 ... * -> $n {...}      # `...` is the "sequence" operator
 for 3, 6, 9 ... * -> $n {...}       # `...` can infer from example list

 # Loop over a range, excluding the start and end points
 for my $i ( $start .. $limit ) { next if $i == $start or $i == $limit; ... }
 for my $i ( ($start+1) .. ($limit-1) ) {...}
 #
 for ($start+1) .. ($limit-1) -> $i {...}
 for $start ^..^ $limit -> $i {...}

Predictions

(Gaze into my crystal ball)

The idioms above are not written in stone. After Christmas, they may be given the boot, eighty-sixed, or traded-in for a newer model. Furthermore, other forces are at work to season this stew. Please indulge these prognostications:

We will see more use of exceptions. They are now cleaner and cheaper and don't need eval(), and users will be introduced to them much sooner due to open()'s new exception-based error handling.
We will see more hash|array slicing. Beginners had to grok another sigil rule to use slices, now they don't! In fact, single-element access now just looks like a degenerate case of multiple-element access.
Lazy lists are cool and efficient. Coding practices will be shaped to allow lists to stay lazy longer.
The new hyper-operators are implicitly parallel. We will see less explicit threading, and more set-at-a-time thinking.
In Perl 5, OO (before Moose) was overly wordy to the point of being grating. Functional programming worked best using the `&` prototype, but we are all told to never use Perl 5's prototypes!
In Perl 6, OO and functional styles are so low-friction that you will hoist code across paradigm boundaries with the same ease that you refactor same-paradigm code today.
`say` will rule the world, with legions of inlined ».methods as its army.

OK, maybe not that last one :)

The crystal ball may be hazy, but the future looks bright!

Day 22 – Operator overloading, revisited

2011-12-22T20:58:11+00:00

Today’s post is a follow-up. Exactly two years ago, Matthew Walton wrote on this blog about overloading operators:

You can exercise further control over the operator’s parsing by adding traits to the definition, such as tighter, equiv and looser, which let you specify the operator’s precedence in relationship to operators which have already been defined. Unfortunately, at the time of writing this is not supported in Rakudo so we will not consider it further today.

Rakudo is still lagging in precedence support (though at this point there are no blockers that I know about to simply going ahead and implementing it). But there’s a new implementation on the block, one that didn’t exist two years ago: Niecza.

Let’s try out operator precedence in Niecza.

$ niecza -e 'sub infix:<mean>($a, $b) { ($a + $b) / 2 }; say 10 mean 4 * 5'
15

Per default, an operator gets the same precedence as infix<+>. This is per spec. (How do we know it got the same precedence as infix<+> above? Well, we know it’s not tighter than multiplication, otherwise we’d have gotten the result 35.)

That’s all well and good, but what if we want to make our mean little operator evaluate tighter than multiplication? Nothing could be simpler:

$ niecza -e 'sub infix:<mean>($a, $b) is tighter(&infix:<*>) { ($a + $b) / 2 }; say 10 mean 4 * 5'
35

See what we did there? is tighter is a trait that we apply to the operator definition. The trait accepts an argument, in this case the language-provided multiplication operator. It all reads quite well, too: “infix mean is tighter [than] infix multiplication”.

Note the explicit use of intuitive naming for the precedence levels. Rather than the inherently confusing terms “higher/lower”, Perl 6 talks about “tighter/looser”, as in “multiplication binds tighter than addition”. Easier to think about precedence that way.

Internally, the precedence levels are stored not as numbers but as strings. Each original precedence level gets a letter of the alphabet and an equals sign (=). Subsequent added precendence levels append either a less-than sign (<) or a greater-than sign (>) to an existing precedence level representation. Using this system, we never “run out” of levels between existing ones (as we could if we were using integers, for example), and tighter levels always come lexigographically before looser ones. Language designers, take heed.

A few last passing notes about operators in Perl 6, while we’re on the subject:

In Perl 6, operators are subroutines. They just happen to have funny names, like prefix:<-> or postfix:<++> or infix:<?? !!>. This actually takes a lot of the hand-wavey magic out of defining them. The traits that we’ve seen applied to operators are really subroutine traits… these just happen to be relevant to operator definitions.
As a consequence, just like subroutines, operators are lexically scoped by default. Lexical scoping is something we like in Perl 6; it keeps popping up in unexpected places as a solid, sound design principle in the language. In practice, this means that if you declare an operator within a given scope, the operator will be visible and usable within that scope. You’re modifying the parser, but you’re doing it locally, within some block or other. (Or within the whole file, of course.)
Likewise, if you want to export your operators, you just use the same exporting mechanism used with subroutines. See how this unification between operators and subroutines keeps making sense? (In Perl 6-land, we say “operators are just funny-looking subroutines”.)
Multiple dispatch in operators works just as with ordinary subroutines. Great if you want to dispatch your operators on different types. As with all other routines in the core library in Perl 6, all operators are declared multi to be able to co-exist peacefully with module extensions to the language.
Operators can be macros, too. This is not an exceptions to the rule that operators are subroutines, because in Perl 6, macros are subroutines. In other words, if you want some syntactic sugar to execute at parse time (which is what a macro does), you can dress it up either as a normal-looking sub, or as an operator.

That’s it for today. Now, go forth and multiply, or even define your own operator that’s either tighter or looser than multiplication.

Native libraries, native objects

2011-12-21T21:42:29+00:00

Last year flussence++ wrote a nice post about writing XMMS bindings for Perl 6 using the Native Call Interface. It has improved a bit since then, (at least NCI, I don’t know about XMMS), so let’s show it off a bit.

To run the examples below you need a NativeCall module installed. Then add use NativeCall; at the top of the file.

Previously, we were carefully writing all the C subs we needed to use and then usually writing some Perl 6 class which wrapped it in a nice, familiar interface. That doesn’t change much, except that now a class is not really an interface for some C-level data structure. Thanks to the new metamodel we can now make our class to actually be a C-level data structure, at least under the hood. Consider a class representing a connection to Music Player Daemon:

    class Connection is repr('CPointer') {
        sub mpd_connection_new(Str $host, Int $port)
            returns Connection
            is native('libmpdclient.so') {}
        sub mpd_connection_free()
            is native('libmpdclient.so') {}
        method new(Str $host, Int $port) {
            self.bless(mpd_connection_new($host, $port))
        }
        method DESTROY {
            mpd_connection_free(self)
        }
    }

The first line does not necesarilly look familiar. The is repr trait tells the compiler that the internal representation of the class Connection is a C pointer. It still is a fully functional Perl 6 type, which we can use in method signatures or wherever (as seen in the lines below).

We then declare some native fuctions we’re going to use. It’s quite convenient to put them inside the class body, so they don’t pollute the namespace and don’t confuse the user. What we are really exposing here is the new method, which uses bless to set the object’s internal representation to what mpd_connection_new has returned. From now on our object is a Perl 6 level object, while under the hood being a mere C pointer. In method DESTROY we just pass self to another native function, mpd_connection_free, without the need to unbox it or whatever. The NativeCall module will just extract its internal representation and pass it around. Ain’t that neat?

Let’s see some bigger example. We’ll use taglib library to extract the metadata about some music files lying around. Let’s see the Tag class first:

    class Tag is repr('CPointer') {
        sub taglib_tag_title(Tag)  returns Str is native('libtag_c.so') {}
        sub taglib_tag_artist(Tag) returns Str is native('libtag_c.so') {}
        sub taglib_tag_album(Tag)  returns Str is native('libtag_c.so') {}
        sub taglib_tag_genre(Tag)  returns Str is native('libtag_c.so') {}
        sub taglib_tag_year(Tag)   returns Int is native('libtag_c.so') {}
        sub taglib_tag_track(Tag)  returns Int is native('libtag_c.so') {}
        sub taglib_tag_free_strings(Tag)       is native('libtag_c.so') {}

        method title  { taglib_tag_title(self)  }
        method artist { taglib_tag_artist(self) }
        method album  { taglib_tag_album(self)  }
        method genre  { taglib_tag_genre(self)  }
        method year   { taglib_tag_year(self)   }
        method track  { taglib_tag_track(self)  }

        method free   { taglib_tag_free_strings(self) }
    }

That one is pretty boring: plenty of native functions, and plenty of methods being exactly the same things. You may have noticed the lack of new: how are we going to get an object and read our precious tags? In taglib, the actual Tag object is obtained from a Tag_File object first. Why didn’t we implement it first? Well, it’s going to have a method returning the Tag object shown above, so it was convenient to declare it first.

    class TagFile is repr('CPointer') {
        sub taglib_file_new(Str) returns TagFile is native('libtag_c.so') {}
        sub taglib_file_free(TagFile)            is native('libtag_c.so') {}
        sub taglib_file_tag(TagFile) returns Tag is native('libtag_c.so') {}
        sub taglib_file_is_valid(TagFile) returns Int
            is native('libtag_c.so') {}

        method new(Str $filename) {
            unless $filename.IO.e {
                die "File '$filename' not found"
            }
            my $self = self.bless(taglib_file_new($filename));
            unless taglib_file_is_valid($self) {
                taglib_file_free(self);
                die "'$filename' is invalid"
            }
            return $self;
        }

        method tag  { taglib_file_tag(self)  }

        method free { taglib_file_free(self) }
    }

Note how we use native functions in new to check for exceptional situations and react in an appropriately Perl 6 way. Now we only have to write a simple MAIN before we can test it on our favourite music files.

    sub MAIN($filename) {
        my $file = TagFile.new($filename);
        my $tag  = $file.tag;
        say 'Artist: ', $tag.artist;
        say 'Title:  ', $tag.title;
        say 'Album:  ', $tag.album;
        say 'Year:   ', $tag.year;

        $tag.free;
        $file.free;
    }

Live demo! Everyone loves live demos.

    $ perl6 taginfo.pl some-track.mp3
    Artist: Diablo Swing Orchestra
    Title:  Balrog Boogie
    Album:  The Butcher's Ballroom
    Year:   2009

Works like a charm. I promise I’ll wrap it up in some nice Audio::Tag module and release it on Github shortly.

Of course there’s more to do with NativeCall than just passing raw pointers around. You could, for example, declare it as a repr('CStruct') and access the struct field directly, as you would in good, old C. This is only partly implemented as for now though, but that shouldn’t stop you from experimenting and seeing what you can do before Christmas. Happy hacking!

Paired up Hashes

2011-12-20T14:31:49+00:00

What is is possible with arrays and lists in Perl 6 is truly remarkable and was demonstrated here several times. But what about hashes?
Superficially not much has changed.
(Following Damian’s rule from PBP to name a hash variable in singular.)

    %song = Panacea => 'found a lover', Photek => 'ni ten ichi ryu';
    say keys %song;   # also %song.keys
    say values %song; # %song.values

Yes, the sigils are now invariant, so you get values with:

    %song{'Panacea'}
    %song{'Panacea', 'Photek'}

That can be shortened, because <> is the new qw():

    %song<Panacea>
    %song<Panacea Photek>

Frankly, almost everything else has changed. Perl 6 can be sometimes hideous, just mimicking to be your good old pal Perl 5 while being a friendly T-X, blasting behind your back your programming problems away. The fat arrow is no longer a fancy comma but an infix operator, creating an object that contains a key-value pair.

    my $song = paniq => 'Godshatter';
    say $song.WHAT; # says: "Pair()"
    $song.key;     # as expected is paniq
    $song.value;  # you guess it

There is another Syntax for that, heavily used in signatures:

    my $song = :paniq('Godshatter');

But what happens if I:

    my @songs = %song; # same as @(%songs)

You maybe predicted it, @songs gets a list of pairs. For the old behaviour, you have to say explicitly: “I want the keys and values as a list.”:

    my @songs = %song.kv; # key 1, value 1, key 2 ....

This new setup of hashes is not only theoretically very pleasing. It also allows iterating over hashes, without the risk of loosing the precious key => value correlation. That’s handy for all kinds of sorting and mashing of data, for which Perl is famous. What else did Randal L. Schwartz once upon a time than creating a list of pairs, sorting them and then picking the needed data bits.

Having pairs as a built-in type helps also subs and methods to handle their parameters. Some of the can be positional, which could be ordered in an array. Some of them are named and could be stored in a hash. But they are actually stored in a “Parcel”, a list that can contain pairs. This way the order of the parameters and the key => value correlations are preserved.

A very similar type is the Capture, which can hold all parameter sent to a routine. Therefore it has to behave more like routine and pass all the named arguments under their names. But if you ask for the positional parameter, you get only them, not the named ones. With a Capture full of values you can ask with a smartmatch if it would pass a certain subroutine and many fine things more. The vaults are going here deeper and deeper, but lets get back to the daylight of everyday hash-usage.

Panacea aka Mathis Mootz had a lot of great tracks. But when I do:

    %song<Panacea> = "state of extacy";

“found a lover” gets overwritten. Nothing new so far, but there are times I just don’t want to loose my data. Then I need to execute some force onto the hash.

    %song.push( Panacea => "state of extacy" );

Whole lists of pairs can be pushed into another hash this way. The result will be (not surprisingly) still a hash but the key ‘Panacea’ now points to an array, containing both song titles. That’s also useful when inverting a hash, that means pulling out a pair-list where key and value are flipped. Pasting that into a hash may lead to collisions, if several keys have the same value. A simple:

    # list with song => artist pairs
    %song = %artist.invert;

might produce losses, but the following does not:

    %song.push( %artist.invert ); # or:
    %song.push: %artist.invert;

While doing some heavy data munging you might regroup the values under a different set of keys. In that case it is likely that several values will end up under the same key. Given you have a sub that recognizes a genre of a song, you might write something like:

    map { %genre.push(genre_of_song($_) => $_) }, %song.values;

But as you already guessed it, there is an easier way to do that:

    %genre = classify { (genre_of_song($_) }, %song.values;

Now you probably say: “That’s unrealistic!”. There are songs for instance from “Magnetic Man” that can be labeled “Drum ‘n Base”, “Dubstep” or even “Pop”. Larry knows that. (This kind of problem of course.) That’s why he pulled out a second hash generating method.

    %genre = categorize { (genre_of_song($_) }, %song.values;

Unlike classify, which expects exactly one value (Called scalar in P5 but item in Perl 6 world), categorize can handle a list of results returned by the closure (block). It also happily accepts a Nil, which means, unlike undef in Perl 5, really nothing. When classify gets a Nil, the song will then not appear in any category, meaning under no hash key.

Imagine a routine of real artificial intelligence that can distinct good from bad songs. (Your definition of good of course!)

    my %quality = classify { good_music($_) ?? 'good' !! 'bad' }, %song.values;

Hence %quality<good> contains all the songs I sent to my music player and %quality<bad> to /dev/null (which is the name of another electronic music artist).

Day 19 – Abstraction and why it’s good

2011-12-19T21:30:09+00:00

Some people are a bit afraid of the word “abstract”, because they’ve heard math teachers say it, and also, abstract art freaks them out. But abstraction is a fine and useful thing, and not so complicated. As programmers, we use it every day in different forms. The term is from Latin and means “to withdraw from” or “to pull away from”, and what we’re pulling away from is the specifics so we can focus on the big picture. That’s often mighty useful.

Here are a few examples:

Variables

If your computer only knew how to handle one specific number at a time, it’d be an abacus. Pretty early on, the programmer guild figured out it made a lot of sense to talk about the memory address of a value, and let that address contain whatever it pleased. They abstracted away from the value, and thus made the program more general.

As time passed, addresses were replaced by names, mostly as a convenience. Some people found it a good idea to give their variables descriptive names, as opposed to things like $grbldf.

Subroutines

Code re-use. We hear so much about it in the OO circles, but it holds equally well for subroutines. You write your code once, and then call it from all over the place. Convenient.

But, as I point out in an announcement pretending to be a computer science professor from an alternate timeline, there’s also the secondary benefit of giving your chunk of code a good mnemonic name, because that act in a sense improves the programming language itself. You’re giving yourself new verbs to program with.

This is especially true in Perl 6, because subroutines are lexically scoped (as opposed to Perl 5) and thus you can easily put a subroutine inside another routine. I use it when writing a Connect 4 game, for example.

Packages and modules

In Perl, packages don’t do much. They pull things together and keep them there. In a sense, what they abstract away is a set of subroutines from the rest of the world.

Perl 5 delivers its whole OO functionality through packages and a bit of dispatch magic on the side. It’s quite a feat, actually, but sometimes a bit too minimal. Moose fixes many of those early issues by providing a full-featured object system. Perl 6 lets packages go back to just being collections of subroutines, but provides a few dedicated abstractions for OO, a kind of built-in Moose. Which brings us to…

Classes

Object-orientation means a lot of different things to different people. To some, it’s the notion of an object, a piece of memory with a given set of operations and a given set of states. In a sense, we’re again in the business of extending the language like with did with subroutines. But this time we’re building new nouns rather than new verbs. One moment the language doesn’t know about a Customer object type; the next, it does.

To others, object-orientation means keeping the operations public and the states private. They refer to this division as encapsulation, because the object is like a little capsule, protecting your data from the big bad world. This is also a kind of abstraction, built on the idea that the rest of the world shouldn’t need to care about the internals of your objects, because some day you may want to refactor them. Don’t talk to the brain, talk to the hand; do your thing through the published operations of the object.

Roles

But class-based OO with inheritance will take you only so far. In the past 10 years or so, people have become increasingly aware of the limitations of inheritance-based class hierarchies. Often there are concerns which cut completely across a conventional inheritance hierarchy.

This is where roles come in; they allow you to apply behaviors in little cute packages here and there, without being tied up by a tree-like structure. In a post about roles I explore how this helps write better programs. But really the best example nowadays is probably the Rakudo compiler and its extensive use of roles; jnthn has been writing about that in an earlier advent post.

If classes abstract away complete sets of behaviors, roles abstract away partial sets of behaviors, or responsibilities.

You can even do so at runtime, using mixins, which are roles that you add to an object as the program executes. Objects changing type during runtime sounds magic almost to the point of recklessness; but it’s all done in a very straightforward manner using anonymous subclasses.

Metaobjects

Sometimes you want extra control over how the object system itself works. The object system in Perl 6, through one of those neat bite-your-own-tail tricks, is written using itself, and is completely modifiable in terms of itself. Basically, a bunch of the complexity has been removed by not having a separate hidden, unreachable system to handle the intricacies of the object system. Instead, there’s a visible API for interacting with the object system.

And, when we feel like it, we can invent new and exotic varieties of object systems. Or just tweak the existing one to our fancy.

Macros

On the way up the abstraction ladder, we’ve abstracted away bigger and bigger chunks of code: values, code, routines, behaviors, responsibilities and object systems. Now we reach the top, and there we find macros. Ah, macros, these magical, inscrutable beasts. What do macros abstract away?

Code.

Well, that’s rather disappointing, isn’t it? Didn’t we already abstract away code with subroutines? Yes, we did. But it turns out there’s so much code in a program that sometimes, it needs to be abstracted away on several levels!

Subroutines abstract away code that can then run in several different ways. You call the routine with other values, and it behaves differently. Macros abstract away code that can then be compiled in several different ways. You write a macro with other values, and it gets compiled into different code, which can then in turn run differently.

Essentially, macros give you a hook into the compiler to help you shape and guide what code it emits during the compilation itself. In a sense, you’re abstracting certain parts of the compilation process, the parsing and the syntax manipulation and the code generation. Again, you’re shaping the language — but this time not inventing new nouns or verbs, but whole ways of expressing yourself.

Macros come in two broad types: textual (a la C) and syntax tree (a la Lisp). The textual ones have a number of known issues stemming from the fact that they’re essentially a big imprecise search-and-replace on your code. The syntax tree ones are hailed as the best thing about Lisp, because it allows Lisp programs to grow and adapt to the needs of the programmer, by inventing new ways of expressing yourself.

Perl 6, being Perl 6, specifies both textual macros and syntax tree macros. I’m currently working on a grant to bring syntax macros to Rakudo Perl 6. There’s a branch where I’m hammering out the syntax and semantics of macros. It’s fun work, and made much more feasible by the past year’s improvements to Rakudo itself.

In conclusion

As an application grows and becomes more complex, it needs more rungs of the abstraction ladder to rest on. It needs more levels of abstraction with which to draw away the specifics and focus on the generalities.

Perl 6 is a new Perl, distinct from Perl 5. Its most distinguishing trait is perhaps that it has more rungs on the abstraction ladder to help you write code that’s more to the point. I like that.

Fourth Grant Report: Structured Error Messages

2011-12-19T18:28:48+00:00

Progress on my grant for error message is slow but steady. Since my last report, I've done the following things:

Merged the nom-exceptions Rakudo branch, so now you can reliably throw Perl 6 objects as exceptions.
Implemented several error classes and roles in Rakudo
Started to throw typed errors both from runtime libraries and from inside the compiler
Hacked the default exception printer Rakudo to be much more flexible, for example you can now write exception classes that supress backtraces from the standard handler.
Wrote tests for typed run time and compile time errors, and at the same time developed a test function that makes it easy to write such tests.

It's time for a quick review of how far I am along the various deliverables in the original grant proposal.

D1: Specification. I think the hard work here is done already, what remains to do is finding good default and how to manipulate them (for example, how to generally switch on/off printing of backtraces?).
D2: Error catalogue, tests: I've not worked on this one too much. The error classes and roles so far mostly served to exercise the implementation; going through the existing errors from the various compilers and formalizing them will be quite a bit of work, but only moderately complicated.
D3: Implementation, documentation. Like D1, the hard part is mostly done. We can now throw errors from within the compiler actions and from the setting, next up will be the grammar. Then all places where errors are thrown need to be changed to use the new error classes. Again that'll be much work, but easy to do. Documentation is still missing.

All in all I feel I'm well on the way, and most complex decisions have been made.

For a more user oriented view of the new exception system I'd like to point you to my Perl 6 advent calendar post on exceptions.

The view from the inside: using meta-programming to implement Rakudo

2011-12-18T22:49:25+00:00

In my previous article for the Perl 6 advent calendar, I looked at how we can use the meta-programming facilities of Rakudo Perl 6 in order to build a range of tools, tweak the object system to our liking or even add major new features “from the outside”. While it’s nice that you can do these things, the Perl 6 object system that you get by default is already very rich and powerful, supporting a wide range of features. Amongst them are:

Classes
Parametric roles
Attributes
Methods (including private ones)
Delegation
Introspection
Subset (aka. refinement) types
Enums

That’s a lot of stuff to implement, but it’s all done by implementing meta-objects, and therefore we can take advantage of OOP – with both classes and roles – to factor it. The only real difference between the meta-programming we saw in my last article and the meta-programming we do while implementing the core Perl 6 object system in Rakudo is that the meta-objects are mostly written in NQP. NQP is a vastly smaller, much more easily optimizable and portable subset of Perl 6. Being able to use it also helps us to avoid many painful bootstrapping issues. Since it is mostly a small subset of Perl 6, it’s relatively easy to get in to.

In this article, I want to take you inside of Rakudo and, through implementing a missing feature, give you a taste of what it’s like to hack on the core language. So, what are we going to implement? Well, one feature of roles is that they can also serve as interfaces. That is, if you write:

role Describable {
    method describe() { ... }
}
class Page does Describable {
}

Then we are meant to get a compile time error, since the class Page does not implement the method “describe”. At the moment, however, there is no error at compile time; we don’t get any kind of failure until we call the describe method at runtime. So, let’s make this work!

One key thing we’re going to need to know is whether a method is just a stub, with a body containing just “…”, “???” or “!!!”. This is available to us by checking its .yada method. So, we have that bit. Question is, where to check it?

Unlike classes, which have the meta-object ClassHOW by default, there isn’t a single RoleHOW. In fact, roles show up in no less than four different forms. The two most worth knowing about are ParametricRoleHOW and ConcreteRoleHOW. Every role is born parametric. Whether you explicitly give it extra parameters or not, it is always parametric on at least the type of the invocant. Before we can ever use a role, it has to be composed into a class. Along the way, we have to specialize it, taking all the parametric things and replacing them with concrete ones. The outcome of this is a role type with a meta-object of type ConcreteRoleHOW, which is now ready for composition into the class.

So that’s roles themselves, but what about composing them? Well, the actual composition is performed by two classes, RoleToClassApplier and RoleToRoleApplier. RoleToClassApplier is actually only capable of applying a single role to a class. This may seem a little odd: classes can do multiple roles, after all. However, it turns out that a neat way to factor this is to always “sum” multiple roles to a single one, and then apply that to the class. Anyway, it would seem that we need to be doing some kind of check in RoleToClassApplier. Looking through, we see this:

my @methods := $to_compose_meta.methods($to_compose, :local(1));
for @methods {
    my $name;
    try { $name := $_.name }
    unless $name { $name := ~$_ }
    unless has_method($target, $name, 1) {
        $target.HOW.add_method($target, $name, $_);
    }
}

OK, so, it’s having a bit of “fun” with, of all things, looking up the name of the method. Actually it’s trying to cope with NQP and Rakudo methods having slightly different ideas about how the name of a method is looked up. But that aside, it’s really just a loop going over the methods in a role and adding them to the class. Seems like a relatively opportune time to spot the yada case, which indicates we require a method rather than want to compose one into the class. So, we change it do this:

my @methods := $to_compose_meta.methods($to_compose, :local(1));
for @methods {
    my $name;
    my $yada := 0;
    try { $name := $_.name }
    unless $name { $name := ~$_ }
    try { $yada := $_.yada }
    if $yada {
        unless has_method($target, $name, 0) {
            pir::die("Method '$name' must be implemented by " ~
            $target.HOW.name($target) ~
            " because it is required by a role");
        }
    }
    elsif !has_method($target, $name, 1) {
        $target.HOW.add_method($target, $name, $_);
    }
}

A couple of notes. The first is that we’re doing binding, because NQP does not have assignment. Binding is easier to analyze and generate code for. Also, the has_method call is passing an argument of 0 or 1, which indicates whether we want to consider methods in just the target class or any of its parents (note that there’s no True/False in NQP). If the class inherits a method then we’ll consider that as good enough: it has it.

So, now we run our program and we get:

===SORRY!===
Method 'describe' must be implemented by Page because it is required by a role

Which is what we were after. Note that the “SORRY!” indicates it is a compile time error. Success!

So, are we done? Not so fast! First, let’s check the inherited method case works out. Here’s an example.

role Describable {
    method describe() { ... }
}
class SiteItem {
    method describe() { say "It's a thingy" }
}
class Page is SiteItem does Describable {
}

And…oh dear. It gives an error. Fail. So, back to RoleToClassApplier. And…aha.

sub has_method($target, $name, $local) {
    my %mt := $target.HOW.method_table($target);
    return nqp::existskey(%mt, $name)
}

Yup. It’s ignoring the $local argument. Seems it was written with the later need to do a required methods check in mind, but never implemented to handle it. OK, that’s an easy fix – we just need to go walking the MRO (that is, the transitive list of parents in dispatch order).

sub has_method($target, $name, $local) {
    if $local {
        my %mt := $target.HOW.method_table($target);
        return nqp::existskey(%mt, $name);
    }
    else {
        for $target.HOW.mro($target) {
            my %mt := $_.HOW.method_table($_);
            if nqp::existskey(%mt, $name) {
                return 1;
            }
        }
        return 0;
    }
}

With that fixed, we’re in better shape. However, you may be able to imagine another case that we didn’t yet handle. What if another role provides the method? Well, first let’s see what the current failure mode is. Here’s the code.

role Describable {
    method describe() { ... }
}
role DefaultStuff {
    method describe() { say "It's a thingy" }
}
class Page does Describable does DefaultStuff {
}

And here’s the failure.

===SORRY!===
Method 'describe' must be resolved by class Page because it exists
in multiple roles (DefaultStuff, Describable)

So, it’s actually considering this as a collision. So where do collisions actually get added? Happily, that just happens in one place: in RoleToRoleApplier. Here’s the code in question.

if +@add_meths == 1 {
    $target.HOW.add_method($target, $name, @add_meths[0]);
}
else {
    # More than one - add to collisions list.
    $target.HOW.add_collision($target, $name, %meth_providers{$name});
}

We needn’t worry if we just have one method and it’s a requirement rather than an actual implementation – it’ll just do the right thing. So it’s just the second branch that needs consideration. Here’s how we change things.

if +@add_meths == 1 {
    $target.HOW.add_method($target, $name, @add_meths[0]);
}
else {
    # Find if any of the methods are actually requirements, not
    # implementations.
    my @impl_meths;
    for @add_meths {
        my $yada := 0;
        try { $yada := $_.yada; }
        unless $yada {
            @impl_meths.push($_);
        }
    }

    # If there's still more than one possible - add to collisions list.
    # If we got down to just one, add it. If they were all requirements,
    # just choose one.
    if +@impl_meths == 1 {
        $target.HOW.add_method($target, $name, @impl_meths[0]);
    }
    elsif +@impl_meths == 0 {
        $target.HOW.add_method($target, $name, @add_meths[0]);
    }
    else {
        $target.HOW.add_collision($target, $name, %meth_providers{$name});
    }
}

Essentially, we filter out those that are implementations of the method rather than just requirements. If we are left with just a single method, then it’s the only implementation, and it satisfies the requirements, so we add it and we don’t need to do anything further. If we discover they are all requirements, then we don’t want to flag up a collision, but instead we just pick any of the required methods and pass it along. They’ll all give the same error. Otherwise, if we have multiple implementations, then it’s a real collision so we add it just as before. And…it works!

So, we run the test suite, things look good…and commit.

3 files changed, 48 insertions(+), 6 deletions(-)

And there we go – Rakudo now supports a part of the spec that it never has before, and it wasn’t terribly much effort to put in. And that just leaves me to go to the fridge and grab a Christmas ale to relax after a little meta-hacking. Cheers!

Day 17: Gtk Mandelbrot

2011-12-17T20:26:00+00:00

Two years ago today, the Advent post was on making Mandelbrot sets in Perl 6. At the time, they were in black and white, slow to produce, Rakudo was prone to crashing, and the only user interface thing you could control was how big the resulting PPM file was.

As they say, that was then. This is now.

The new gtk-mandelbrot.pl script is 423 lines of Perl 6 code — targeted at Niecza, threaded, and using the GtkSharp library. It allows you to move and resize the windows, zoom in (left mouse button, drag across image to define zoom boundaries), create Julia set images (right click on a Mandelbrot set image), increase the number of iterations (press ‘m’), and output a PPM file for a window (press ‘s’).

The threading doesn’t actually improve performance on my MacBook Pro (still looking into why) but it does make the script much more responsive.

It would be far too long to go through all the code, but lets hit on the highlights. The core is almost unchanged:

        sub julia(Complex $c, Complex $z0) {
            my $z = $z0;
            my $i;
            loop ($i = 0; $i < $max_iterations; $i++) {
                if $z.abs > 2 {
                    return $i + 1;
                }
                $z = $z * $z + $c;
            }
            return 0;
        }

It’s named julia instead of mandel now, because it is more general. If you call it with $z0 set to 0, it calculates the same thing the old mandel did. Allowing $z0 to vary allows you to calculate Julia sets as well.

The code around it is very different, though! Stefan O’Rear wrote the threading code, using Niecza’s Threads library, which is a thin wrapper on C#’s threading libraries, and probably not very close to what Perl 6′s built-in threading will look like when it is ready to go. He establishes a WorkQueue with a list of the work that needs to be done, and then starts N running threads, where N comes from the environment variable THREADS if it is present, and the reported processor count otherwise:

for ^(%*ENV<THREADS> // CLR::System::Environment.ProcessorCount) {
    Thread.new({ WorkQueue.run });
}

WorkQueue.run is pretty simple:

    method run() {
        loop {
            my $item = self.shift;
            next if $item.cancelled;
            $item.run.();
            $item.mark-done;
        }
    }

This is an infinite loop that starts by getting the next WorkItem off the queue, checks to see if it has been cancelled, and if it hasn’t, calls the .run Callable attribute and then the mark-done method.

The WorkItems on the queue look like this:

class WorkItem {
    has Bool $!done = False;
    has Bool $!cancelled = False;

    has Callable &.run;
    has Callable &.done-cb;

    method is-done() { WorkQueue.monitor.lock({ $!done }) }
    method mark-done() {
        &.done-cb.() unless WorkQueue.monitor.lock({ $!done++ })
    }

    method cancelled() { WorkQueue.monitor.lock({ $!cancelled }) }
    method cancel() { WorkQueue.monitor.lock({ $!cancelled = True }) }
}

Each WorkItem has two flags, $!done and $!cancelled, and two Callable attributes, &.run, already mentioned as what is called by WorkQueue.run, and &.done-cb, which is the callback function to be called when the &.run method finishes.

The two methods (for now) we use in our WorkItem are relatively simple:

            sub row() {
                my $row = @rows[$y];
                my $counter = 0;
                my $counter_end = $counter + 3 * $width;
                my $c = $ur - $y * $delta * i;

                while $counter < $counter_end {
                    my $value = $is-julia ?? julia($julia-z0, $c) !! julia($c, 0i);
                    $row.Set($counter++, @red[$value % 72]);
                    $row.Set($counter++, @green[$value % 72]);
                    $row.Set($counter++, @blue[$value % 72]);
                    $c += $delta;
                }
            }

            sub done() {
                Application.Invoke(-> $ , $ {
                    $.draw-area.QueueDrawArea(0, $y, $width, 1);
                });
            }

            my $wi = WorkItem.new(run => &row, done-cb => &done);
            WorkQueue.push($wi);
            push @.line-work-items, $wi;

As you might expect, row calculates one line of the set we are working on. It may look like it is using global variables, but these subs are actually local to the FractalSet.start-work method and the variables are local to it from there. The done invokes a Gtk function noting that a portion of the window needs to be redrawn (namely the portion we just calculated).

The above block of code is called once for each row of the fractal window being generated, which has the effect of queuing up all of the fractal to be handled as there are available threads.

Moving upward in the code's organization, each fractal window we generate is managed by an instance of the FractalSet class.

class FractalSet {
    has Bool $.is-julia;
    has Complex $.upper-right;
    has Real $.delta;
    has Int $.width;
    has Int $.height;
    has Int $.max_iterations;
    has Complex $.c;
    has @.rows;
    has @.line-work-items;
    has $.new-upper-right;

    has $.draw-area;

$.is-julia and $.max_iterations are self-explanatory. $.upper-right is the fixed complex number anchoring the image. $.delta is the amount of change in the previous number per-pixel; we assume the pixels are square. $.width and $.height are the size of the window in pixels. $.c only has meaning for Julia sets, where it is the value $c in the equation $new-z = $z * $z + $c. @.rows the pixel information generated by the row sub above; @.line-work-items saves a reference to all of the WorkItems generating those rows. $.new-upper-right is temporary used during the zoom mouse operation. $.draw-area is the Gtk.DrawingArea for the related window.

Once all that is set up, the rest of the code is pretty straightforward. The Gtk windowing code is set up in FractalSet.build-window:

    method build-window()
    {
        my $index = +@windows;
        @windows.push(self);
        self.start-work;

        my $window = Window.new($.is-julia ?? "julia $index" !! "mandelbrot $index");
        $window.SetData("Id", SystemIntPtr.new($index));
        $window.Resize($.width, $.height);  # TODO: resize at runtime NYI

        my $event-box = GtkEventBox.new;
        $event-box.SetData("Id", SystemIntPtr.new($index));
        $event-box.add_ButtonPressEvent(&ButtonPressEvent);
        $event-box.add_ButtonReleaseEvent(&ButtonReleaseEvent);

        my $drawingarea = $.draw-area = GtkDrawingArea.new;
        $drawingarea.SetData("Id", SystemIntPtr.new($index));
        $drawingarea.add_ExposeEvent(&ExposeEvent);
        $window.add_DeleteEvent(&DeleteEvent);
        $event-box.Add($drawingarea);

        $window.Add($event-box);
        $window.add_KeyReleaseEvent(&KeyReleaseEvent);
        $window.ShowAll;
    }

We store a global array @windows tracking all the FractalSets in play. Each of the different objects here gets the "Id" data set to this set's index into the @windows array so we can easily look up the FractalSet from callback functions. The rest of the method is just plugging the right callback into each component -- simple conceptually but it took this Gtk novice a lot of work figuring it all out.

As an example, consider the KeyReleaseEvent callback, which responds to presses on the keyboard.

sub KeyReleaseEvent($obj, $args) {
    my $index = $obj.GetData("Id").ToInt32();
    my $set = @windows[$index];
    
    given $args.Event.Key {
        when 'm' | 'M' {
            $set.increase-max-iterations;
        }
        when 's' | 'S' {
            $set.write-file;
        }
    }
}

First we lookup the index into @windows, then we get the $set we're looking at. Then we just call the appropriate FractalSet method, for instance

    method increase-max-iterations() {
        self.stop-work;
        $.max_iterations += 32;
        self.start-work;
    }

.stop-work cancels all the pending operations for this FractalSet, then we bump up the number of iterations, and then we .start-work again to queue up a new set of rows with the new values.

The full source code is here. As of this writing it agrees with the code here, but this is an active project, and probably will change again in the not-too-distant future. Right now my biggest goals are figuring out how to get the threading to actually improve performance on my MacBook Pro and cleaning up the code. Both suggestions and questions are welcome.

Where Have All The References Gone?

2011-12-16T14:00:59+00:00

Perl 5 programmers that start to learn Perl 6 often ask me how to take a reference to something, and my answers usually aren’t really helpful. In Perl 6, everything that can be held in a variable is an object, and objects are passed by reference everywhere (though you don’t always notice that, because objects like strings and numbers are immutable, so there’s no difference to passing by value). So, everything is already treated as a reference in some sense, and there’s no point in explicitly taking references.

But people aren’t happy with that answer, because it doesn’t explain how to get stuff done that involved references in Perl 5. So here are a few typical use cases of references, and how Perl 6 handles them.

Creating Objects

In Perl 5, an object is really just a reference to a blessed value (but people usually say "blessed reference", because you virtually never use the blessed value without going through a reference).

So, in Perl 5 you’d write

 package My::Class;
 # constructor
 sub new { bless {}, shift };
 # an accessor
 sub foo {
     my $self = shift;
     # the ->{} dereferences $self as a hash
     $self->{foo} // 5;
 }

 # use the object:
 say My::Class->new->foo;

In Perl 6, you just don’t think about references; classes are much more declarative, and there’s no need for dereferencing anything anywhere:

 class My::Class {
     # attribute with accessor (indicated by the dot)
     # and default value
     has $.foo = 5;
 }

 # use it:
 say My::Class.new.foo

If you don’t like the default constructor, you can still use bless explicitly, but even then you don’t have to think about references:

 method new() {
     # the * specifies the storage, and means "default storage"
     self.bless(*);
 }

So, no explicit reference handling when dealing with OO. Great.

Nested Data Structures

In both Perl 5 and Perl 6, lists flatten automatically by default. So if you write

 my @a = (1, 2);
 my @b = (3, 4);
 push @a, @b

then @a ends up with the four elements 1, 2, 3, 4, not with three elements of which the third is an array.

In Perl 5, nesting the data structure happens by taking a reference to @b:

 push @a, \@b;

In Perl 6, item context replaces this use of references. It is best illustrated by a rather clumsy method to achieve the same thing:

 my $temp = @b;
 push @a, $temp;  # does not flatten the two items in $temp,
                 # because $temp is a scalar

Of course there are shortcuts; the following lines work too:

 push @a, $(@b);
 push @a, item @b;

(As a side note, push @a, $@b is currently not allowed, it tries to catch a p5ism; I will also try to persuade Larry and the other language designers to allow it, and have it mean the same thing as the other two).

On the flip side you need explicit dereferencing to get the values out of item context:

 my @a = 1, 2;
 my $scalar = @a;
 for @a {
     # two iterations
 }
 for $scalar {
     # one iteration only
 }
 for @$scalar {
     # two iterations again
 }

This explicit use of scalar and list context is the closest analogy to Perl 5 references, because it requires explicit context annotations in the same places where referencing and dereferencing is used in Perl 5.

But it’s not really the same, because there are cases where Perl 5 needs references, but Perl 6 can deduce the item context all on its own:

 @a[3] = @b; # automatically puts @b in item context

Mutating Arguments

Another use references in Perl 5 is for passing data to routines that should be modified inside the routine:

 sub set_five; {
     my $x = shift;
     # explicit dereferencing with another $:
     $$x = 5;
 }
 my $var;
 # explicit taking of a reference
 set_five \$var;

In Perl 6, there is a separate mechanism for this use case:

 sub set_five($x is rw) {
     # no dereferencing
     $x = 5;
 }
 my $var;
 # no explicit reference taking
 set_five $var;

So again a use case of Perl 5 references is realized by another mechanism in Perl 6 (signature binding, or binding in general).

Summary

Nearly everything is a reference in Perl 6, but you still don’t see them, unless you take a very close look. The control of list flattening with item and list context is the one area where Perl 5′s referencing and dereferencing shines through the most.

Day 15 – Something Exceptional

2011-12-15T12:39:31+00:00

The Perl 6 exception system is currently in development; here is a small example demonstrating a part of the current state:

 use v6;

 sub might_die(Real $x) {
     die "negative" if $x < 0;
     $x.sqrt;
 }

 for 5, 0, -3, 1+2i -> $n {
     say "The square root of $n is ", might_die($n);

     CATCH {
         # CATCH sets $_ to the error object,
         # and then checks the various cases:
         when 'negative' {
             # note that $n is still in scope,
             # since the CATCH block is *inside* the
             # to-be-handled block
             say "Cannot take square root of $n: negative"
         }
         default {
             say "Other error: $_";
         }
     }
 }

This produces the following output under rakudo:

 The square root of 5 is 2.23606797749979
 The square root of 0 is 0
 Cannot take square root of -3: negative
 Other error: Nominal type check failed for parameter '$x'; expected Real but got Complex instead

A few interesting points: the presence of a CATCH block automatically makes the surrounding block catch exceptions. Inside the CATCH block, all lexical variables from the outside are normally accessible, so all the interesting information is available for error processing.

Inside the CATCH block, the error object is available in the $_ variable, on the outside it is available in $!. If an exception is thrown inside a CATCH block, it is not caught — unless there is a second, inner CATCH that handles it.

The insides of a CATCH block typically consists of when clauses, and sometimes a default clause. If any of those matches the error object, the error is considered to be handled. If no clause matches (and no default block is present), the exception is rethrown.

Comparing the output from rakudo to the one that niecza produces for the same code, one can see that the last line differs:

 Other error: Nominal type check failed in binding Real $x in might_die; got Complex, needed Real

This higlights a problem in the current state: The wording of error messages is not yet specified, and thus differs among implementations.

I am working on rectifying that situation, and also throwing interesting types of error objects. In the past week, I have managed to start throwing specific error objects from within the Rakudo compiler. Here is an example:

 $ ./perl6 -e 'try eval q[ class A { $!x } ]; say "error: $!"; say $!.perl'
 error: Attribute $!x not declared in class A
 X::Attribute::Undeclared.new(
         name => "\$!x",
         package-type => "class",
         package-name => "A", filename => "",
         line => 1,
         column => Any,
         message => "Attribute \$!x not declared in class A"
 )
 # output reformatted for clarity

The string that is passed to eval is not a valid Perl 6 program, because it accesses an attribute that wasn’t declared in class A. The exception thrown is of type X::Attribute::Undeclared, and it contains several details: the name of the attribute, the type of package it was missing in (could be class, module, grammar and maybe others), the name of the package, the actual error message and information about the source of the error (line, cfile name (empty because eval() operates on a string, not on a file), and column, though column isn’t set to a useful value yet).

X::Attribute::Undeclared inherits from type X::Comp, which is the common superclass for all compile time errors. Once all compile time errors in Rakudo are switched to X::Comp objects, one will be able to check if errors were produced at run time or at compile with code like

 eval $some-string;
 CATCH {
     when X::Comp { say 'compile time' }
     default      { say 'run time'     }
 }

The when block smart-matches the error object against the X::Comp type object, which succeeds whenever the error object conforms to that type (so, is of that type or a subclas of X::Comp).

Writing and using new error classes is quite easy:

 class X::PermissionDenied is X::Base {
     has $.reason;
     method message() { "Permission denied: $.reason" };
 }

 # and using it somewhere:
 die X::PermissionDenied.new( reason => "I don't like your nose");

So Perl 6 has a rather flexible error handling mechanism, and libraries and applications can choose to throw error objects with rich information. The plan is to have the Perl 6 compilers throw such easily introspectable error objects too, and at the same time unify their error messages.

Many thanks go to Ian Hague and The Perl Foundation for funding my work on exceptions.

Meta-programming: what, why and how

2011-12-14T21:55:48+00:00

Sometimes, it’s good to take ones understanding of a topic, throw it away and try to build a new mental model of it from scratch. I did that in the last couple of years with object orientation. Some things feel ever so slightly strange to let go of and re-evaluate. For many people, an object really is “an instance of a class” and inheritance really is a core building block of OOP. I suspect many people who read this post will at this point be thinking, “huh, of course they really are” – and if so, that’s totally fair enough. Most people’s view of OOP will, naturally, be based around the languages they’ve applied object orientation in, and most of the mainstream languages really do have objects that are instances of classes and really do have inheritance as a core principle.

Step back and look around, however, and things get a bit more blurry. JavaScript doesn’t have any notion of classes. CLOS (the Common Lisp Object System) does have classes, but they don’t have methods. And even if we do just stick with the languages that have classes with methods, there’s a dizzying array of “extras” playing their part in the language’s OO world view; amongst them are interfaces, mixins and roles.

Roles – more often known as traits in the literature – are a relatively recent arrival on the OO scene, and they serve as an important reminder than object orientation is not finished yet. It’s a living, breathing paradigm, undergoing its own evolution just as our programming languages in general are.

And that brings me nicely on to Perl 6 – a language that from the start has set out to be able to evolve. At a syntax level, that’s done by opening up the grammar to mutation – in a very carefully controlled way, such that you always know what language any given lexical scope is in. Meta-programming plays that same role, but in the object orientation and type system space.

So what is a meta-object? A meta-object is simply an object that describes how a piece of our language works. What sorts of things in Perl 6 have meta-objects? Here’s a partial list.

Classes
Roles
Subsets
Enumerations
Attributes
Subroutines
Methods
Signatures
Parameters

So that’s meta-objects, but what about the protocol? You can read protocol as “API” or “interface”. It’s an agreed set of methods that a meta-object will provide if it wants to expose certain features. Let’s consider the API for anything that can have methods, such as classes and roles. At a minimum, it will provide:

add_method – adds a method to the object
methods – enables introspection of the methods that the object has
method_table – provides a hash of the methods in this type, excluding any that may be inherited

What about something that you can call a method on? It just has to provide one thing:

find_method – takes an object and a name, and returns the method if one exists

By now you may be thinking, “wait a moment, is there something that you can call a method on, but that does not have methods”? And the answer is – yes. For example, an enum has values that you can call a method on – the methods that the underlying type of the enumeration provides. You can’t actually add a method to an enum itself, however.

What’s striking about this is that we are now doing object oriented programming…to implement our object oriented language features. And this in turn means that we can tweak and extend our language – perhaps by subclassing an existing meta-object, or even by writing a new one from scratch. To demonstrate this, we’ll do a simple example, then a trickier one.

Suppose we wanted to forbid multiple inheritance. Here’s the code that we need to write.

my class SingleInheritanceClassHOW
    is Mu is Metamodel::ClassHOW
{
    method add_parent(Mu $obj, Mu $parent) {
        if +self.parents($obj, :local) > 0 {
            die "Multiple inheritance is forbidden!";
        }
        callsame;
    }
}
my module EXPORTHOW { }
EXPORTHOW.WHO.<class> = SingleInheritanceClassHOW;

What are we doing here? First, we inherit from the standard Perl 6 implementation of classes, which is defined by the class Metamodel::ClassHOW. (For now, we also inherit from Mu, since meta-objects currently consider themselves outside of the standard type hierarchy. This may change.) We then override the add_parent method, which is called whenever we want to add a parent to a class. We check the current number of (local) parents that a class has; if it already has one, then we die. Otherwise, we use callsame in order to just call the normal add_parent method, which actually adds the parent.

You may wonder what the $obj parameter that we’re taking is, and why it is needed. It is there because if we were implementing a prototype model of OOP, then adding a method to an object would operate on the individual object, rather than stashing the method away in the meta-object.

Finally, we need to export our new meta-object to anything that uses our module, so that it will be used in place of the “class” package declarator. Do do this, we stick it in the EXPORTHOW module, under the name “class”. The importer pays special attention to this module, if it exists. So, here it is in action, assuming we put our code in a module si.pm. This program works as usual:

use si;
class A { }
class B is A { }

While this one:

class A { }
class B { }
class C is A is B { }

Will die with:

===SORRY!===
Multiple inheritance is forbidden!

At compile time.

Now for the trickier one. Let’s do a really, really simple implementation of aspect oriented programming. We’ll write an aspects module. First, we declare a class that we’ll use to mark aspects.

my class MethodBoundaryAspect is export {
}

Next, when a class is declared with “is SomeAspect”, where SomeAspect inherits from MethodBoundaryAspect, we don’t want to treat it as inheritance. Instead, we’d like to add it to a list of aspects. Here’s an extra trait modifier to do that.

multi trait_mod:(Mu:U $type, MethodBoundaryAspect:U $aspect) is export {
    $aspect === MethodBoundaryAspect ??
        $type.HOW.add_parent($type, $aspect) !!
        $type.HOW.add_aspect($type, $aspect);
}

We take care to make sure that the declaration of aspects themselves – which will directly derive from this class – still works out by continuing to call add_parent for those. Otherwise, we call a method add_aspect, which we’ll define in a moment.

Supposing that our aspects work by optionally implementing entry and exit methods, which get passed the details of the call, here’s our custom meta-class, and the code to export it, just as before.

my class ClassWithAspectsHOW
    is Mu is Metamodel::ClassHOW
{
    has @!aspects;
    method add_aspect(Mu $obj, MethodBoundaryAspect:U $aspect) {
        @!aspects.push($aspect);
    }
    method compose(Mu $obj) {
        for @!aspects -> $a {
        for self.methods($obj, :local) -> $m {
            $m.wrap(-> $obj, |$args {
                $a.?entry($m.name, $obj, $args);
                my $result := callsame;
                $a.?exit($m.name, $obj, $args, $result);
                $result
            });
        }
        }
        callsame;
    }
}
my module EXPORTHOW { }
EXPORTHOW.WHO.<class> = ClassWithAspectsHOW;

Here, we see how add_aspect is implemented – it just pushes the aspect onto a list. The magic all happens at class composition time. The compose method is called after we’ve parsed the closing curly of a class declaration, and is the point at which we finalize things relating to the class declaration. Ahead of that, we loop over any aspects we have, and the wrap each method declared in the class body up so that it will make the call to the entry and exit methods.

Here’s an example of the module in use.

use aspects;
class LoggingAspect is MethodBoundaryAspect {
    method entry($method, $obj, $args) {
        say "Called $method with $args";
    }
    method exit($method, $obj, $args, $result) {
        say "$method returned with $result.perl()";
    }
}
class Example is LoggingAspect {
    method double($x) { $x * 2 }
    method square($x) { $x ** 2 }
}
say Example.double(3);
say Example.square(3);

And the output is:

Called double with 3
double returned with 6
6
Called square with 3
square returned with 9
9

So, a module providing basic aspect orientation support in 30 or so lines. Not so bad.

As you can imagine, we can go a long way with meta-programming, whether we want to create policies, development tools (like Grammar::Debugger) or try to add entirely new concepts to our language. Happy meta-hacking.

Bailador — A small Dancer clone

2011-12-13T18:58:49+00:00

Today we’ll write a simple Dancer clone in Perl 6. Simple also means Very Minimal — it will only recognize basic GET requests. Let’s look at how the simplest Dancer app possible looks like:

    get '/' => sub {
        "Hello World!"
    };
    dance;

So we need something to add routes to our app, and something to run it. Let’s take care of adding routes first. We’ll create an array to store all those, and thus get() will just add them to it.

    my @routes;
    sub get(Pair $x) is export {
        @routes.push: $x;
    }

In case you’re not familiar with the Pair thing, in Perl 6 a fat comma (=>) creates an actual data structure containing a key and a value. In this case, the key is just a string ‘/’, and the value is a subroutine.

Having @routes being a simple array of keys and values we can now write a simple dispatcher:

    sub dispatch($env) {
        for @routes -> $r {
            if $env<REQUEST_URI> ~~ $r.key {
                return $r.value.();
            }
        }
        return "404";
    }

dispatch() takes a hash representing our environment, which contains the REQUEST_URI string, basing on which we’ll try to find an appropriate candidate to dispatch to.

The above example is also cheating a bit: it just returns a ’404′ string instead of creating a proper HTTP response. Making it respond properly is left as an exercise for the reader (not the last one in this short article, I assure you :)).

Since we got that far already, writing our own dance() is a piece of cake. We’re going to call it baile() though. Why do we write all this in Spanish? If you can guess on which classes I was bored and wrote this thing on a piece of paper, then on the next YAPC I’ll show you the fastest possible way to tie a shoe. No kidding! But let’s finish this thing first.

Luckily we don’t need to write our own web server now. We have HTTP::Server::Simple::PSGI in Perl 6, which will do most of the hard work for us. The only thing we have to do is to create a PSGI app. In case you’ve never heard of it, a PSGI app is a mere subroutine, taking the environment as an argument, and returning an array of three things: an HTTP response code, an array of HTTP headers and a response body. Once we have our PSGI app ready, we just feed HTTP::Server::Simple::PSGI with it, and we’re good to go.

    sub baile is export {
        my $app = sub($env) {
            my $res = dispatch($env);
            return ['200', [ 'Content-Type' => 'text/plain' ], $res];
        }

        given HTTP::Server::Simple.PSGI.new {
            .host = 'localhost';
            .app($app);
            .run;
        }
    }

Yep, we’re cheating again and returning 200 no matter what. Remember the part about “an exercise for the reader?” You can think about it while celebrating a working Dancer clone.

But wait, there’s more!

Let’s look at our dispatch() once again:

    sub dispatch($env) {
        for @routes -> $r {
            if $env<REQUEST_URI> ~~ $r.key {
                return $r.value.();
            }
        }
        return "404";
    }

You probably noticed that we’ve used ~~ — a smartmatching operator. Thanks to that, we can match REQUEST_URI against strings, but not only. Junctions will work fine too:

    get any('/h', '/help', '/halp') => sub {
        "A helpful help message"
    }

And regexes:

    get /greet\/(.+)/ => sub ($x) {
        "Welcome $x"
    }

The last one will need a bit of tweaking in dispatch(). Yes, ~~ does the regex matching for us, but we have to take care of passing the match results to the callback function. Let’s modify the if body then:

    sub dispatch($env) {
        for @routes -> $r {
            if $env<REQUEST_URI> ~~ $r.key {
                if $/ {
                    return $r.value.(|$/.list);
                } else {
                    return $r.value.();
                }
            }
        }
        return "404";
    }

The if $/ part checks whether the match resulted in setting the Match object in the $/ variable. If it did, we flatten the Match to a list, and pass it to the callback function. We need a | prefix, so it becomes expanded to a parameter list instead of being passed as a mere array. From now on, the above example with greet will Just Work. Yay!

The Bailador code presented here is available in the Github repository. If you feel challenged by the “exercises for the reader”, or just want to make it a bit more proper Dancer port, you’re welcome to hack on it a bit and contribute. I hope I showed you how simple it is to write a simple, yet useful thing, and going with those simple steps we can hopefully get to something close to a full-blown Dancer port. Happy hacking, and have an appropriate amount of fun!

Privacy and OOP

2011-12-11T21:31:55+00:00

There are a number of ways in which Perl 6 encourages you to restrict the scope of elements of your program. By doing so, you can better understand how they are used and will be able to refactor them more easily later, potentially aiding agility. Lexical scoping is one such mechanism, and subroutines are by default lexically scoped.

Let’s take a look at a class that demonstrates some of the object oriented related privacy mechanisms.

    class Order {
        my class Item {
            has $.name;
            has $.price;
        }

        has Item @!items;

        method add_item($name, $price) {
            @!items.push(Item.new(:$name, :$price))
        }

        method discount() {
            self!compute_discount()
        }

        method total() {
            self!compute_subtotal() - self!compute_discount();
        }

        method !compute_subtotal() {
            [+] @!items>>.price
        }

        method !compute_discount() {
            my $sum = self!compute_subtotal();
            if $sum >= 1000 {
                $sum * 0.15
            }
            elsif $sum >= 100 {
                $sum * 0.1
            }
            else {
                0
            }
        }
    }

Taking a look at this, the first thing we notice is that Item is a lexical class. A class declared with “my” scope can never be referenced outside of the scope it is declared within. In our case, we never leak instances of it outside of our Order class either. This makes our class an example of the aggregate pattern: it prevents outside code from holding direct references to the things inside of it. Should we ever decide to change the way that our class represents its items on the inside, we have complete freedom to do so.

The other example of a privacy mechanism at work in this class is the use of private methods. A private method is declared just like an ordinary method, but with an exclamation mark appearing before its name. This gives it the same visibility as an attribute (which, you’ll note, are also declared with an exclamation mark – a nice bit of consistency). It also means you need to call it differently, using the exclamation mark instead of the dot.

Private methods are non-virtual. This may seem a little odd at first, but is consistent: attributes are also not visible to subclasses. By being non-virtual, we also get some other benefits. The latest Rakudo, with its optimizer cranked up to its highest level, optimizes calls to private methods and complains about missing ones at compile time. Thus a typo:

    self!compite_subtotal() - self!compute_discount();

Will get us a compile time error:

    ===SORRY!===
    CHECK FAILED:
    Undefined private method 'compite_subtotal' called (line 18)

You may worry a little over the fact that we now can’t subclass the discount computation, but that’s likely not a good design anyway; for one, we’d need to also expose the list of items, breaking our aggregate boundary. If we do want pluggable discount mechanisms we’d probably be better implementing the strategy pattern.

Private methods can, of course, not be called from outside of the class, which is also a compile time error. First, if you try:

    say $order!compute_discount;

You’ll be informed:

    ===SORRY!===
    Private method call to 'compute_discount' must be fully qualified
    with the package containing the method

Which isn’t so surprising, given they are non-virtual. But even if we do:

    say $o!Order::compute_discount;

Our encapsulation-busting efforts just get us:

    ===SORRY!===
    Cannot call private method 'compute_discount' on package Order
    because it does not trust GLOBAL

This does, however, hint at the get-out clause for private methods: a class may choose to trust another one (or, indeed, any other package) to be able to call its private methods. Critically, this is the decision of the class itself; if the class declaration didn’t decide to trust you, you’re out of luck. Generally, you won’t need “trusts”, but occasionally you may be in a situation where you have two very closely coupled classes. That’s usually undesirable in itself, though. Don’t trust too readily. :-)

So, lexical classes, private methods and some nice compiler support to help catch mistakes. Have an agile advent. :-)

Documenting Perl 6

2011-12-10T18:41:08+00:00

A wise man once said that programs must be written for people to read, and only incidentally for machines to execute. But aside from being read, your code is also going to be used by people, who don’t really want to dive into it to understand what it does. That’s where the documentation comes in.

In Perl 5 we had POD, which stands for Plain Old Documentation. In Perl 6 we have Pod, which is not really an abbreviation of anything. As its specification says, “Perl 6′s Pod is much more uniform, somewhat more compact, and considerably more expressive”. It has changed slightly compared to Perl 5 Pod, but most of the stuff remains the same, or at least very similar.

There are three main types of Pod blocks in Perl 6. Delimited blocks are probably the most obvious and simple ones:

    =begin pod

    <whatever Pod content we want>

    =end pod

Paragraph blocks are a bit more magical. They begin with =for, and end on the nearest blank line (as the name, Paragraph, suggest):

    my $piece = 'of perl 6 code'

    =for comment
    Here we put whatever we want.
    The compiler will not notice anyway.

    our $perl6 = 'code continues';

Abbreviated blocks are similar to Paragraph blocks. The leading = is followed immediately by a Pod block identifier and the content. Sounds familiar?

    =head1 Shoulders, Knees and Toes

That’s right, =head is nothing magical in Perl 6 Pod. That means you can write it also as a paragraph block

    =for head1
    Longer header
    than we usually write.

Or a full-blown delimited block

    =begin head1

    This header is longer than it should be

    =end head1

Any block can be written as a delimited block, paragraph block, or abbreviated block. No magic. Not all blocks are created equal, of course. =head will be treated differently than plain =pod. By whom? By the Pod renderer, of course, but also by the Perl 6 compiler itself. In Perl 6, Pod is not a second-class citizen, ignored during the program compilation. Pod in Perl 6 is a part of the code; along with parsing and constructing AST of the code to be executed, the compiler also parses and builds AST of all Pod blocks. They are then kept in the special $=POD variable, and can be inspected by the runtime:

    =begin pod

    Some pod content

    =end pod

    say $=POD[0].content[0].content;

The say line may look a little complicated. Content, of a content, of a what? What really happens, is that ‘Some pod content’ is parsed as an ordinary paragraph, and kept in the Pod::Block::Para object. The delimited block started with =begin pod becomes a Pod::Block::Named, and it can contain any number of child blocks. It’s also a first block in our example code, so it ends up in $=POD[0].

You now probably think “geez, how ugly is that. Who’s going to use it in this form”. Don’t worry. Frankly, I don’t expect anyone to use the AST directly. That’s what Pod renderers are useful for. Take for example Pod::To::Text:

    =begin pod

    =head1 A Heading!

    A paragraph! With many lines!

        An implicit code block!
        my $a = 5;

    =item A list!
    =item Of various things!

    =end pod

    DOC INIT {
        use Pod::To::Text;
        pod2text($=POD);
    }

Ran as it is, the code doesn’t produce any output. Why is it so? The DOC INIT block looks a little special. It gets run with every other INIT block, but also only when the --doc flag is passed to the compiler. Let’s take a look:

    $ perl6 --doc foo.pl
    A Heading!

    A paragraph! With many lines!

        An implicit code block!
        my $a = 5;

     * A list!

     * Of various things!

Actually, when no DOC INIT block exists in the code, the compiler generates a default DOC INIT, identical to the one in the example above. So you could really omit it, leaving only the Pod in the file, and perl6 --doc will produce exactly the same result.

But wait, there’s more!

I wrote about 3 types of Pod blocks, but there’s another one I didn’t talk about before. They are Declarator blocks, and they single purpose is to document the actual Perl 6 objects. Take a look.

    #= it's a sheep! really!
    class Sheep {

        #= produces a funny sound
        method bark {
            say "Actually, I don't think sheeps bark"
        }
    }

Every declarator block gets attached to the object which comes after it. It’s available in the .WHY attribute, so we can use it like this:

    say Sheep.WHY.content;                      # it's a sheep! really!
    say Sheep.^find_method('bark').WHY.content; # produces a funny sound

In this case we also don’t need to care about doing a ^find_method and all this for every piece of documentation we want to read. The mighty Pod::To::Text takes care about it too. If we run the Sheep code with --doc flag, we get the following:

    class Sheep: it's a sheep! really!

    method bark: produces a funny sound

The specification says it’s possible to document all the class attributes and all the arguments that methods or subroutines take. Unfortunately no Perl 6 implementation (that I know of) implements those yet.

There are dozens of Pod features that are not covered by this post, for example the formatting codes (<, > and so), or tables. If you’re interested take a look at Synopsis 26 (http://perlcabal.org/syn/S26.html). It’s actually written in Pod 6, and rendered by Pod::To::HTML. Not all features it describes are implemented yet, but most of them are (see the test suite linked below), and some modules are actually documented with it (Term::ANSIColor for example).

Some useful links:

Synopsis 26
Pod::To::Text source code
Term::ANSIColor documentation
Pod test suite (shows what Pod in Rakudo is capable of)

Happy documenting!

Day 9: Contributing to Perl 6

2011-12-09T14:27:40+00:00

This time, instead of sharing some cool feature of Perl 6, I’d like to talk about how easy it is to contribute usefully to the project. So I’m going to walk you through the process of making a change to Niecza. It does require a bit of domain knowledge (which the fine folks on #perl6 will be happy to help you with) but it’s definitely not rocket science. It’s not even particularly deep computer science, for the most part.

A few days ago, Radvendii asked on #perl6 if there was a round function in the core. The correct answer is “There should be one”, and it lead to a couple of bug fixes in Rakudo. But it got me to thinking — is Niecza supporting round (and its relatives ceiling, floor, and truncate) correctly?

Perl 6 has a huge suite of tests to see if an implementation is conforming to the spec, including a file for the round tests, S32-num/rounders.t. My first step then was to check the spectests currently being run by Niecza. Just like in Rakudo, this is stored in a file named t/spectest.data. So

Wynne:niecza colomon$ grep round t/spectest.data 
Wynne:niecza colomon$

Okay, clearly we’re not running the S32-num/rounders.t test file. (Note, in case you’re getting confused — the links in this post are to the latest versions of the files, which include all the changes I made writing this post.) That’s a sign that something is not properly supported yet. So let’s go ahead and run it by hand to see what happens. Both Niecza and Rakudo use a fudging process, allowing you to mark the bits of a test file that don’t work yet in a particular compiler. So we need to use a special fudging tool to run the code:

Wynne:niecza colomon$ t/fudgeandrun t/spec/S32-num/rounders.t
1..108
not ok 1 - floor(NaN) is NaN
# /Users/colomon/tools/niecza/t/spec/S32-num/rounders.t line 16
#    Failed test
#           got: -269653970229347386159395778618353710042696546841345985910145121736599013708251444699062715983611304031680170819807090036488184653221624933739271145959211186566651840137298227914453329401869141179179624428127508653257226023513694322210869665811240855745025766026879447359920868907719574457253034494436336205824

That’s followed by about 15 similar errors, then

Unhandled exception: Unable to resolve method truncate in class Num
  at /Users/colomon/tools/niecza/t/spec/S32-num/rounders.t line 34 (mainline @ 32) 
  at /Users/colomon/tools/niecza/lib/CORE.setting line 2224 (ANON @ 2) 
  at /Users/colomon/tools/niecza/lib/CORE.setting line 2225 (module-CORE @ 58) 
  at /Users/colomon/tools/niecza/lib/CORE.setting line 2225 (mainline @ 1) 
  at <unknown> line 0 (ExitRunloop @ 0)

Okay, so that’s at least two errors that need fixing.

We’ll go in order here, even though it means tackling what is most likely the most complicated error first. (If you do think this part of the problem is too hard to tackle, please skip ahead, because the last few improvements I made really were incredibly easy to do.) Opening src/CORE.setting, we find the following definition for round:

sub round($x, $scale=1) { floor($x / $scale + 0.5) * $scale }

Okay, so the real problem is in floor:

sub floor($x) { Q:CgOp { (floor {$x}) } }

What the heck does Q:CgOp mean? It means floor is actually implemented in C#. So we open up lib/Builtins.cs and search for floor, eventually finding public static Variable floor(Variable a1). I won’t print the full source code here, because it is on the long side, with a case for each of the different number types. We’re only interested in the floating point case here:

        if (r1 == NR_FLOAT) {
            double v1 = PromoteToFloat(r1, n1);
            ulong bits = (ulong)BitConverter.DoubleToInt64Bits(v1);
            BigInteger big = (bits & ((1UL << 52) - 1)) + (1UL << 52);
            int power = ((int)((bits >> 52) & 0x7FF)) - 0x433;
            // note: >>= has flooring semantics for signed values
            if ((bits & (1UL << 63)) != 0) big = -big;
            if (power > 0) big <<= power;
            else big >>= -power;
            return MakeInt(big);
        }

We don’t actually need to understand how all that works to fix this problem. The important bit is the PromoteToFloat line, which sets v1 to the floating point value which is the input to our floor. If we add a trap right after that, it should fix this bug. A quick C# websearch shows me that Double has member functions IsNaN, IsNegativeInfinity, and IsPositiveInfinity. Looking a bit around the Niecza source shows there is a MakeFloat function for returning floating point values. Let’s try:

if (Double.IsNaN(v1) || Double.IsNegativeInfinity(v1) || Double.IsPositiveInfinity(v1)) {
    return MakeFloat(v1);
}

One quick call to make later, I can try the test file again:

Wynne:niecza colomon$ t/fudgeandrun t/spec/S32-num/rounders.t
1..108
ok 1 - floor(NaN) is NaN
ok 2 - round(NaN) is NaN
ok 3 - ceiling(NaN) is NaN
not ok 4 - truncate(NaN) is NaN
# /Users/colomon/tools/niecza/t/spec/S32-num/rounders.t line 19
#    Failed test
#           got: -269653970229347386159395778618353710042696546841345985910145121736599013708251444699062715983611304031680170819807090036488184653221624933739271145959211186566651840137298227914453329401869141179179624428127508653257226023513694322210869665811240855745025766026879447359920868907719574457253034494436336205824

Progress! Apparently truncate uses a separate method, so we’ll have to fix it separately.

sub truncate($x) { $x.Int }
method Int() { Q:CgOp { (coerce_to_int {self}) } }

    public static Variable coerce_to_int(Variable a1) {
        int small; BigInteger big;
        return GetAsInteger(a1, out small, out big) ?
            MakeInt(big) : MakeInt(small);
    }

Oooo, this is perhaps a little bit trickier. Still a basic variant on the previous method, grabbing boilerplate code from a nearby function:

        int r1;
        P6any o1 = a1.Fetch();
        P6any n1 = GetNumber(a1, o1, out r1);

        if (r1 == NR_FLOAT) {
            double v1 = PromoteToFloat(r1, n1);
            if (Double.IsNaN(v1) || Double.IsNegativeInfinity(v1) || Double.IsPositiveInfinity(v1)) {
                return MakeFloat(v1);
            }
        }

I skipped the HandleSpecial2 bit in the boilerplate, because I’m never quite sure how that works. Luckily, we have the spectests to check and see if I have broken something by doing this.

Now the first 15 tests in rounders.t pass, leaving us with the

Unhandled exception: Unable to resolve method truncate in class Num

error. That should be easy to handle! If we go back to lib/CORE.setting and search for ceiling, we see it appears two times: in the catch-all base class Cool and as a stand-alone sub. If we look at the neighboring subs, we see floor, ceiling, round, and truncate are all defined. If we look in Cool, however, only floor, ceiling, and round defined. That’s the source of our trouble!

The method definitions of the others in Cool are really simple; all they do is forward to the sub versions. It’s very easy to add a truncate that does that:

    method truncate() { truncate self }

And poof! This time when we run rounders.t, we pass all 108 tests.

At this point we’ve got three things left to do. First, now that rounders.t passes, we need to add it to t/spectest.data. The list of tests there is ordered, so I just find the S32-num section and add S32-num/rounders.t in alphabetical order.

Next I will commit all the changes to my copy of the git repo. (I won’t explain how to do that, there are lots of git tutorials on the web.) Then I run make spectest to make sure I haven’t broken anything with these changes. (Hmm… actually a few TODO passing, bugs elsewhere that this patch has fixed! Oh, and one test broken, but it’s one which we were only passing by accident before, so I won’t feel bad about fudging it.)

Once that is done, you need to send the patch on to the Niecza developers; I believe the easiest way to do this is via github.

I’ve got one more little change to make that popped into my head while I was working on this. One naive way of implementing, say floor would be to convert the input into a floating point value (a Num in Perl 6) and then do Num.floor. That doesn’t work for all numbers, however, as most of the other number types are capable of storing numbers larger than will fit in a standing floating point double. So we probably need tests in the test suite to check for these cases. Let’s add them.

The tests in rounders.t are weirdly organized for my taste. But hey, we can always add our tests at the bottom.

{
    my $big-int = 1234567890123456789012345678903;
    is $big-int.floor, $big-int, "floor passes bigints unchanged";
    is $big-int.ceiling, $big-int, "ceiling passes bigints unchanged";
    is $big-int.round, $big-int, "round passes bigints unchanged";
    is $big-int.truncate, $big-int, "truncate passes bigints unchanged";
}

That passes okay in Niecza. (Probably out of courtesy we should check it on Rakudo as well and fudge it appropriately to make sure we’re not breaking their spectest!) We need to remember to add the count of new tests to the plan at the top of the test file. And then we can push that fix to github as well.

In conclusion, contributing to Perl 6 is easy. Anyone who tries writing Perl 6 code and reports problems they have to #perl6 is helping in a very real way. If you can write even fairly simple Perl 6 code, then you can write useful spec tests. It’s only marginally harder to write new methods for the setting in Perl 6. And even when you have to get down and dirty and start dealing with the language the compiler is implemented in, it’s still quite possible to do useful work without any deep understanding of how the compiler works.

Lexicality and Optimizability

2011-12-08T13:32:17+00:00

Traditional optimizations in compilers rely on compile-time knowledge about the program. Usually statically typed langauges like Java and C are rather good at that, and dynamic languages like Perl 5, ruby and python are not.

Perl 6 offers the flexibility of dynamic languages, but tries to provide much optimizability nonetheless by gradual typing, that is offering optional static type annotations.

But even in the presence of type annotations, another piece is needed for compile time dispatch decision and inlining: the knowledge about the available routines (and in the case of multi subs, the available candidates).

To provide that knowledge, Perl 6 installs subroutine in lexical scopes (and not packages / symbol tables, as in Perl 5), and lexical scopes are immutable at run time. (Variables inside the lexical scopes are still mutable, you just cannot add or remove entries at run time).

To provide the necessary flexibility, Perl 6 allows code to run at compile time. A typical way to run code at compile time is with the use directive:

 {
    use Test;  # imports routines into the current
               # lexical scope, at compile time
    plan 1;
    ok 1, 'success';
 }
 # plan() and ok() are not available here,
 # outside the scope into which the routines has been imported to.

The upside is that a sufficiently smart compiler can complain before runtime about missing routines and dispatches that are bound to fail. Current Rakudo does that, though there are a certainly cases that rakudo does not detect yet, but which are possible to detect.

 sub f(Int $x) {
          say $x * 2;
           }
 say "got here";
 f('some string');

produces this output with current Rakudo:

 ===SORRY!===
 CHECK FAILED:
 Calling 'f' will never work with argument types (str) (line 5)
     Expected: :(Int $x)

Since built-in routines are provided in an outer scope to the user’s program, all built-in routines are automatically subjected to all the same rules and optimizations as user-provided routines.

Note that this has other implications: require, which loads modules at run time, now needs a list of symbols to stub in at compile time, which are later wired up to the symbols loaded from the module.

The days are past where "a sufficiently smart compiler" was a legend; these days we have a compiler that can provide measurable speed-ups. There is still room for improvements, but we are now seeing the benefits from static knowledge and lexical scoping.

Adventures in writing a simple grammar profiler

2011-12-07T00:00:22+00:00

Inspired by jnthn’s earlier post on Grammar::Debugger, I wondered how hard it would be to implement a simple Perl 6 grammar profiler. Turns out it wasn’t that hard at all.

As far as profiling goes, all I wanted was counts of how many times each rule was executed and the cumulative time each rule took to execute. The interface I had in mind was something simple–a multi-level hash with names of grammars at the first level then, at the second level, names of the individual rules within the grammar, and finally the actual timing information. The timing information would be accessed thusly:

say "MyGrammar::MyRule was called " ~ %timing<MyGrammar><MyRule><calls> ~ "times";
say "and took " ~ %timing<MyGrammar><MyRule><time> ~ " seconds to execute";

But first I had to figure out what jnthn’s code was doing.

From the outside looking in, the basic technique is to replace the normal grammar meta-object with a custom meta-object that inherits most of the behavior of the normal grammar meta-object but replaces the normal method lookup with a custom one that returns a routine that collects the timing information while calling the original method.

Looking at jnthn’s code, I see that if the method name starts with ! or is any one of “parse”, “CREATE”, “Bool”, “defined” or “MATCH”, we just return the original method without modification. This is so that we don’t trace private methods or accidentally trace methods that aren’t directly part of the grammar but are used by it. In my simple profiler, I need to get the name of the grammar, which I do by calling my $grammar = $obj.WHAT.perl. So it looks like I need to add “perl” to that list of methods to pass through unscathed. Otherwise, I get an infinite recursion.

Anyway, for those method names that don’t match the aforementioned criteria, we return a custom built routine that accumulates the execution time and increments a counter for the number of calls. Seems straight-forward enough … below is the code (somewhat untested):

my %timing;

my class ProfiledGrammarHOW is Metamodel::GrammarHOW is Mu {

    method find_method($obj, $name) {
        my $meth := callsame;
        substr($name, 0, 1) eq '!' || $name eq any(<parse CREATE Bool defined MATCH perl>) ??
            $meth !!
            -> $c, |$args {
                my $grammar = $obj.WHAT.perl;
                %timing{$grammar} //= {};                   # Vivify grammar hash
                %timing{$grammar}{$meth.name} //= {};       # Vivify method hash
                my %t := %timing{$grammar}{$meth.name};
                my $start = now;                            # get start time
                my $result := $meth($obj, |$args);          # Call original method
                %t<time> += now - $start;             # accumulate execution time
                %t<calls>++;
                $result
            }
    }

    method publish_method_cache($obj) {
        # no caching, so we always hit find_method
    }
}

sub get-timing is export { %timing }
sub reset-timing is export { %timing = {} }

my module EXPORTHOW { }
EXPORTHOW.WHO.<grammar> = ProfiledGrammarHOW;

Assuming the above code was saved in file called “GrammarProfiler.pm”, you’d use it by adding the line use GrammarProfiler; to the top of any program that makes grammar declarations. Then after you parse your grammar, you can call get-timing() to obtain the hash that has the timing information for the individual rules that were executed during the parse or reset-timing() to clear the timing information.

Of course, a more full-fledged profiler would do much more work and provide many more profiling options, but this isn’t bad for a quick hack and it just might be useful too.

Tetris on Niecza

2011-12-05T23:45:26+00:00

Niecza, the Other Perl 6 Implementation on Mono and .NET, recently gained the ability to call almost any Common Language Runtime library. In Niecza’s examples directory, a simple 30 line script called gtk1.pl shows how to use gtk-sharp, and thus Gtk and Gdk, the graphical basis of Gnome. Here is gtk1′s central working part:

my $btn = Button.new("Hello World");
$btn.add_Clicked: sub ($obj, $args) { #OK
    # runs when the button is clicked.
    say "Hello World";
    Application.Quit;
};

The add_Clicked method defines a callback routine, essential to process user input. Running gtk1.pl makes the following resizeable button in a window, and it closes when clicked:

From gtk1 to Tetris is not far, see the source also in niecza/examples. Two extra ingredients make it possible: a timer tick callback routine to animate the falling pieces, and non blocking keyboard input to give the user the illusion of control. Add some simple physics and Cairo graphics and you have a playable game (modulo scoring and similar low hanging fruit) in under 170 lines of Perl 6.

Animation by timer tick works by causing the whole window to be redrawn by an ExposeEvent at regular intervals. The redraw tries to move the falling piece downwards, and if the physics says no, it adds a new piece at the top instead. (Bug: that should eventually fail with a full pile of pieces.) GLibTimeout sets up the timer callback handler which invokes .QueueDraw. The default interval is 300 milliseconds, and if the game engine wants to speed that up, it can adjust $newInterval which will replace the GLibTimeout on the next tick (sorry about the line wrap):

my $oldInterval = 300;
my $newInterval = 300;
...
GLibTimeout.Add($newInterval, &TimeoutEvent);
...
sub TimeoutEvent()
{
    $drawingarea.QueueDraw;
    my $intervalSame = ($newInterval == $oldInterval);
    unless $intervalSame { GLibTimeout.Add($newInterval, &TimeoutEvent); }
    return $intervalSame; # True means continue calling this timeout handler
}

Thanks to the excellent way Gtk handles typing, the keystroke event handler is fairly self documenting. The Piece subroutines do the physics ($colorindex 4 is the square block that does not rotate):

$drawingarea.add_KeyPressEvent(&KeyPressEvent);
...
sub KeyPressEvent($sender, $eventargs) #OK not used
{
    given $eventargs.Event.Key {
        when 'Up' { if $colorindex != 4 { TryRotatePiece() } }
        when 'Down' { while CanMovePiece(0,1) {++$pieceY;} }
        when 'Left' { if CanMovePiece(-1,0) {--$pieceX;} }
        when 'Right' { if CanMovePiece( 1,0) {++$pieceX;} }
    }
    return True; # means this keypress is now handled
}

With a bit more glue added, here is the result:

This post has glossed over other details such as the drawing of the graphics, because a later Perl 6 Advent post will cover that, even showing off some beautiful fractals, so keep following this blog! The above software was presented at the London Perl Workshop 2011.

The Flip-Flop operator

2011-12-05T07:00:59+00:00

Perl 5 has a binary operator called flip-flop that is false until its first argument evaluates to true and it stays true (flips) until the second argument evaluates to true at which point it becomes false again (flops). This is such a useful operator that Perl 6 also has flip-flop, only it’s spelled ff and has a few variants:

    ff
    ff^
    ^ff
    ^ff^

The circumflex means to skip the end point on that end.

Perhaps some examples are in order …

    for 1..20 { .say if $_ == 9  ff  $_ == 13; }     # 9 10 11 12 13
    for 1..20 { .say if $_ == 9  ff^ $_ == 13; }     # 9 10 11 12
    for 1..20 { .say if $_ == 9 ^ff  $_ == 13; }     #   10 11 12 13
    for 1..20 { .say if $_ == 9 ^ff^ $_ == 13; }     #   10 11 12

In each example we’re iterating over the range of numbers from 1 to 20 and output those numbers where the flip-flop returns true. Both the right hand side of the flip-flop ($_ == 9) and left hand side of the flip-flop ($_ == 13) are evaluated on each iteration of the loop. (I’ve used simple numeric comparison on both sides of the flip-flop operators here but, in general, any boolean expression could be used.)

Each instance of the flip-flop operator maintains it’s own little bit of internal state to decide when to return True or False. All flip-flop operators are born with their internal state set to return False waiting for the moment they can be flipped and start returning True.

In the first and second examples when $_ == 9, the flip-flop operators flips their internal state to True and immediately return True. In the third and fourth examples when $_ == 9 the flip-flop operators set their internal state to True but they return False on that iteration because of the leading circumflex.

Similarly, in the first and third examples above, once the RHS evaluates to True, the flip-flop operators flop their internal state back to False on next evaluation and return True. In the third and fourth examples, the flip-flops operators flop sooner by returning False immediately upon evaluating the RHS True.

To make the flip-flop operator flip, but never flop, use a * on the RHS:

    for 1..20 { .say if $_ == 15 ff *; }     # 15 16 17 18 19 20

Perl 6 has another set of flip-flop operators that function similar to the ones mentioned above, except the RHS isn’t evaluted when the LHS becomes true. This is particularly important when both the RHS and the LHS of the flip-flop could evaluate to True at the same time. These operators are spelled fff, fff^, ^fff, and ^fff^.

Traits — Meta Data With Character

2011-12-04T07:00:15+00:00

Traits are a nice, extensible way to attach meta data to all sorts of objects in Perl 6.

An example is the is cached trait that automatically caches the functions return value, based on the argument(s) passed to it.

Here is a simple implementation of that trait:

 # this gets called when 'is cached' is added
 # to a routine
 multi sub trait_mod:<is>(Routine $r, :$cached!) {
     my %cache;
     #wrap the routine in a block that..
     $r.wrap(-> $arg {
         # looks up the argument in the cache
         %cache.exists($arg)
             ?? %cache{$arg}
             # ... and calls the original, if it
             # is not found in the cache
             !! (%cache{$arg} = callwith($arg))
         }
     );
 }

 # example aplication:
 sub fib($x) is cached {
     say("fib($x)");
     $x <= 1 ?? 1 !! fib($x - 1) + fib($x - 2);
 }

 # only one call for each value from 0 to 10
 say fib(10);

A trait is applied with a verb, here is. That verb appears in the routine name that handles the trait, here trait_mod:<is>. The arguments to that handler are the object on which the trait is applied, and the name of the trait (here cached) as a named argument.

Note that a production grade is cached would need to handle multiple arguments, and maybe things like limiting the cache size.

In this example, the .wrap method is called on the routine, but of course you can do whatever you want. Common applications are mixing roles into the routine or adding them to a dispatch table.

Traits can not only be applied to routines, but also to parameters, attributes and variables. For example writable accessors are realized with the is rw trait:

 class Book {
     has @.pages is rw;
     ...
 }

Traits are also used to attach documentation to classes and attributes (stay tuned for an addvent calendar post on Pod6), marking routine parameters as writable and declaring class inheritance and role application.

This flexibility makes them ideal for writing libraries that make the user code look like a domain-specific language, and supplying meta data in a safe way.

Buffers and Binary IO

2011-12-03T07:30:36+00:00

Perl 5 is known to have very good Unicode support (starting from version 5.8, the later the better), but people still complain that it is hard to use. The most important reason for that is that the programmer needs to keep track of which strings have been decoded, and which are meant to be treated as binary strings. And there is no way to reliably introspect variables to find out if they are binary or text strings.

In Perl 6, this problem has been addressed by introducing separate types. Str holds text strings. String literals in Perl 6 are of type Str. Binary data is stored in Buf objects. There is no way to confuse the two. Converting back and forth is done with the encode and decode methods.

    my $buf = Buf.new(0x6d, 0xc3, 0xb8, 0xc3, 0xbe, 0x0a);
    $*OUT.write($buf);

    my $str = $buf.decode('UTF-8');
    print $str;

Both of those output operations have the same effect, and print møþ to the standard output stream, followed by a newline. Buf.new(...) takes a list of integers between 0 and 255, which are the byte values from which the new byte buffer is constructed. $*OUT.write($buf) writes the $buf buffer to standard output.

$buf.decode('UTF-8') decodes the buffer, and returns a Str object (or dies if the buffer doesn’t consistute valid UTF-8). The reverse operation is $Buf.encode($encoding). A Str can simply be printed with print.

Of course print also needs to convert the string to a binary representation somewhere in the process. There is a default encoding for this and other operations, and it is UTF-8. The Perl 6 specification allows the user to change the default, but no compiler implements that yet.

For reading, you can use the .read($no-of-bytes) methods to read a Buf, and .get for reading a line as a Str.

The read and write methods are also present on sockets, not just on the ordinary file and stream handles.

One of the particularly nasty things you can accidentally do in Perl 5 is
concatenating text and binary strings, or combine them in another way (like with join or string interpolation). The result of such an operation is a string that happens to be broken, but only if the binary string contains any bytes above 127 — which can be a nightmare to debug.

In Perl 6, you get Cannot use a Buf as a string when you try that, avoiding that trap.

The existing Perl 6 compilers do not yet provide the same level of Unicode support as Perl 5 does, but the bits that are there are much harder to misuse.

Grammar::Tracer and Grammar::Debugger

2011-12-02T12:37:00+00:00

Grammars are, for many people, one of the most exciting features of Perl 6. They unify parsing with object orientation, with each production rule in your grammar being represented by a method. These methods are a little special: they are declared using the keywords “regex”, “rule” or “token”, each of which gives you different defaults on backtracking and whitespace handling. In common is that they lead to the body of the method being parsed using the Perl 6 rule syntax. Under the hood, however, they really are just methods, and production rules that refer to others are really just method calls.

Perl 6 grammars also give you a seamless way to combine declarative and imperative parsing. This means efficient mechanisms, such as NFAs and DFAs, may be used to handle the declarative parts – the things that your tokens tend to be made up of – while a more imperative mechanism drives the parsing of larger structures. This in turn means that you don’t need to write a tokenizer; it can be derived from the rules that you write in the grammar.

So what is the result of parsing some text with a grammar? Well, provided it’s able to match your input, you get back a parse tree. This data structure – made up of Match objects – captures the structure of the input. You can treat each Match node a little bit like a hash, indexing in to it to look at the values that its production rules matched. While you can build up your own tree or other data structure while parsing, sometimes the Match tree you get back by default will be convenient enough to extract the information you need.

That’s wonderful, but there was a key clause in all of this: “provided it’s able to match”. In the case that the grammar fails to match your input, then it tells you so – by giving back an empty Match object that, in boolean context, is false. It’s at this point that many people stop feeling the wonder of grammars and start feeling the pain of trying to figure out why on earth their seemingly fine grammar did not accept the input they gave it. Often, it’s something silly – but in a grammar of dozens of production rules – or sometimes even just ten – the place where things go wrong can be elusive.

Thankfully, help is now at hand, in the form of two modules: Grammar::Tracer, which gives you a tree-like trace output of your grammar, and Grammar::Debugger, which gives the same trace output but also enables you to set breakpoints and single step through the grammar.

A picture is worth a thousand words, so here’s how Grammar::Tracer looks in action!

What we’re seeing here is a tree representation of the production rules that were called, starting at “TOP”, next trying to parse a production rule called “country”, which in turn wants to parse a name, two “num”s and an “integer”. The green indicates a successful match, and next to it we see the snippet of text that was captured.

So what happens when things go wrong? In that case, we see something like this:

Here, we see that something happened during the parse that caused a cascade of failures all the way back up to the “TOP” production rule, which meant that the parse failed overall. Happily, though, we now have a really good clue where to look. Here is the text my grammar was trying to match at the time:

Russia
	Ulan Ude : 51.833333,107.600000 : 1
	Moscow : 55.75000,37.616667 : 4

Looking at this, we see that the “name” rule appears to have picked up “Ulan”, but actually the place in question is “Ulan Ude”. This leads us directly to the name production in our grammar:

token name { \w+ }

Just a smattering of regex fu is enough to spot the problem here: we don’t parse names that happen to have spaces in them. Happily, that’s an easy fix.

token name { \w+ [\h+ \w+]* }

So how do we turn on the tracing? Actually, that’s easy: just take the file containing the grammar you wish to trace, and add at the top:

use Grammar::Tracer;

And that’s it; now whenever you use the grammar, it will be traced. Note that this statement has lexical effect, so if you’re using modules that also happen to have grammars – which you likely don’t care about – they will not end up getting the tracing behavior.

You can also do this:

use Grammar::Debugger;

The debugger is the tracer’s big sister, and knows a few more tricks. Here’s an example of it in action.

Instead of getting the full trace, now as soon as we hit the TOP production rule the program execution breaks and we get a prompt. Pressing enter allows you to step rule by rule through the parse. For some people, this may be preferable; others prefer to get the full trace output and analyze it. However, there are a few more tricks. In the example above, I added a breakpoint on the “name” rule. Using “r” informs the debugger to keep running through the production rules until it hits one called “name”, at which point it breaks. It is also possible to add breakpoints in code, for more extended debugging sessions with many runs. There’s one additional feature in code, which is to set a conditional breakpoint.

Sound interesting? You can get modules from GitHub, and if you want to see a live demo of a grammar being debugged using it, then there is a video of my Debugging Perl 6 Grammars talk from YAPC::Europe 2011; slides are also available to make the sample code more clear than it is on the video. Note that the modules need one of the compiler releases from the Rakudo “nom” development branch; we’ll be making a distribution release later this month based on that, though, and these modules will come with it.

You may also be thinking: I bet these are complex modules doing lots of guts stuff! In fact, they are 44 lines (Grammar::Tracer) and 171 lines (Grammar::Debugger), and written in Perl 6. They are built using the meta-programming support we’ve been working on in the Rakudo Perl 6 compiler during the course of the last year – and if you want to know more about that, be sure to check out my meta-programming post coming up later on in this year’s advent calendar.

Macros progress report: a bit of D1

2011-12-01T22:17:28+00:00

I've not been making much noise about it, but work on macros is progressing nicely. D1 is about providing a macro declarator to parallel the sub declarator, and a quasi construct which creates ASTs... all of that works already, in a branch of Rakudo. Try it out! Be the first one on your block to run macros in Perl 6!

November has been a busy month for me, $dayjob-wise. I knew that was going to happen, even though it ended up being even more busy than I had imagined. Now I'm taking a well-deserved two-week vacation, and then I'll be back and actually have some time for Perl 6 hacking. Looking forward to that. 哈哈

A bit of the part of D1 that doesn't work yet: it turns out that there are "interesting" things happening with lexical lookup and quasiquotes. It's actually nothing very complicated, but it requires some extra wiring. So it's actually possible to cause Null PMC accesses right now because variable lookups from inside of the quasiquote end up confused.

I know how to solve this, in theory. ASTs have to start carrying around their own lexical environment. But I haven't had time to actually sit down and type out the solution. Will write more when I've done that. Till then, feel free to play around with the parts of macros that don't do too wild lexical lookups.

Beyond that, I'd like to give D1 a bit of test coverage in roast, and then I think we can merge the D1 work into nom. Looking forward to that.

Day 1: Catching Up With Perl 6

2011-12-01T14:40:26+00:00

When we started the Perl 6 Advent Calendar back in 2009, Rakudo was really the only game in town if you wanted to play with Perl 6. But Perl 6 was intended from the start to be a language with multiple implementations, and at the moment there are four different Perl 6 implementations of interest. Because there are so many implementations, I’m not going to give instructions for getting each; instead I’m linking to those instructions.

The most stable and complete implementation is Rakudo Star. This is currently based on the last major revision of Rakudo. It’s been frozen since July, and so lags a bit behind the current Perl 6 spec. It’s slow. But it’s also pretty reliable.

The current Rakudo development version is called “Nom”. It’s full of great improvements over the last Rakudo Star release, notably native types, improved performance, and a much better metamodel. (For example, check out the Grammar::Tracer module, which takes advantage of the new metamodel to add regex tracing in just 44 lines of code.) It’s not quite ready for prime time yet, as it still misses some features that work in Rakudo Star, but progress has been incredible, and it’s quite possible a new Rakudo Star based on Nom will be released during this month.

Stefan O’Rear’s Niecza was just a fledging compiler during last year’s Advent calendar, but it’s a serious contender these days. Built to run on the CLR (.NET and Mono), it is relatively zippy, implements a significant portion of Perl 6, and works easily with existing CLR libraries.

Lastly, ingy and Mäsak have plans afoot to revive Pugs, the original Perl 6 implementation in Haskell. So far they’ve just got it building again on current Haskell compilers, but the long-term goal is to get it running on the spec tests again and bring it closer to the current spec.

Which implementation should you use? If you’re looking for a stable, fairly complete Perl 6, Rakudo Star is it. If you just want to explore the language, try Rakudo Nom — you will probably run into bugs, but it’s significantly more advanced than Rakudo Star, and exposing the bugs is a big help to Rakudo’s development. If you have an idea which would benefit from being able to use CLR libraries, Niecza is fantastic. There’s a handy comparison chart of the different features available.

Personally, I have all three of these installed on my machine, and have different projects underway on each of them.

Finally, please don’t hesitate to ask for help, either in the comments here or on the #perl6 IRC channel on Freenode. The Perl 6 community is very friendly.

Perl 6 Advent Calendar 2011

2011-12-01T00:19:32+00:00

For the third year in a row, we are going to post something about Perl 6 every day until Christmas. This post will serve as a table of contents for the entire month.

Day 1: Catching Up With Perl 6
Day 2: Grammar::Tracer and Grammar::Debugger
Day 3: Buffers and Binary IO
Day 4: Traits — Meta Data with Character
Day 5: The Flip-Flop operator
Day 6: Tetris on Niecza
Day 7: Adventures in writing a simple grammar profiler
Day 8: Lexicality and Optimizability
Day 9: Contributing to Perl 6

Announce: Niecza Perl 6 v12 by Stefan O'Rear

2011-11-28T20:56:52+00:00

Announce: Niecza Perl 6 v12

This is the twelth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. I've not had very much time for Niecza
this month. This release marks one year since the first public release,
and about a year and a half since the project began. It's hard to
believe I've been at it this long.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.

List of changes

[Minor features]

:BASE<DIGITS> literals are now supported. (Solomon Foster)

[Bug fixes]

sqrt(-1) is NaN, complex math is strictly opt-in (#64).

Undefined values can be passed correctly to CLR interop (#66).

-I is now mentioned by --help (#52).

Constants, including 'our constant's, are now truly immutable (#46).

$@ and $^X now properly handled as the last characters in the file (#63).

Left side of xx is now autothunkified so [] xx 4 works correctly (#70).

[Other]

The internals documentation has been largely rewritten to match the
refactor in v11.

Added a copy of Martin Berends' London Perl Workshop slides.

Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

My current priorities are:
1. Make regexes much more feature-complete, including general Unicode
properties and grapheme mode
2. Prototype the debugger
3. 6model convergence work, including roles/native types
4. Figure out how modules and S11 stuff should work in Niecza. Do it.

[1] https://github.com/downloads/sorear/niecza/niecza-12.zip
[2] https://github.com/sorear/niecza