🔬68. The smartness of the sequence operator in Perl 6, part 1

In Perl 6, you can ask the sequence operator to build a desired sequence for you. It can be arithmetic or geometric progression. All you need is to show the beginning of the sequence to Perl, for example:

.say for 3, 5 ... 11;

This prints numbers 3, 5, 7, 9, and 11. Or:

.say for 2, 4, 8 ... 64;

This code prints powers of 2 from 2 to 64: 2, 4, 8, 16, 32, and 64.

I am going to try understanding how that works in Rakudo. First of all, look into the src/core/operators.pm file, which keeps a lot of different operators, including a few versions of the ... operator. The one we need looks really simple:

multi sub infix:<...>(\a, Mu \b) {
    Seq.new(SEQUENCE(a, b).iterator)
}

Now, the main work is done inside the SEQUENCE sub. Before we dive there, it is important to understand what its arguments a and b receive.

In the case of, say, 3, 5 ... 11, the first argument is a list 3, 5, and the second argument is a single value 11.

These values land in the parameters of the routine:

sub SEQUENCE(\left, Mu \right, :$exclude_end) {
    . . .
}

What happens next is not that easy to grasp. Here is a screenshot of the complete function:

sequence

It contains about 350 lines of code and includes a couple of functions. Nevertheless, let’s try.

What you see first, is creating iterators for both left and right operands:

my \righti := (nqp::iscont(right) ?? right !! [right]).iterator;
my \lefti := left.iterator;

Then, the code loops over the left operand and builds an array @tail out of its data:

while !((my \value := lefti.pull-one) =:= IterationEnd) {
    $looped = True;
    if nqp::istype(value,Code) { $code = value; last }
    if $end_code_arity != 0 {
        @end_tail.push(value);
        if +@end_tail >= $end_code_arity {
            @end_tail.shift xx (@end_tail.elems - $end_code_arity)
                unless $end_code_arity ~~ -Inf;

            if $endpoint(|@end_tail) {
                $stop = 1;
                @tail.push(value) unless $exclude_end;
                last;
            }
        }
    }
    elsif value ~~ $endpoint {
        $stop = 1;
        @tail.push(value) unless $exclude_end;
        last;
    }
    @tail.push(value);
}

I leave you reading and understand this piece of code as an exercise, but for the given example, the @tail array will just contain two values: 3 and 5.

> .say for 3,5...11;
multi sub infix:<...>(\a, Mu \b)
List    # nqp::say(a.^name);
~~3     # nqp::say('~~' ~ value);
~~5     # nqp::say('~~' ~ value);
elems=2 # nqp::say('elems='~@tail.elems);
0=3     # nqp::say('0='~@tail[0]);
1=5     # nqp::say('1='~@tail[1]);

This output shows some debug data print outs that I added to the source code to see how it works. The green comments show the corresponding print instructions.

That’s it for today. See you tomorrow with more stuff from the sequence operator. Tomorrow, we have to understand how the list 3, 5 tells Perl 6 to generate increasing values with step 1.

🔬67. Redeclaration of a symbol in Perl 6

Today, we will see how Perl 6 helps to keep our programs better.

Redeclaration of a variable

Examine the following program:

my $x = 1;
my $x = 2;
say $x;

You can immediately see that this program is not entirely correct. Either we meant to assign a new value to $x or to create a new variable with a different name. In either case, compiler has no idea and complains:

$ perl6 redecl.pl 
Potential difficulties:
    Redeclaration of symbol '$x'
    at /Users/ash/redecl.pl:2
    ------> my $x⏏ = 2;
2

You see a runtime warning, while the program does not stop. Let us find out where it happens in the source code.

When you declare a variable, the grammar matches the corresponding text and calls the variable_declarator action method. It is quite compact but nevertheless I will not quote it completely.

class Perl6::Actions is HLL::Actions does STDActions {
    . . .

    method variable_declarator($/) {
        . . .
    }

    . . .
}

By the way, you can see here how Perl 6 treats a variable name:

 my $past := $<variable>.ast;
 my $sigil := $<variable><sigil>;
 my $twigil := $<variable><twigil>;
 my $desigilname := ~$<variable><desigilname>;
 my $name := $sigil ~ $twigil ~ $desigilname;

The name of a variable is a concatenation of a sigil, a twigil and an identifier (which is called desigiled name in the code).

Then, if we’ve got a proper variable name, check it against an existing lexpad:

if $<variable><desigilname> {
    my $lex := $*W.cur_lexpad();
    if $lex.symbol($name) {
        $/.typed_worry('X::Redeclaration', symbol => $name);
    }

If the name is known, generate a warning. If everything is fine, create a variable declaration:

make declare_variable($/, $past, ~$sigil, ~$twigil, $desigilname,
                      $<trait>, $<semilist>, :@post);

Redeclaration of a routine

Now, let us try to re-create a subroutine:

sub f() {}
sub f() {}

This may only be OK if the subs are declared as multi-subs. With the given code, the program will not even compile:

===SORRY!=== Error while compiling /Users/ash/redecl.pl
Redeclaration of routine 'f' (did you mean to declare a multi-sub?)
at /Users/ash/redecl.pl:6
------> sub f() {}⏏<EOL>

This time, it happens in a much more complicated method, routine_def:

method routine_def($/) {
     . . .

     my $predeclared := $outer.symbol($name);
     if $predeclared {
         my $Routine := $*W.find_symbol(['Routine'], :setting-only);
         unless nqp::istype( $predeclared<value>, $Routine)
                && nqp::getattr_i($predeclared<value>, $Routine, '$!yada') {
              $*W.throw($/, ['X', 'Redeclaration'],
                        symbol => ~$<deflongname>.ast,
                        what => 'routine',
              );
         }
     }

The exception

The code of the exception is rather simple. Here it is:

my class X::Redeclaration does X::Comp {
    has $.symbol;
    has $.postfix = '';
    has $.what = 'symbol';
    method message() {
        "Redeclaration of $.what '$.symbol'"
        ~ (" $.postfix" if $.postfix)
        ~ (" (did you mean to declare a multi-sub?)" if $.what eq 'routine');
    }
}

As you see, depending on the value of $.what, it prints either a short message or adds a suggestion to use the multi keyword.

🦋 66. Atomic operations in Perl 6

N. B. The examples below require a fresh Rakudo compiler, at least of the version 2017.09.

Discussing parallel computing earlier or later leads to solving race conditions. Let us look at a simple counter that is incremented by two parallel threads:

my $c = 0;

await do for 1..10 {
    start {
        $c++ for 1 .. 1_000_000
    }
}

say $c;

If you run the program a few times, you will immediately see that the results are very different:

$ perl6 atomic-1.pl 
3141187
$ perl6 atomic-1.pl 
3211980
$ perl6 atomic-1.pl 
3174944
$ perl6 atomic-1.pl 
3271573

Of course, the idea was to increase the counter by 1 million in all of the ten threads, but about ⅓ of the steps were lost. It is quite easy to understand why that happens: the parallel threads read the variable and write to it ignoring the presence of other threads and not thinking that the value can be changed in-between. Thus, some of the threads work with an outdated value of the counter.

Perl 6 offers a solution: atomic operations. The syntax of the language is equipped with the Atom Symbol (U+0x269B) ⚛ character (no idea of why it is displayed in that purple colour). Instead of $c++, you should type $c⚛++.

my atomicint $c = 0;

await do for 1..10 {
    start {
        $c⚛++ for 1 .. 1_000_000
    }
}

say $c;

And before thinking of the necessity to use a Unicode character, let us look at the result of the updated program:

$ perl6 atomic-2.pl 
10000000

This is exactly the result we wanted!

Notice also, that the variable is declared as a variable of the atomicint type. That is a synonym for int, which is a native integer (unlike Int, which is a data type represented by a Perl 6 class).

It is not possible to ask a regular value to be atomic. That attempt will be rejected by the compiler:

$ perl6 -e'my $c; $c⚛++'
Expected a modifiable native int argument for '$target'
  in block  at -e line 1

A few other operators can be atomic, for example, prefix and postfix increments and decrements ++ and --, or += and -=. There are also atomic versions of the assignment operator = and the one for reading: ⚛ (sic!).

If you need atomic operations in your code, you are not forced to use the ⚛ character. There exists a bunch of alternative functions that you can use instead of the operators:

my atomicint $c = 1;

my $x = ⚛$c;  $x = atomic-fetch($c);
$c ⚛= $x;     atomic-assign($c, $x);
$c⚛++;        atomic-fetch-inc($c);
$c⚛--;        atomic-fetch-dec($c);
++⚛$c;        atomic-inc-fetch($c);
--⚛$c;        atomic-dec-fetch($c);
$c ⚛+= $x;    atomic-fetch-add($c,$x);

say $x; # 1
say $c; # 3

🔬65. The EVAL routine in Perl 6, part 2

Welcome back! As you might notice, there was a small gap in the daily post flow.

Before we are back to the Rakudo internals, a couple of words about some changes here. First of all, every post is now marked with either 🦋 or 🔬 (or with indistinguishable rectangles □ if your browser cannot display an emoji :-). These characters mean two categories of posts here: a butterfly stands for Perl 6 syntax, while a microscope is for Perl 6 internals. In the first category, only user-level aspects or Perl 6 are discussed. In the second, we dig into the source codes of Rakudo. All the past post are updated accordingly.

The second change is that I will occasionally post more articles in the Perl 6 syntax category because I found out that non-Russian speakers often like my Russian blog posts. Those posts are mostly short texts explaining interesting features of Perl 6, such as the =~= operator or promises.

* * *

OK, now we have to talk about the EVAL routine. It is defined in the src/core/ForeignCode.pm file as a multi-function. Let us see at their signatures:

proto sub EVAL($code is copy where Blob|Cool, 
               Str() :$lang = 'perl6',
               PseudoStash :$context, *%n)

multi sub EVAL($code, 
               Str :$lang where { ($lang // '') eq 'Perl5' },
               PseudoStash :$context)

Notice that one of the function is a proto, while another is the only multi-candidate. Unlike many other cases that you can see in the sources of Rakudo, this proto routine contains code. Refer to one of the recent blog posts to see how it works.

We start with an example from the first part of the article.

EVAL('say 123');

Here, the passed value is Str, and it is caught by the proto sub, as its first argument can be Cool.

The sub creates a compiler for the given language (which is Perl 6 by default).

my $compiler := nqp::getcomp($lang);

The next step in the sub is to make a string out of $code. In this first example, this task is trivial.

$code = nqp::istype($code,Blob) ?? $code.decode(
    $compiler.cli-options<encoding> // 'utf8'
) !! $code.Str;

Finally, the string is compiled:

my $compiled := $compiler.compile:
    $code,
    :outer_ctx($eval_ctx),
    :global(GLOBAL),
    :mast_frames(mast_frames),
    |(:optimize($_) with nqp::getcomp('perl6').cli-options<optimize>),
    |(%(:grammar($LANG<MAIN>), :actions($LANG<MAIN-actions>)) if $LANG);

After compilation, you get an object of the ForeignCode type. This class is a child class of Callable, so the object can be called and returned (actually, it’s not quite clear how it happens):

$compiled();

Now you can understand that single quotes in the second example with curly braces still create an executable code:

EVAL('say {456}');

Here, the whole string is compiled as it was a Perl 6 code, and the code block there is a code block, which Perl should execute, and thus say gets a code block, so it calls its gist method to prepare the output:

> {456}.gist
-> ;; $_? is raw { #`(Block|140388575216888) ... }

 

🦋64. What does gist do in Perl 6?

When you print an object, say, as say $x, Perl 6 calls the gist method. This method is defined for all built-in types: for some of them, it calls the Str method, for some the perl method, for some types it makes the string representation somehow differently.

Let us see how you can use the method to create your own variant:

class X {
    has $.value;

    method gist {
        '[' ~ $!value ~ ']'
    }
}

my $x = X.new(value => 42);

say $x; # [42]
$x.say; # [42]

When you call say, the program prints a number in square brackets: [42].

Please notice that the interpolation inside double-quoted strings is using Str, not gist. You can see it here:

say $x.Str; # X<140586830040512>
say "$x";   # X<140586830040512>

If you need a custom interpolation, redefine the Str method:

class X {
    has $.value;

    method gist {
        '[' ~ $!value ~ ']'
    }
    method Str {
        '"' ~ $!value ~ '"'
    }
}

my $x = X.new(value => 42);

say $x;     # [42]
$x.say;     # [42]

say $x.Str; # "42"
say "$x";   # "42"

Now, we got the desired behaviour.

🦋63. More on the proto keyword in Perl 6

Before digging into the details of the EVAL routine, we have to reveal some more information about protos and multiple dispatch. Examine the following program:

proto sub f($x) {
    say "proto f($x)";
}

multi sub f($x) {
    say "f($x)"
}

multi sub f(Int $x) {
    say "f(Int $x)"
}

multi sub f(Str $x) {
    say "f(Str $x)"
}

f(2);
f('2');
f(3);
f('3');

Here, there are three multi-candidates of the function plus a function declared with the proto keyword. Earlier, we only saw such proto-functions with empty body, such as:

proto sub f($x) {*}

But this is not a necessity. The function can carry a regular load, as we see in the example:

proto sub f($x) {
    say "proto f($x)";
}

Run the program:

proto f(2)
proto f(2)
proto f(3)
proto f(3)

All the calls were caught by the proto-candidate. Now, update it and return the {*} block for some dedicated values;

proto sub f($x) {
    if $x.Str eq '3' {
        return {*}
    }
    say "proto f($x)";
}

The if check triggers its block for the last two function calls:

f(3);
f('3');

In these cases, the proto-function returns {*}, which makes Perl 6 trying other candidates. As we have enough candidates for both integer and string arguments, the compiler can easily choose one of them:

proto f(2)
proto f(2)
f(Int 3)
f(Str 3)

🦋62. The EVAL routine in Perl 6, part 1

The EVAL routine in Perl 6 compiles and executes the code that it gets as an argument.  Today, we will see some potential use cases that you may try in your practice. Tomorrow, we will dig into Rakudo sources to see how it works and why it breaks sometimes.

1

Let us start with evaluating a simple program:

EVAL('say 123');

This program prints 123, there’s no surprise here.

2

There are, though, more complicated cases. What, do you think, does the following program print?

EVAL('say {456}');

I guess it prints not what you expected:

-> ;; $_? is raw { #`(Block|140570649867712) ... }

It parses the content between the curly braces as a pointy block.

3

What if you try double quotes?

EVAL("say {789}");

Now it even refuses to compile:

===SORRY!=== Error while compiling eval.pl
EVAL is a very dangerous function!!! (use the MONKEY-SEE-NO-EVAL pragma to override this error,
but only if you're VERY sure your data contains no injection attacks)
at eval.pl:6
------> EVAL("say {789}")⏏;

4

We can fix the code by adding a few magic words:

use MONKEY-SEE-NO-EVAL;

EVAL("say {789}");

This time, it prints 789.

5

The code is executed (we don’t know yet when exactly, that is the topic of tomorrow’s post), so you can make some calculations, for example:

use MONKEY-SEE-NO-EVAL;

EVAL("say {7 / 8 + 9}"); # 9.875

6

Finally, if you try passing a code block directly, you also cannot achieve the goal, even with a blind monkey:

use MONKEY-SEE-NO-EVAL;

EVAL {say 123};

The error happens at runtime:

Constraint type check failed in binding to parameter '$code';
expected anonymous constraint to be met but got 
-> ;; $_? is raw { #`...
  in block <unit> at eval.pl line 10

This message looks cryptic, but at least we see once again that we got an anonymous pointy block passed to the function.

7

And before we wrap up for today, an attempt to use Perl 5 syntax:

eval('say 42');

There is no such function in Perl 6, and we get a standard error message:

===SORRY!=== Error while compiling eval2.pl
Undeclared routine:
  eval used at line 5. Did you mean 'EVAL', 'val'?

It looks OK but it can be better.

Stay tuned, tomorrow we will try to understand how all these examples work in Rakudo.