34. Delimiters of embedded comments in Perl 6

In Perl 6, you can leave a comment inside the line of code using the #`(...) syntax:

my Int #`(Integer variable) $i = #`(initialising the variable) 42;
say #`( Printing it ) $i;

This program is equivalent to the following:

my Int $i = 42;
say $i;

The body of the embedded comment is located between the pair of brackets, and you are not limited to parentheses only. For example, both {} or [] perfectly work:

my Int #`(Integer variable) $i = #`[initialising the variable] 42;
say #`{ Printing it } $i;

Perl 6 allows even more. The following program is also a correct Perl 6 program with the same inline comments that in the above samples:

my Int #`﹝Integer variable﹞ $i = #`⸨initialising the variable⸩ 42;
say #`∈ Printing it ∋ $i;

Let us see which other brackets are available for the programmer and where it is defined in the source code.

So, here is a place for extracting an embedded comment in the Grammar (src/core/Grammar.nqp):

proto token comment { <...> }

. . .

token comment:sym<#`(...)> {
    '#`' <?opener> {}
    [ <.quibble(self.slang_grammar('Quote'))> || <.typed_panic: 'X::Syntax::Comment::Embedded'> ]
}

The comment token expects the #` sequence followed by the opener token. What is an opener? Here we are (still in src/Perl6/Grammar.nqp):

role STD {
    token opener {
        <[
        \x0028 \x003C \x005B \x007B \x00AB \x0F3A \x0F3C \x169B \x2018 \x201A \x201B
        \x201C \x201E \x201F \x2039 \x2045 \x207D \x208D \x2208 \x2209 \x220A \x2215
        \x223C \x2243 \x2252 \x2254 \x2264 \x2266 \x2268 \x226A \x226E \x2270 \x2272
        \x2274 \x2276 \x2278 \x227A \x227C \x227E \x2280 \x2282 \x2284 \x2286 \x2288
        \x228A \x228F \x2291 \x2298 \x22A2 \x22A6 \x22A8 \x22A9 \x22AB \x22B0 \x22B2
        \x22B4 \x22B6 \x22C9 \x22CB \x22D0 \x22D6 \x22D8 \x22DA \x22DC \x22DE \x22E0
        \x22E2 \x22E4 \x22E6 \x22E8 \x22EA \x22EC \x22F0 \x22F2 \x22F3 \x22F4 \x22F6
        \x22F7 \x2308 \x230A \x2329 \x23B4 \x2768 \x276A \x276C \x276E \x2770 \x2772
        \x2774 \x27C3 \x27C5 \x27D5 \x27DD \x27E2 \x27E4 \x27E6 \x27E8 \x27EA \x2983
        \x2985 \x2987 \x2989 \x298B \x298D \x298F \x2991 \x2993 \x2995 \x2997 \x29C0
        \x29C4 \x29CF \x29D1 \x29D4 \x29D8 \x29DA \x29F8 \x29FC \x2A2B \x2A2D \x2A34
        \x2A3C \x2A64 \x2A79 \x2A7D \x2A7F \x2A81 \x2A83 \x2A8B \x2A91 \x2A93 \x2A95
        \x2A97 \x2A99 \x2A9B \x2AA1 \x2AA6 \x2AA8 \x2AAA \x2AAC \x2AAF \x2AB3 \x2ABB
        \x2ABD \x2ABF \x2AC1 \x2AC3 \x2AC5 \x2ACD \x2ACF \x2AD1 \x2AD3 \x2AD5 \x2AEC
        \x2AF7 \x2AF9 \x2E02 \x2E04 \x2E09 \x2E0C \x2E1C \x2E20 \x2E28 \x3008 \x300A
        \x300C \x300E \x3010 \x3014 \x3016 \x3018 \x301A \x301D \xFD3E \xFE17 \xFE35
        \xFE37 \xFE39 \xFE3B \xFE3D \xFE3F \xFE41 \xFE43 \xFE47 \xFE59 \xFE5B \xFE5D
        \xFF08 \xFF1C \xFF3B \xFF5B \xFF5F \xFF62
    ]>
}

Wow, quite a lot! Notice that these are only opening brackets. Let’s print them (just copy and paste the list and pass it to say):

( < [ { « ༺ ༼ ᚛ ‘ ‚ ‛
“ „ ‟ ‹ ⁅ ⁽ ₍ ∈ ∉ ∊ ∕
∼ ≃ ≒ ≔ ≤ ≦ ≨ ≪ ≮ ≰ ≲
≴ ≶ ≸ ≺ ≼ ≾ ⊀ ⊂ ⊄ ⊆ ⊈
⊊ ⊏ ⊑ ⊘ ⊢ ⊦ ⊨ ⊩ ⊫ ⊰ ⊲
⊴ ⊶ ⋉ ⋋ ⋐ ⋖ ⋘ ⋚ ⋜ ⋞ ⋠
⋢ ⋤ ⋦ ⋨ ⋪ ⋬ ⋰ ⋲ ⋳ ⋴ ⋶
⋷ ⌈ ⌊ 〈 ⎴ ❨ ❪ ❬ ❮ ❰ ❲
❴ ⟃ ⟅ ⟕ ⟝ ⟢ ⟤ ⟦ ⟨ ⟪ ⦃
⦅ ⦇ ⦉ ⦋ ⦍ ⦏ ⦑ ⦓ ⦕ ⦗ ⧀
⧄ ⧏ ⧑ ⧔ ⧘ ⧚ ⧸ ⧼ ⨫ ⨭ ⨴
⨼ ⩤ ⩹ ⩽ ⩿ ⪁ ⪃ ⪋ ⪑ ⪓ ⪕
⪗ ⪙ ⪛ ⪡ ⪦ ⪨ ⪪ ⪬ ⪯ ⪳ ⪻
⪽ ⪿ ⫁ ⫃ ⫅ ⫍ ⫏ ⫑ ⫓ ⫕ ⫬
⫷ ⫹ ⸂ ⸄ ⸉ ⸌ ⸜ ⸠ ⸨ 〈 《
「 『 【 〔 〖 〘 〚 〝 ﴾ ︗ ︵
︷ ︹ ︻ ︽ ︿ ﹁ ﹃ ﹇ ﹙ ﹛ ﹝
( < [ { ⦅ 「

I am almost sure nobody will ever use all this variety in the same program, neither you may be familiar with some characters from the list. Nevertheless, there are not only pure brackets here but also other characters such as less or equal, which have the corresponding pair in the Unicode space.

Let us now find the closing brackets. There is a list in nqp/src/HLL/Grammar.pm:

grammar HLL::Grammar {
    my $brackets := '<>[](){}' ~ "\x[0028]\x[0029]\x[003C]\x[003E]\x[005B]\x[005D]" ~
    "\x[007B]\x[007D]\x[00AB]\x[00BB]\x[0F3A]\x[0F3B]\x[0F3C]\x[0F3D]\x[169B]\x[169C]" ~
    "\x[2018]\x[2019]\x[201A]\x[2019]\x[201B]\x[2019]\x[201C]\x[201D]\x[201E]\x[201D]" ~
    "\x[201F]\x[201D]\x[2039]\x[203A]\x[2045]\x[2046]\x[207D]\x[207E]\x[208D]\x[208E]" ~
    "\x[2208]\x[220B]\x[2209]\x[220C]\x[220A]\x[220D]\x[2215]\x[29F5]\x[223C]\x[223D]" ~
    "\x[2243]\x[22CD]\x[2252]\x[2253]\x[2254]\x[2255]\x[2264]\x[2265]\x[2266]\x[2267]" ~
    "\x[2268]\x[2269]\x[226A]\x[226B]\x[226E]\x[226F]\x[2270]\x[2271]\x[2272]\x[2273]" ~
    "\x[2274]\x[2275]\x[2276]\x[2277]\x[2278]\x[2279]\x[227A]\x[227B]\x[227C]\x[227D]" ~
    "\x[227E]\x[227F]\x[2280]\x[2281]\x[2282]\x[2283]\x[2284]\x[2285]\x[2286]\x[2287]" ~
    "\x[2288]\x[2289]\x[228A]\x[228B]\x[228F]\x[2290]\x[2291]\x[2292]\x[2298]\x[29B8]" ~
    "\x[22A2]\x[22A3]\x[22A6]\x[2ADE]\x[22A8]\x[2AE4]\x[22A9]\x[2AE3]\x[22AB]\x[2AE5]" ~
    "\x[22B0]\x[22B1]\x[22B2]\x[22B3]\x[22B4]\x[22B5]\x[22B6]\x[22B7]\x[22C9]\x[22CA]" ~
    "\x[22CB]\x[22CC]\x[22D0]\x[22D1]\x[22D6]\x[22D7]\x[22D8]\x[22D9]\x[22DA]\x[22DB]" ~
    "\x[22DC]\x[22DD]\x[22DE]\x[22DF]\x[22E0]\x[22E1]\x[22E2]\x[22E3]\x[22E4]\x[22E5]" ~
    "\x[22E6]\x[22E7]\x[22E8]\x[22E9]\x[22EA]\x[22EB]\x[22EC]\x[22ED]\x[22F0]\x[22F1]" ~
    "\x[22F2]\x[22FA]\x[22F3]\x[22FB]\x[22F4]\x[22FC]\x[22F6]\x[22FD]\x[22F7]\x[22FE]" ~
    "\x[2308]\x[2309]\x[230A]\x[230B]\x[2329]\x[232A]\x[23B4]\x[23B5]\x[2768]\x[2769]" ~
    "\x[276A]\x[276B]\x[276C]\x[276D]\x[276E]\x[276F]\x[2770]\x[2771]\x[2772]\x[2773]" ~
    "\x[2774]\x[2775]\x[27C3]\x[27C4]\x[27C5]\x[27C6]\x[27D5]\x[27D6]\x[27DD]\x[27DE]" ~
    "\x[27E2]\x[27E3]\x[27E4]\x[27E5]\x[27E6]\x[27E7]\x[27E8]\x[27E9]\x[27EA]\x[27EB]" ~
    "\x[2983]\x[2984]\x[2985]\x[2986]\x[2987]\x[2988]\x[2989]\x[298A]\x[298B]\x[298C]" ~
    "\x[298D]\x[2990]\x[298F]\x[298E]\x[2991]\x[2992]\x[2993]\x[2994]\x[2995]\x[2996]" ~
    "\x[2997]\x[2998]\x[29C0]\x[29C1]\x[29C4]\x[29C5]\x[29CF]\x[29D0]\x[29D1]\x[29D2]" ~
    "\x[29D4]\x[29D5]\x[29D8]\x[29D9]\x[29DA]\x[29DB]\x[29F8]\x[29F9]\x[29FC]\x[29FD]" ~
    "\x[2A2B]\x[2A2C]\x[2A2D]\x[2A2E]\x[2A34]\x[2A35]\x[2A3C]\x[2A3D]\x[2A64]\x[2A65]" ~
    "\x[2A79]\x[2A7A]\x[2A7D]\x[2A7E]\x[2A7F]\x[2A80]\x[2A81]\x[2A82]\x[2A83]\x[2A84]" ~
    "\x[2A8B]\x[2A8C]\x[2A91]\x[2A92]\x[2A93]\x[2A94]\x[2A95]\x[2A96]\x[2A97]\x[2A98]" ~
    "\x[2A99]\x[2A9A]\x[2A9B]\x[2A9C]\x[2AA1]\x[2AA2]\x[2AA6]\x[2AA7]\x[2AA8]\x[2AA9]" ~
    "\x[2AAA]\x[2AAB]\x[2AAC]\x[2AAD]\x[2AAF]\x[2AB0]\x[2AB3]\x[2AB4]\x[2ABB]\x[2ABC]" ~
    "\x[2ABD]\x[2ABE]\x[2ABF]\x[2AC0]\x[2AC1]\x[2AC2]\x[2AC3]\x[2AC4]\x[2AC5]\x[2AC6]" ~
    "\x[2ACD]\x[2ACE]\x[2ACF]\x[2AD0]\x[2AD1]\x[2AD2]\x[2AD3]\x[2AD4]\x[2AD5]\x[2AD6]" ~
    "\x[2AEC]\x[2AED]\x[2AF7]\x[2AF8]\x[2AF9]\x[2AFA]\x[2E02]\x[2E03]\x[2E04]\x[2E05]" ~
    "\x[2E09]\x[2E0A]\x[2E0C]\x[2E0D]\x[2E1C]\x[2E1D]\x[2E20]\x[2E21]\x[2E28]\x[2E29]" ~
    "\x[3008]\x[3009]\x[300A]\x[300B]\x[300C]\x[300D]\x[300E]\x[300F]\x[3010]\x[3011]" ~
    "\x[3014]\x[3015]\x[3016]\x[3017]\x[3018]\x[3019]\x[301A]\x[301B]\x[301D]\x[301E]" ~
    "\x[FE17]\x[FE18]\x[FE35]\x[FE36]\x[FE37]\x[FE38]\x[FE39]\x[FE3A]\x[FE3B]\x[FE3C]" ~
    "\x[FE3D]\x[FE3E]\x[FE3F]\x[FE40]\x[FE41]\x[FE42]\x[FE43]\x[FE44]\x[FE47]\x[FE48]" ~
    "\x[FE59]\x[FE5A]\x[FE5B]\x[FE5C]\x[FE5D]\x[FE5E]\x[FF08]\x[FF09]\x[FF1C]\x[FF1E]" ~
    "\x[FF3B]\x[FF3D]\x[FF5B]\x[FF5D]\x[FF5F]\x[FF60]\x[FF62]\x[FF63]\x[27EE]\x[27EF]" ~
    "\x[2E24]\x[2E25]\x[27EC]\x[27ED]\x[2E22]\x[2E23]\x[2E26]\x[2E27]"
#?if js
    ~ nqp::chr(0x2329) ~ nqp::chr(0x232A)
#?endif
    ;

Here is its visualisation:

<>[](){}()<>[]{}«»༺༻༼༽᚛᚜‘’‚’‛’“”„”‟”‹›⁅⁆
⁽⁾₍₎∈∋∉∌∊∍∕⧵∼∽≃⋍≒≓≔≕≤≥≦≧≨≩≪≫≮≯≰≱≲≳≴≵≶≷≸≹≺≻≼≽
≾≿⊀⊁⊂⊃⊄⊅⊆⊇⊈⊉⊊⊋⊏⊐⊑⊒⊘⦸⊢⊣⊦⫞⊨⫤⊩⫣⊫⫥⊰⊱⊲⊳⊴⊵⊶⊷⋉⋊
⋋⋌⋐⋑⋖⋗⋘⋙⋚⋛⋜⋝⋞⋟⋠⋡⋢⋣⋤⋥⋦⋧⋨⋩⋪⋫⋬⋭⋰⋱⋲⋺⋳⋻⋴⋼⋶⋽
⋷⋾⌈⌉⌊⌋〈〉⎴⎵❨❩❪❫❬❭❮❯❰❱❲❳❴❵⟃⟄⟅⟆⟕⟖⟝⟞⟢⟣⟤⟥⟦⟧⟨⟩⟪⟫
⦃⦄⦅⦆⦇⦈⦉⦊⦋⦌⦍⦐⦏⦎⦑⦒⦓⦔⦕⦖⦗⦘⧀⧁⧄⧅⧏⧐⧑⧒⧔⧕⧘⧙⧚⧛⧸⧹⧼⧽⨫⨬⨭⨮⨴⨵⨼⨽⩤⩥
⩹⩺⩽⩾⩿⪀⪁⪂⪃⪄⪋⪌⪑⪒⪓⪔⪕⪖⪗⪘⪙⪚⪛⪜⪡⪢⪦⪧⪨⪩⪪⪫⪬⪭⪯⪰⪳⪴⪻⪼⪽⪾⪿⫀⫁⫂
⫃⫄⫅⫆⫍⫎⫏⫐⫑⫒⫓⫔⫕⫖⫬⫭⫷⫸⫹⫺⸂⸃⸄⸅⸉⸊⸌⸍⸜⸝⸠⸡⸨⸩〈〉《》「」『』【】
〔〕〖〗〘〙〚〛〝〞︗︘︵︶︷︸︹︺︻︼︽︾︿﹀﹁﹂
﹃﹄﹇﹈﹙﹚﹛﹜﹝﹞()<>[]{}⦅⦆「」⟮⟯⸤⸥⟬⟭⸢⸣⸦⸧

So just pick a pair you like the most and amaze your colleagues.

In the peek_delimiters method of the HLL::Grammar, you may see the following way of finding the closing pair:

my str $stop := $start;
my int $brac := nqp::index($brackets, $start);
if $brac >= 0 {
    # if it's a closing bracket, that's an error also
    if $brac % 2 {
        self.panic('Use of a closing delimiter for an opener is reserved');
    }

    # it's an opener, so get the closing bracket
    $stop := nqp::substr($brackets, $brac + 1, 1);

It finds the opening character in the $brackets string and takes the next one as its closing pair. As the most obvious brackets are located in the beginning of the string, the presence of the rest should not influence the performance at all.

༺ And that’s all for today. ༻

33. The cmp infix in Perl 6

In Perl 6, there is an infix operator called cmp. Despite its simple name and some connotations with its counter partner in Perl 5, its semantic is not trivial.

From the documentation, we read:

Generic, “smart” three-way comparator.

Compares strings with string semantics, numbers with number semantics, Pair objects first by key and then by value etc.

As we have access to the source codes, let us directly look inside and allow us to begin with strings, so go to src/core/Str.pm.

multi sub infix:<cmp>(Str:D \a, Str:D \b --> Order:D) {
    ORDER(nqp::cmp_s(nqp::unbox_s(a), nqp::unbox_s(b)))
}
multi sub infix:<cmp>(str $a, str $b --> Order:D) {
    ORDER(nqp::cmp_s($a, $b))
}

There is a method operating two objects of the Str type and another method for the lower-cased type str, which is a native type, which we skip for now; just look at its definition in src/core/natives.pm:

my native str is repr('P6str') is Str { }

For the Perl 6 type, the objects are first converted to native strings via nqp::unbox_s.

Then both methods delegate the comparison to the nqp::cmp_s function. It returns 1, 0, or -1, which is fine in NQP but not enough for Perl 6, where the result should be of the Order type—you can see the expected return type Order:D in the signature of the methods.

Go to src/core/Order.pm to see that the Order type is an enumeration with the above three values:

my enum Order (:Less(-1), :Same(0), :More(1));

In the same file, there is a function that acts as a constructor coercing an integer to Order:

sub ORDER(int $i) {
    nqp::iseq_i($i,0) ?? Same !! nqp::islt_i($i,0) ?? Less !! More
}

So, the result of cmp is either Same, or Less, or More.

We covered the hardest part already. The rest of the smartness of the cmp operator is due to multiple dispatching.

For example, for the two given integers, the following functions are triggered (also defined in src/core/Order.pm):

multi sub infix:<cmp>(Int:D \a, Int:D \b) {
    ORDER(nqp::cmp_I(nqp::decont(a), nqp::decont(b)))
}
multi sub infix:<cmp>(int $a, int $b) {
    ORDER(nqp::cmp_i($a, $b))
}

Here, there is not much difference from the string implementation. You may notice the different suffixes in the NQP methods.

Then, step by step, variety rises. For example, integers and rationals:

multi sub infix:<cmp>(Int:D \a, Rational:D \b) {
    a.isNaN || b.isNaN ?? a.Num cmp b.Num !! a <=> b
}
multi sub infix:<cmp>(Rational:D \a, Int:D \b) {
    a.isNaN || b.isNaN ?? a.Num cmp b.Num !! a <=> b
}

Again, the implementation is simple but of course it is different from what was needed for two integers or two strings.

It gets more complicated for Real numbers:

multi sub infix:<cmp>(Real:D \a, Real:D \b) {
       (nqp::istype(a, Rational) && nqp::isfalse(a.denominator))
    || (nqp::istype(b, Rational) && nqp::isfalse(b.denominator))
    ?? a.Bridge cmp b.Bridge
    !! a === -Inf || b === Inf
        ?? Less
        !! a === Inf || b === -Inf
            ?? More
            !! a.Bridge cmp b.Bridge
}

I leave parsing the algorithms to the reader as an exercise but would like to pay attention to the use of the Bridge method, which is a polymorphic method that we already saw as part of the Int type.

There are separate methods for comparing complex numbers, dates, lists, ranges, and even version numbers (which looks quite complicated, by the way, see it in src/core/Version.pm).

At the bottom (or at the top, as you define what is more and less important—base or children classes), there are a few methods that deal with Mu:

proto sub infix:<cmp>(Mu $, Mu $) is pure {*}
multi sub infix:<cmp>(\a, \b) {
    nqp::eqaddr(a,b)
      ?? Same
      !! a.Stringy cmp b.Stringy
}
multi sub infix:<cmp>(Real:D \a, \b) {
    a === -Inf
      ?? Less
      !! a === Inf
        ?? More
        !! a.Stringy cmp b.Stringy
}
multi sub infix:<cmp>(\a, Real:D \b) {
    b === Inf
      ?? Less
      !! b === -Inf
        ?? More
        !! a.Stringy cmp b.Stringy
}

That’s all for today, see you tomorrow!

32. It’s time for optimism

Today, we will not talk about the internals of Rakudo, NQP, or MoarVM; let me pause for a bit and talk about some random things related to Perl 6 and its future.

Efficiency

If you follow my blog, I have an update to the post about optimising MoarVM, and it appears that when making calculations with many big numbers (e.g., 3000 ** 4000), Perl 6 is ~15% faster than Perl 5.

Not to mention that the code is more expressive. Just compare visually.

Perl 5:

use v5.10;
use bigint;

for (my $i = 1; $i <= 5000; $i++) { # Ranges do not work with bigint
    say $i ** $i;
}

Perl 6:

for 1 .. 5000 -> $i {
    say $i ** $i;
}

Built-in support for big numbers is pretty awesome (something like the full Unicode support).

Antihype

If you follow the Perl groups on Facebook (or just visit reddit time to time), you might notice a new wave of semi-haters saying the Perl 6 stole the name, blocked the Perl 5 development, cut Perl jobs in general and even blocked some money flows (was there any?). I was very surprised that that is happening now. It was OK ten years ago, where it could be difficult to believe that Perl 6 would ever be a reality.

Today, although there are still many difficulties, it is so much closer to the language of a dream, and all you need is just work every day in a few areas: compiler improvements, educational activities, writing documentation. No one will bring a new shiny Perl 6 to you; just act and serve yourself.

FOSDEM ahead

In a few days, on 3-4 February, there’s a great chance to see the Perl 6 people at FOSDEM, the biggest European open source conference. There will be a booth at the venue (building K, I believe) and a full day of talks in the Perl Programming Languages devroom (notice the plural form).

Online materials

I can understand a bit why people complain about the lack of documentation. There are tons of outdated materials, which are, actually, not often pop up in Google. But for me, such complains rather indicate human resistance to learn a new language because it is so huge.

There are at least two resources which are quite structural and contain a lot:

Perl 6 Documentation — docs.perl6.org

Perl 6 Introduction — perl6intro.com

I do not suggest everyone reading sources of the compiler but it is also a great thing to do 🙂

Compiler at home

There’s nothing easier than installing Rakudo Star on your computer. It immediately gives you both a ready-to-use compiler and a set of some modules.

Don’t want downloading any software? Go online and try Perl 6 at glot.io/new/perl6.

Offline materials

In 2017, seven Perl 6 books were published. They cover many aspects and aim to different audiences. Just grab one, either on paper or as an e-book.

Perl 6 at a Glance — a brief introduction about new features of Perl 6. A good start if you don’t want to spend weeks on reading a book.

Perl 6 Deep Dive — a textbook covering almost everything you need to know about the language. For sure, much more than you need to start.

Using Perl 6 — an exercise book with solutions of 100 programming challenges. Just copy-and-paste, or, even better, think through and solve it better.

Think Perl 6 — a Perl 6-based tutorial on programming in general. With very few effort from your side, you get through the whole language and learn both programming and Perl 6.

Perl 6 Fundamentals — a book with practical examples of the Perl 6 code that you can start using already today. Not only you get the working code, but you also benefit from understanding Perl 6.

Parsing with Perl 6 Regexes and Grammars — a book that will guide you through another set of programs using regexes and grammars, real killer features of Perl 6.

Learning to Program in Perl 6 without leaving the command line — a small book that you can read in a couple of hours; it demonstrates many little things that can be very useful for daily use.

You can take a look at the books at FOSDEM, too.

Is Perl 6 Perl?

If you doubt, you are either a pessimist or overloaded with your day job, or just love to complain. Is Perl 6 Perl? Of course it is!

If someone tells you that to create a class method you need to write the following code, ignore it or say that this is an exaggeration:

method from-ingredients(::?CLASS:U $pizza: @ingredients)

It was a real comment on reddit. Even if you can do that formally, why on Earth you should do it that way instead of writing a simple and clean code.

Fortunately or not, we are all now at the point in time when Perl 6 has all chances to shine. Even if you cannot find a Perl 6 job, just continue your daily work in other languages. If you slowly start adding Perl 6 bits, you will help to bootstrap it. What is needed from us to Perl 6 today is just some additional love to this great language.

31. The opcode dispatching loop in MoarVM

Let me make some correction work following the previous post. I felt that something escaped from my view because changing the code did not bring any advances in speed.

Jonathan Worthington left a brilliant comment saying that I was trying to optimise the JIT compiler. Bang! Indeed, I thought that JIT should have had some influence, but I did not notice how I fell to the src/jit directory of MoarVM.

Today, we are looking into a different place where the opcodes are handled. I hope you will enjoy it as much as I did.

Computed goto

Before we continue, let us take a look at the so-called computed goto mechanism available in some of the C compilers, for example, in GCC.

The following program illustrates the simplest virtual machine. First, it generates a random program and stores it in the prog array. Then, the program is being read opcode by opcode, and the switch statement chooses the path that is executed in response to the opcode.

#include <stdio.h>
#include <time.h>
#include <stdlib.h>

const int size = 10;

int prog[size];

void gen_prog() {
    for (int c = 0; c != size; c++) {
        prog[c] = rand() % 3;
    }
}

void exec_prog() {
    for (int c = 0; c != size; c++) {
        switch(prog[c]) {
            case 0:
                printf("0\n");
                break;
            case 1:
                printf("1\n");
                break;
            case 2:
                printf("2\n");
                break;
        }
    }
}

int main() {
    srand(time(NULL));
    
    gen_prog();
    exec_prog();
}

There are three different opcodes and thus three different actions that prints 0, 1, or 2. The program contains ten random commands.

This first program is not optimised (we do not take into account the optimisation that a compiler can do for us), and it should check all the branches before it finds the one for the given opcode.

Our next step: move the actions to separate functions:

void op0() {
    printf("0\n");
}

void op1() {
    printf("1\n");
}

void op2() {
    printf("2\n");
}

Having that done, call the functions in switch/case:

void exec_prog() {
    for (int c = 0; c != 10; c++) {
        switch(prog[c]) {
            case 0:
                op0();
                break;
            case 1:
                op1();
                break;
            case 2:
                op2();
                break;
        }
    }
}

At this point, the dispatching code looks uniform and can be replaced with an array containing pointers to the opX functions:

void exec_prog() {
    void (*ops[])() = {op0, op1, op2};
 
    for (int c = 0; c != 10; c++) { 
        ops[prog[c]]();
    }
}

Now, it is really simple and transparent. Opcodes are indices of the array and directly lead to the desired function. Now, we can introduce computed goto. Here is an updated exec_prog function:

void exec_prog() {
    prog[size] = 3;
 
    void* ops[] = {&&op0, &&op1, &&op2, &&eop};

    int c = 0;
    goto *ops[prog[c++]];
    op0:
        printf("0\n");
        goto *ops[prog[c++]];
    op1:
        printf("1\n");
        goto *ops[prog[c++]];
    op2:
        printf("2\n");
        goto *ops[prog[c++]];
    eop:
        NULL;
}

What’s new here? First of all, there is no explicit loop. Also, all the functions are inlined as it was in the first program. The ops array contains now the addresses of the labels. They can be used as arguments of goto to jump directly to the place you need. From one hand, this is similar to function calls, from another, the code looks like the switch/case sequence and has no function calls.

The switch statement is also gone. On each step, the command pointer is incremented and thus the program jumps to different labels until the program is completely consumed. We intentionally added an end-of-program opcode (value = 3) so that it can stop the command loop.

Back to MoarVM

It’s time to return to the sources of MoarVM. In src/core/interp.c, you can find our friend, the switch statement that dispatches control:

DISPATCH(NEXT_OP) {
    OP(no_op):
        goto NEXT;
    OP(const_i8):
    OP(const_i16):
    OP(const_i32):
        MVM_exception_throw_adhoc(tc, "const_iX NYI");
    OP(const_i64):
        GET_REG(cur_op, 0).i64 = MVM_BC_get_I64(cur_op, 2);
        cur_op += 10;
        goto NEXT;
    OP(const_n32):
        MVM_exception_throw_adhoc(tc, "const_n32 NYI");

And it continues for other hundreds of opcodes. The capitalised names are macros depending on the MVM_CGOTO flag:

#define NEXT_OP (op = *(MVMuint16 *)(cur_op), cur_op += 2, op)

#if MVM_CGOTO
#define DISPATCH(op)
#define OP(name) OP_ ## name
#define NEXT *LABELS[NEXT_OP]
#else
#define DISPATCH(op) switch (op)
#define OP(name) case MVM_OP_ ## name
#define NEXT runloop
#endif

If the compiler is able to do computed gotos, these macros generate the following code:

{
    OP_no_op:
        goto *LABELS[(op = *(MVMuint16 *)(cur_op), cur_op += 2, op)];
    OP_const_i8:
    OP_const_i16:
    OP_const_i32:
        MVM_exception_throw_adhoc(tc, "const_iX NYI");
        cur_op += 10;
        goto *LABELS[(op = *(MVMuint16 *)(cur_op), cur_op += 2, op)];
    OP_const_n32:
        MVM_exception_throw_adhoc(tc, "const_n32 NYI");

For this case, the LABELS are loaded from src/core/oplabels.h:

static const void * const LABELS[] = {
    &&OP_no_op,
    &&OP_const_i8,
    &&OP_const_i16,
    &&OP_const_i32,
    &&OP_const_i64,
    &&OP_const_n32,
    &&OP_const_n64,
    &&OP_const_s,

When the compiler does not support it, macros help to generate a traditional switch/case sequence:

runloop: {
    . . .
 
    switch ((op = *(MVMuint16 *)(cur_op), cur_op += 2, op)) {
        case MVM_OP_no_op:
            goto runloop;
        case MVM_OP_const_i8:
        case MVM_OP_const_i16:
        case MVM_OP_onst_i32:
            MVM_exception_throw_adhoc(tc, "const_iX NYI");
        case MVM_OP_const_i64:
            GET_REG(cur_op, 0).i64 = MVM_BC_get_I64(cur_op, 2);
            cur_op += 10;
            goto runloop;
        case MVM_OP_const_n32:
            MVM_exception_throw_adhoc(tc, "const_n32 NYI");

And that’s all for today. It was a lot of C code but I hope it was quite useful to lurk into such deep level of the Perl 6 compiler system.

30. How I was optimising MoarVM

A Friday story for your pleasure. An attentive reader could notice yesterday that MoarVM is using many switch/case choices when it processes the bytecode.

In src/jit/graph.c, there is code which returns the address of the function corresponding to the given opcode:

static void * op_to_func(MVMThreadContext *tc, MVMint16 opcode) {
    switch(opcode) {
        case MVM_OP_checkarity: return MVM_args_checkarity;
        case MVM_OP_say: return MVM_string_say;
        case MVM_OP_print: return MVM_string_print;
        case MVM_OP_isnull: return MVM_is_null;

In the same file, another function:

static MVMint32 consume_invoke(MVMThreadContext *tc, MVMJitGraph *jg,
                               MVMSpeshIterator *iter, MVMSpeshIns *ins) {

. . .

    while ((ins = ins->next)) {
        switch(ins->info->opcode) {
            case MVM_OP_arg_i:
            case MVM_OP_arg_n:
            case MVM_OP_arg_s:
            case MVM_OP_arg_o:
            case MVM_OP_argconst_i:
            case MVM_OP_argconst_n:
            case MVM_OP_argconst_s:
                MVM_jit_log(tc, "Invoke arg: <%s>\n", ins->info->name);
                arg_ins[i++] = ins;
                break;
            case MVM_OP_invoke_v:
                return_type = MVM_RETURN_VOID;
                return_register = -1;
                code_register = ins->operands[0].reg.orig;
                spesh_cand = -1;
                is_fast = 0;
                goto checkargs;
            case MVM_OP_invoke_i:
                return_type = MVM_RETURN_INT;
                return_register = ins->operands[0].reg.orig;
                code_register = ins->operands[1].reg.orig;
                spesh_cand = -1;
                is_fast = 0;
                goto checkargs;

(By the way, notice the presence of goto in place of break.)

Similar things happen inside two more functions, consume_ins and comsume_reprop, in the same file. Each switch/case set contains hundreds of cases. There are more than 800 different opcodes currently, and many of them have their own branch in every switch/case.

It looks inefficient. Although the GCC compiler can optimise such sequences, what if we replace everything with arrays so that we can index it directly?

The easiest candidate for this operation is the op_to_func function: its only job is to return a pointer to the function in response to the given opcode value. So, write a small Perl script that transforms the C source:

my %func2opcode;
my %opcode2func;
for my $f (keys %f) {
    my $list_of_lists = $f{$f};

    my @opcodes;
    for my $list (@$list_of_lists) {
        for my $opcode (@$list) {
            push @opcodes, $opcode;
            $opcode2func{$opcode} = $f;
        }
    }

    $func2opcode{$f} = [@opcodes];
}

my @opcodes = ();
for my $opcode_name (keys %opcode2func) {
    my $opcode_value = $opcodes{$opcode_name};
    $opcodes[$opcode_value] = $opcode2func{$opcode_name};
}

say 'void* opcode2func[] = {';
for my $func (@opcodes) {
    $func //= 'NULL';
    say "\t$func,";
}
say '};';

The script generates an array, where the index of the element is the opcode, and the value is the pointer to a function:

void* opcode2func[] = {
	NULL,
	NULL,
	NULL,
        . . .
	MVM_frame_getdynlex,
	MVM_frame_binddynlex,
	NULL,
	NULL,
	MVM_args_set_result_int,
	MVM_args_set_result_num,
	MVM_args_set_result_str,
	MVM_args_set_result_obj,
	MVM_args_assert_void_return_ok,
        . . .

Now, our initial function is extremely short and clear:

static void * op_to_func(MVMThreadContext *tc, MVMint16 opcode) {
    return opcode2func[opcode];
}

(The thread context variable is used for error reporting, which I ignored here).

The idea behind this change is, among the potential gain of direct indexing, is the observation that the function is called for every opcode that the VM reads from the source. I did not take into account any performance improvements that JIT could add to it.

Anyway, the other switch/case places look less attractive to change. First, you should create many small functions for each branch, and after you have that done, you will lose performance by the need to call a function (push arguments on stack, etc.).

To test the changes, I ran the following program:

for 1..100 {my @a; @a[$_] = $_ ** $_ for 1..5000;}

Before the change, it took 41 seconds on my laptop. After the change, it became 44 🙂

Ah, wait, let’s in-place accessing the array in all 142 places where the function is called:

- jg_append_call_c(tc, jg, op_to_func(tc, op), 3, args, MVM_JIT_RV_PTR, dst);
+ jg_append_call_c(tc, jg, opcode2func[op], 3, args, MVM_JIT_RV_PTR, dst);

This change resulted in 41 seconds again, as it was with the original code.

OK, so a naïve ad hoc attempt to find the bottleneck by just looking at the source code failed. It would be a good idea to try profiling the code next time.

I could conclude here with the statement that MoarVM is already well optimised but I would still want 10x speeding up. The same program (with the corresponding tiny syntax changes) takes only 4 seconds on the same computer when run by Perl 5.

Update 1

I slipped down to the JIT source in this blog post, so check out my next post too.

Update 2

In Perl 5, with the bigint module, you cannot use the range 1..5000, so after some number, all $_ ** $_ became a bare ‘inf’. Here’s the updated program that you may want to use for testing both Perl 5 and Perl 6. To prevent any misleads, it prints the results so that you can check it and compare the two versions.

Perl 5:

use v5.10;
use bigint;

for (my $i = 1; $i <= 5000; $i++) {
    say $i ** $i;
}

Perl 6:

for 1 .. 5000 -> $i {
    say $i ** $i;
}

On my laptop, the programs with maximum of 1000 took 3.9 and 4.1 seconds (Perl 5 and Perl 6), and for 5000 values, it took 12m45s for Perl 5 and 11m3s for Perl 6. Notice that these times also include disk I/O.

29. Exploring the Int type in Perl 6, part 2

Today, the journey to the internals of Int continues and first let’s make a deep dive into one on of the fascinating methods, is-prime. If you never used it, this is a great method that implements the logic, which is usually not built-in in a programming language. It is also great because primality is an attribute of an integer and you get it for free for all integer numbers in Perl 6.

So, this is how the method is defined in src/core/Int.pm:

method is-prime(--> Bool:D) {
    nqp::p6bool(nqp::isprime_I(self,100))
}

As in many other cases, a Perl module only passes execution to some underlying NQP function. Let us try to track it down. The constant 100 in the call is a number of tests, which I will not touch here: you may get the details of the algorithm on Wikipedia, for example.

In the NQP sources, you are asked (src/vm/moar/QAST/QASTOperationsMAST.nqp) to go one level further, to MoarVM:

QAST::MASTOperations.add_core_moarop_mapping('isprime_I', 'isprime_I');

For the JVM backend, the implementation is a bit easier to reach (src/vm/jvm/runtime/org/perl6/nqp/runtime/Ops.java):

public static long isprime_I(SixModelObject a, long certainty, ThreadContext tc) {
    BigInteger bi = getBI(tc, a);
    if (bi.compareTo(BigInteger.valueOf(1)) <= 0) {
        return 0;
    }
    return bi.isProbablePrime((int)certainty) ? 1 : 0;
}

It is a surprise that detecting if the number is prime is not 100% precise (the more accuracy you want, the slower it works, and ideally the number of rounds should depend on the tested number, while I hope the chosen number of rounds should give more than enough that you ever need).

For completeness, look briefly at the JavaScript function (src/vm/js/nqp-runtime/bignum.js):

op.isprime_I = function(n) {
     return intishBool(sslBignum(getBI(n).toString()).probPrime(50));
};

For some reason, the number of tests is only 50 here.

Nevertheless, MoarVM is my primary goal, so let us try exploring it further, even if it is not that easy.

There are a few mappings and switches that select between different operations, such as (generated file src/core/ops.h):

#define MVM_OP_lcm_I 447
#define MVM_OP_expmod_I 448
#define MVM_OP_isprime_I 449
#define MVM_OP_rand_I 450
#define MVM_OP_coerce_In 451

An interesting fact about this generated list is that it is prepared by the script tools/update_ops.p6, which is a script in Perl 6:

#!/usr/bin/env perl6
# This script processes the op list into a C header file that contains
# info about the opcodes.

The numbers are, actually, opcodes. And then we finally get to the place where it looks like a function call is prepared (src/jit/graph.c):

case MVM_OP_isprime_I: {
    MVMint16 dst = ins->operands[0].reg.orig;
    MVMint32 invocant = ins->operands[1].reg.orig;
    MVMint32 rounds = ins->operands[2].reg.orig;
    MVMJitCallArg args[] = { { MVM_JIT_INTERP_VAR, MVM_JIT_INTERP_TC },
                             { MVM_JIT_REG_VAL, invocant },
                             { MVM_JIT_REG_VAL, rounds } };
    jg_append_call_c(tc, jg, op_to_func(tc, op), 3, args, MVM_JIT_RV_INT, dst);
    break;
}

In the list of arguments, you can see that the number of tests (rounds) is also passed. The op_to_func function takes the thread context and the opcode. The function is defined in the same file and is a set of another switch/case sequence:

static void * op_to_func(MVMThreadContext *tc, MVMint16 opcode) {
    switch(opcode) {
        case MVM_OP_checkarity: return MVM_args_checkarity;
        case MVM_OP_say: return MVM_string_say;
        case MVM_OP_print: return MVM_string_print;
        . . .
        case MVM_OP_isprime_I: return MVM_bigint_is_prime;

It returns a pointer to a function (void *) that should finally do the job, so let’s look for MVM_bigint_is_prime. It is found in src/math/bigintops.c in the MoarVM repository:

MVMint64 MVM_bigint_is_prime(MVMThreadContext *tc, MVMObject *a, MVMint64 b) {
    /* mp_prime_is_prime returns True for 1, and I think
     * it's worth special-casing this particular number :-)
     */
     MVMP6bigintBody *ba = get_bigint_body(tc, a);

    if (MVM_BIGINT_IS_BIG(ba) || ba->u.smallint.value != 1) {
        mp_int *tmp[1] = { NULL };
        mp_int *ia = force_bigint(ba, tmp);
        if (mp_cmp_d(ia, 1) == MP_EQ) {
            clear_temp_bigints(tmp, 1);
            return 0;
        }
        else {
            int result;
            mp_prime_is_prime(ia, b, &result);
            clear_temp_bigints(tmp, 1);
            return result;
        }
    } else {
        /* we only reach this if we have a smallint that's equal to 1.
         * which we define as not-prime. */
        return 0;
    }
}

This function passes the job further to a library function mp_prime_is_prime.

If you followed the adventures of the method internals in this blog post, you might be quite exhausted. So let me stop for today because there’s a lot more to say about today’s topic.

28. Exploring the Int type in Perl 6, part 1

Actually, we already started looking at the internals of the Int data type two days ago, but today we’ll start from the very beginning.

So, the Int data type. It is a great data type for practical programming and it is also widely used in Rakudo itself. For example, the Rational object is an object that keeps two Int values (it may be an Int plus uint one day but let us not focus on that today).

On a big scale, an Int is a Real:

my class Int does Real {
    . . .
}

At this point, I was always confused. I am not sure if I have this paradox only in my mind, but I always treated integers as being less rich data type than real numbers. On the other side, all properties of integers also exist for real numbers and it would be strange not to inherit them. (Well, as a side story, the object-oriented terminology is extremely vague if you say subclass and superclass instead of child and parent.)

The actual value is contained in the private attribute $!value, which is defined somewhere on a deeper level but is directly used in src/core/Int.pm. The value is set in one of the constructors:

proto method new(|) {*}
multi method new( \value) { self.new: value.Int }
multi method new(int \value) {
    # rebox the value, so we get rid of any potential mixins
    nqp::fromI_I(nqp::decont(value), self)
}
multi method new(Int:D \value = 0) {
    # rebox the value, so we get rid of any potential mixins
    nqp::fromI_I(nqp::decont(value), self)
}

Then, a bunch of simple but useful methods for type conversion follows.

multi method Bool(Int:D:) {
    nqp::p6bool(nqp::bool_I(self));
}

method Int() { self }

multi method Str(Int:D:) {
    nqp::p6box_s(nqp::tostr_I(self));
}

method Num(Int:D:) {
    nqp::p6box_n(nqp::tonum_I(self));
}

method Rat(Int:D: $?) {
    Rat.new(self, 1);
}
method FatRat(Int:D: $?) {
    FatRat.new(self, 1);
}

All these methods operate with the only Int object, so you may be confused by the fact that most of the methods still take an argument. The colon after the type means that this is not a regular function parameter but an invocant, i.e. the object on which you call the given method.

The following test program should clarify the syntax:

class X {
    has $.value;

    method a() {
        say $!value
    }
    method b(X $x) {
        say $x.value
    }
    method c(X $x:) {
        say $x.value
    }
}

my X $x = X.new(value => 42);
my X $y = X.new(value => 43);

$x.a();   # 42
$x.b($y); # 43
$x.c();   # 42

The three methods print the value of the only attribute. In the first case, the method has no parameters and $!value refers to the attribute of the object in hand. In the second case, the argument of the method is a different variable, which is not connected with the object on which the method is called. Finally, the third method demonstrates how you introduce an invocant in the method signature. This method behaves exactly like the first one.

So, return to the Int class. There are no questions about the logic of the methods. Some of them are implemented via NQP functions. The most charming method in the series is Int(), which just returns self. (Homework: re-write the method using an invocant in the signature.)

Moving further.

method Bridge(Int:D:) {
    nqp::p6box_n(nqp::tonum_I(self));
}

This is another very interesting method. If you grep for its name, you will see that the method is used as a polymorphic method:

src/core/Real.pm: method sqrt() { self.Bridge.sqrt }
src/core/Real.pm: method rand() { self.Bridge.rand }
src/core/Real.pm: method sin()  { self.Bridge.sin }
src/core/Real.pm: method asin() { self.Bridge.asin }
src/core/Real.pm: method cos()  { self.Bridge.cos }
src/core/Real.pm: method acos() { self.Bridge.acos }
src/core/Real.pm: method tan()  { self.Bridge.tan }
src/core/Real.pm: method atan() { self.Bridge.atan }

It is implemented in other classes, too. For example, in the Num class, which is also a descendant of Real:

my class Num does Real { 
    method Bridge(Num:D:) { self }
}

OK, enough of the easy stuff for today 🙂 Let us dig deeper tomorrow.