The State of LFE

A Lisp Flavoured Smörgåsbord

Erlang User Conference, Stockholm 2014

Duncan McGreggor

Overview

  • Introduction
  • New Developments
  • A Tour of LFE
  • Special Projects
  • Language Interop
  • Wrap-up

Introduction

Why Lisp: Preview

  • A Historical Glance
  • Why not 'Just Erlang'?
  • The Best of Two Worlds

Why Lisp?

Why Lisp?

Do we really want to code in something so old?

Why Lisp?

Do we really want to code in something so old?

Fortunately, we don't have to.

Why Lisp?

  • A lot has changed since 1958... even for Lisp: it now has even more to offer.
  • It's a programmable programming language.
  • As such, it's an excellent language for exploratory programming.
  • Many are drawn to the beauty of the near syntaxlessness of the language.
  • Due to it's venerable age, there is an enormous corpus of code to draw from.

Why Lisp?

Most importantly, however, it's because the universe is cons's all the way down:

Why in Erlang?

The obvious answer:

One now has easy access to the same idioms that have made Lisp so powerful for so long,

without having to give up OTP.

Why in Erlang?

A terrifyingly powerful combination:

  1. Lisp's homoiconicity and macro system, coupled with
  2. Erlang's fault-tolerant distributed systems capabilities.


It is the perfect language lab for the multi-core world.

Why in Erlang?

Couldn't you just have left Erlang well enough alone?

Yes.

But Lisp is such a simple syntax; it was bound to happen, sooner or later. In fact ...

From 2007 to 2008, three major languages/VMs got a Lisp:

  • JVM got Clojure.
  • Haskell got Liskell.
  • Erlang got LFE.

Why in Erlang?

    But there are subtler and more important considerations:

    • Diversity brings new ways of thinking and new discoveries.
    • Increased options for the Erlang VM bring more programmers to the community. (Witness Elixir, LFE, Joxa, Erlog, luerl, Concurrent Schemer.)
    • Living by these principles ensures the wider success of the Erlang VM.
    • More people benefit from the excellence that is Erlang.

New Developments

New Developments: Preview

  • Released Features
  • New Example Code
  • New Documentation
  • Current Work
  • LFE Roadmap
  • Community Projects

Released Features

  • Maps have landed in LFE.
  • A new (run ...) REPL command for running LFE shell commands.
  • It is now possible to define functions and macros in the REPL.
  • Strings no longer need to be quoted.
  • (mod:func ...) syntax has been added.

Released Features (cont'd)

  • LFE now has Common Lisp macros (prog1 ...) and (prog2 ...).
  • New (fields-name) macro for easier usage of Mnesia from LFE.
  • #| ... |# Common Lisp-style multiline comments now supported.
  • Improved handling of .hrl files.

New Example Code

  • Ring benchmark.
  • Joe Armstrong's favorite program.
  • Another message-passing example.
  • Async HTTP example.
  • Guessing game example :-)

New Sites

  • LFE has a new main site+landing page: http://lfe.io
  • The old language site has been converted to a docs-only site and lives at a new subdomain: http://docs.lfe.io
  • There is a new updates-oriented micro-blogging site for LFE: http://blog.lfe.io

New Documentation

  • The Erlang "Programming Rules and Conventions" was ported to LFE.
  • The Common Lisp Style Guide was ported to LFE.
  • The Quick Start Guide was rewritten.
  • A new "Counter Application" tutorial on docs.lfe.io.
  • "Setting up a Development Environment" section was added to the User Guide.

New Documentation In-Progress

  • An LFE port of "Land of Lisp" is under way.
  • Mneisa tutorial on docs.lfe.io.
  • Java Interop Section of the User Guide.

Current Work

  • Improved scripting options for LFE (for those wanting to use LFE as a systems-programming language, i.e., eat Go-lang's lunch).
  • First candidate module for LFE stdlib has an open pull request.
  • Several tasks to improve the use of Erjang and Java from LFE.

LFE Roadmap

  • Spec and type support, pending either the OTP team's work on .beam files or some interesting hacks by Robert :-)
  • Submitting more candidates for inclusion in LFE stdlib.
  • Improvements to LFE internals.
  • Reader macros.
  • APIs for data type functions.
  • Defining the release process.
  • Preparing for an LFE 1.0 release!

LFE Docs Roadmap

  • Creating an "Erlang Idioms in LFE" Guide.
  • Starting an LFE Cookbook.
  • Completing the Java Interop documentation.
  • Finishing the LFE User Guide.

Community Projects of Note

  • Monads for LFE (calrissian; more later)
  • LFE and Elli
  • LFE and YAWS
  • Reveal.js for LFE and YAWS (lfe-reveal-js)
  • HTML as S-expressions in LFE (exemplar)
  • lfetool (more later)
  • LFE on Erjang (more later)

Quick Tour of LFE

Quick Tour of LFE: Preview

  • Origins of LFE
  • LFE Basics
  • Data Types
  • Erlang Idioms in LFE
  • Sample Code

Lisp Flavored Erlang

Who wants a cup?

Beginnings

LFE was born on Robert Virding's Dell XPS in 2007.

Initially released in March of 2008.

The intent was to modify the Lisp to suit the Erlang VM.

Motivation

  • Was curious see how a Lisp would run on and integrate with Erlang.
  • Wanted to explore generating Core Erlang, plugging it into the backend of the Erlang compiler.
  • Likes implementing languages.
  • Thought it would be a fun problem to solve, as a solution would be comprised of many different parts and theproblem space was quite open-ended.

LFE Basics

LFE is built on Core Erlang and supports the usual good stuff:

  • No mutable data.
  • Pattern matching and guards.
  • Same types as Erlang.
  • Erlang modules and functions.
  • Compiler, interpreter, and REPL.
  • Seamless interoperability with Erlang/OTP.

Data Types

Data Types

Numbers


      > (* 2 (+ 1 2 3 4 5 6))
      42
      > (/ 36 7)
      5.142857142857143
      > (+ #b101010 #o52 #x2a #36r16)
      168
      > (integer_to_list (+ #b1001 #b100 #b10) 2)
      "1111"
      > (div 11 2)
      5
      > (rem 11 2)
      1
      

Data Types

Atoms and Strings


      > 'Vogon
      Vogon
      > '|and now with hyperspace bypasses|
      |and now with hyperspace bypasses|
      > "Don't Panic."
      "Don't Panic."
      > (list 68 111 110 39 116 32 80 97 110 105 99 46)
      "Don't Panic."
      

Data Types

Lists


      > (list 1 3 9 27)
      (1 3 9 27)
      > (length '(1 2 3 4 5 6 7))
      7
      > (lists:append '(1 2) '(3 4))
      (1 2 3 4)
      > (car '(1 2 3 4 5 6))
      1
      > (cdr '(1 2 3 4 5 6))
      (2 3 4 5 6)
      > (cadr '(1 2 3 4 5 6))
      2
      > (cddr '(1 2 3 4 5 6))
      (3 4 5 6)
      > (cons '(1 2 3) '(4 5 6))
      ((1 2 3) 4 5 6)
      

Data Types

Tuples


      > (tuple)
      #()
      > (set data (tuple 'odds "5 to 1 against"))
      #(odds "5 to 1 against")
      > (tuple_size data)
      2
      > (element 2 data)
      "5 to 1 against"
      

Data Types

Records


      > (defrecord person
          name
          address
          age)
      ()
      > (set ford
          (make-person name "Ford Prefect"
                       address "Betelgeuse Seven"
                       age 234))
      #(person "Ford Prefect" "Betelgeuse Seven" 234)
      > (person-name ford)
      "Ford Prefect"
      > (set ford
          (set-person-age ford 244))
      #(person "Ford Prefect" "Betelgeuse Seven" 244)
      > (person-age ford)
      244
      

Data Types

Maps


    > (set data (map 'ship "Heart of Gold"
                     'hitchhiker "Arthur Dent"))
    #M(hitchhiker "Arthur Dent"
       ship "Heart of Gold")
    > (set data (set-map data 'captain "Zaphod"))
    #M(captain "Zaphod"
       hitchhiker "Arthur Dent"
       ship "Heart of Gold")
    > (set data (update-map data 'hitchhiker "Ford Prefect"))
    #M(captain "Zaphod"
       hitchhiker "Ford Prefect"
       ship "Heart of Gold")
      

Data Types

Maps

Bonus: Common Lisp-based Map Macros


      > (mref 'ship data)
      "Heart of Gold"
      > (set data (mupd 'ship "Golgafrinchan Ark Fleet Ship B"
                        'captain "Captain"
                        data))
      #M(captain "Captain"
         hitchhiker "Arthur Dent"
         ship "Golgafrinchan Ark Fleet Ship B")
      > (set data (mset 'art-director "Mella" data))
      #M(art-director "Mella"
         captain "Captain"
         hitchhiker "Arthur Dent"
         ship "Golgafrinchan Ark Fleet Ship B")
      

Erlang Idioms in LFE

Erlang Idioms in LFE

Pattern Matching

Erlang


      1> {Len,Status,Msg} = {8,ok,"Trillian"}.
      {8,ok,"Trillian"}
      2> Msg.
      "Trillian"
      

LFE


      > (set (tuple len status msg) #(8 ok "Trillian"))
      #(8 ok "Trillian")
      > msg
      "Trillian"
      

Erlang Idioms in LFE

List Comprehensions

Erlang


      1> trunc(math:pow(3,X)) || X <- [0,1,2,3]].
      [1,3,9,27]
      

LFE


      > (list-comp
          ((<- x '(0 1 2 3)))
          (trunc (math:pow 3 x)))
      (1 3 9 27)
      

      > (lists:map
          (lambda (x) (trunc (math:pow 3 x)))
          '(0 1 2 3))
      (1 3 9 27)
      

Erlang Idioms in LFE

Guards

Erlang


      right_number(X) when X == 42; X == 276709 ->
      true;
      right_number(_) ->
      false.
      

LFE


      (defun right-number?
        ((x) (when (orelse (== x 42) (== x 276709)))
          'true)
        ((_) 'false))
      

Erlang Idioms in LFE

cons'ing in Function Heads

Erlang


      sum(L) -> sum(L,0).
      sum([], Total) -> Total;
      sum([H|T], Total) -> sum(T, H+Total).
      

LFE


      (defun sum (l) (sum l 0))
      (defun sum
        (('() total) total)
        (((cons h t) total) (sum t (+ h total))))
      

Erlang Idioms in LFE

Matching Records in Function Heads

Erlang


  handle_info(ping, #state {remote_pid = undefined} = S) ->
      gen_server:cast(self(), ping),
      {noreply, S};
  handle_info(ping, S) ->
      {noreply, S};
      

LFE


  (defun handle_info
    (('ping (= (match-state remote-pid 'undefined) state))
      (gen_server:cast (self) 'ping)
      `#(noreply ,state))
    (('ping state)
     `#(noreply ,state)))
      

Erlang Idioms in LFE

Receiving Messages

Erlang


      universal_server() ->
          receive
              {become, Func} ->
                  Func()
          end.
      

LFE


      (defun universal-server ()
        (receive
          ((tuple 'become func)
           (funcall func))))
      

Sample Code

Sample Code

Dot Product Function


      (defun dot-product (a b)
        (lists:foldl #'+/2 0
          (lists:zipwith #'*/2 a b)))
      

Sample Code

Tail-Recursive Factorial Function


      (defun factorial (n)
        (factorial n 1))

      (defun factorial
        ((0 acc) acc)
        ((n acc) (when (> n 0))
          (factorial (- n 1) (* n acc))))
      

Sample Code

Partial Function Application in the REPL


      > (defun partial (f arg)
          (lambda (x)
            (apply f
              (list arg x))))
      partial
      > (set applied (partial #'+/2 10))
      #Fun<lfe_eval.14.16114431>
      > (funcall applied 10)
      20
      > (funcall applied 100)
      110
      > (funcall applied -20)
      -10
      

Sample Code

Ring Benchmark Module


      (defmodule ring
        (export
          (main 1)
          (roundtrip 2)))

      (defun main (args)
        (apply
          #'start-ring/2
          (lists:map #'list_to_integer/1 args)))

      (defun start-ring (process-count traversal-count)
        (let ((batch (make-processes process-count
                                     traversal-count)))
          (! batch traversal-count)
          (roundtrip 1 batch)))
      

Sample Code

Ring Benchmark Module (cont'd)


      (defun make-processes (process-count traversal-count)
        (lists:foldl
          #'make-process/2
          (self)
          (lists:seq process-count 2 -1)))

      (defun make-process (id pid)
        (spawn 'ring 'roundtrip (list id pid)))

      (defun roundtrip (id pid)
        (receive
          (1
            (io:fwrite "Result: ~b~n" (list id))
            (erlang:halt))
          (data
            (! pid (- data 1))
            (roundtrip id pid))))
      

Special Projects

Special Projects: Preview

  • Calrissian
  • LFE + Elli Web Framework
  • LFE + YAWS
  • lfetool
  • jlfe

Monads

Monad

Calrissian

  • Brand new.
  • Inspired by erlando project (RabbitMQ).
  • Only supports maybe monad so far.
  • https://github.com/correl/calrissian

Calrissian


      (include-lib "deps/calrissian/include/monads.lfe")

      (defun dostuff ()
        (do-m error-monad
           (input <- (fetch-input))
           (parsed <- (parse-input input))
           (store-data parsed)))
        

Calrissian


      (defun dostuff ()
        (case (fetch-input)
          ((tuple 'error reason)
           (tuple 'error reason))
          ((tuple 'ok input)
           (case (parse-input input)
                 ((tuple 'error reason)
                  (tuple 'error reason))
                 ((tuple 'ok parsed)
                  (store-data parsed))))))
        

LFE + Elli Web Framework

  • Run inside your Erlang application.
  • Expose HTTP APIs.
  • Do it from LFE!
  • https://github.com/knutin/elli
  • https://github.com/lfe/lfe-elli-example

LFE + YAWS

  • YAWS: still cranking out the hits.
  • lfetool currently generates 2 kinds of YAWS projects.
  • Dead-simple with LFE: just define an out function in a module.
  • Even better: use Exemplar to generate HTML S-expressions.
  • Great for REST services.

LFE + YAWS + Bootstrap

via lfetool

LFE + YAWS + Exemplar

This is the index function for this presentation:


      (defun build-index (deck-func arg-data)
        (list
          (!doctype 'html)
          (html '(lang "en")
            (list
              (build-head arg-data)
              (body
                (list
                  (funcall deck-func arg-data)
                  (reveal-js-content:build-javascript
                    arg-data)))))))
        

LFE + YAWS

  • https://github.com/klacke/yaws
  • https://github.com/lfe/lfe-reveal-js
  • https://github.com/lfe/yaws-rest-starter

lfetool

lfetool

  • This one's a beast.
  • Not enough time to cover everything.
  • It's basically your one-stop-shop for LFE development.

lfetool

Goodies include:

  • Installing kerl, erlang, rebar, relx, erjang, and others.
  • Building skeleton libraries, OTP apps, YAWS projects, scripts, etc.
  • Colored output test-runner.
  • Bash auto-completion for feature-discovery.

lfetool

lfetool

  • Getting regular community contributions.
  • New features landing almost every week.
  • https://github.com/lfe/lfetool

jlfe

jlfe

  • Super experimental.
  • Working on Java call syntax from LFE.
  • Lots of macro work.
  • TONS of fun!

jlfe

Sample calls:


      > (.Double 42)
      42.0
      > (.Math:sin 0.5)
      0.479425538604203
      > (.Math:PI)
      3.141592653589793
      >
      > (.java.math.BigDecimal:ROUND_CEILING)
      2
      > (jlfe-types:value-of bool)
      true
      

jlfe

  • Looking at Clojure a great deal.
  • Currently working on solution for instance methods.
  • Also sorting out evaluated expressions in calls.
  • https://github.com/trifork/erjang
  • https://github.com/lfe/jlfe

Language Interop

Language Interop: Preview

  • LFE & Erlang
  • LFE & Elixir
  • LFE & Java/Erjang
  • LFE & Clojure

The Erlang VM

We're seeing a renaissance in Erlang these days.

Lots of languages on the Erlang VM.

Several ways to interact with Erlang: ports, drivers, JInterface.

What does this look like from the LFE perspective?

LFE & Erlang

LFE & Erlang

LFE & Erlang

  • This one's a no-brainer.
  • Included for completeness :-)
  • Define modules & functions.
  • Make calls to Erlang functions from LFE functions.

LFE & Erlang

Classic LFE call syntax:


        (: lists map ... )
        

New call syntax:


        (lists:map ... )
        

LFE & Elixir

LFE & Elixir


  • Almost as easy as using Erlang from LFE.
  • Add Elixir to the rebar deps of your LFE project.

LFE & Elixir

Build it and start LFE:


        $ make get-deps && make compile
        $ lfetool repl lfe \
            -pa ./deps/elixir/lib/elixir/ebin/
        

LFE & Elixir

Sample calls:


      > (Elixir.IO:puts "hello\nworld")
      hello
      world
      ok
      > (set stream (Elixir.Stream:cycle '(1 2 3)))
      #Fun<Elixir.Stream.15.23582569>
      > (Elixir.Enum:take stream 10)
      (1 2 3 1 2 3 1 2 3 1)
      > (Elixir.Enum:take stream 20)
      (1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2)
      

LFE & Elixir

Things to try

  • Working with Elixir's built-ins.
  • Elixir Protocols.
  • Exploring Elixir's Unicode support.

LFE & Java/Erjang

LFE & Java/Erjang

  • jlfe - an experimental project providing syntax for LFE/Erjang.
  • Right now, it patches a local copy of LFE.
  • The patch adds support for parsing calls that start with . (dot).
  • Wrapper functions for providing convenient syntax.
  • Boiler plate java.lfe module creation.
  • Utility functions for working with Java objects.

LFE & Java/Erjang

Outstanding Issues

  • Constructors with args in the jlfe syntax isn't supported yet.
  • jlfe doesn't currently support a special syntax for instance methods.
  • LFE has a problem handling the Java String object.
  • jlfe provides a work-around for this, but it isn't pretty.
  • Quitting with ^D results in ugly traceback.

LFE & Java/Erjang

Comparison: Static Method Calls

LFE/Erjang


      > (java:call 'java.lang.String 'getName '() '())
      "java.lang.String"
      > (call 'java.lang.Math 'sin 0.5)
      0.479425538604203
      

jlfe


      > (.String:getName)
      "java.lang.String"
      > (.Math:sin 0.5)
      0.479425538604203
      

LFE & Java/Erjang

Comparison: Static Field Variable Access

LFE/Erjang


    > (java:get_static 'java.lang.Math 'PI)
    3.141592653589793
    > (java:get_static 'java.math.BigDecimal 'ROUND_CEILING)
    2
      

jlfe


    > (.Math:PI)
    3.141592653589793
    >
    > (.java.math.BigDecimal:ROUND_CEILING)
    2
      

LFE & Java/Erjang

Comparison: Constructors

LFE/Erjang


  > (set st (call 'java.lang.String 'new))
  ()
  > (set hm (call 'java.util.HashMap 'new))
  ()
  > (set se (java.util.AbstractMap$SimpleEntry 'new "a" "b"))
  #B()
      

jlfe


  > (set st (.String))
  ()
  > (set hm (.java.util.HashMap))
  ()
  > (set se (.java.util.AbstractMap$SimpleEntry "a" "b"))
  #B()
      

LFE & Java/Erjang

Instance Members

LFE/Erjang


      > (call se 'getKey)
      '"a"
      > (call se 'getValue)
      '"b"
      

No jlfe syntax yet.

LFE & Java/Erjang

Values of Java Types


      > (set bool (.Boolean true))
      #B()
      > (set flt (.Float 42))
      #B()
      > (set bigdec (.java.math.BigDecimal 42))
      #B()
      

Convenience functions:


      > (jlfe-types:value-of bool)
      true
      > (jlfe-types:value-of flt)
      42.0
      > (jlfe-types:value-of bigdec)
      42.0
      

LFE & Java/Erjang

Areas to Explore

  • Discovering a nice macro for unifying object access syntax.
  • Coming up with better ways to handle arguments in constructors.
  • Building Erjang with additional/3rd party .jar files.
  • Working with Robert to find a way to make jlfe pluggable with LFE.

LFE & Clojure

LFE & Clojure

  • Create 2 nodes: an LFE one, and a Clojure one.
  • LFE/OTP starts up Clojure java process
  • Uses JInterface in Clojure to handle Erlang data.
  • Ported from Erlang and Clojre work originally by Maxim Molchanov.

Erlang & Clojure

Preparations


      $ git clone https://github.com/lfe/lfecljapp.git
      $ make compile
      $ make repl
      

Erlang & Clojure

Starting


  (lfenode@cahwsx01)> (lfeclj-app:start)
  ok
  (lfenode@cahwsx01)> lfecljapp.lfe:187:<0.44.0>:...
  INFO: Starting clojure app with cmd "java -Dnode=... "
  (lfenode@cahwsx01)> lfecljapp.lfe:118:<0.42.0>:...
  INFO: Connection to java node established, pid: <6709.1.0>
      

Erlang & Clojure

Trying It Out

Send a message:


  (lfenode@cahwsx01)> (lfecljapp:ping "clj-node@cahwsx01"
                                      "clj-mbox")
  #(ping <0.32.0>)
      

Check for a response:


      (lfenode@cahwsx01)> (c:flush)
      Shell got {pong,<6709.1.0>}
      ok
      

LFE & Clojure

What's Happening

  • We've got an LFE/OTP app,
  • That's calling out to the shell to start up a Clojure/JInterface app.
  • Messages are sent by the LFE node to the Clojure node,
  • And then processed by Clojure code,
  • Finally sending a response back to the LFE node (in this case, the REPL).

LFE & Clojure

Future Work

  • Run number-crunching apps as Erlang nodes.
  • Do machine learning!
  • Try it with Erjang and jlfe instead of Clojure ...

Wrap Up

Topics Covered

  • Introduction
  • New Developments
  • A Tour of LFE
  • Special Projects
  • Language Interop

Wrap Up

Highlights

  • LFE gives us the power of a Lisp language lab.
  • Erlang gives us the power of a means for building reliable, distributed systems.
  • With just these two, unimagined new possibilies are at our fingertips.
  • But it goes beyond that.

Wrap Up

Highlights

  • We have access to all the other works in the BEAM community.
  • We can easily (mostly) translate decades of Lisp code to LFE.
  • We have multiple ways of working with the endless options provided by the JVM world.
  • All of this from the best Actor model language on the market :-)

Wrap Up

The power is mind-blowing.

Wrap Up

...Great power involves great responsibility.
- Franklin D. Roosevelt

Computing has its beginnings in the darkness of
war:

ENIAC was built with the purpose of calculating artillery trajectories.

After von Neumann's involvement, it was used on the Manhattan Project.

Wrap Up

... With great power there must also come -- great responsibility!
- Amazing Fantasy #15

In contrast to that heaviness, Lisp was created to explore intelligence.

Erlang was created in a playful manner to explore reliable systems (train sets!).

LFE was created in the same, playful spirit upheld at Ericsson's CSL.

Wrap Up

Un grand pouvoir implique une grande responsabilité
- François-Marie Arouet (Voltaire)?

As socially responsible hackers, we need to remember the power of our tools.

It's easy to forget in the excitement of the moment.

Or in the frustrated boredom of a job.

Wrap Up

Le plus digne du pouvoir est celui qui en connaît la responsabilité.
- Louis François Sosthène de La Rochefoucauld-Doudeauville

We've got OTP and Lisp macros ...

Screw surveillance and digital bombs.

Let's make mind-blowing software that helps people.

Wrap Up


"I think we should care because we write programs for the whole population, and we should have a fair representation of that population creating [these programs]. The technology we produce shapes our world and our future , and we want it to be shaped in a way that reflects the interests of all, not just an empowered few."

- Katie Miller, EUC 2014

Thanks!

Contact

LFE Info


@ErlangLisp on Twitter

#erlang-lisp on Freenode IRC

lisp-flavoured-erlang@googlegroups.com

http://lfe.github.io/

Contact

My Info


@oubiwann on Twitter, Github, Freenode, etc

oubiwann@cogitat.io

http://cogitat.io/

Contact

AdRoll


We're hiring!

@AdRoll & @AdRollEng on Twitter

SemanticSugar on Github

duncan@adroll.com

http://adroll.com/

References

Projects

References

Projects

References

Projects

References

Papers, books, etc.

References

Quotes