Spoofax 2.0.0 Migration Guide

This migration guide describes the differences between Spoofax 1.5 and 2.0 and describes the steps to convert a Spoofax 1.5 project to Spoofax 2.0 project.

To gather the required knowledge for migrating a language project, go through the documentation in the following order:

  1. Language Development Getting Started, to install and get familiar with Spoofax 2.0.
  2. Spoofax 2.0 Release Notes, for a general overview of the changes in Spoofax 2.0.
  3. This document, for the concrete differences and steps to convert your Spoofax project.

Differences

Concepts

Spoofax 2.0 introduces several new concepts and terminology.

A language specification is the specification of a language using meta-languages.

A language specification project specifies a language component. When the specification is compiled, the result is a component which can be loaded into Spoofax. A language component has specifications and implementations for parts of a language, such as its parser, pretty-printer, analysis, transformations, editor services, etc.

A component contributes these parts to a language implementation. Multiple components can contribute to the same language implementation, and components can contribute to multiple language implementations. In the most simple case, a single component contributes all parts of the language to a single implementation.

Language components can depend on other language components to depend on parts of a language. Currently, there are two kinds of dependencies: compile and source dependencies.

A compile dependency on a language component is used to compile source files of that language component. For example, a compile dependency on NaBL will ensure that all .nab files are compiled into .str files.

A source dependency is used to depend on source files of a language component. Source dependencies are used to depend on libraries, for example to depend on a Stratego library for name and type analysis. They are also used to compose multiple language components into a single language component, for example to do language extension.

A language is the more abstract notion of a language, which has multiple language implementations. For example, the Java language has the JDK7 and JDK8 implementations, which each have front-end and back-end components.

An end-user project is a project with programs of an end-user language, in contrast to a language specification project which has programs of meta-languages. For example, a Java project is a Java end-user project, whereas the JDK project is a language specification project.

Project structure

The project structure of language specification projects has significantly changed. The biggest change is that these projects are no longer Eclipse (plugin) projects, so that they can be used outside of the Eclipse platform as well. Ant build files have also been removed since we do not use Ant to build projects any more. Many ESV files have been deprecated, and all generated ESV files in the editor directory have been removed.

The following files and directories are no longer part of the project structure:

  • Ant build: .externalToolBuilders, build.generated.xml, build.main.xml
  • Eclipse plugin: plugin.xml, META-INF
  • Eclipse project: .settings, .classpath, .project, build.properties
  • Refactoring: lib/refactor-common.generated.str
  • Deprecated ESV files: editor/langname-Completions.esv, editor/langname-Folding.esv, editor/langname-Outliner.str, editor/langname-Refactorings.esv
  • Generated ESV files: editor/langname-*.generated.esv, editor/langname-Outliner.generated.str
  • The RTG and signatures files are no longer generated for SDF3 projects, since SDF3 generates its own signatures.
  • The generated box pp files are no longer generated, and box pp files are no longer converted into pp.af files.

The following files and directories have been moved:

  • ESV
    • Main ESV file must be at editor/Main.esv. If it does not exist, no packed ESV file will be generated.
    • Packed ESV file: target/metaborg/editor.esv.af
  • SDF
    • Definition: src-gen/syntax/[LanguageName].def
    • Permissive definition: src-gen/syntax/[LanguageName]-permissive.def
    • Parenthesizer: src-gen/pp/[LanguageName]-parenthesize.str
    • Parse table: target/metaborg/sdf.tbl
  • Stratego
    • ‘editor-common.generated’ file: src-gen/stratego/metaborg.str
    • Ctree: target/metaborg/stratego.ctree
    • Generated Java files: src-gen/stratego-java
    • JAR: target/metaborg/stratego.jar
    • Java strategies: src/main/strategies
    • Java strategies JAR: target/metaborg/stratego-javastrat.jar
    • Build cache: target/stratego-cache
  • DynSem
    • Manual Java: src/main/ds
    • Generated Java: src-gen/ds-java

The following generated files and directories still exist, but should not be published to source control any more:

  • lib/editor-common.generated.str or stratego/metaborg.str
  • src-gen

When importing a language specification project into Eclipse or IntelliJ, several platform-specific files will be generated. These files should not be published to source control to keep projects as platform-agnostic as possible.

Eclipse

Importing

To import a language specification project into Eclipse, use Import… ‣ Maven ‣ Existing Maven Projects. We use Maven to set up the correct Java dependencies, which is why there is no special ‘Existing Spoofax Projects’ importer.

Builds

Eclipse has the concept of incremental project builders, which incrementally parse, analyze, and compile files inside a project. An example of such a project builder is the Eclipse JDT builder which incrementally builds Java files. Spoofax 1.5 did not use this functionality, but the new Eclipse plugin in Spoofax 2.0 does.

The project builder for Spoofax parses, analyses, executes transformations, and shows all error markers, for all language files (Stratego files, SDF3 files, files of your language, etc.) in the project. If the project is opened for the first time, a full build will occur, building all language files. When changes occur in the project, an incremental build occurs, building only changed files.

The commands under the Project menu in Eclipse now also affect Spoofax projects. Executing Project ‣ Build… (or pressing Ctrl/Cmd+Alt+B) will build the project.

Executing Project ‣ Clean… will delete the .cache directory, reset the index and task engine, remove all error markers, and reanalyze and rebuild all files in the project. This makes the Reset and Reanalyze builder unnecessary, since this is now properly integrated with Eclipse.

Automatic building can also be turned off by disabling Project ‣ Build Automatically. Builds will then only occur if Project ‣ Build Project is executed or if Ctrl/Cmd+Alt+B is pressed.

Furthermore, the language specification build is no longer written in Ant, but in Java using the Pluto incremental build system.

Natures

The Spoofax language specification project builder is not enabled by default, to enable it a ‘Spoofax meta nature’ must be added to the project. A nature in Eclipse is a project tag which enables functionality for that project. To add the Spoofax nature to a project, right click the project, and choose Spoofax (meta) ‣ Add Spoofax meta nature. When importing a language specification, this nature is automatically added.

For end-user projects, right click the project, and choose Spoofax ‣ Add Spoofax nature to add a nature for end-user projects.

Editors will parse and analyze files regardless of there being a Spoofax nature, but the on-save handler will not be called.

Builders

Builders for the open editor are now located in the Spoofax main menu instead of buttons on the tool bar. Builders wait for the analyzed AST if needed, so the issue where builders are sometimes not executed on the analyzed AST should be solved now.

Builders can also be executed on a set of files by selecting the files in the project or package explorer, right clicking the files, selecting the language name from the menu, and then selecting a builder.

Cancellation

Editor updates can now be cancelled by clicking the red stop button in the progress view. If the progress view is not visible, you can open it by choosing Window ‣ Show View ‣ Progress. If the editor update is not responsive (it is looping for example), the thread running the editor update will be killed after a while.

Killing a thread during analysis may leave the index and task engine in an inconsistent state. If this happens, clean the project using Project ‣ Clean… to force a full build, which makes the state consistent again. Killing a thread is not very well supported in Java and may break Eclipse or even the JVM, which then requires a restart.

Project builds and transformations can also be cancelled in the progress view.

Console logging

Console logging in the new plugin is more prominent so that we can diagnose problems more easily. If the console is not visible, you can open it by choosing Window ‣ Show View ‣ Console. The console does not automatically pop-up when there is a message any more, so it can also be hidden by just closing it.

All warning and error messages are also sent to Eclipse’s error log. The error log can sometimes contain more information about exceptions and stack traces in errors. If the error log is not visible, you can open it by choosing Window ‣ Show View ‣ Error Log.

Manually loading/unloading a language

A language can be manually loaded or reloaded by right clicking a project and choosing Spoofax (meta) ‣ Load language, and unloaded with Spoofax (meta) ‣ Unload language.

External dependencies

The new plugin does not depend on a modified version of IMP, making it possible to install the Rascal plugin alongside the Spoofax plugin. All other external dependencies are limited to the Spoofax plugin, which should prevent conflicts with other Eclipse plugins.

Converting a project

If your project is simple (e.g. it only has syntax and a few transformations), the easiest way to convert your project is to create a new Spoofax language specification project, and to copy your files into that project.

Otherwise, Spoofax 2.0 supports converting an old Spoofax project into a new Spoofax project, but some conversions need to be done manually.

Warning

Converting a Spoofax project is a destructive operation, some files will be deleted, replaced, or changed. Make a backup of your projects before doing any conversions!

Automatic conversion

First, import your existing Spoofax project into Eclipse using File ‣ Import… ‣ Existing Projects into Workspace. Right click the project, and choose Spoofax (meta) ‣ Upgrade language project. A wizard screen will pop up where you have to fill in some details about your language.

If a packed.esv file was found, Spoofax will try to fill in some fields for you. If not, all fields need to be filled in manually. The id and name of your language can be found in the main ESV file. For group id, use a Maven identifier for your organization (e.g. org.metaborg), and as version: 1.0.0-SNAPSHOT.

Warning

Make sure that the id and name fields match exactly with the fields in your ESV file, otherwise the conversion will go wrong.

Press finish to convert the language project.

Manual conversion

Unfortunately, not all required conversions can be done automatically. Do the following conversions manually.

Project configuration

Most of the project configuration is now in the metaborg.yaml file. The manual page on configuration lists all configuration options.

  • Add/remove compile and source dependencies as needed.
  • Add build configuration, such as Stratego compiler arguments, SDF compiler arguments, external def files, and external JAR files.

Imports

In Stratego, SDF2, SDF3, NaBL, and TS files:

  • Remove src-gen, lib, and trans, from module names and imports. These paths are now on the source path of the SDF and Stratego compilers.

In Stratego, NaBL, and TS files:

  • Instead of importing lib/editor-common.generated, import stratego/metaborg.
  • Instead of importing include/<langname>-parenthesize, import pp/<langname>-parenthesize.
  • If you’re using SDF3:
    • Instead of importing the signatures from include/<langname>, import them from signatures/<langname>-sig. These signatures are spread over multiple files, import all the required files to fix errors, since the Stratego editor does not handle transitive imports. You can also use the wildcard import signatures/- to import all signature files, if your syntax definition is not spread over multiple directories.
  • If you’re using SDF2 or an external definition file:
    • Instead of importing the signatures from include/<langname>, import them from signatures/<langname>.

SDF

If you are still using SDF2 instead of SDF3, add the following setting to the metaborg.yaml file:

language:
  sdf:
    version: sdf2

NaBL and TS

If you’re using a NaBL/TS based analysis, perform the following changes:

  • NaBL files are now generated into src-gen/names, fix imports to NaBL files, delete old generated NaBL files.
  • TS files are now generated into src-gen/types, fix imports to TS files, delete old generated TS files.
  • The editor-analyze calls have been changed. Remove analysis-single-default-interface, analysis-multiple-default-interface, and editor-analyze. Replace it with:
editor-analyze = analyze-all(pre-analysis, post-analysis, pp-message|<language>)

with the pre-analysis, post-analysis, and pp-message arguments that you were using before. Also make sure the observer (in your esv) has the (multifile) property.

  • The editor-save call to analysis-save-default(|<language>) has been removed, remove that call. You can remove editor-save entirely if you don’t do any generation, also remove the on save strategy from ESV if you do. If you do generation but do not return a (file, text) tuple from editor-save, be sure to return a !None() to tell Spoofax that you’re returning nothing.
  • The index-setup and task-setup strategies have been removed, Spoofax Core does this for you now. Remove all calls to these strategies.
  • Remove the path argument to analysis-resolve in editor-resolve.
  • Remove the path argument to analysis-propose-completions in editor-complete.
  • Remove the debug-reanalyze strategy, and remove it from your menu in ESV. You can reanalyze by cleaning the project.

ESV

  • The following ESV files are now deprecated, delete and remove any imports to these files:
    • editor/langname-Completions.esv
    • editor/langname-Folding.esv
    • editor/langname-Refactorings.esv
  • Previously generated ESV files in the editor directory are not generated any more. Delete the generated files and remove the imports to generated files.
  • The colorer ESV file is now generated to src-gen/editor/Colorer.generated.esv, import it with imports editor/Colorer.generated in an ESV file.
  • The generated syntax ESV file is no longer generated. If you were using the defaults from the generated file, add them to an ESV file:
language

  line comment  : "//"
  block comment : "/*" * "*/"
  fences        : [ ] ( ) { }

  • The outliner (editor/langname-Outliner.str) must be moved to the trans directory. Rename it to trans/outline.str, change its module to outline, and fix imports of the outliner.

  • Change the file name of the main ESV file to Main.esv, and change its module to Main.

  • In the main ESV file:

    • Change the parse table:

      table : target/metaborg/sdf.tbl

    • Change the Stratego providers

      • For ctree projects:

        provider : target/metaborg/stratego.ctree

      • For jar projects:

        provider : target/metaborg/stratego.jar

      • For projects with Java strategies:

        provider : target/metaborg/stratego.jar
provider : target/metaborg/stratego-javastrat.jar

Java strategies

If your project has Java strategies:

  • Create the src/main/strategies directory.
  • Move Java strategies from editor/java into the src/main/strategies directory. Be sure to preserve the existing Java package structure.
  • Perform a Maven update by right clicking the project and choosing Maven ‣ Update Project…, to update the Java source directories of the project.

DynSem

If your project has manual DynSem Java files:

  • Create the src/main/ds directory.
  • Move manual DynSem Java files from editor/java into the src/main/ds directory. Be sure to preserve the existing Java package structure.
  • Perform a Maven update by right clicking the project and choosing Maven ‣ Update Project…, to update the Java source directories of the project.

Ant build customization

Language specification builds do not use Ant any more, so any customizations to the build.main.xml are lost. To perform an Ant task before and after the build, add the following configuration option to your metaborg.yaml file:

build:
  ant:
  - phase: preCompile
    file: ${path:root}/ant.xml
    target: generate-xml
  - phase: postCompile
    file: ${path:root}/ant.xml
    target: package-xml

See the manual page on configuration for more information about configuring Ant build steps.

Eclipse plugin

Language specification projects are not Eclipse plugins any more. To create an Eclipse plugin for your language, follow the guide for exporting a language as an Eclipse plugin.

Git

If you’re using Git, the .gitignore file is replaced with a new one, add entries that you need again. All generated files that were previously not ignored, are ignored now. To delete all ignored files from the Git index (the files will remain on the filesystem but Git will forget about them), make sure all your useful changes are committed and pushed, then run the following commands:

git rm -r --cached .
git add .
git commit -am "Remove ignored files"

Building

When you are done with converting the project, build it with Cmd+Shift+B or Project ‣ Build. If the build does not work, try cleaning the project first with Project ‣ Clean, and then building again. Also make sure that Project ‣ Build Automatically is turned on.