Add RGA compiler support to HLSL (#3961)

* Add RGA compiler support to HLSL

AMD's Radeon GPU Analyzer (RGA) tool is an indispensible tool for
graphics programmers to inspect the instructions that will be
compiled by the driver for a given shader and GPU. The integration of
this compiler into CE is slightly unconventional for reasons to be
explained here:

When trying to compile HLSL directly to ISA, RGA requires a full
pipeline state description (specified for graphics pipelines using a
gpso file), and a root signature, since changes to either affect how
resources and data within the shader are accessed or written to.
Specifying both is difficult to the extent that it would add significant
usage friction to the tool. An informal survey among other senior
graphics programming practitioners suggested unanimous agreement to
infer both pipeline state and root signature where possible, with the
expectation that if more accurate ISA code was needed, RGA could be used
directly offline.

Fortunately, RGA supplies an alternative workflow, wherein SPIR-V code
can be compiled directly to the approximate ISA, bypassing both the
pipeline and root signature requirement. To use RGA, the following steps
are performed:

1. Use the default DXC compiler to emit SPIR-V as text to a temporary
   file in the selected temp folder.
2. Compile the ISA using RGA, consuming the output of step 1
3. Rename the resulting file to the output file CE expects

In addition, a non-standard argument --asic is added to the user
options. This argument is filtered for DXC, but for RGA is forwarded as
the selected ASIC to emit ISA for.

These steps are performed by a single `rga.js` script, which is invoked
as an executable. This chaining could have been added within CE library
code directly, but being an atypical flow, felt more appropriate as a
separate script (which has the benefit of faster iteration, due to being
loaded on each compilation request). The paths to DXC and RGA are
supplied as arguments through a simple CLI interface.

NOTE: This commit also adds `-Zi` and `-Qembed_debug` flags to both DXC
and RGA, which provides line association data for DXIL.

Signed-off-by: Jeremy Ong <jeremycong@gmail.com>

* Incorporate PR feedback (see commit body)

This commit introduces several changes:

- HLSL compiler is now Typescript
- A new RGACompiler Typescript class is used to invoke RGA
- The multi-phase compilation that was previously done using a Node
  script now leverages the existing sandboxed execution facilities
- The DXC compiler is configurable as a property on the RGA
  configuration, and an example is provided in hlsl.defaults.properties

As there are several steps needed to compile HLSL to AMD ISA as before,
as steps are done asynchronously so the runtime can continue to service
other requests during compilation (either DXC or RGA) or file I/O.

Signed-off-by: Jeremy Ong <jeremycong@gmail.com>

Signed-off-by: Jeremy Ong <jeremycong@gmail.com>
Co-authored-by: Rubén Rincón Blanco <ruben@rinconblanco.es>
5 files changed
tree: 70afa75dc599f0ccb5c58917c8fa18aecc0838cf
  1. .eslint-license-header.yml
  2. .eslintignore
  3. .eslintrc.yml
  4. .git-blame-ignore-revs
  5. .gitattributes
  6. .github/
  7. .gitignore
  8. .husky/
  9. .idea/
  10. .mocharc-min.yml
  11. .mocharc.yml
  12. .nycrc.yml
  13. .prettierignore
  14. .prettierrc.js
  15. AUTHORS.md
  16. CODE_OF_CONDUCT.md
  17. CONTRIBUTING.md
  18. CONTRIBUTORS.md
  19. LICENSE
  20. Makefile
  21. PULL_REQUEST_TEMPLATE.md
  22. README.md
  23. SECURITY.md
  24. app.js
  25. codecov.yml
  26. cypress.json
  27. cypress/
  28. docs/
  29. etc/
  30. examples/
  31. lib/
  32. package-lock.json
  33. package.json
  34. static/
  35. test/
  36. tsconfig.backend.json
  37. tsconfig.base.json
  38. tsconfig.frontend.json
  39. tsconfig.json
  40. types/
  41. views/
  42. webpack.config.esm.js
README.md

Build Status codecov

Compiler Explorer

Compiler Explorer

Compiler Explorer is an interactive compiler exploration website. Edit code in C, C++, C#, F#, Rust, Go, D, Haskell, Swift, Pascal, ispc, Python, Java or in any of the other 30+ supported languages, and see how that code looks after being compiled in real time. Multiple compilers are supported for each language, many different tools and visualisations are available, and the UI layout is configurable (thanks to GoldenLayout).

Try out at godbolt.org, or run your own local instance.

Compiler Explorer follows a Code of Conduct which aims to foster an open and welcoming environment.

Compiler Explorer was started in 2012 to show how C++ constructs translated to assembly code. It started out as a tmux session with vi running in one pane and watch gcc -S foo.cc -o - running in the other.

Since then, it has become a public website serving around 3,000,000 compilations per week.

You can financially support this project on Patreon, GitHub, Paypal, or by buying cool gear on the Compiler Explorer store.

Using Compiler Explorer

FAQ

There is now a FAQ section in the repository wiki. If your question is not present, please contact us as described below, so we can help you. If you find that the FAQ is lacking some important point, please free to contribute to it and/or ask us to clarify it.

Videos

There are a number of videos that showcase some features of Compiler Explorer:

A Road map is available which gives a little insight into the future plans for Compiler Explorer.

Developing

Compiler Explorer is written in Node.js.

Assuming you have a compatible version of node installed, on Linux simply running make ought to get you up and running with an Explorer running on port 10240 on your local machine: http://localhost:10240/. If this doesn't work for you, please contact us, as we consider it important you can quickly and easily get running. Currently, Compiler Explorer requires node 16 (LTS version) installed, either on the path or at NODE_DIR (an environment variable or make parameter).

Running with make EXTRA_ARGS='--language LANG' will allow you to load LANG exclusively, where LANG is one for the language ids/aliases defined in lib/languages.ts. For example, to only run Compiler Explorer with C++ support, you'd run make EXTRA_ARGS='--language c++'. The Makefile will automatically install all the third party libraries needed to run; using npm to install server-side and client side components.

For development, we suggest using make dev to enable some useful features, such as automatic reloading on file changes and shorter startup times.

You can also use npm run dev to run if make dev doesn't work on your machine.

Some languages need extra tools to demangle them, e.g. rust, d, or haskell. Such tools are kept separately in the tools repo.

Configuring compiler explorer is achieved via configuration files in the etc/config directory. Values are key=value. Options in a {type}.local.properties file (where {type} is c++ or similar) override anything in the {type}.defaults.properties file. There is a .gitignore file to ignore *.local.* files, so these won‘t be checked into git, and you won’t find yourself fighting with updated versions when you git pull. For more information see Adding a Compiler.

Check CONTRIBUTING.md for detailed information about how you can contribute to Compiler Explorer, and the docs folder for specific details regarding various things you might want to do, such as how to add new compilers or languages to the site.

Running a local instance

If you want to point it at your own GCC or similar binaries, either edit the etc/config/LANG.defaults.properties or else make a new one with the name LANG.local.properties, substituting LANG as needed. *.local.properties files have the highest priority when loading properties.

When running in a corporate setting the URL shortening service can be replaced by an internal one if the default storage driver isn't appropriate for your environment. To do this, add a new module in lib/shortener/myservice.js and set the urlShortenService variable in configuration. This module should export a single function, see the tinyurl module for an example.

RESTful API

There's a simple restful API that can be used to do compiles to asm and to list compilers.

You can find the API documentation here.

Contact us

We run a Compiler Explorer Discord, which is a place to discuss using or developing Compiler Explorer. We also have a presence on the cpplang Slack channel #compiler_explorer and we have a public mailing list.

There's a development channel on the discord, and also a development mailing list.

Feel free to raise an issue on github or email Matt directly for more help.

Credits

Compiler Explorer is maintained by the awesome people listed in the AUTHORS file.

We would like to thank the contributors listed in the CONTRIBUTORS file, who have helped shape Compiler Explorer.

We would also like to specially thank these people for their contributions to Compiler Explorer:

A number of amazing sponsors, both individuals and companies, have helped fund and promote Compiler Explorer.