diff --git a/docs/conf.py b/docs/conf.py index 50e41501db8f..b93930bfb1c8 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -20,7 +20,7 @@ # -- Project information ----------------------------------------------------- -project = u'Xen' +project = u'The Xen Project' copyright = u'2019, The Xen development community' author = u'The Xen development community' diff --git a/docs/contribute-xen.rst b/docs/contribute-xen.rst new file mode 100644 index 000000000000..e93dae037e10 --- /dev/null +++ b/docs/contribute-xen.rst @@ -0,0 +1,60 @@ +******************* +Contributing to Xen +******************* + +There are many ways to contribute to the Xen Project. + +As with any open source project, providing concrete contributions will demonstrate to the project leaders that you are serious about being involved, which helps us separate the people who talk about contributing from the people who actually contribute to the project in a meaningful way. This builds trust in your work, and the people who consistently contribute good work will be given more and more responsibility. + +Our goal is to maintain an environment of professionalism, respect, and innovation within Xen Project development. Please adhere to the following guidelines as you participate in the Xen Project community: + +Be professional in all communications. Please avoid flaming, profanity, vulgarity, SHOUTING IN ALL CAPS. Keep personal discussion for the IRC channel. There is never any reason to publicly discuss gender, ethnicity, orientation, lifestyle, or religion in our project. +Stay on topic. If you have a comment that is not relevant to the current discussion, please start a new thread. +Keep it legal. Please use project resources for their intended purpose. +Don’t feed trolls. Every open project attracts unwelcome commentary from individuals trying to get a rise out of someone. When this happens, just ignore them. + +================= +Contributing Code +================= + +Because every open source project has slightly different policies for accepting code contributions, here are some specific guidelines for contributing to The Xen Project codebase. + +Contributions to the Xen Project codebase follow the conventions as they are known in the Linux Kernel community. In summary, contributions are made through patches that are reviewed by the community. The project does not require community members to sign contribution or committer agreements. We do require contributors to sign contributions using the sign-off feature of the code repository, following the same approach as the Linux Kernel does (see Developer Certificate Of Origin). + +In addition, each Xen Project subproject (also called team) may have its own mailing list, submission guidelines, IRC channel, and culture. Please check the team portal to learn more. + +The majority of our projects have some shared conventions: + +Before submitting a patch please read the following guides: + +- CONTRIBUTING, COPYING, and CODING_STYLE files (these are in most, but not all our our git repositories). +- Guide to submitting patches via e-mail workflow. Note that XAPI, Mirage OS and XCP-ng follow a standard Github workflow. +- Guide to asking developer questions + +You can also ask questions on the `xen-devel@lists.xenproject.org` mailing list or any of the subproject lists and irc channels. + +============================================== +Submitting Patches to the Xen Project Codebase +============================================== + +Please first check the submission process for the Xen Sub-Project, and send an email to the proper mailing list with [PATCH] as the first word in the subject line. Each patch should perform a single function. Patches sent to the mailing lists should be broken up into several email messages of less than 100KB each, with only one patch per email. + +Please include a description of why you want the change made (not just the “what”) and why it is important for the team to make this change. Your patch will need to include a signed-off-by tag, author’s name, and other information. + +For details about what to include in your patch, you should start with the patch submission documentation. + +========== +Escalation +========== + +In a project of this size, patches sometimes slip through the cracks. If you submitted a patch to the xen-devel mailing list or bugzilla and did not receive a response within 5 business days, please send an email to xen-devel and in the first line of that email, include this phrase “Patch escalation: no response for x days”. + +This is one case where you should “top post” to make sure that the escalation text is read. + +====================== +Code Security Scanning +====================== + +The Xen Project is registered with the “Coverity Scan” service which applies Coverity’s static analyser to the Open Source projects. The tool can and does find flaws in the source code which can include security issues. Currently only the Xen Project Hypervisor (i.e. xen.git) is covered by these scans. Triaging and proposing solutions for the flaws found by Coverity is a useful way in which Community members can contribute to the Xen Project. + +Members of the community may request access to the Coverity database. However, Coverity requires that you create an account and apply for Xen Project membership by searching for the Xen Project and then requesting to be added to the project. We typically will approve requests within a few days, but reserve rejecting requests from accounts who never engaged with the project (aka never posted to a mailing list) or which look like spam accounts. \ No newline at end of file diff --git a/docs/contribute-xen/contribute-xen.rst b/docs/contribute-xen/contribute-xen.rst new file mode 100644 index 000000000000..e93dae037e10 --- /dev/null +++ b/docs/contribute-xen/contribute-xen.rst @@ -0,0 +1,60 @@ +******************* +Contributing to Xen +******************* + +There are many ways to contribute to the Xen Project. + +As with any open source project, providing concrete contributions will demonstrate to the project leaders that you are serious about being involved, which helps us separate the people who talk about contributing from the people who actually contribute to the project in a meaningful way. This builds trust in your work, and the people who consistently contribute good work will be given more and more responsibility. + +Our goal is to maintain an environment of professionalism, respect, and innovation within Xen Project development. Please adhere to the following guidelines as you participate in the Xen Project community: + +Be professional in all communications. Please avoid flaming, profanity, vulgarity, SHOUTING IN ALL CAPS. Keep personal discussion for the IRC channel. There is never any reason to publicly discuss gender, ethnicity, orientation, lifestyle, or religion in our project. +Stay on topic. If you have a comment that is not relevant to the current discussion, please start a new thread. +Keep it legal. Please use project resources for their intended purpose. +Don’t feed trolls. Every open project attracts unwelcome commentary from individuals trying to get a rise out of someone. When this happens, just ignore them. + +================= +Contributing Code +================= + +Because every open source project has slightly different policies for accepting code contributions, here are some specific guidelines for contributing to The Xen Project codebase. + +Contributions to the Xen Project codebase follow the conventions as they are known in the Linux Kernel community. In summary, contributions are made through patches that are reviewed by the community. The project does not require community members to sign contribution or committer agreements. We do require contributors to sign contributions using the sign-off feature of the code repository, following the same approach as the Linux Kernel does (see Developer Certificate Of Origin). + +In addition, each Xen Project subproject (also called team) may have its own mailing list, submission guidelines, IRC channel, and culture. Please check the team portal to learn more. + +The majority of our projects have some shared conventions: + +Before submitting a patch please read the following guides: + +- CONTRIBUTING, COPYING, and CODING_STYLE files (these are in most, but not all our our git repositories). +- Guide to submitting patches via e-mail workflow. Note that XAPI, Mirage OS and XCP-ng follow a standard Github workflow. +- Guide to asking developer questions + +You can also ask questions on the `xen-devel@lists.xenproject.org` mailing list or any of the subproject lists and irc channels. + +============================================== +Submitting Patches to the Xen Project Codebase +============================================== + +Please first check the submission process for the Xen Sub-Project, and send an email to the proper mailing list with [PATCH] as the first word in the subject line. Each patch should perform a single function. Patches sent to the mailing lists should be broken up into several email messages of less than 100KB each, with only one patch per email. + +Please include a description of why you want the change made (not just the “what”) and why it is important for the team to make this change. Your patch will need to include a signed-off-by tag, author’s name, and other information. + +For details about what to include in your patch, you should start with the patch submission documentation. + +========== +Escalation +========== + +In a project of this size, patches sometimes slip through the cracks. If you submitted a patch to the xen-devel mailing list or bugzilla and did not receive a response within 5 business days, please send an email to xen-devel and in the first line of that email, include this phrase “Patch escalation: no response for x days”. + +This is one case where you should “top post” to make sure that the escalation text is read. + +====================== +Code Security Scanning +====================== + +The Xen Project is registered with the “Coverity Scan” service which applies Coverity’s static analyser to the Open Source projects. The tool can and does find flaws in the source code which can include security issues. Currently only the Xen Project Hypervisor (i.e. xen.git) is covered by these scans. Triaging and proposing solutions for the flaws found by Coverity is a useful way in which Community members can contribute to the Xen Project. + +Members of the community may request access to the Coverity database. However, Coverity requires that you create an account and apply for Xen Project membership by searching for the Xen Project and then requesting to be added to the project. We typically will approve requests within a few days, but reserve rejecting requests from accounts who never engaged with the project (aka never posted to a mailing list) or which look like spam accounts. \ No newline at end of file diff --git a/docs/contribute-xen/submit-patch.rst b/docs/contribute-xen/submit-patch.rst new file mode 100644 index 000000000000..eaf50aa362e3 --- /dev/null +++ b/docs/contribute-xen/submit-patch.rst @@ -0,0 +1,399 @@ +************************* +Submitting Patches to Xen +************************* + +Changes to the Xen Project are submitted to a mailing list either as an individual patch or as a patch series. A patch series is a series of patches that are related and broken into a logical series of individual patches. Patches, if accepted, will turn into commits in the git source tree. However, frequently multiple versions of patches will have to be posted, before the patch is accepted. + +Sending the patches to the list +The xen-devel mailing list is moderated for non-subscribers. It is not mandatory to subscribe but it can help avoid this delay. It is possible to subscribe and then disable delivery in the mailman options so as to avoid moderation but not receive list traffic if that is what you would prefer. + +========================= +Setting up git send-email +========================= + +The most robust way to send files is by using the git send-email command. (If you're using mercurial, please see our submitting_Xen_Patches_-_mercurial). + +To do this, first set configure your information: + +.. code-block:: + + git config --global sendemail.from "YOUR NAME " + git config --global sendemail.smtpserver imap.example.org + git config --global sendemail.smtpuser USER + # depending on your config you may also set: + git config --global sendemail.smtpencryption tls + +If you don't want to enter password for your SMTP server all the time: + +.. code-block:: + + git config --global sendemail.smtppass = PASS + + +Important Settings +~~~~~~~~~~~~~~~~~~ + +There are a number of settings you may want to set globally or per repository as follows: + +# Allows you to drop --thread from git format-patch ... + +.. code-block:: + + git config --global format.thread true + +# Allows you to use -s or --signoff when committing a patch + +.. code-block:: + + git config --global user.name "YOUR NAME" + git config --global user.email "" + +Subject Prefix +~~~~~~~~~~~~~~ + +Patches and patch series are posted to a mailing list: when using the git format-patch command a subject prefix is added to a patch or patch series, which helps disambiguate normal discussions from posted patches and code reviews. + +git format-patch creates mails with subject lines such as + +.. code-block:: + + # Single patch + [] + # Patch series + [<SUBJECT-PREFIX> 0/n] <Title of cover letter/patch series> + [<SUBJECT-PREFIX> 1/n] <Title of 1st patch in git> + ... + [<SUBJECT-PREFIX> n/n] <Title of nth patch in git> + +Canonical Subject Prefix +~~~~~~~~~~~~~~~~~~~~~~~~ + +As patches for different git repositories can be posted (or CC'ed) to different mailing lists, some conventions have emerged over time which help code reviewers identify which patch is intended for which git tree. We refer to these as canonical subject prefix. The table below shows recommended default Subject Prefixes, which either need to be passed using the `--subject-prefix=<Subject-Prefix>` option when calling git format-patch, or can be set as defaults per git repository as described here + +The following table lists common canonical subject prefixes used in the project: + ++----------------------+----------------------+----------------------+ +| **repo** | **Subject Prefix** | **List** | ++======================+======================+======================+ +| git default | PATCH | | ++----------------------+----------------------+----------------------+ +| `xen | PATCH | xen-devel@ | +| .git <http://xenbits | or XEN | | +| .xen.org/gitweb/?p=x | PATCH (recommended) | | +| en.git;a=summary>`__ | | | ++----------------------+----------------------+----------------------+ +| `osstest.git | OSSTEST PATCH | xen-devel@ | +| <http://xenbits.xen | | | +| .org/gitweb/?p=osste | | | +| st.git;a=summary>`__ | | | ++----------------------+----------------------+----------------------+ +| `xtf | XTF PATCH | xen-devel@ | +| .git <http://xenbits | | | +| .xen.org/gitweb/?p=x | | | +| tf.git;a=summary>`__ | | | ++----------------------+----------------------+----------------------+ +| `xenalyze.git | XENALYZE PATCH | xen-devel@ | +| <http://xenbits.xen. | | | +| org/gitweb/?p=xenaly | | | +| ze.git;a=summary>`__ | | | ++----------------------+----------------------+----------------------+ +| `livepatch-build-too | L | xen-devel@ | +| ls.git <http://xenbi | IVEPATCH-BUILD-TOOLS | | +| ts.xen.org/gitweb/?p | PATCH | | +| =livepatch-build-too | | | +| ls.git;a=summary>`__ | | | ++----------------------+----------------------+----------------------+ +| `mini-os.git | PATCH | minios-devel@ but | +| <http://xenbits.xen | or MINI-OS | sometimes xen-devel@ | +| .org/gitweb/?p=mini- | PATCH (recommended) | is CC'ed | +| os.git;a=summary>`__ | | | ++----------------------+----------------------+----------------------+ +| `unikraft/un | UNIKRAFT PATCH | minios-devel@ | +| ikraft.git <http://x | | | +| enbits.xen.org/gitwe | | | +| b/?p=unikraft/unikra | | | +| ft.git;a=summary>`__ | | | ++----------------------+----------------------+----------------------+ +| `unikra | UNIKRAFT/<REPO> | minios-devel@ | +| ft/.../<REPO>.git <h | PATCH | | +| ttp://xenbits.xen.or | | | +| g/gitweb/?a=project_ | | | +| list;pf=unikraft>`__ | | | ++----------------------+----------------------+----------------------+ + +.. important:: If you do not use a canonical subject prefix some existing and future automated tooling may not work correctly: consider for example a situation in which a patch for livepatch-build-tools.git is sent to xen-devel@ and that this triggers a build test in different environments. In this case, the build test would try and apply the patch to xen.git and might fail in unexpected ways. + +Common modifications to the Subject Prefix +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In addition, the following modifications to Subject Prefixes are commonly used and can also be used in combination + ++----------------------+----------------------+----------------------+ +| **Subject Prefix | **Example** | **When to use** | +| Modification** | | | ++======================+======================+======================+ +| **RFC** ... | **RFC** XTF PATCH | RFC means "Request | +| | | For Comments"; use | +| | | this when sending an | +| | | experimental patch | +| | | for discussion | +| | | rather than | +| | | application. | ++----------------------+----------------------+----------------------+ +| ... **RESEND** | XTF PATCH **RESEND** | Typically used if | +| | | your patch has not | +| | | been reviewed for | +| | | some time and you | +| | | want to remind | +| | | people of the patch | ++----------------------+----------------------+----------------------+ +| . | XEN | Typically | +| .. **for-<release>** | PATCH **for-4.13** | used **after the | +| | | master xen.git | +| | | branch has been | +| | | feature frozen** and | +| | | you want to | +| | | highlight a patch | +| | | that is destined for | +| | | the release which is | +| | | currently being | +| | | developed | ++----------------------+----------------------+----------------------+ +| ... **for-next** | XEN | Typically | +| | PATCH **for-next** | used **towards the | +| | | end of a development | +| | | cycle** when you | +| | | want to highlight a | +| | | patch that is | +| | | destined for | +| | | the **next** release | ++----------------------+----------------------+----------------------+ + +The `git format-patch` contains the following options to help with these: + +* `--rfc` will mark a patch as RFC, but will always lead to RFC PATCH +* `--subject-prefix=<Subject-Prefix>` allows you to set a subject prefix, but will overwrite any defaults you may have set + +Changing git defaults to use the Canonical Subject Prefix +--------------------------------------------------------- + +To change the setting perform + +.. code-block:: + + cd <local-git-tree> + git config --local format.subjectPrefix "<canonical subject prefix>" + +====================== +Sending a Patch Series +====================== + +Step 1: Create Patches using git format-patch +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Git format patch allows you to create and save formatted patches in a directory. By default it will create them in the root of your git directory, but you probably want to direct this into a ../patches/feature-version directory (in the examples below ../patches/feature-v2 would contain only v2 of that series). It is also possible to store several versions of a patch, e.g. v1, v2, etc in the same ../patches/feature directory). Let's say the last two commits of your head are part of your series you want to send out. In this case, the command line would look like + +.. code-block:: + + $ git format-patch --reroll-count=2 --thread --cover-letter -o ../patches/feature-v2 -2 + +This will create three files, such as + +`v2-0000-cover-letter.patch` +`v2-0002-Patch-to-do-bar.patch` +`v2-0001-Patch-to-do-foo.patch` + +.. notes:: + + * You will need to edit the subject and body of `v2-0000-cover-letter.patch`. + * You must always use the `--thread` and `--cover-letter` options. If you omit `--thread`, any automatic tooling and patch checking that is triggered by sending a mail to one of our mailing lists may not work without using this option. You can set `--thread` as a default, as outlined here. + * `--rfc` and `--subject-prefix=Subject-Prefix`: please read this document before using these options. + +Step 2: Use `add_maintainers.pl` (or `get_maintainer.pl`) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Option 1: Use `add_maintainers.pl` +-------------------------------- + +.. code-block:: + + $ ./scripts/add_maintainers.pl -d ../patches/feature-v2 + +Then follow the instructions (which are essentially Step 3, with correct command line options based on what you pass to add_maintainers.pl). Note that the add_maintainers.pl script works around a limitation of git send-email ... --cc-cmd="./scripts/get_maintainer.pl" ..., which does not automatically update the CC list of your cover letter. + +Other useful options include: + +* `--reroll-count|-v` (e.g. -v 2): If you store your patches in one directory with different versions in it generated by --reroll-count=2 you want to use this option +* `--patchcc|-p LOCATION`: Inserts CC's into a specific location (see --help ) to `*.patch` files +* `--covercc|-c LOCATION`: Inserts CC's into a specific location (see --help ) to the cover letter +* `--tags|-t`: Adds people who have *-by tags to the mail header (git send-email does not do this itself) +* `--tagscc`: Adds people who have *-by tags to the CC list +* `--arg|-a` (e.g. -a "<argument1 with space>" -a <argument2> ... for arguments you want to pass to `./scripts/get_maintainer.pl`) + +Common LOCATION combinations include: + +* -p commit: copy CC blocks into the *.patch body (CC's will become part of the commit message) +* -p none -c header: copy CC blocks from *.patch files in the cover letter without modifying *.patch files. This is useful in rerolled patches, where you do not want to modify the CC blocks. -p ccbody behaves similarly, only that any new CCs that may come from an updated MAINTAINERS file or from new files that have been added to *.patch files will be added to *.patch files. +* -p comment -c end: copy CC blocks after the *.patch body (CC's will not be committed) and into the body of the cover letter + +Option 2: Use `get_maintainer.pl` Manually +---------------------------------------- + +Foreach <v2 patchfile> in ../patches/feature (with XXXX > 0000): + +- Run: ./scripts/get_maintainer.pl <v2 patchfile> +- Review the e-mail list and prefix each line with Cc: +- Copy the e-mail CC block into patchfile +- Do anything else you want to do manually + +For `v2-0000-cover-letter.patch` take the superset of all the files generated previously and merge, then copy the e-mail CC block into `0000-cover-letter.patch`. + +Step 3: Send Patches using git send-email +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Send the patches using: + +.. code-block:: + + $ git send-email --to xen-devel@lists.xenproject.org ../patches/feature-v2/*.patch + +Other useful options include: + +--dry-run +--cc if you need to CC additional reviewers (e.g. from within your team) + +=========================================================== +Submitting Patches against Point Releases, such as Xen 4.10 +=========================================================== + +When you develop against older Xen versions you are likely going to use an outdated version of the MAINTAINERS file. Typically only the files on staging or master are up-to-date. To work around this, use the following slightly modified workflow: + +.. code-block: + + $ git format-patch ... -o <patchdir> ... + $ checkout master + $ ./scripts/add_maintainers.pl -d <patchdir> + $ checkout <original branch> + $ git send-email ... <patchdir> + +This ensures that you use the latest set of tools and the latest MAINTAINERS file. + +=========================== +Sending an Individual Patch +=========================== + +You can use the same workflow as outlined in the previous section, without generating a cover letter. In this case, the git-format-patch command line in step 1 would look like + +.. code-block:: + + $ git format-patch --thread -o ../patches/bugfix-v2 -1 + +This is followed by steps 2 and 3. However, for single patches without a cover letter, using git-send-email alone, is quite a reasonable option. In this case, you can use the following command line, which will get the CC list from the ./scripts/get_maintainer.pl script, which allows you to fold all 3 steps into 1. + +.. code-block:: + + $ git send-email --to xen-devel@lists.xenproject.org --cc-cmd="./scripts/get_maintainer.pl" -1 + +Other useful options include: + +* --dry-run will go through all the motions of preparing the patchbomb, but instead of sending a mail, will just output the mails it would have sent. Useful for testing. +* --cc if you need to CC additional reviewers (e.g. from within your team) +* --reroll-count=N allows you to change the revision of the patch +* --rfc and --subject-prefix=Subject-Prefix allow you to change the subject prefix: please read this document before using these options + +======================================================================= +Using add_maintainers.pl (or get_maintainer.pl) from Outside of xen.git +======================================================================= + +You can use `add_maintainers.pl` or `get_maintainer.pl` on any Xen Project git repository with a compatible MAINTAINERS file in the root of its tree. An example is livepatch-build-tools.git. In this case simply replace ./scripts/add_maintainers.pl or ./scripts/get_maintainer.pl with the full path to the script. + +# You are in the xen.git sister repository (e.g. livepatch-build-tools) + +.. code-block:: + + $ git format-patch ... -o <patchdir> ... + $ $LOCATION-OF-XEN-GIT/scripts/add_maintainers.pl -d <patchdir> + $ git send-email ... <patchdir> + +A minimum template for such a MAINTAINERS file can be found below. + +.. code-block:: + + This file follows the same conventions as outlined in + xen.git:MAINTAINERS. Please refer to the file in xen.git + for more information. + + THE REST + M: MAINTAINER1 <maintainer1@email.com> + M: MAINTAINER2 <maintainer2@email.com> + L: xen-devel@lists.xenproject.org + S: Supported + F: * + F: */ + V: xen-maintainers-1 + +=============================== +Reviewing and Resending Patches +=============================== + +After posting your patches you will hopefully see some response in the form of comments, patch review and eventually commit. + +Code Review +~~~~~~~~~~~ + +The form of the review may often be quite direct and to the point which may be daunting for some people. However bear in mind that detailed criticism of a patch usually means that the reviewer is interested in your patch and wants to see it go in! + +Once you have addressed any review you should normally resend the patch. It is normally expected that you address all review comments before reposting. This often means code changes in your patch but could also mean simply responding to the review comments explaining you reasoning or giving reasons why something should be the way it is. + +You should also rebase your change against the project's development tree before sending out a new version, such that when it is approved it applies cleanly. The only reason not to do that is if you don't expect it to be approved, and want feedback on what you have. In that case you can do the rebase later. + +The relevant command line options in git-format-patch, ./scripts/add_maintainers.pl and git-send-email (if used standalone) for reposting new revisions is `--reroll-count|-v` + +Highlight Changes in the New Version +------------------------------------ + +When resending a patch you should normally include a note of the changes between the current and last version of the patch. Common practice is to include these notes after your Signed-off-by separated by a triple-dash (---). This indicates patch commentary specific to the posting which need not be included in the final changelog (although you should also remember to update the changelog if necessary). You should also include a "V2" (V3, V4 etc) tag in the subject line (if you are using the git send-email command then the `--reroll-count=N` option is helpful here, or for older git versions --subject-prefix='PATCH vN'). + +Update Tags +----------- + +If someone replies to your patch with a tag of the form Acked-by: <Developer>, Reviewed-by:, Tested-by: etc then, assuming you have not significantly reworked the patch, you should include these tags in any reposting after your own Signed-off-by line. This lets people know that the patch has been seen and that someone approves of it and also serves to prevent reviewers wasting time re-reviewing a patch which is not significantly different to last time. The developers with commit access also like to see postings with such tags since it means they are likely to be much easier to deal with and commit. + +An example of a new Patch Version +An example of a resend of the example patch from above might be: + +.. code-block:: + + Subject: [PATCH v2] foobar: Add a new trondle calls + + Add a some new trondle calls to the foobar interface to support + the new zot feature. + + Signed-off-by: Joe Smith <joe.smith@citrix.com> + Acked-by: Jane Doe <jane.doe@example.com> + + --- + Changed since v1: + * fix coding style + * be sure to treadle the trondle in the error case. + + diff -r 63531e640828 tools/libxc/Makefile + --- a/tools/libxc/Makefile Mon Dec 07 17:01:11 2009 +0000 + +++ b/tools/libxc/Makefile Mon Dec 21 11:45:00 2009 +0000 + ... + +Resending Patches +~~~~~~~~~~~~~~~~~ + +If you do not get any response to your patch or you got lots of Acked-by's but the patch has not been committed (remember that reviewers and maintainers are busy people too and sometimes things may fall through the cracks) then after some time, perhaps 2-4 weeks (guidelines), you should resend the patch, perhaps including [RESEND] in the subject line to alert people that the last mail was dropped. Before resending you should: + +Check that the patch has not been applied to the staging branch, since committers do not always send a notification when they apply a patch. Consider if there is anything you can do to improve the commit message or subject line of your patch to better attract the attention of the relevant people. Remember to include any Acked-by/Reviewed-by which you received in response to the previous post. + +How to Know when a Patch has been Committed +------------------------------------------- +Changes committed to Xen Project by the committers are immediately available in the "staging" branch of the main xen.git tree. They are then automatically tested, and if the tests pass the changes are propagated to the "master" branch. + +After your Patch is Committed +----------------------------- + +If your patch turns out to break something you will be expected to respond promptly to help diagnose and fix the problem. This assumes that you in particular keep an eye on the osstest flight reports sent to xen-devel. If it can't be fixed quickly, your change may be reverted. \ No newline at end of file diff --git a/docs/faq/faq.rst b/docs/faq/faq.rst new file mode 100644 index 000000000000..1d345d842ada --- /dev/null +++ b/docs/faq/faq.rst @@ -0,0 +1,3 @@ +************************** +Frequently Asked Questions +************************** \ No newline at end of file diff --git a/docs/features/feature-levelling.pandoc b/docs/features/feature-levelling.pandoc deleted file mode 100644 index 171a42c35840..000000000000 --- a/docs/features/feature-levelling.pandoc +++ /dev/null @@ -1,216 +0,0 @@ -% Feature Levelling -% Revision 1 - -\clearpage - -# Basics - ----------------- ---------------------------------------------------- - Status: **Supported** - - Architecture: x86 - - Component: Hypervisor, toolstack, guest ----------------- ---------------------------------------------------- - - -# Overview - -On native hardware, a kernel will boot, detect features, typically optimise -certain codepaths based on the available features, and expect the features to -remain available until it shuts down. - -The same expectation exists for virtual machines, and it is up to the -hypervisor/toolstack to fulfill this expectation for the lifetime of the -virtual machine, including across migrate/suspend/resume. - - -# User details - -Many factors affect the featureset which a VM may use: - -* The CPU itself -* The BIOS/firmware/microcode version and settings -* The hypervisor version and command line settings -* Further restrictions the toolstack chooses to apply - -A firmware or software upgrade might reduce the available set of features -(e.g. Intel disabling TSX in a microcode update for certain Haswell/Broadwell -processors), as may editing the settings. - -It is unsafe to make any assumption about features remaining consistent across -a host reboot. Xen recalculates all information from scratch each boot, and -provides the information for the toolstack to consume. - -`xl` currently has no facilities to help the user collect appropriate feature -information from relevant hosts and compute appropriate feature specifications -for use in host or domain configurations. (`xl` being a single-host -toolstack, it would in any case need external support for accessing remote -hosts eg via ssh, in the form of automation software like GNU parallel or -ansible.) - -# Technical details - -The `CPUID` instruction is used by software to query for features. In the -virtualisation usecase, guest software should query Xen rather than hardware -directly. However, `CPUID` is an unprivileged instruction which doesn't -fault, complicating the task of hiding hardware features from guests. - -Important files: - -* Hypervisor - * `xen/arch/x86/cpu/*.c` - * `xen/arch/x86/cpuid.c` - * `xen/include/asm-x86/cpuid-autogen.h` - * `xen/include/public/arch-x86/cpufeatureset.h` - * `xen/tools/gen-cpuid.py` -* `libxc` - * `tools/libxc/xc_cpuid_x86.c` - -## Ability to control CPUID - -### HVM - -HVM guests (using `Intel VT-x` or `AMD SVM`) will unconditionally exit to Xen -on all `CPUID` instructions, allowing Xen full control over all information. - -### PV - -The `CPUID` instruction is unprivileged, so executing it in a PV guest will -not trap, leaving Xen no direct ability to control the information returned. - -### Xen Forced Emulation Prefix - -Xen-aware PV software can make use of the 'Forced Emulation Prefix' - -> `ud2a; .ascii 'xen'; cpuid` - -which Xen recognises as a deliberate attempt to get the fully-controlled -`CPUID` information rather than the hardware-reported information. This only -works with cooperative software. - -### Masking and Override MSRs - -AMD CPUs from the `K8` onwards support _Feature Override_ MSRs, which allow -direct control of the values returned for certain `CPUID` leaves. These MSRs -allow any result to be returned, including the ability to advertise features -which are not actually supported. - -Intel CPUs between `Nehalem` and `SandyBridge` have differing numbers of -_Feature Mask_ MSRs, which are a simple AND-mask applied to all `CPUID` -instructions requesting specific feature bitmap sets. The exact MSRs, and -which feature bitmap sets they affect are hardware specific. These MSRs allow -features to be hidden by clearing the appropriate bit in the mask, but does -not allow unsupported features to be advertised. - -### CPUID Faulting - -Intel CPUs from `IvyBridge` onwards have _CPUID Faulting_, which allows Xen to -cause `CPUID` instruction executed in PV guests to fault. This allows Xen -full control over all information, exactly like HVM guests. - -## Compile time - -As some features depend on other features, it is important that, when -disabling a certain feature, we disable all features which depend on it. This -allows runtime logic to be simplified, by being able to rely on testing only -the single appropriate feature, rather than the entire feature dependency -chain. - -To speed up runtime calculation of feature dependencies, the dependency chain -is calculated and flattened by `xen/tools/gen-cpuid.py` to create -`xen/include/asm-x86/cpuid-autogen.h` from -`xen/include/public/arch-x86/cpufeatureset.h`, allowing the runtime code to -disable all dependent features of a specific disabled feature in constant -time. - -## Host boot - -As Xen boots, it will enumerate the features it can see. This is stored as -the *raw_featureset*. - -Errata checks and command line arguments are then taken into account to reduce -the *raw_featureset* into the *host_featureset*, which is the set of -features Xen uses. On hardware with masking/override MSRs, the default MSR -values are picked from the *host_featureset*. - -The *host_featureset* is then used to calculate the *pv_featureset* and -*hvm_featureset*, which are the maximum featuresets Xen is willing to offer -to PV and HVM guests respectively. - -In addition, Xen will calculate how much control it has over non-cooperative -PV `CPUID` instructions, storing this information as *levelling_caps*. - -## Domain creation - -The toolstack can query each of the calculated featureset via -`XEN_SYSCTL_get_cpu_featureset`, and query for the levelling caps via -`XEN_SYSCTL_get_cpu_levelling_caps`. - -These data should be used by the toolstack when choosing the eventual -featureset to offer to the guest. - -Once a featureset has been chosen, it is set (implicitly or explicitly) via -`XEN_DOMCTL_set_cpuid`. Xen will clamp the toolstacks choice to the -appropriate PV or HVM featureset. On hardware with masking/override MSRs, the -guest cpuid policy is reflected in the MSRs, which are context switched with -other vcpu state. - -# Limitations - -A guest which ignores the provided feature information and manually probes for -features will be able to find some of them. e.g. There is no way of forcibly -preventing a guest from using 1GB superpages if the hardware supports it. - -Some information simply cannot be hidden from guests. There is no way to -control certain behaviour such as the hardware MXCSR_MASK or x87 FPU exception -behaviour. - - -# Testing - -Feature levelling is a very wide area, and used all over the hypervisor. -Please ask on xen-devel for help identifying more specific tests which could -be of use. - - -# Known issues / Areas for improvement - -The feature querying and levelling functions should exposed in a -convenient-to-use way by `xl`. - -Xen currently has no concept of per-{socket,core,thread} CPUID information. -As a result, details such as APIC IDs, topology and cache information do not -match real hardware, and do not match the documented expectations in the Intel -and AMD system manuals. - -The CPU feature flags are the only information which the toolstack has a -sensible interface for querying and levelling. Other information in the CPUID -policy is important and should be levelled (e.g. maxphysaddr). - -The CPUID policy is currently regenerated from scratch by the receiving side, -once memory and vcpu content has been restored. This means that the receiving -Xen cannot verify the memory/vcpu content against the CPUID policy, and can -end up running a guest which will subsequently crash. The CPUID policy should -be at the head of the migration stream. - -MSRs are another source of features for guests. There is no general provision -for controlling the available MSRs. E.g. 64bit versions of Windows notice -changes in IA32_MISC_ENABLE, and suffer a BSOD 0x109 (Critical Structure -Corruption) - - -# References - -[Intel Flexmigration](http://www.intel.co.uk/content/dam/www/public/us/en/documents/application-notes/virtualization-technology-flexmigration-application-note.pdf) - -[AMD Extended Migration Technology](http://developer.amd.com/wordpress/media/2012/10/43781-3.00-PUB_Live-Virtual-Machine-Migration-on-AMD-processors.pdf) - - -# History - ------------------------------------------------------------------------- -Date Revision Version Notes ----------- -------- -------- ------------------------------------------- -2016-05-31 1 Xen 4.7 Document written ----------- -------- -------- ------------------------------------------- diff --git a/docs/how-to/code-coverage.rst b/docs/how-to/code-coverage.rst new file mode 100644 index 000000000000..49c4a8ad3bda --- /dev/null +++ b/docs/how-to/code-coverage.rst @@ -0,0 +1,103 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +Code Coverage +============= + +Xen can be compiled with coverage support. When configured, Xen will record +the coverage of its own basic blocks. Being a piece of system software rather +than a userspace, it can't automatically write coverage out to the filesystem, +so some extra steps are required to collect and process the data. + +.. warning:: + + ARM doesn't currently boot when the final binary exceeds 2MB in size, + and the coverage build tends to exceed this limit. + + +Compiling Xen +------------- + +Coverage support is dependent on the compiler and toolchain used. As Xen +isn't a userspace application, it can't use the compiler supplied library, and +instead has to provide some parts of the implementation itself. + +For x86, coverage support was introduced with GCC 3.4 or later, and Clang 3.9 +or later, and Xen is compatible with these. However, the compiler internal +formats do change occasionally, and this may involve adjustments to Xen. +While we do our best to keep up with these changes, Xen may not be compatible +with bleeding edge compilers. + +To build with coverage support, enable ``CONFIG_COVERAGE`` in Kconfig. The +build system will automatically select the appropriate format based on the +compiler in use. + +The resulting binary will record its own coverage while running. + + +Accessing the raw coverage data +------------------------------- + +The ``SYSCTL_coverage_op`` hypercall is used to interact with the coverage +data. A dom0 userspace helper, ``xenconv`` is provided as well, which thinly +wraps this hypercall. + +The ``read`` subcommand can be used to obtain the raw coverage data:: + + [root@host ~]# xencov read > coverage.dat + +This is toolchain-specific data and needs to be fed back to the appropriate +programs to post-process. + +Alternatively, the ``reset`` subcommand can be used reset all counters back to +0:: + + [root@host ~]# xencov reset + + +GCC coverage +------------ + +A build using GCC's coverage will result in ``*.gcno`` artefact for every +object file. The raw coverage data needs splitting to form the matching +``*.gcda`` files. + +An example of how to view the data is as follows. It uses ``lcov`` which is a +graphical frontend to ``gcov``. + +* Obtain the raw coverage data from the test host, and pull it back to the + build working tree. +* Use ``xencov_split`` to extract the ``*.gcda`` files. Note that full build + paths are used by the tools, so splitting needs to output relative to ``/``. +* Use ``geninfo`` to post-process the raw data. +* Use ``genhtml`` to render the results as HTML. +* View the results in a browser. + +:: + + xen.git/xen$ ssh root@host xencov read > coverage.dat + xen.git/xen$ ../tools/xencov_split coverage.dat --output-dir=/ + xen.git/xen$ geninfo . -o cov.info + xen.git/xen$ genhtml cov.info -o cov/ + xen.git/xen$ $BROWSER cov/index.html + +Clang coverage +-------------- + +An example of how to view the data is as follows. + +* Obtain the raw coverage data from the test host, and pull it back to the + build working tree. +* Use ``llvm-profdata`` to post-process the raw data. +* Use ``llvm-cov show`` in combination with ``xen-syms`` from the build to + render the results as HTML. +* View the results in a browser. + +:: + + xen.git/xen$ ssh root@host xencov read > xen.profraw + xen.git/xen$ llvm-profdata merge xen.profraw -o xen.profdata + xen.git/xen$ llvm-cov show -format=html -output-dir=cov/ xen-syms -instr-profile=xen.profdata + xen.git/xen$ $BROWSER cov/index.html + +Full documentation on Clang's coverage capabilities can be found at: +https://clang.llvm.org/docs/SourceBasedCodeCoverage.html diff --git a/docs/how-to/how-xen-boots.rst b/docs/how-to/how-xen-boots.rst new file mode 100644 index 000000000000..ca77d7c8a333 --- /dev/null +++ b/docs/how-to/how-xen-boots.rst @@ -0,0 +1,101 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +How Xen Boots +============= + +This is an at-a-glance reference of Xen's booting capabilities and +expectations. + + +Build +----- + +A build of xen produces ``xen.gz`` and optionally ``xen.efi`` as final +artefacts. + + * For BIOS, Xen supports the Multiboot 1 and 2 protocols. + + * For EFI, Xen supports Multiboot 2 with EFI extensions, and native EFI64. + + * For virtualisation, Xen supports starting directly with the PVH boot + protocol. + + +Objects +~~~~~~~ + +To begin with, most object files are compiled and linked. This includes the +Multiboot 1 and 2 headers and entrypoints, including the Multiboot 2 tags for +EFI extensions. When ``CONFIG_PVH_GUEST`` is selected at build time, this +includes the PVH entrypoint and associated ELF notes. + +Depending on whether the compiler supports ``__attribute__((__ms_abi__))`` or +not, either an EFI stub is included which nops/fails applicable setup and +runtime calls, or full EFI support is included. + + +Protocols and entrypoints +~~~~~~~~~~~~~~~~~~~~~~~~~ + +All headers and tags are built in ``xen/arch/x86/boot/head.S`` + +The Multiboot 1 headers request aligned modules and memory information. Entry +is via the start of the binary image, which is the ``start`` symbol. This +entrypoint must be started in 32bit mode. + +The Multiboot 2 headers are more flexible, and in addition request that the +image be loaded as high as possible below the 4G boundary, with 2M alignment. +Entry is still via the ``start`` symbol as with MB1, and still in 32bit mode. + +Headers for the EFI MB2 extensions are also present. These request that +``ExitBootServices()`` not be called, and register ``__efi_mb2_start`` as an +alternative entrypoint, entered in 64bit mode. + +If ``CONFIG_PVH_GUEST`` was selected at build time, an Elf note is included +which indicates the ability to use the PVH boot protocol, and registers +``__pvh_start`` as the entrypoint, entered in 32bit mode. + + +xen.gz +~~~~~~ + +The objects are linked together to form ``xen-syms`` which is an ELF64 +executable with full debugging symbols. ``xen.gz`` is formed by stripping +``xen-syms``, then repackaging the result as an ELF32 object with a single +load section at 2MB, and ``gzip``-ing the result. Despite the ELF32 having a +fixed load address, its contents are relocatable. + +Any bootloader which unzips the binary and follows the ELF headers will place +it at the 2M boundary and jump to ``start`` which is the identified entry +point. However, Xen depends on being entered with the MB1 or MB2 protocols, +and will terminate otherwise. + +The MB2+EFI entrypoint depends on being entered with the MB2 protocol, and +will terminate if the entry protocol is wrong, or if EFI details aren't +provided, or if EFI Boot Services are not available. + + +xen.efi +~~~~~~~ + +When a PEI-capable toolchain is found, the objects are linked together and a +PE32+ binary is created. It can be run directly from the EFI shell, and has +``efi_start`` as its entry symbol. + +.. note:: + + xen.efi does contain all MB1/MB2/PVH tags included in the rest of the + build. However, entry via anything other than the EFI64 protocol is + unsupported, and won't work. + + +Boot +---- + +Xen, once loaded into memory, identifies its position in order to relocate +system structures. For 32bit entrypoints, this necessarily requires a call +instruction, and therefore a stack, but none of the ABIs provide one. + +Overall, given that on a BIOS-based system, the IVT and BDA occupy the first +5/16ths of the first page of RAM, with the rest free to use, Xen assumes the +top of the page is safe to use. diff --git a/docs/index.rst b/docs/index.rst index b75487a05d0b..2d785a09fad5 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,11 +1,11 @@ .. SPDX-License-Identifier: CC-BY-4.0 -The Xen Hypervisor documentation +The Xen Hypervisor Documentation ================================ .. note:: - Xen's Sphinx/RST documentation is a work in progress. The existing + Xen's Sphinx/RST documentation is a work in progress. The existing documentation can be found at https://xenbits.xen.org/docs/ Xen is an open source, bare metal hypervisor. It runs as the most privileged @@ -13,49 +13,61 @@ piece of software on the system, and shares the resources of the hardware between virtual machines. See :doc:`admin-guide/introduction` for an introduction to a Xen system. -User documentation ------------------- +.. toctree:: + :maxdepth: 4 + :caption: Introduction to Xen + :hidden: + + introduction/introduction + introduction/architecture + introduction/installation + introduction/features/index -This is documentation for an administrator of a Xen system. It is intended -for someone who is not necesserily a developer, has installed Xen from their -preferred distribution, and is attempting to run virtual machines and -configure the system. .. toctree:: - :maxdepth: 2 + :maxdepth: 4 + :caption: Tutorials + :hidden: + + tutorials/configure-networking + tutorials/reduce-latency + tutorials/running-xen-on-arm/index - admin-guide/index +.. toctree:: + :maxdepth: 4 + :caption: How-to Guides + :hidden: -Guest documentation -------------------- + how-to/how-xen-boots + how-to/code-coverage + admin-guide/microcode-loading -This documentation concerns the APIs and ABIs available to guests. It is -intended for OS developers trying to use a Xen feature, and for Xen developers -to avoid breaking things. .. toctree:: - :maxdepth: 3 + :maxdepth: 4 + :caption: References + :hidden: - guest-guide/index + reference/hypercall-abi -Hypervisor developer documentation ----------------------------------- +.. toctree:: + :caption: Glossary + :hidden: + + glossary -This is documentation for a hypervisor developer. It is intended for someone -who is building Xen from source, and is running the new hypervisor in some -kind of development environment. .. toctree:: :maxdepth: 2 + :caption: FAQ - hypervisor-guide/index - - -Miscellanea ------------ + faq/faq .. toctree:: + :maxdepth: 2 + :caption: Contributing to the Xen Project - glossary + contribute-xen/contribute-xen + contribute-xen/submit-patch diff --git a/docs/introduction/architecture.rst b/docs/introduction/architecture.rst new file mode 100644 index 000000000000..a1acb1f088eb --- /dev/null +++ b/docs/introduction/architecture.rst @@ -0,0 +1,13 @@ +**************** +Xen Architecture +**************** + +Xen is type-1 hypervisor: it runs directly on the hardware, everything else in the system is running as a virtual machine on top of Xen, including Dom0, the first virtual machine. Dom0 is created by Xen, is privileged and drives the devices on the platform. Xen virtualizes CPU, memory, interrupts and timers, providing virtual machines with one or more virtual CPUs, a fraction of the memory of the system, a virtual interrupt controller and a virtual timer. Xen assigns devices such as SATA controllers and network cards to Dom0, taking care of remapping MMIO regions and IRQs. Dom0 (typically Linux, but it could also be FreeBSD or other operating systems) runs the same device drivers for these devices that would be using on a native execution. + +Dom0 also runs a set of drivers called paravirtualized backends to give access to disk, network, etc, to the other unprivileged virtual machines. The operating system running as DomU (unprivileged guest in Xen terminology) gets access to a set of generic virtual devices by running the corresponding paravirtualized frontend drivers. A single backend services multiple frontends. A pair of paravirtualized drivers exist for all the most common classes of devices: disk, network, console, framebuffer, mouse, keyboard, etc. They usually live in the operating system kernel, i.e. Linux. A few PV backends can also run in userspace in QEMU. The frontends connect to the backends using a simple ring protocol over a shared page in memory. Xen provides all the tools for discovery and to setup the initial communication. Xen also provides a mechanism for the frontend and the backend to share additional pages and notify each other via software interrupts. + +.. image:: ./media/Xen_arch1.png + +Even though it is the most common configuration, there is no reasons to run all the device drivers and all the paravirtualized backends in Dom0. The Xen architecture allows driver domains: unprivileged virtual machines with the only purpose of running the driver and the paravirtualized backend for one class of devices. For example you can have a disk driver domain, with the SATA controller assigned, running the driver for it and the disk paravirtualized backend. You can have a network driver domain with the network card assigned, running the driver for it and the network paravirtualized backend. As driver domains are regular unprivileged guests, they make the system more secure because they allow large pieces of code, such as the entire network stack, to run unprivileged. Even if a malicious guest manages to take over the paravirtualized network backend and the network driver domain, it would not be able to take over the entire system. Driver domains also improve isolation and resilience: the network driver domain is fully isolated from the disk driver domains and Dom0. If the network driver crashes it would not be able to take down the entire system, only the network. It is possible to reboot just the network driver domains while everything else remains online. Finally driver domains allow Xen users to disaggregate and componentize the system in ways that would not be possible otherwise. For example they allow users to run a real-time operating system alongside the main OS to drive a device that has real time constraints. They allow users to run a legacy OS to drive old devices that do not have any new drivers in modern operating systems. They allow users to separate and isolate critical functionalities from less critical ones. For example they allow to run an OS such as QNX to drive most devices on the platform alongside Android for the user interface. + +.. image:: ./media/Xen_arch2.png \ No newline at end of file diff --git a/docs/features/dom0less.pandoc b/docs/introduction/features/dom0less.rst similarity index 93% rename from docs/features/dom0less.pandoc rename to docs/introduction/features/dom0less.rst index d798596cdfa2..5cf6e30bf43e 100644 --- a/docs/features/dom0less.pandoc +++ b/docs/introduction/features/dom0less.rst @@ -1,3 +1,4 @@ +======== Dom0less ======== @@ -6,8 +7,9 @@ system without an control domain (often referred to as "dom0"). Each feature can be used independently from the others, unless otherwise stated. +***************************************** Booting Multiple Domains from Device Tree ------------------------------------------ +***************************************** This feature enables Xen to create a set of DomUs at boot time. Information about the DomUs to be created by Xen is passed to the @@ -23,16 +25,18 @@ booted and the Xen tools to become available, domains created by Xen this way are started right away in parallel. Hence, their boot time is typically much shorter. - Configuration -------------- +~~~~~~~~~~~~~ -### Loading binaries into memory ### +Loading Binaries into Memory +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ U-Boot needs to load not just Xen, the device tree binary, the dom0 kernel and ramdisk. It also needs to load the kernel and ramdisk of any additional domains to boot. For example if this is the bootcmd for Xen and Dom0: +.. code-block:: + tftpb 0x1280000 xen.dtb tftpb 0x0x80000 xen-Image tftpb 0x1400000 xen.ub @@ -43,6 +47,8 @@ to boot. For example if this is the bootcmd for Xen and Dom0: If we want to add one DomU with Image-DomU as the DomU kernel and ramdisk-DomU as DomU ramdisk: +.. code-block:: + tftpb 0x1280000 xen.dtb tftpb 0x80000 xen-Image tftpb 0x1400000 xen.ub @@ -54,12 +60,15 @@ and ramdisk-DomU as DomU ramdisk: bootm 0x1400000 0x9000000 0x1280000 -### Device Tree configuration ### +Device Tree Configuration +^^^^^^^^^^^^^^^^^^^^^^^^^ In addition to loading the necessary binaries, we also need to advertise the presence of the additional VM and its configuration. It is done via device tree adding a node under /chosen as follows: +.. code-block:: + domU1 { #address-cells = <1>; #size-cells = <1>; @@ -90,7 +99,7 @@ Otherwise, they may be unusable in Xen (for instance if they are compressed). See docs/misc/arm/device-tree/booting.txt for more information. Limitations ------------ +~~~~~~~~~~~ Domains started by Xen at boot time currently have the following limitations: @@ -118,9 +127,9 @@ limitations: configured. Notes ------ +~~~~~ -- 'xl console' command will not attach to the domain's console in case +- `xl console` command will not attach to the domain's console in case of dom0less. DomU are domains created by Xen (similar to Dom0) and therefore they are all managed by Xen and some of the commands may not work. diff --git a/docs/introduction/features/feature-levelling.rst b/docs/introduction/features/feature-levelling.rst new file mode 100644 index 000000000000..3e66864ca8a8 --- /dev/null +++ b/docs/introduction/features/feature-levelling.rst @@ -0,0 +1,159 @@ +***************** +Feature Levelling +***************** + +- Status: **Supported** +- Architecture: x86 +- Components: Hypervisor, toolstack, guest + +======== +Overview +======== + +On native hardware, a kernel boots, detects features, and typically optimizes certain codepaths based on the available features, and expect the features to remain available until the kernel shuts down. The same expectation exists for virtual machines, and it is up to the hypervisor/toolstack to fulfill this expectation for the lifetime of the virtual machine, including across migrate/suspend/resume. + + +============ +User Details +============ + +Many factors affect the set of features which a VM may use: + +* The CPU itself +* The BIOS/firmware/microcode version and settings +* The hypervisor version and command line settings +* Further restrictions the toolstack chooses to apply + +A firmware or software upgrade might reduce the available set of features, for example, Intel |reg| disabling TSX in a microcode update for certain Haswell/Broadwell processors). The available set of features may also be reduced by editing the settings. + +It is unsafe to make any assumptions about features remaining consistent across a host reboot. Xen recalculates all information from scratch at each boot, and provides the information for the toolstack to consume. + +`xl` currently has no facilities to help the user collect appropriate feature information from relevant hosts and compute appropriate feature specifications for use in host or domain configurations. (`xl` being a single-host toolstack, it would in any case need external support for accessing remote hosts, for example, via SSH, in the form of automation software like GNU parallel or ansible.) + +================= +Technical Details +================= + +The `CPUID` instruction is used by softwares to query for features. In the virtualisation usecase, guest software should query Xen rather than hardware directly. However, `CPUID` is an unprivileged instruction which does not fault, complicating the task of hiding hardware features from guests. + +Important files: + +* Hypervisor + * `xen/arch/x86/cpu/*.c` + * `xen/arch/x86/cpuid.c` + * `xen/include/asm-x86/cpuid-autogen.h` + * `xen/include/public/arch-x86/cpufeatureset.h` + * `xen/tools/gen-cpuid.py` +* `libxc` + * `tools/libxc/xc_cpuid_x86.c` + + +Controlling CPUID +~~~~~~~~~~~~~~~~~ + +HVM +--- + +HVM guests (using `Intel VT-x` or `AMD SVM`) will unconditionally exit to Xen on all `CPUID` instructions, allowing Xen full control over all information. + +PV +--- + +The `CPUID` instruction is unprivileged, so executing it in a PV guest will not trap, leaving Xen no direct ability to control the information returned. + +Xen Forced Emulation Prefix +--------------------------- + +Xen-aware PV software can make use of the 'Forced Emulation Prefix' + +> `ud2a; .ascii 'xen'; cpuid` + +which Xen recognises as a deliberate attempt to get the fully-controlled `CPUID` information rather than the hardware-reported information. This only works with cooperative software. + +Masking and Override MSRs +------------------------- + +AMD CPUs from the `K8` onwards support _Feature Override_ MSRs, which allow direct control of the values returned for certain `CPUID` leaves. These MSRs allow any result to be returned, including the ability to advertise features which are not actually supported. + +Intel CPUs between `Nehalem` and `SandyBridge` have differing numbers of _Feature Mask_ MSRs, which are a simple AND-mask applied to all `CPUID` instructions requesting specific feature bitmap sets. The exact MSRs, and which feature bitmap sets they affect are hardware specific. These MSRs allow features to be hidden by clearing the appropriate bit in the mask, but does not allow unsupported features to be advertised. + +CPUID Faulting +-------------- + +Intel CPUs from `IvyBridge` onwards have _CPUID Faulting_, which allows Xen to cause `CPUID` instruction executed in PV guests to fault. This allows Xen full control over all information, exactly like HVM guests. + +Compile time +------------ + +As some features depend on other features, it is important that, when disabling a certain feature, we disable all features which depend on it. This allows runtime logic to be simplified, by being able to rely on testing only the single appropriate feature, rather than the entire feature dependency chain. + +To speed up runtime calculation of feature dependencies, the dependency chain is calculated and flattened by `xen/tools/gen-cpuid.py` to create `xen/include/asm-x86/cpuid-autogen.h` from `xen/include/public/arch-x86/cpufeatureset.h`, allowing the runtime code to disable all dependent features of a specific disabled feature in constant time. + +========= +Host Boot +========= + +As Xen boots, it will enumerate the features it can see. This is stored as the *raw_featureset*. + +Errata checks and command line arguments are then taken into account to reduce the *raw_featureset* into the *host_featureset*, which is the set of features Xen uses. On hardware with masking/override MSRs, the default MSR values are picked from the *host_featureset*. + +The *host_featureset* is then used to calculate the *pv_featureset* and *hvm_featureset*, which are the maximum featuresets Xen is willing to offer to PV and HVM guests respectively. + +In addition, Xen will calculate how much control it has over non-cooperative PV `CPUID` instructions, storing this information as *levelling_caps*. + +=============== +Domain Creation +=============== + +The toolstack can query each of the calculated featureset via `XEN_SYSCTL_get_cpu_featureset`, and query for the levelling caps via +`XEN_SYSCTL_get_cpu_levelling_caps`. + +These data should be used by the toolstack when choosing the eventual featureset to offer to the guest. + +Once a featureset has been chosen, it is set (implicitly or explicitly) via `XEN_DOMCTL_set_cpuid`. Xen will clamp the toolstacks choice to the appropriate PV or HVM featureset. On hardware with masking/override MSRs, the guest cpuid policy is reflected in the MSRs, which are context switched with other vcpu state. + +=========== +Limitations +=========== + +A guest which ignores the provided feature information and manually probes for features will be able to find some of them. e.g. There is no way of forcibly preventing a guest from using 1GB superpages if the hardware supports it. + +Some information simply cannot be hidden from guests. There is no way to control certain behaviour such as the hardware MXCSR_MASK or x87 FPU exception behaviour. + +======= +Testing +======= + +Feature levelling is a very wide area, and used all over the hypervisor. Ask on xen-devel for help identifying more specific tests which could be of use. + +==================================== +Known issues / Areas for improvement +==================================== + +The feature querying and levelling functions should exposed in a convenient-to-use way by `xl`. + +Xen currently has no concept of per-{socket,core,thread} CPUID information. As a result, details such as APIC IDs, topology and cache information do not match real hardware, and do not match the documented expectations in the Intel and AMD system manuals. + +The CPU feature flags are the only information which the toolstack has a sensible interface for querying and levelling. Other information in the CPUID policy is important and should be levelled (for example, maxphysaddr). + +The CPUID policy is currently regenerated from scratch by the receiving side, once memory and vcpu content has been restored. This means that the receiving Xen cannot verify the memory/vcpu content against the CPUID policy, and can end up running a guest which will subsequently crash. The CPUID policy should be at the head of the migration stream. + +MSRs are another source of features for guests. There is no general provision for controlling the available MSRs, for example, 64-bit versions of Windows notice changes in IA32_MISC_ENABLE, and suffer a BSOD 0x109 (Critical Structure Corruption). + + +========== +References +========== + +- `Intel Flexmigration <http://www.intel.co.uk/content/dam/www/public/us/en/documents/application-notes/virtualization-technology-flexmigration-application-note.pdf>`__ +- `AMD Extended Migration Technology <http://developer.amd.com/wordpress/media/2012/10/43781-3.00-PUB_Live-Virtual-Machine-Migration-on-AMD-processors.pdf>`__ + +========= +Changelog +========= + ++-------------+----------------------------+-------------------------------------+ +| Date | Revision Version | Notes | ++=============+============================+=====================================+ +| 2016-05-31 | Xen 4.7 | Document written | ++-------------+----------------------------+-------------------------------------+ diff --git a/docs/features/hypervisorfs.pandoc b/docs/introduction/features/hypervisorfs.rst similarity index 68% rename from docs/features/hypervisorfs.pandoc rename to docs/introduction/features/hypervisorfs.rst index a0a0ead05735..660fa17145c3 100644 --- a/docs/features/hypervisorfs.pandoc +++ b/docs/introduction/features/hypervisorfs.rst @@ -1,18 +1,14 @@ -% Hypervisor FS -% Revision 1 +************* +Hypervisor FS +************* -\clearpage +- Status: **Supported** +- Architectures: all +- Components: Hypervisor, toolstack -# Basics ----------------- --------------------- - Status: **Supported** - - Architectures: all - - Components: Hypervisor, toolstack ----------------- --------------------- - -# Overview +======== +Overview +======== The Hypervisor FS is a hierarchical name-value store for reporting information to guests, especially dom0. It is similar to the Linux @@ -20,7 +16,9 @@ kernel's sysfs. Entries and directories are created by the hypervisor, while the toolstack is able to use a hypercall to query the entry values or (if allowed by the hypervisor) to modify them. -# User details +============ +User Details +============ With: @@ -42,7 +40,9 @@ the complete Hypervisor FS entry tree can be printed. The FS paths are documented in `docs/misc/hypfs-paths.pandoc`. -# Technical details +================= +Technical Details +================= Access to the hypervisor filesystem is done via the stable new hypercall __HYPERVISOR_filesystem_op. This hypercall supports a sub-command @@ -65,28 +65,37 @@ version of the interface to use). * path documentation * `docs/misc/hypfs-paths.pandoc` -# Testing +======= +Testing +======= Any new parameters or hardware mitigations should be verified to show up correctly in the filesystem. -# Areas for improvement +Areas for Improvement +~~~~~~~~~~~~~~~~~~~~~ * More detailed access rights * Entries per domain and/or per cpupool -# Known issues +============ +Known Issues +============ * None -# References +========== +References +========== * None -# History +========= +Changelog +========= ------------------------------------------------------------------------- -Date Revision Version Notes ----------- -------- -------- ------------------------------------------- -2020-01-23 1 Xen 4.14 Document written ----------- -------- -------- ------------------------------------------- ++-------------+----------------------------+-------------------------------------+ +| Date | Revision Version | Notes | ++=============+============================+=====================================+ +| 2020-01-23 | Xen 4.14 | Document written | ++-------------+----------------------------+-------------------------------------+ diff --git a/docs/introduction/features/index.rst b/docs/introduction/features/index.rst new file mode 100644 index 000000000000..1b2bb8e1723f --- /dev/null +++ b/docs/introduction/features/index.rst @@ -0,0 +1,18 @@ +******** +Features +******** + +.. toctree:: + :maxdepth: 2 + :hidden: + + dom0less + feature-levelling + intel_psr_cat_cdp + intel_psr_mba + livepatch + migration + qemu-deprivilege + sched_credit + sched_credit2 + sched_rtds \ No newline at end of file diff --git a/docs/features/intel_psr_cat_cdp.pandoc b/docs/introduction/features/intel_psr_cat_cdp.rst similarity index 71% rename from docs/features/intel_psr_cat_cdp.pandoc rename to docs/introduction/features/intel_psr_cat_cdp.rst index 1fb02ea1e674..cd2568a79e1d 100644 --- a/docs/features/intel_psr_cat_cdp.pandoc +++ b/docs/introduction/features/intel_psr_cat_cdp.rst @@ -1,23 +1,17 @@ -% Intel Cache Allocation Technology and Code and Data Prioritization Features -% Revision 1.17 +*************************************************************************** +Intel Cache Allocation Technology and Code and Data Prioritization Features +*************************************************************************** -\clearpage +- Status: **Tech Preview** +- Architecture(s): Intel x86 +- Component(s): Hypervisor, toolstack +- Hardware: L3 CAT: Haswell and beyond CPUs + CDP : Broadwell and beyond CPUs + L2 CAT: Atom codename Goldmont and beyond CPUs -# Basics - ----------------- ---------------------------------------------------- - Status: **Tech Preview** - -Architecture(s): Intel x86 - - Component(s): Hypervisor, toolstack - - Hardware: L3 CAT: Haswell and beyond CPUs - CDP : Broadwell and beyond CPUs - L2 CAT: Atom codename Goldmont and beyond CPUs ----------------- ---------------------------------------------------- - -# Terminology +=========== +Terminology +=========== * CAT Cache Allocation Technology * CBM Capacity BitMasks @@ -27,7 +21,9 @@ Architecture(s): Intel x86 * MSRs Machine Specific Registers * PSR Intel Platform Shared Resource -# Overview +======== +Overview +======== Intel provides a set of allocation capabilities including Cache Allocatation Technology (CAT) and Code and Data Prioritization (CDP). @@ -48,14 +44,13 @@ prioritization and tuning of cache capacity to the characteristics of the workload. CDP extends CAT by providing separate code and data masks per Class of Service (COS). When SW configures to enable CDP, L3 CAT is disabled. -# User details +============ +User Details +============ -* Feature Enabling: +* Feature Enabling: Add "psr=cat" to boot line parameter to enable all supported level CAT features. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW. - Add "psr=cat" to boot line parameter to enable all supported level CAT - features. Add "psr=cdp" to enable L3 CDP but disables L3 CAT by SW. - -* xl interfaces: +* xl Interfaces: 1. `psr-cat-show [OPTIONS] domain-id`: @@ -101,17 +96,20 @@ of Service (COS). When SW configures to enable CDP, L3 CAT is disabled. Option `-a, --cat`: Show CAT/CDP hardware info. -# Technical details +================= +Technical Details +================= L3 CAT/CDP and L2 CAT are all members of Intel PSR features, they share the base PSR infrastructure in Xen. -## Hardware perspective +Hardware Perspective +~~~~~~~~~~~~~~~~~~~~ CAT/CDP defines a range of MSRs to assign different cache access patterns which are known as CBMs, each CBM is associated with a COS. -E.g. L2 CAT: +For example, L2 CAT: +----------------------------+----------------+ IA32_PQR_ASSOC | MSR (per socket) | Address | @@ -175,7 +173,8 @@ The COS\[0\] always stores the default value without change. There is a `IA32_PQR_ASSOC` register which stores the COS ID of the VCPU. HW enforces cache allocation according to the corresponding CBM. -## The relationship between L3 CAT/CDP and L2 CAT +Relationship between L3 CAT/CDP and L2 CAT +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ HW may support all features. By default, CDP is disabled on the processor. If the L3 CAT MSRs are used without enabling CDP, the processor operates in @@ -204,7 +203,8 @@ CAT is 16, when a domain is assigned 9 as COS, the L3 CAT CBM associated to COS 9 would be enforced, but for L2 CAT, the HW works as default value is set since COS 9 is beyond the max COS (8) of L2 CAT. -## Design Overview +Design Overview +~~~~~~~~~~~~~~~ * Core COS/CBM association @@ -242,7 +242,8 @@ set since COS 9 is beyond the max COS (8) of L2 CAT. is migrated from socket 1 to 2, the COS ID used is 7, that means 0x3f is the CBM to work for this domain 1 now. -## Implementation Description +Implementation Description +~~~~~~~~~~~~~~~~~~~~~~~~~~ * Hypervisor interfaces: @@ -378,7 +379,7 @@ set since COS 9 is beyond the max COS (8) of L2 CAT. * Member `feat_init` - feat_init` is a flag, to indicate whether the CPU init on a socket + `feat_init` is a flag, to indicate whether the CPU init on a socket has been done. * Member `features` @@ -404,13 +405,17 @@ set since COS 9 is beyond the max COS (8) of L2 CAT. domain_id. It is used to help restore the cos_id of the domain to 0 when a socket is offline and then online again. -# Limitations +=========== +Limitations +=========== CAT/CDP can only work on HW which enables it(check by CPUID). So far, there is no HW which enables both L2 CAT and L3 CAT/CDP. But SW implementation has considered such scenario to enable both L2 CAT and L3 CAT/CDP. -# Testing +======= +Testing +======= We can execute above xl commands to verify L2 CAT and L3 CAT/CDP on different HWs support them. @@ -432,7 +437,9 @@ For example: ID NAME CBM 1 ubuntu14 0x7f -# Areas for improvement +===================== +Areas for Improvement +===================== A hexadecimal number is used to set/show CBM for a domain now. Although this is convenient to cover overlap/isolated bitmask requirement, it is not @@ -442,71 +449,91 @@ To improve this, the libxl interfaces can be wrapped in libvirt to provide more user-friendly interfaces to user, e.g. a percentage number of the cache to set and show. -# Known issues +============ +Known Issues +============ N/A -# References - -"INTEL RESOURCE DIRECTOR TECHNOLOGY (INTEL RDT) ALLOCATION FEATURES" [Intel 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html) - -# History - ------------------------------------------------------------------------- -Date Revision Version Notes ----------- -------- -------- ------------------------------------------- -2016-08-12 1.0 Xen 4.9 Design document written - -2017-02-13 1.7 Xen 4.9 Changes: - - 1. Modify the design document to cover L3 - CAT/CDP and L2 CAT; - - 2. Fix typos; - - 3. Amend description of `feat_mask` to make - it clearer; - - 4. Other minor changes. - -2017-02-15 1.8 Xen 4.9 Changes: - - 1. Add content in 'Areas for improvement'; - - 2. Adjust revision number. - -2017-03-16 1.9 Xen 4.9 Changes: - - 1. Add 'CMT' in 'Terminology'; - - 2. Change 'feature list' to 'feature array'. - - 3. Modify data structure descriptions. - - 4. Adjust revision number. - -2017-05-03 1.11 Xen 4.9 Changes: - - 1. Modify data structure descriptions. - - 2. Adjust revision number. - -2017-07-13 1.14 Xen 4.10 Changes: - - 1. Fix a typo. - -2017-08-01 1.15 Xen 4.10 Changes: - - 1. Add 'alt_type' in 'feat_props' structure. - -2017-08-04 1.16 Xen 4.10 Changes: - - 1. Remove special character which may cause - html creation failure. - -2018-07-10 1.17 Xen 4.12 Changes: - - 1. Reformat complete document to enable PDF - creation. - ----------- -------- -------- ------------------------------------------- +========== +References +========== + +`INTEL RESOURCE DIRECTOR TECHNOLOGY (INTEL RDT) ALLOCATION FEATURES [Intel 64 and IA-32 Architectures Software Developer Manuals, vol3 <http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html>`__ + +========= +Changelog +========= + ++--------+-------+-------+--------------------------------------+ +| Date | Rev | Ve | Notes | +| | ision | rsion | | ++========+=======+=======+======================================+ +| 2016 | 1.0 | Xen | Design document written | +| -08-12 | | 4.9 | | ++--------+-------+-------+--------------------------------------+ +| 2017 | 1.7 | Xen | Changes: | +| -02-13 | | 4.9 | | ++--------+-------+-------+--------------------------------------+ +| | | | 1. Modify the design document to | +| | | | cover L3 CAT/CDP and L2 CAT; | ++--------+-------+-------+--------------------------------------+ +| | | | 2. Fix typos; | ++--------+-------+-------+--------------------------------------+ +| | | | 3. Amend description of | +| | | | ``feat_mask`` to make it clearer; | ++--------+-------+-------+--------------------------------------+ +| | | | 4. Other minor changes. | ++--------+-------+-------+--------------------------------------+ +| 2017 | 1.8 | Xen | Changes: | +| -02-15 | | 4.9 | | ++--------+-------+-------+--------------------------------------+ +| | | | 1. Add content in ‘Areas for | +| | | | improvement’; | ++--------+-------+-------+--------------------------------------+ +| | | | 2. Adjust revision number. | ++--------+-------+-------+--------------------------------------+ +| 2017 | 1.9 | Xen | Changes: | +| -03-16 | | 4.9 | | ++--------+-------+-------+--------------------------------------+ +| | | | 1. Add ‘CMT’ in ‘Terminology’; | ++--------+-------+-------+--------------------------------------+ +| | | | 2. Change ‘feature list’ to ‘feature | +| | | | array’. | ++--------+-------+-------+--------------------------------------+ +| | | | 3. Modify data structure | +| | | | descriptions. | ++--------+-------+-------+--------------------------------------+ +| | | | 4. Adjust revision number. | ++--------+-------+-------+--------------------------------------+ +| 2017 | 1.11 | Xen | Changes: | +| -05-03 | | 4.9 | | ++--------+-------+-------+--------------------------------------+ +| | | | 1. Modify data structure | +| | | | descriptions. | ++--------+-------+-------+--------------------------------------+ +| | | | 2. Adjust revision number. | ++--------+-------+-------+--------------------------------------+ +| 2017 | 1.14 | Xen | Changes: | +| -07-13 | | 4.10 | | ++--------+-------+-------+--------------------------------------+ +| | | | 1. Fix a typo. | ++--------+-------+-------+--------------------------------------+ +| 2017 | 1.15 | Xen | Changes: | +| -08-01 | | 4.10 | | ++--------+-------+-------+--------------------------------------+ +| | | | 1. Add ‘alt_type’ in ‘feat_props’ | +| | | | structure. | ++--------+-------+-------+--------------------------------------+ +| 2017 | 1.16 | Xen | Changes: | +| -08-04 | | 4.10 | | ++--------+-------+-------+--------------------------------------+ +| | | | 1. Remove special character which | +| | | | may cause html creation failure. | ++--------+-------+-------+--------------------------------------+ +| 2018 | 1.17 | Xen | Changes: | +| -07-10 | | 4.12 | | ++--------+-------+-------+--------------------------------------+ +| | | | 1. Reformat complete document to | +| | | | enable PDF creation. | ++--------+-------+-------+--------------------------------------+ diff --git a/docs/features/intel_psr_mba.pandoc b/docs/introduction/features/intel_psr_mba.rst similarity index 90% rename from docs/features/intel_psr_mba.pandoc rename to docs/introduction/features/intel_psr_mba.rst index 86df661ba886..254f82485683 100644 --- a/docs/features/intel_psr_mba.pandoc +++ b/docs/introduction/features/intel_psr_mba.rst @@ -1,21 +1,15 @@ -% Intel Memory Bandwidth Allocation (MBA) Feature -% Revision 1.8 +*********************************************** +Intel Memory Bandwidth Allocation (MBA) Feature +*********************************************** -\clearpage +- Status: **Tech Preview** +- Architecture(s): Intel x86 +- Component(s): Hypervisor, toolstack +- Hardware: MBA is supported on Skylake Server and beyond -# Basics - ----------------- ---------------------------------------------------- - Status: **Tech Preview** - -Architecture(s): Intel x86 - - Component(s): Hypervisor, toolstack - - Hardware: MBA is supported on Skylake Server and beyond ----------------- ---------------------------------------------------- - -# Terminology +=========== +Terminology +=========== * CAT Cache Allocation Technology * CBM Capacity BitMasks @@ -27,14 +21,18 @@ Architecture(s): Intel x86 * PSR Intel Platform Shared Resource * THRTL Throttle value or delay value -# Overview +======== +Overview +======== The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate control over memory bandwidth available per-core. This feature provides OS/ hypervisor the ability to slow misbehaving apps/domains by using a credit-based throttling mechanism. -# User details +============ +User Details +============ * Feature Enabling: @@ -75,12 +73,15 @@ throttling mechanism. obtained through CPUID inside hypervisor. Users can fetch the MBA_MAX value using the `psr-hwinfo` xl command. -# Technical details +================= +Technical Details +================= MBA is a member of Intel PSR features, it shares the base PSR infrastructure in Xen. -## Hardware perspective +Hardware Perspective +~~~~~~~~~~~~~~~~~~~~ MBA defines a range of MSRs to support specifying a delay value (Thrtl) per COS, with details below. @@ -101,7 +102,8 @@ in Xen. thread MSR `IA32_PQR_ASSOC`, and then hardware enforces bandwidth allocation according to the throttling value stored in the Thrtl MSR register. -## The relationship between MBA and CAT/CDP +Relationship between MBA and CAT/CDP +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Generally speaking, MBA is completely independent of CAT/CDP, and any combination may be applied at any time, e.g. enabling MBA with CAT @@ -117,7 +119,8 @@ in Xen. the HW works as default value is set since COS 9 is beyond the max COS (8) of MBA. -## Design Overview +Design Overview +~~~~~~~~~~~~~~~ * Core COS/Thrtl association @@ -145,7 +148,8 @@ in Xen. This part reuses CAT COS infrastructure. -## Implementation Description +Implementation Description +~~~~~~~~~~~~~~~~~~~~~~~~~~ * Hypervisor interfaces: @@ -197,11 +201,15 @@ in Xen. property for all features. So, for other data structure details, please refer to 'intel_psr_cat_cdp.pandoc'. -# Limitations +=========== +Limitations +=========== MBA can only work on HW which supports it (check CPUID). -# Testing +======= +Testing +======= We can execute these commands to verify MBA on different HWs supporting them. @@ -232,19 +240,27 @@ For example: ID NAME THRTL 1 ubuntu14 10 -# Areas for improvement +===================== +Areas for Improvement +===================== N/A -# Known issues +============ +Known Issues +============ N/A -# References +========== +References +========== -"INTEL RESOURCE DIRECTOR TECHNOLOGY (INTEL RDT) ALLOCATION FEATURES" [Intel 64 and IA-32 Architectures Software Developer Manuals, vol3](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html) +`INTEL RESOURCE DIRECTOR TECHNOLOGY (INTEL RDT) ALLOCATION FEATURES [Intel 64 and IA-32 Architectures Software Developer Manuals, vol3]<http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html>`__ -# History +========= +Changelog +========= ------------------------------------------------------------------------ Date Revision Version Notes diff --git a/docs/features/livepatch.pandoc b/docs/introduction/features/livepatch.rst similarity index 89% rename from docs/features/livepatch.pandoc rename to docs/introduction/features/livepatch.rst index 17f1cd0d0597..4da28ea4d3f2 100644 --- a/docs/features/livepatch.pandoc +++ b/docs/introduction/features/livepatch.rst @@ -1,24 +1,16 @@ -% Live Patching -% Revision 1 +************* +Live Patching +************* -\clearpage +- Status: **Supported** +- Architecture: x86 +- Status: **Tech Preview/Experimental** +- Architecture: ARM +- Component(s): Hypervisor, toolstack -# Basics - ----------------- ---------------------------------------------------- - Status: **Supported** - - Architecture: x86 - - Status: **Tech Preview/Experimental** - - Architecture: ARM - - Component: Hypervisor, toolstack ----------------- ---------------------------------------------------- - - -# Details +======= +Details +======= Xen Live Patching has been available as tech preview feature since Xen 4.7 and has now had a couple of releases to stabilize. Xen Live patching @@ -72,7 +64,7 @@ worth detailing the scope of security support: of a live patch despite pausing it (xl pause ...), it shall be treated as a security issue. -Note: It is expected that live patches are tested in a test environment +.. note:: It is expected that live patches are tested in a test environment before being used in production to avoid unexpected issues. In particular, to avoid the issues described by (3), (4), & (5). diff --git a/docs/features/migration.pandoc b/docs/introduction/features/migration.rst similarity index 65% rename from docs/features/migration.pandoc rename to docs/introduction/features/migration.rst index 719925818e25..1942b0158cc5 100644 --- a/docs/features/migration.pandoc +++ b/docs/introduction/features/migration.rst @@ -1,31 +1,29 @@ -% Migration -% Revision 2 +********* +Migration +********* -\clearpage +- Status: **Supported** +- Architecture: x86 +- Component: Toolstack -# Basics ---------------- ------------- - Status: **Supported** - - Architecture: x86 - - Component: Toolstack ---------------- ------------- - -# Overview +======== +Overview +======== Migration is a mechanism to move a virtual machine while the VM is running. Live migration moves a running virtual machine between two physical servers, but the same mechanism can be used for non-live migration (pause and copy) and suspend/resume from disk. -# User details +============ +User Details +============ No hardware requirements, although hypervisor logdirty support is required for live migration. From the command line, `xl migrate/save/restore` are the top level -interactions. e.g. +interactions. For example: xl create my-vm.cfg xl migrate my-vm localhost @@ -39,9 +37,11 @@ or Xen 4.6 sees the introduction of Migration v2. There is no change for people using `xl`, although the `libxl` API has had an extension. -# Technical details +================= +Technical Details +================= -Migration is formed of several layers. `libxc` is responsible for the +Migration is formed of several layers. `libxc` is responsible for the contents of the VM (ram, vcpus, etc) and the live migration loop, while `libxl` is responsible for items such as emulator state. @@ -63,14 +63,17 @@ legacy stream into a migration v2 stream. * `tools/python/scripts/convert-legacy-stream` * `tools/python/scripts/verify-stream-v2` -## libxl +libxl +~~~~~ With migration v2 support, LIBXL_HAVE_SRM_V2 and LIBXL_HAVE_SRM_V1 are introduced to indicate support. `domain_restore_params` gains a new parameter, `stream_version`, which is used to distinguish between legacy and v2 migration streams, and hence whether legacy conversion is required. -# Limitations +=========== +Limitations +=========== Hypervisor logdirty support is incompatible with hardware passthrough, as IOMMU faults cannot be used to track writes. @@ -80,24 +83,29 @@ changes in cpuid information, and cpuid levelling support currently has its issues. Extreme care should be taken when migrating VMs between non-identical CPUs until the cpuid levelling improvements are complete. -# Testing +======= +Testing +======= Changes in libxc should be tested with every guest type (32bit PV, 64bit PV, HVM), while changes in libxl should test HVM guests with both qemu-traditional and qemu-upstream. -In general, testing can be done on a single host using `xl -save/restore` or `xl migrate $VM localhost`. +In general, testing can be done on a single host using `xl save/restore` or `xl migrate $VM localhost`. Any changes to the conversion script should be tested in all upgrade scenarios, which will involve starting with VMs from Xen 4.5 -# Areas for improvement +===================== +Areas for Improvement +===================== * Arm support * Live looping parameters -# Known issues +============ +Known Issues +============ * x86 HVM guest physmap operations (not reflected in logdirty bitmap) * x86 HVM with PoD pages (attempts to map cause PoD allocations) @@ -107,18 +115,24 @@ scenarios, which will involve starting with VMs from Xen 4.5 * x86 PV P2M structure changes (not noticed, stale mappings used) for guests not using the linear p2m layout -# References - -Xen Developer Summit 2015 Presentation -[video](https://www.youtube.com/watch?v=RwiDeG21lrc) and -[slides](http://events.linuxfoundation.org/sites/events/files/slides/migv2.pdf) -for Migration v2 - -# History - ------------------------------------------------------------------------- -Date Revision Version Notes ----------- -------- -------- ------------------------------------------- -2015-10-24 1 Xen 4.6 Document written -2015-12-11 2 Xen 4.7 Support of linear p2m list ----------- -------- -------- ------------------------------------------- +========== +References +========== + +- `Xen Developer Summit 2015 Presentation [video] <https://www.youtube.com/watch?v=RwiDeG21lrc>`__ +- `Xen Developer Summit 2015 Presentation [slides](http://events.linuxfoundation.org/sites/events/files/slides/migv2.pdf)>`__ + +========= +Changelog +========= + ++--------+-------+-------+--------------------------------------+ +| Date | Rev | Ve | Notes | +| | ision | rsion | | ++========+=======+=======+======================================+ +| 2015 | 1 | Xen | Document written | +| -10-24 | | 4.6 | | ++--------+-------+-------+--------------------------------------+ +| 2015 | 2 | Xen | Support of linear p2m list | +| -12-11 | | 4.7 | | ++--------+-------+-------+--------------------------------------+ diff --git a/docs/features/qemu-deprivilege.pandoc b/docs/introduction/features/qemu-deprivilege.rst similarity index 75% rename from docs/features/qemu-deprivilege.pandoc rename to docs/introduction/features/qemu-deprivilege.rst index 4ef119c8214a..511dc3a3c2b9 100644 --- a/docs/features/qemu-deprivilege.pandoc +++ b/docs/introduction/features/qemu-deprivilege.rst @@ -1,20 +1,14 @@ -% QEMU Deprivileging / dm_restrict -% Revision 1 +******************************** +QEMU Deprivileging / dm_restrict +******************************** -\clearpage +- Status: **Tech Preview** +- Architecture(s): x86 +- Component(s): toolstack -# Basics - ----------------- ---------------------------------------------------- - Status: **Tech Preview** - -Architecture(s): x86 - - Component(s): toolstack - ----------------- ---------------------------------------------------- - -# Overview +======== +Overview +======== By default, the QEMU device model is run in domain 0. If an attacker can gain control of a QEMU process, it could easily take control of a @@ -27,15 +21,18 @@ dm_restrict is a set of operations to restrict QEMU running in domain domain 2. Mechanisms to restruct QEMU's ability to interact with domain 0. -# User details - -## Getting the right versions of software +============ +User Details +============ -Linux: 4.11+ +Getting the Right Versions of Software +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Qemu: 3.0+ (Or the version that comes with Xen 4.12+) +- Linux: 4.11+ +- Qemu: 3.0+ (Or the version that comes with Xen 4.12+) -## Setting up a group and userid range +Setting up a Group and UserID Range +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For maximum security, libxl needs to run the devicemodel for each domain under a user id (UID) corresponding to its domain id. There @@ -64,6 +61,7 @@ interpreted by such systems as a single container ID. Another, less-secure way is to run all QEMUs as the same UID. To do this, create a user named `xen-qemuuser-shared`; for example: +:: adduser --no-create-home --system xen-qemuuser-shared A final way to set up a separate process for qemus is to allocate one @@ -71,18 +69,21 @@ UID per VM, and set the UID in the domain config file with the `device_model_user` argument. For example, suppose you have a VM named `c6-01`. You might do the following: +:: adduser --system --no-create-home --group xen-qemuuser-c6-01 And then in your config file, the following line: +:: device_model_user="xen-qemuuser-c6-01" If you use this method, you should also allocate one "reaper" user to be used for killing device models: +:: adduser --system --no-create-home --group xen-qemuuser-reaper -NOTE: It is important when using `device_model_user` that EACH VM HAVE +.. note:: It is important when using `device_model_user` that EACH VM HAVE A SEPARATE UID, and that none of these UIDs map to root. xl will throw an error a uid maps to zero, but not if multiple VMs have the same uid. Multiple VMs with the same device model uid will cause @@ -92,7 +93,8 @@ It is also important that `xen-qemuuser-reaper` not have any processes associated with it, as they will be destroyed when deprivileged qemu processes are destroyed. -## Domain config changes +Domain Config Changes +~~~~~~~~~~~~~~~~~~~~~ The core domain config change is to add the following line to the domain configuration: @@ -102,11 +104,15 @@ domain configuration: This will perform a number of restrictions, outlined below in the 'Technical details' section. -# Technical details +================= +Technical Details +================= See docs/design/qemu-deprivilege.md for technical details. -# Limitations +=========== +Limitations +=========== The following features still need to be implemented: @@ -127,10 +133,14 @@ implemented at some point but is not a near-term priority. See SUPPORT.md for security support status. -# History - ------------------------------------------------------------------------- -Date Revision Version Notes ----------- -------- -------- ------------------------------------------- -2018-09-14 1 Xen 4.12 Imported from docs/misc ----------- -------- -------- ------------------------------------------- +========= +Changelog +========= + ++--------+-------+-------+--------------------------------------+ +| Date | Rev | Ve | Notes | +| | ision | rsion | | ++========+=======+=======+======================================+ +| 2018 | 1 | Xen | Imported from docs/misc | +| -09-14 | | 4.12 | | ++--------+-------+-------+--------------------------------------+ diff --git a/docs/features/sched_credit.pandoc b/docs/introduction/features/sched_credit.rst similarity index 58% rename from docs/features/sched_credit.pandoc rename to docs/introduction/features/sched_credit.rst index 706513de6e44..67d190b6f679 100644 --- a/docs/features/sched_credit.pandoc +++ b/docs/introduction/features/sched_credit.rst @@ -1,23 +1,22 @@ -% Credit Scheduler -% Revision 1 +**************** +Credit Scheduler +**************** -\clearpage +- Status: **Supported** +- Component: Hypervisor -# Basics ----------------- ---------------------------------------------------- - Status: **Supported** - - Component: Hypervisor ----------------- ---------------------------------------------------- - -# Overview +======== +Overview +======== Credit (also known as Credit1) is the old virtual CPU (vCPU) scheduler of the Xen hypervisor. It is a general purpose, weighted fair-share scheduler. -# User details +============ +User Details +============ Xen supports multiple schedulers. Credit is no longer the default. In order to use it as the Xen scheduler the following parameter should be @@ -42,13 +41,17 @@ Two kind of interactions with the scheduler are possible: * `xl sched-credit -d vm1` * `xl sched-credit -d vm1 -w 512` -# Technical details +================= +Technical Details +================= Implementation entirely lives in the hypervisor. Xen has a pluggable, hook based, architecture for schedulers. Thanks to this, Credit code is all contained in `xen/common/sched_credit.c`. -# Limitations +=========== +Limitations +=========== In Credit, a vCPU has a priority, a status (i.e., active or inactive), a weight and some credits... and all these things interact in a rather @@ -65,7 +68,9 @@ with mixed workloads and VMs with low latency requirements are other known problems. [4] For all these reasons, effort is ongoing to have Credit2 become the new default scheduler. -# Testing +======= +Testing +======= Any change to Credit code must to be tested by doing at least the following: @@ -80,21 +85,29 @@ by also doing the following: * move the virtual machines between cpupools. -# References - -* [potential non-ideal behavior on hyperthreaded systems](https://lists.xenproject.org/archives/html/xen-devel/2014-07/msg01848.html) [1] -* [long standing BOOST vs. migration bug](https://lists.xen.org/archives/html/xen-devel/2015-10/msg02851.html) [2] -* [priority handling issues](https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg01362.html) [3] -* "Scheduler development update", XenSummit Asia 2009 [whitepaper](http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf) [4] -* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf) -* "The Credit Scheduler" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Credit_Scheduler) -* "Xen Project Schedulers" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers) - -# History - ------------------------------------------------------------------------- -Date Revision Version Notes ----------- -------- -------- ------------------------------------------- -2016-10-14 1 Xen 4.8 Document written -2019-02-7 3 Xen 4.12 No longer default scheduler ----------- -------- -------- ------------------------------------------- +========== +References +========== + +- `Potential non-ideal behavior on hyperthreaded systems <https://lists.xenproject.org/archives/html/xen-devel/2014-07/msg01848.html>`__ +- `Long standing BOOST vs. migration bug <https://lists.xen.org/archives/html/xen-devel/2015-10/msg02851.html>`__ +- `Priority handling issues <https://lists.xenproject.org/archives/html/xen-devel/2016-05/msg01362.html>`__ +- `"Scheduler development update", XenSummit Asia 2009 [whitepaper] <http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf>`__ +- `"Scheduling in Xen" [XPDS15 Presentation] <http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf>`__ +- `"The Credit Scheduler" [on the Xen-Project wiki] <https://wiki.xenproject.org/wiki/Credit_Scheduler>`__ +- `"Xen Project Schedulers" [on the Xen-Project wiki] <https://wiki.xenproject.org/wiki/Xen_Project_Schedulers>`__ + +========= +Changelog +========= + ++--------+-------+-------+--------------------------------------+ +| Date | Rev | Ve | Notes | +| | ision | rsion | | ++========+=======+=======+======================================+ +| 2016 | 1 | Xen | Document written | +| -10-14 | | 4.8 | | ++--------+-------+-------+--------------------------------------+ +| 201 | 3 | Xen | No longer default scheduler | +| 9-02-7 | | 4.12 | | ++--------+-------+-------+--------------------------------------+ diff --git a/docs/features/sched_credit2.pandoc b/docs/introduction/features/sched_credit2.rst similarity index 58% rename from docs/features/sched_credit2.pandoc rename to docs/introduction/features/sched_credit2.rst index 436ff9f8f6ad..bac0cf139bc0 100644 --- a/docs/features/sched_credit2.pandoc +++ b/docs/introduction/features/sched_credit2.rst @@ -1,16 +1,13 @@ -% Credit2 Scheduler -% Revision 2 +***************** +Credit2 Scheduler +***************** -\clearpage +- Status: **Supported** +- Component: Hypervisor -# Basics ----------------- ---------------------------------------------------- - Status: **Supported** - - Component: Hypervisor ----------------- ---------------------------------------------------- - -# Overview +======== +Overview +======== Credit2 is the default virtual CPU (vCPU) scheduler available in the Xen hypervisor. @@ -19,7 +16,9 @@ Credit2 was designed as a general purpose scheduler, with particular focus on improving handling of mixed workloads, scalability and support for low latency applications inside VMs. -# User details +============ +User Details +============ Xen supports multiple schedulers. As said, Credit2 is the default, so it is used automatically, unless the `sched=$SCHED` (with `$SCHED` @@ -47,7 +46,9 @@ Two kind of interactions with the scheduler are possible: * `xl sched-credit2 -d vm1` * `xl sched-credit2 -d vm1 -w 1024` -# Technical details +================= +Technical Details +================= Implementation entirely lives in the hypervisor. Xen has a pluggable, hook based, architecture for schedulers. Thanks to this, Credit2 code @@ -58,7 +59,9 @@ limiting, is only available from Xen 4.8 onward. In libxl, the LIBXL_HAVE_SCHED_CREDIT2_PARAMS symbol is introduced to indicate their availability. -# Testing +======= +Testing +======= Any change done in Credit2 wants to be tested by doing at least the following: @@ -83,26 +86,38 @@ following: parameters (`sched_credit2_migrate_resist`, `credit2_balance_over`, `credit2_balance_under`) -# Known issues +============ +Known Issues +============ * I/O oriented benchmarks (like network and disk throughput) have given contradictory and non-conclusive results so far. Need to run more of those. -# References - -* "Scheduler development update", XenSummit Asia 2009 [whitepaper](http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf) -* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf) -* "Scope and Performance of Credit-2 Scheduler" [XPDS16 Presentation](http://www.slideshare.net/xen_com_mgr/xpds16-scope-and-performance-of-credit2-scheduler-anshul-makkar-ctirix-systems-uk-ltd) -* "The Credit2 Scheduler" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Credit2_Scheduler_Development) -* "Xen Project Schedulers" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers) - -# History - ------------------------------------------------------------------------- -Date Revision Version Notes ----------- -------- -------- ------------------------------------------- -2016-10-14 1 Xen 4.8 Document written -2017-11-6 2 Xen 4.10 Soft-affinity and caps implemented -2019-02-7 3 Xen 4.12 Made the default scheduler ----------- -------- -------- ------------------------------------------- +========== +References +========== + +* `"Scheduler development update", XenSummit Asia 2009 [whitepaper] <http://www-archive.xenproject.org/files/xensummit_intel09/George_Dunlap.pdf>`__ +* `"Scheduling in Xen" [XPDS15 Presentation] <http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf>`__ +* `"Scope and Performance of Credit-2 Scheduler" [XPDS16 Presentation] <http://www.slideshare.net/xen_com_mgr/xpds16-scope-and-performance-of-credit2-scheduler-anshul-makkar-ctirix-systems-uk-ltd>`__ +* `"The Credit2 Scheduler" [on the Xen-Project wiki] <https://wiki.xenproject.org/wiki/Credit2_Scheduler_Development>`__ +* `"Xen Project Schedulers" [on the Xen-Project wiki] <https://wiki.xenproject.org/wiki/Xen_Project_Schedulers>`__ + +========= +Changelog +========= + ++--------+-------+-------+--------------------------------------+ +| Date | Rev | Ve | Notes | +| | ision | rsion | | ++========+=======+=======+======================================+ +| 2016 | 1 | Xen | Document written | +| -10-14 | | 4.8 | | ++--------+-------+-------+--------------------------------------+ +| 201 | 2 | Xen | Soft-affinity and caps implemented | +| 7-11-6 | | 4.10 | | ++--------+-------+-------+--------------------------------------+ +| 201 | 3 | Xen | Made the default scheduler | +| 9-02-7 | | 4.12 | | ++--------+-------+-------+--------------------------------------+ diff --git a/docs/features/sched_rtds.pandoc b/docs/introduction/features/sched_rtds.rst similarity index 65% rename from docs/features/sched_rtds.pandoc rename to docs/introduction/features/sched_rtds.rst index 4ccf704b53d0..422084f61f7b 100644 --- a/docs/features/sched_rtds.pandoc +++ b/docs/introduction/features/sched_rtds.rst @@ -1,16 +1,13 @@ -% RTDS Scheduler -% Revision 1 +************** +RTDS Scheduler +************** -\clearpage +- Status: **Experimental** +- Component: Hypervisor -# Basics ----------------- ---------------------------------------------------- - Status: **Experimental** - - Component: Hypervisor ----------------- ---------------------------------------------------- - -# Overview +======== +Overview +======== RTDS is one of the virtual CPU (vCPU) scheduler available in the Xen hypervisor. @@ -19,7 +16,9 @@ RTDS is a real-time scheduler, so its purpose is enabling **deterministic** scheduling of the virtual machine's vCPUs. It has been originally developed in the context of the RT-Xen project. -# User details +============ +User Details +============ RTDS is not in use by default. In order to use it as the Xen scheduler the following parameter should be passed to the hypervisor at boot: @@ -42,7 +41,9 @@ It is possible, for a multiple vCPUs VM, to change the parameters of each vCPU individually: * `xl sched-rtds -d vm-rt -v 0 -p 20000 -b 10000 -e 1 -v 1 -p 45000 -b 12000 -e 0` -# Technical details +================= +Technical Details +================= Implementation entirely lives in the hypervisor. Xen has a pluggable, hook based, architecture for schedulers. Thanks to this, RTDS code @@ -56,7 +57,9 @@ introduced later, and is available if the following symbols are defined: * LIBXL_HAVE_SCHED_RTDS_VCPU_PARAMS, * LIBXL_HAVE_SCHED_RTDS_VCPU_EXTRA. -# Limitations +=========== +Limitations +=========== RTDS is a special purpose scheduling. This is by design, and not at all a limitation, but it is certainly something to keep in mind when @@ -70,7 +73,9 @@ is definitely possible, but the vCPU scheduling parameters (of both Domain0 and of the various VMs) would probably require tweaking, with respect to their default values. -# Testing +======= +Testing +======= Any change done in RTDS must be tested by doing the following: @@ -94,29 +99,41 @@ at a macroscopic level), the following should be done: * check with `xentop` (in Domain0) that the VM is getting no more than 50% pCPU time. -# Areas for improvement +===================== +Areas for Improvement +===================== * performance assessment, especially focusing on what level of real-time behavior the scheduler enables. -# Known issues +============ +Known Issues +============ * OSSTest reports occasional failures on ARM. -# References - -* "RT-Xen: Real-Time Virtualization" [XPDS14 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/2014_Xen_Developer_Summit_0.pdf) -* "Scheduling in Xen" [XPDS15 Presentation](http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf) -* [RT-Xen Project](https://sites.google.com/site/realtimexen/) -* [RTDS-Based-Scheduler](https://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) -* "The RTDS Scheduler" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/RTDS-Based-Scheduler) -* "Xen Project Schedulers" [on the Xen-Project wiki](https://wiki.xenproject.org/wiki/Xen_Project_Schedulers) - -# History - ------------------------------------------------------------------------- -Date Revision Version Notes ----------- -------- -------- ------------------------------------------- -2016-10-14 1 Xen 4.8 Document written -2017-08-31 2 Xen 4.10 Revise for work conserving feature ----------- -------- -------- ------------------------------------------- +========== +References +========== + +* `"RT-Xen: Real-Time Virtualization" [XPDS14 Presentation] <http://events.linuxfoundation.org/sites/events/files/slides/2014_Xen_Developer_Summit_0.pdf>`__ +* `"Scheduling in Xen" [XPDS15 Presentation] <http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf>`__ +* `[RT-Xen Project] <https://sites.google.com/site/realtimexen/>`__ +* `[RTDS-Based-Scheduler] <https://wiki.xenproject.org/wiki/RTDS-Based-Scheduler>`__ +* `"The RTDS Scheduler" [on the Xen-Project wiki] <https://wiki.xenproject.org/wiki/RTDS-Based-Scheduler>`__ +* `"Xen Project Schedulers" [on the Xen-Project wiki] <https://wiki.xenproject.org/wiki/Xen_Project_Schedulers>`__ + +========= +Changelog +========= + ++--------+-------+-------+--------------------------------------+ +| Date | Rev | Ve | Notes | +| | ision | rsion | | ++========+=======+=======+======================================+ +| 2016 | 1 | Xen | Document written | +| -10-14 | | 4.8 | | ++--------+-------+-------+--------------------------------------+ +| 2017 | 2 | Xen | Revise for work conserving feature | +| -08-31 | | 4.10 | | ++--------+-------+-------+--------------------------------------+ diff --git a/docs/features/template.pandoc b/docs/introduction/features/template.pandoc similarity index 100% rename from docs/features/template.pandoc rename to docs/introduction/features/template.pandoc diff --git a/docs/introduction/installation.rst b/docs/introduction/installation.rst new file mode 100644 index 000000000000..14b598fc9f57 --- /dev/null +++ b/docs/introduction/installation.rst @@ -0,0 +1,331 @@ +************ +Installation +************ + +============= +Prerequisites +============= + + + +============================================= +Installing the Xen Project Software on Distro +============================================= + +The Debian Xen Project packages consist primarily of a Xen Project-enabled Linux kernel, the hypervisor itself, a modified version of QEMU that support the hypervisor’s HVM mode and a set of userland tools. + +All of this can be installed via an Apt meta-package called xen-linux-system. A meta-package is basically a way of installing a group of packages automatically. Apt will of course resolve all dependencies and bring in all the extra libraries we need. + +Let's install the xen-linux-system meta-package: + +.. code-block:: + + apt-get install xen-system-amd64 + +Now we have a Xen Project hypervisor, a Xen Project kernel and the userland tools installed. When you next boot the system, the boot menu should include entries for starting Debian with the Xen hypervisor. One of them should be highlighted, to start Xen by default. Do that now, logging in as root again. + +Next, let's check to see if virtualization is enabled in the BIOS. There are a few ways to do that. + +The most comprehensive is to review the Xen section of dmesg created during the boot process. This will be your first use of xl, the very versatile Xen tool, which we will come back to shortly to create and manage domUs: + +.. code-block:: + + xl dmesg + +Included in the output will be references to the CPU flags set in the BIOS to enable virtualization: 'vmx' for Intel, 'svm' for AMD. It will also detail other hardware virtualization extensions: VT-d features, Hardware Assisted Paging (HAP), I/O Virtualization and so on. + +Another way is to check the flags set in the CPU on boot: + +.. code-block:: + + egrep '(vmx|svm|hypervisor)' /proc/cpuinfo + +egrep will return any line containing one or more of those same text fragments (vmx/svm or more recently, just 'hypervisor'). If nothing comes back and you think it should, you may wish to look through the flags yourself: + +.. code-block:: + + cat /proc/cpuinfo + +If the virtualization extensions don't appear, take a closer look at the BIOS settings. A few round-trips through the BIOS are often required to get all the bits working right. + +====================================== +Installing the Xen Project from Source +====================================== + +Prerequisites +~~~~~~~~~~~~~ + +A number of prerequisites are required for building a Xen source release. Ensure that you have all the following installed, either by visiting the project webpage or installing a pre-built package provided by your OS distributor: + + * GNU Make v3.80 or later + * C compiler and linker: + - For x86: + - GCC 4.1.2_20070115 or later + - GNU Binutils 2.16.91.0.5 or later + or + - Clang/LLVM 3.5 or later + - For ARM: + - GCC 4.8 or later + - GNU Binutils 2.24 or later + * Development install of zlib (e.g., zlib-dev) + * Development install of Python 2.6 or later (e.g., python-dev) + * Development install of curses (e.g., libncurses-dev) + * Development install of openssl (e.g., openssl-dev) + * Development install of x11 (e.g. xorg-x11-dev) + * Development install of uuid (e.g. uuid-dev) + * Development install of yajl (e.g. libyajl-dev) + * Development install of libaio (e.g. libaio-dev) version 0.3.107 or greater. + * Development install of GLib v2.0 (e.g. libglib2.0-dev) + * Development install of Pixman (e.g. libpixman-1-dev) + * pkg-config + * bridge-utils package (/sbin/brctl) + * iproute package (/sbin/ip) + * GNU bison and GNU flex + * GNU gettext + * ACPI ASL compiler (iasl) + +In addition to the above, there are a number of optional build prerequisites. Omitting these will cause the related features to be disabled at compile time: + + * Development install of Ocaml (e.g. ocaml-nox and ocaml-findlib). Required to build ocaml components which includes the alternative ocaml xenstored. + * cmake (if building vtpm stub domains) + * pandoc, transfig, pod2{man,html,text} for rendering various pieces of documentation into alternative formats + * figlet (for generating the traditional Xen start of day banner) + * systemd daemon development files + * Development install of libnl3 (e.g., libnl-3-200, libnl-3-dev, etc). Required if network buffering is desired when using Remus with libxl. See docs/README.remus for detailed information. + * 16-bit x86 assembler, loader and compiler for qemu-traditional / rombios (dev86 rpm or bin86 & bcc debs) + * Development install of liblzma for rombios + * Development install of libbz2, liblzma, liblzo2, and libzstd for DomU kernel decompression. + +.. note:: You need to acquire a suitable kernel for use in domain 0. If possible, use a kernel provided by your OS distributor. If no suitable kernel is available from your OS distributor then refer to `Xen Dom0 Kernels <https://wiki.xen.org/wiki/XenDom0Kernels>`__ for suggestions for suitable kernels to use. If you are looking to compile a Dom0 kernel from source, see `XenParavirtOps <https://wiki.xen.org/wiki XenParavirtOps>`__. + +Building and Installing Xen +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. important:: Unless noted otherwise, all the following steps should be +performed with root privileges. + +1. Download and untar the source tarball file. This will be a + file named xen-unstable-src.tgz, or xen-$version-src.tgz. + You can also pull the current version from the git or mercurial + repositories at `https://xenbits.xen.org/ <https://xenbits.xen.org/>`__. + + .. code-block:: + + # tar xzf xen-unstable-src.tgz + + Assuming you are using the unstable tree, this will untar into xen-unstable. The rest of the instructions + use the unstable tree as an example, substitute the version for unstable. + +2. cd to xen-unstable (or, whatever you have renamed it to). + +3. For the very first build, or if you want to destroy build trees, perform the following steps: + + .. code-block:: + + # ./configure + # make world + # make install + + See the documentation in the INSTALL file for more information. + + This will create and install onto the local machine. It will build + the xen binary (xen.gz), the tools and the documentation. + + You can override the destination for make install by setting DESTDIR + to some value. + +4. To rebuild an existing tree without modifying the config: + + .. code-block:: + + # make dist + + This will build and install xen, tools, and docs into the local dist/ + directory. + + You can override the destination for make install by setting DISTDIR + to some value. + + make install and make dist differ in that make install does the + right things for your local machine (installing the appropriate + version of udev scripts, for example), but make dist includes all + versions of those scripts, so that you can copy the dist directory + to another machine and install from that distribution. + +xenstore: xenstored and oxenstored +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Xen uses a configuration database called xenstore [0] to maintain configuration and status information shared between domains. A daemon is implemented as part of xenstore to act as an interface for access to the database for dom0 and guests. Two xenstored daemons are supported, one written in C which we refer +to as the xenstored (sometimes referred to as cxenstored), and another written in Ocaml called oxenstored. Details for xenstore and the different implementations can be found on the wiki's xenstore reference guide [1] and the xenstored [2] page. You can choose which xenstore you want to enable as default on a system through configure: + + ./configure --with-xenstored=xenstored + ./configure --with-xenstored=oxenstored + +By default oxenstored will be used if the ocaml development tools are found. +If you enable oxenstored the xenstored will still be built and installed, +the xenstored used can be changed through the configuration file: + +/etc/sysconfig/xencommons +or +/etc/default/xencommons + +You can change the preferred xenstored you want to use in the configuration +but since we cannot stop the daemon a reboot will be required to make the +change take effect. + +[0] https://wiki.xen.org/wiki/XenStore +[1] https://wiki.xen.org/wiki/XenStoreReference +[2] https://wiki.xen.org/wiki/Xenstored + +Python Runtime Libraries +~~~~~~~~~~~~~~~~~~~~~~~~ + +Various tools, such as pygrub, have the following runtime dependencies: + + * Python 2.6 or later. + URL: http://www.python.org/ + Debian: python + +Note that the build system expects `python` to be available. If your system only has `python2` or `python3` but not `python` (as in Linux From Scratch), you will need to create a symlink for it, or specify PYTHON= when invoking make, like (note the position of PYTHON= matters): + + # make PYTHON=/usr/bin/python3 + +Intel(R) Trusted Execution Technology Support +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Intel's technology for safer computing, Intel(R) Trusted Execution Technology (Intel(R) TXT), defines platform-level enhancements that provide the building blocks for creating trusted platforms. For more information, see `http://www.intel.com/technology/security/ <http://www.intel.com/technology/security/>`__. + +Intel(R) TXT support is provided by the Trusted Boot (tboot) module in conjunction with minimal logic in the Xen hypervisor. + +Tboot is an open source, pre- kernel/VMM module that uses Intel(R) TXT to perform a measured and verified launch of an OS kernel/VMM. + +The Trusted Boot module is available from `http://sourceforge.net/projects/tboot <http://sourceforge.net/projects/tboot>`__. This project hosts the code in a mercurial repo at `http://tboot.sourceforge.net/hg/tboot.hg <http://tboot.sourceforge.net/hg/tboot.hg>`__ and contains tarballs of the source. Instructions in the tboot README describe how to modify `grub.conf` to use tboot to launch Xen. + +=============================================== +Installing the Xen Project Software using Yocto +=============================================== + +Follow these instructions to cross-compile a hypervisor and minimal Dom0 filesystem image containing the Xen tools for the ARM64 QEMU emulator. The instructions are similar for other ARM64 platforms. + +Obtaining the Source Code +~~~~~~~~~~~~~~~~~~~~~~~~~ + +We will use the Yocto stable release, "Dunfell": + +.. code-block:: + + $ git clone -b dunfell http://git.yoctoproject.org/git/poky + $ cd poky + $ git clone -b dunfell http://git.openembedded.org/meta-openembeddded + $ git clone -b dunfell http://git.yoctoproject.org/git/meta-virtualization + +Preparing the Build Tree +~~~~~~~~~~~~~~~~~~~~~~~~ + +Initialize your shell to be ready to build - this will generate basic configuration files when you do this the first time: + +.. code-block:: + + $ source ./oe-init-build-env + +Edit `conf/bblayers.conf`, to add the source code layers needed. /scratch/repos/poky is the directory where you cloned the poky source code to - you will need to adjust the example paths here to match your directory layout: + +.. code-block:: + + BBLAYERS ?= " \ + /scratch/repos/poky/meta \ + /scratch/repos/poky/meta-poky \ + /scratch/repos/poky/meta-yocto-bsp \ + /scratch/repos/poky/meta-openembedded/meta-oe \ + /scratch/repos/poky/meta-openembedded/meta-filesystems \ + /scratch/repos/poky/meta-openembedded/meta-python \ + /scratch/repos/poky/meta-openembedded/meta-networking \ + /scratch/repos/poky/meta-virtualization \ + " + +The `conf/local.conf` file contains instructions for the variables that it sets. You should review and make sure to set: + +.. code-block:: + + DL_DIR -- set this to a local download directory for retrieved tarballs or other source code files + SSTATE_DIR -- set to a local directory for build cache files to speed up subsequent builds + PACKAGE_CLASSES -- package_ipk can be a good choice for package format + +Then add the following to the same file, or make sure that the values here match if the variables are already present in the file: + +.. code-block:: + + MACHINE = "qemuarm64" + DISTRO = "poky" + IMAGE_FSTYPES += "cpio.gz" + QEMU_TARGETS = "i386 aarch64" + DISTRO_FEATURES += " virtualization xen" + BUILD_REPRODUCIBLE_BINARIES = "1" + +This configuration will enable OpenEmbedded's support for reproducible builds. It also reduces the number of emulation platforms for QEMU to significantly reduce build time. + +If you would like to build QEMU to provide PV backends, such as disk and 9pfs, then you need to add: + +.. code-block:: + + PACKAGECONFIG_pn-qemu += " virtfs xen fdt" + +Sdl is enabled by default in the Xen build of QEMU but it is not actually necessary and can be disabled with: + +.. code-block:: + + PACKAGECONFIG_remove_pn-qemu += " sdl" + +Building +~~~~~~~~~ + +.. code-block:: + + $ bitbake xen-image-minimal + +When the build is completed, the output image file will be in build/tmp/deploy/images. + +Targeting Hardware Platforms +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you are targetting a specific ARM hardware platform, you will likely need to add the "BSP layer", which contains the MACHINE definition for the hardware, and any closely related software such as bootloader configuration. The MACHINE variable needs to be set to match the hardware definition. You may also add additional layers containing related support software, if any. + +TODO: instructions in this section are still pending validation for Dunfell + +eg. For the Raspberry Pi 4: + +In your poky directory, add the BSP layer: + +.. code-block:: + + $ git clone -b dunfell https://git.yoctoproject.org/git/meta-raspberrypi + +In `local.conf`, set: + +.. code-block:: + + MACHINE = "raspberrypi4-64" + +In bblayers.conf, add: + +.. code-block:: + + BBLAYERS_append = " /scratch/repos/poky/meta-raspberrypi" + +Then, perform your build as before. + +Building with a Local Copy of the Xen Source Code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +TODO: instructions in this section are still pending validation for Dunfell + +If you are building a local copy of a Xen source tree, you can add to `local.conf`: + +.. code-block:: + + INHERIT_pn-xen += "externalsrc" + INHERIT_pn-xen-tools += "externalsrc" + EXTERNALSRC_pn-xen = "/scratch/repos/xen" + EXTERNALSRC_pn-xen-tools = "/scratch/repos/xen" + EXTERNALSRC_BUILD_pn-xen = "/scratch/repos/xen" + EXTERNALSRC_BUILD_pn-xen-tools = "/scratch/repos/xen" \ No newline at end of file diff --git a/docs/admin-guide/introduction.rst b/docs/introduction/introduction.rst similarity index 97% rename from docs/admin-guide/introduction.rst rename to docs/introduction/introduction.rst index 6da2758d7099..62cc88f94a31 100644 --- a/docs/admin-guide/introduction.rst +++ b/docs/introduction/introduction.rst @@ -14,7 +14,7 @@ When Xen boots, dom0 is automatically started as well. Dom0 is a virtual machine which, by default, is granted full permissions [1]_. A typical setup might be: -.. image:: xen-overview.drawio.svg +.. image:: ./media/xen-overview.drawio.svg Dom0 takes the role of :term:`control domain`, responsible for creating and managing other virtual machines, and the role of :term:`hardware domain`, diff --git a/docs/introduction/media/Xen_arch1.png b/docs/introduction/media/Xen_arch1.png new file mode 100644 index 000000000000..927e0940b92b Binary files /dev/null and b/docs/introduction/media/Xen_arch1.png differ diff --git a/docs/introduction/media/Xen_arch2.png b/docs/introduction/media/Xen_arch2.png new file mode 100644 index 000000000000..a0e4b06c13a4 Binary files /dev/null and b/docs/introduction/media/Xen_arch2.png differ diff --git a/docs/admin-guide/xen-overview.drawio.svg b/docs/introduction/media/xen-overview.drawio.svg similarity index 100% rename from docs/admin-guide/xen-overview.drawio.svg rename to docs/introduction/media/xen-overview.drawio.svg diff --git a/docs/media/Xen_arch1.png b/docs/media/Xen_arch1.png new file mode 100644 index 000000000000..927e0940b92b Binary files /dev/null and b/docs/media/Xen_arch1.png differ diff --git a/docs/media/xen-overview.drawio.svg b/docs/media/xen-overview.drawio.svg new file mode 100644 index 000000000000..f120cdf77a05 --- /dev/null +++ b/docs/media/xen-overview.drawio.svg @@ -0,0 +1,97 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="701px" height="461px" viewBox="-0.5 -0.5 701 461" content="<mxfile modified="2019-08-04T17:05:55.267Z" host="" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/11.1.1 Chrome/76.0.3809.88 Electron/6.0.0 Safari/537.36" etag="M7ISh4Ny83I7m1UfK1F2" version="11.1.1" type="device"><diagram id="7q-U8ZVDCMAbtjTOF8Vq" name="Page-1">7Zpbd5s4EMc/jR/Tw/3y6FuadNu0jde72X3pwSAua4w4smzjfPoVRhiD5DvGbM42D0WDkMRv5q8ZDB25P0s+Iyv2v0EHhB1JcJKOPOhIkigKEvkvtawziyGamcFDgUM7FYZR8A6oUaDWReCAeakjhjDEQVw22jCKgI1LNgshuCp3c2FYnjW2PMAYRrYVstY/Awf79C5UobA/gcDz8faG6ZmZlXemhrlvOXC1Y5KHHbmPIMTZ0SzpgzCFl3PJrnvcc3a7MAQifMoFySP67j/+MuJ+8nUIfo3D/it4oN5ZWuGC3vAbiOh68TqHgOAickA6jtCRe24Qhn0YQrQ5KbuGDWyb2OcYwSnYOTMxVEXdXAEjTF0rpe0lQDggjLth4EXEOAscJ52rZ1EDyoj2fIiCd3KxFRKjmM4RW3YQea+UuFqYaIulQkGlc4Jkx0QpfQZwBjBaky40RGWZemxVOFzP3ervOFujNovGmLcdqnADOaCeOMMrCuOVAZwJ57nFUYHhKDy3GNJE1rQd2iFwMccrGMYc310DeAfoVha7QCXjVkRVHtFxR9IID7G4rb1wT0NV8YAFDJcrDM02wMStGS49Kyls9N4OtjcZP78mzz/hj/fk53T8Pv2+/v1BZHeVj0pbMZqk/fJtjKDyNogfu8/CQDKXttV9kBnYTxZyVuSujm0Yl+7K1QTguhI/ATjaRFO1enZlRWhwV+aC1hjQL8/9MxnnKG0CAqCaFE9vOAclsqAMDiflVpx0Vv3BfNoCUOIdSRlfxoH79OWP0Ys2nI49bRz+veJUX6P1HIPZnBhHAC0Dm5S+14k42yvrgFdhx9n3eGoUxVvBY/e9bhyH5N5xAKPWYJPUtnFji8s2clPMtnFjS8jfAIrSx90aiJ3/TFWpkepXuKg1iZxbSLLIXwA+nbeoNZNxt+0G8giXE1uZ9EJon5Fyb0aqmnKbRMVVMYvqg6m4mnCalTGXOVsQfjDm1WTVgq2TrY5asnVqLds72XKoLXun3DZUIsuqHUGlCm0jxVYubYkqrXWsRIYVgwk4HhjRJkTYhx6MrHBYWHtlkEWfrzDNEpt3B/8AjNcUr7XAkJNi5r4Vpw03BEk3fXtEbAQ1Wr+lA39S8+ZfdMhNY5DQWbPWusN/TaKmf9zfJzf/NmcshOm0g8kmXsigSYCz2XWVNrezk+Ni8rSRz71913TwZ845XCAbHHANjWKyLA/g48km9dLBEEQgJE+ZS1BaBS+g6KU/YECWXGyIRjl0lWr2zBZKryrCkhmoMo5aje0MDDMOcY213ukWpx3m+9dbmUYyhYOrkqvKLPcnB9kCCqlt0V6hPrY4a5X67qmiS5V/mvqOqipPtzeXVTXuqr+gXygrZpz7yGqr7iZlJbGlUqtkVVNSA6KjAp0nR1PTZateOe5V0s0FYpqfdFMWBVlRDElXjXI8qeWzpnyZekSdP86J09QkLrGiLlk8krRM81D/G6mLLa9bpa57quRSZd9NXZr8v7r2rVpTDva/Wl3c97dmY1ry8SzcieEsaqWLYl0qBbtwJNhLFdopZacLtD2fOOjmRBD2KXivqC559Dr0UUJ9Ir3q1X/+QHZW7PD4/6fjiRcJ5wfU5bGjnRg7ekOxQ5rFZ67ZLlV8LCwP/wU=</diagram></mxfile>"> + <defs/> + <g> + <rect x="0" y="330" width="700" height="60" fill="#f8cecc" stroke="#b85450" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="end" font-size="20px"> + <text x="688.5" y="367.5">Xen</text> + </g> + <rect x="0" y="0" width="220" height="280" fill="#d5e8d4" stroke="#82b366" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" font-size="20px"> + <text x="2.5" y="26.5">Dom0</text> + </g> + <rect x="240" y="0" width="220" height="280" fill="#dae8fc" stroke="#6c8ebf" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" font-size="20px"> + <text x="242.5" y="26.5">DomU</text> + </g> + <rect x="480" y="0" width="220" height="280" fill="#dae8fc" stroke="#6c8ebf" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" font-size="20px"> + <text x="482.5" y="26.5">DomU</text> + </g> + <rect x="0" y="400" width="700" height="60" fill="#fff2cc" stroke="#d6b656" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="end" font-size="20px"> + <text x="696.5" y="437.5">Hardware</text> + </g> + <rect x="20" y="410" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="20px"> + <text x="59.5" y="437.5">NIC</text> + </g> + <rect x="120" y="410" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="20px"> + <text x="159.5" y="437.5">Disk</text> + </g> + <rect x="10" y="40" width="200" height="110" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="20px"> + <text x="109.5" y="66.5">Systems Services</text> + </g> + <rect x="250" y="40" width="200" height="110" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="20px"> + <text x="349.5" y="66.5">Applications</text> + </g> + <rect x="490" y="40" width="200" height="110" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="20px"> + <text x="589.5" y="66.5">Applications</text> + </g> + <rect x="10" y="160" width="200" height="110" fill="#f8cecc" stroke="#b85450" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" font-size="20px"> + <text x="12.5" y="186.5">Kernel</text> + </g> + <rect x="20" y="220" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="16px"> + <text x="59.5" y="245.5">Net</text> + </g> + <rect x="120" y="220" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="16px"> + <text x="159.5" y="245.5">Block</text> + </g> + <rect x="250" y="160" width="200" height="110" fill="#f8cecc" stroke="#b85450" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" font-size="20px"> + <text x="252.5" y="186.5">Kernel</text> + </g> + <rect x="490" y="160" width="200" height="110" fill="#f8cecc" stroke="#b85450" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" font-size="20px"> + <text x="492.5" y="186.5">Kernel</text> + </g> + <rect x="260" y="220" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="16px"> + <text x="299.5" y="245.5">Net</text> + </g> + <rect x="360" y="220" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="16px"> + <text x="399.5" y="245.5">Block</text> + </g> + <rect x="500" y="220" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="16px"> + <text x="539.5" y="245.5">Net</text> + </g> + <rect x="600" y="220" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="none"/> + <g fill="#000000" font-family="Helvetica" text-anchor="middle" font-size="16px"> + <text x="639.5" y="245.5">Block</text> + </g> + <path d="M 75 279.5 L 64.5 279.5 L 80 260.5 L 95.5 279.5 L 85 279.5 L 85 285 L 295 285 L 295 279.5 L 284.5 279.5 L 300 260.5 L 315.5 279.5 L 305 279.5 L 305 295 L 75 295 Z" fill="#f5f5f5" stroke="#666666" stroke-miterlimit="1.42" pointer-events="none"/> + <path d="M 75 279.5 L 64.5 279.5 L 80 260.5 L 95.5 279.5 L 85 279.5" fill="none" stroke="#666666" stroke-miterlimit="4" pointer-events="none"/> + <path d="M 295 279.5 L 284.5 279.5 L 300 260.5 L 315.5 279.5 L 305 279.5" fill="none" stroke="#666666" stroke-miterlimit="4" pointer-events="none"/> + <path d="M 75 279.5 L 64.5 279.5 L 80 260.5 L 95.5 279.5 L 85 279.5 L 85 285 L 535 285 L 535 279.5 L 524.5 279.5 L 540 260.5 L 555.5 279.5 L 545 279.5 L 545 295 L 75 295 Z" fill="#f5f5f5" stroke="#666666" stroke-miterlimit="1.42" pointer-events="none"/> + <path d="M 75 279.5 L 64.5 279.5 L 80 260.5 L 95.5 279.5 L 85 279.5" fill="none" stroke="#666666" stroke-miterlimit="4" pointer-events="none"/> + <path d="M 535 279.5 L 524.5 279.5 L 540 260.5 L 555.5 279.5 L 545 279.5" fill="none" stroke="#666666" stroke-miterlimit="4" pointer-events="none"/> + <path d="M 175 279.5 L 164.5 279.5 L 180 260.5 L 195.5 279.5 L 185 279.5 L 185 305 L 395 305 L 395 279.5 L 384.5 279.5 L 400 260.5 L 415.5 279.5 L 405 279.5 L 405 315 L 175 315 Z" fill="#e1d5e7" stroke="#9673a6" stroke-miterlimit="1.42" pointer-events="none"/> + <path d="M 175 279.5 L 164.5 279.5 L 180 260.5 L 195.5 279.5 L 185 279.5" fill="none" stroke="#9673a6" stroke-miterlimit="4" pointer-events="none"/> + <path d="M 395 279.5 L 384.5 279.5 L 400 260.5 L 415.5 279.5 L 405 279.5" fill="none" stroke="#9673a6" stroke-miterlimit="4" pointer-events="none"/> + <path d="M 175 279.5 L 164.5 279.5 L 180 260.5 L 195.5 279.5 L 185 279.5 L 185 305 L 635 305 L 635 279.5 L 624.5 279.5 L 640 260.5 L 655.5 279.5 L 645 279.5 L 645 315 L 175 315 Z" fill="#e1d5e7" stroke="#9673a6" stroke-miterlimit="1.42" pointer-events="none"/> + <path d="M 175 279.5 L 164.5 279.5 L 180 260.5 L 195.5 279.5 L 185 279.5" fill="none" stroke="#9673a6" stroke-miterlimit="4" pointer-events="none"/> + <path d="M 635 279.5 L 624.5 279.5 L 640 260.5 L 655.5 279.5 L 645 279.5" fill="none" stroke="#9673a6" stroke-miterlimit="4" pointer-events="none"/> + <path d="M 35 279.5 L 24.5 279.5 L 40 260.5 L 55.5 279.5 L 45 279.5 L 45 390.5 L 55.5 390.5 L 40 409.5 L 24.5 390.5 L 35 390.5 Z" fill="#ffe6cc" stroke="#d79b00" stroke-miterlimit="10" pointer-events="none"/> + <path d="M 135 279.5 L 124.5 279.5 L 140 260.5 L 155.5 279.5 L 145 279.5 L 145 390.5 L 155.5 390.5 L 140 409.5 L 124.5 390.5 L 135 390.5 Z" fill="#ffe6cc" stroke="#d79b00" stroke-miterlimit="10" pointer-events="none"/> + </g> +</svg> diff --git a/docs/reference/hypercall-abi.rst b/docs/reference/hypercall-abi.rst new file mode 100644 index 000000000000..14c48929d748 --- /dev/null +++ b/docs/reference/hypercall-abi.rst @@ -0,0 +1,129 @@ +.. SPDX-License-Identifier: CC-BY-4.0 + +Hypercall ABI +============= + +Hypercalls are system calls to Xen. Two modes of guest operation are +supported, and up to 6 individual parameters are supported. + +Hypercalls may only be issued by kernel-level software [1]_. + +Registers +--------- + +The registers used for hypercalls depends on the operating mode of the guest. + +.. list-table:: + :header-rows: 1 + + * - ABI + - Hypercall Index + - Parameters (1 - 6) + - Result + + * - 64bit + - RAX + - RDI RSI RDX R10 R8 R9 + - RAX + + * - 32bit + - EAX + - EBX ECX EDX ESI EDI EBP + - EAX + +32 and 64bit PV guests have an ABI fixed by their guest type. The ABI for an +HVM guest depends on whether the vCPU is operating in a 64bit segment or not +[2]_. + + +Parameters +---------- + +Different hypercalls take a different number of parameters. Each hypercall +potentially clobbers each of its parameter registers; a guest may not rely on +the parameter registers staying the same. A debug build of Xen checks this by +deliberately poisoning the parameter registers before returning back to the +guest. + + +Mode transfer +------------- + +The exact sequence of instructions required to issue a hypercall differs +between virtualisation mode and hardware vendor. + +.. list-table:: + :header-rows: 1 + + * - Guest + - Transfer instruction + + * - 32bit PV + - INT 0x82 + + * - 64bit PV + - SYSCALL + + * - Intel HVM + - VMCALL + + * - AMD HVM + - VMMCALL + +To abstract away the details, Xen implements an interface known as the +Hypercall Page. This allows a guest to make a hypercall without needing to +perform mode-specific or vendor-specific setup. + + +Hypercall Page +============== + +The hypercall page is a page of guest RAM into which Xen will write suitable +transfer stubs. + +Creating a hypercall page is an isolated operation from Xen's point of view. +It is the guests responsibility to ensure that the hypercall page, once +written by Xen, is mapped with executable permissions so it may be used. +Multiple hypercall pages may be created by the guest, if it wishes. + +The stubs are arranged by hypercall index, and start on 32-byte boundaries. +To invoke a specific hypercall, ``call`` the relevant stub [3]_: + +.. code-block:: none + + call hypercall_page + index * 32 + +There result is an ABI which is invariant of the exact operating mode or +hardware vendor. This is intended to simplify guest kernel interfaces by +abstracting away the details of how it is currently running. + + +Creating Hypercall Pages +------------------------ + +Guests which are started using the PV boot protocol may set set +``XEN_ELFNOTE_HYPERCALL_PAGE`` to have the nominated page written as a +hypercall page during construction. This mechanism is common for PV guests, +and allows hypercalls to be issued with no additional setup. + +Any guest can locate the Xen CPUID leaves and read the *hypercall transfer +page* information, which specifies an MSR that can be used to create +additional hypercall pages. When a guest physical address is written to the +MSR, Xen writes a hypercall page into the nominated guest page. This +mechanism is common for HVM guests which are typically started via legacy +means. + + +.. rubric:: Footnotes + +.. [1] For HVM guests, ``HVMOP_guest_request_vm_event`` may be configured to + be usable from userspace, but this behaviour is not default. + +.. [2] While it is possible to use compatibility mode segments in a 64bit + kernel, hypercalls issues from such a mode will be interpreted with the + 32bit ABI. Such a setup is not expected in production scenarios. + +.. [3] ``HYPERCALL_iret`` is special. It is only implemented for PV guests + and takes all its parameters on the stack. This stub should be + ``jmp``'d to, rather than ``call``'d. HVM guests have this stub + implemented as ``ud2a`` to prevent accidental use. diff --git a/docs/reference/imagebuilder.rst b/docs/reference/imagebuilder.rst new file mode 100644 index 000000000000..a2f17203db9d --- /dev/null +++ b/docs/reference/imagebuilder.rst @@ -0,0 +1,98 @@ +************ +ImageBuilder +************ + +Building an embedded virtualized system with anything more than one domain can be difficult, error prone and time consuming. +ImageBuilder, an Open Source collection of scripts, changes all that. + +ImageBuilder generates a U-Boot script that can be used to load all of the binaries automatically and boot the full system fast. Given a +collection of binaries such as Xen, Dom0 and a number of Dom0-less DomUs, ImageBuilder takes care of calculating all loading addresses, +editing device tree with the necessary information, and even pre-configuring a disk image with kernels and rootfses. + +ImageBuilder has been tested on Xilinx Zynq UltraScale+ MPSoC boards. + +ImageBuilder can be invoked as a container for build automation, but its useful scripts can also be called manually. Specifically, `script/uboot-script-gen` generates a U-Boot script that loads all the necessary binaries and automatically adds the required entries to device tree at boot time. + +The ImageBuilder script that generates a u-boot script to load all your binaries for a Xen Dom0-less setup is `scripts/uboot-script-gen`. To use it, first write a config file like config: + +.. code-block:: + + MEMORY_START="0x0" + MEMORY_END="0x80000000" + + DEVICE_TREE="mpsoc.dtb" + XEN="xen" + XEN_CMD="console=dtuart dtuart=serial0 dom0_mem=1G dom0_max_vcpus=1 bootscrub=0 vwfi=native sched=null" + DOM0_KERNEL="Image-dom0" + DOM0_CMD="console=hvc0 earlycon=xen earlyprintk=xen clk_ignore_unused" + DOM0_RAMDISK="dom0-ramdisk.cpio" + + NUM_DOMUS=2 + DOMU_KERNEL[0]="zynqmp-dom1/Image-domU" + DOMU_RAMDISK[0]="zynqmp-dom1/domU-ramdisk.cpio" + DOMU_PASSTHROUGH_DTB[0]="zynqmp-dom1/passthrough-example-part.dtb" + DOMU_KERNEL[1]="zynqmp-dom2/Image-domU" + DOMU_RAMDISK[1]="zynqmp-dom2/domU-ramdisk.cpio" + DOMU_MEM[1]=512 + DOMU_VCPUS[1]=1 + + UBOOT_SOURCE="boot.source" + UBOOT_SCRIPT="boot.scr" + +where, + +- MEMORY_START and MEMORY_END specify the start and end of RAM. +- DEVICE_TREE specifies the DTB file to load. +- XEN specifies the Xen hypervisor binary to load. Note that it has to be a regular Xen binary, not a u-boot binary. +- XEN_CMD specifies the command line arguments used for Xen. If not set, the default one will be used. +- DOM0_KERNEL specifies the Dom0 kernel file to load. +- DOM0_CMD specifies the command line arguments for Dom0's Linux kernel. If `"root="` isn't set, imagebuilder will try to determine it If not set at all, the default one is used. +- DOM0_RAMDISK specifies the Dom0 ramdisk to use. Note that it should be a regular ramdisk `cpio.gz` file, not a u-boot binary. +- NUM_DOMUS specifies how many Dom0-less DomUs to load. +- DOMU_KERNEL[number] specifies the DomU kernel to use. +- DOMU_RAMDISK[number] specifies the DomU ramdisk to use. +- DOMU_PASSTHROUGH_DTB[number] specifies the device assignment configuration, see `xen.git:docs/misc/arm/passthrough.txt`. +- DOMU_MEM[number] is the amount of memory for the VM in MB, default 512MB +- DOMU_VCPUS[number] is the number of vcpus for the VM, default 1. +- UBOOT_SOURCE and UBOOT_SCRIPT specify the output. They are optional as you can pass -o FILENAME to uboot-script-gen as a command line parameter. + +======================== +scripts/uboot-script-gen +======================== + +Once you have the config file, invoke uboot-script-gen as follows: + +.. code-block:: + + $ bash ./scripts/uboot-script-gen -c /path/to/config-file -d . -t tftp -o bootscript + +where, + +- `-c` specifies the path to the config file to use +- `-d` specifies the "root" directory (paths in the config file are relative to it), this is not a working directory (any output file locations are specified in the config and any temporary files are in `/tmp`) +- `-t` specifies the u-boot command to load the binaries. "tftp", "sd" and "scsi" are shorthands for "tftpb", "load mmc 0:1" and "load scsi 0:1", but actually any arbitrary command can be used, for instance `-t` "fatload" is valid. +- `-o` specifies the output filename for the uboot script and its source. + +`uboot-script-gen` generates a U-Boot script named `boot.scr` that will load all your binaries automatically using tftp. All the path specified in the config file are relative to the directory passed as an argument to `-d`. In this case, Image-dom0 and `dom1/Image-domU` must be relative to the current directory, because we are passing `-d .` to `uboot-script-gen`. + +================== +scripts/disk_image +================== + +The ImageBuilder script that generates a disk image file to load on a SD or SATA drive. This creates two partitions: boot partition where the boot files from working directory (`-c` option) are, and the dom0 ramdisk uncompressed into the root FS partition. + +After you've generated the u-boot scripts using the `uboot-script-gen` script, `disk_image` is run as follows: + +.. code-block:: + + $ sudo bash ./scripts/disk_image -c /path/to/config-file -d . \ + -w /path/to/tmp/dir \ + -o /path/to/output/disk.img + +where, + +- `-c` specifies the path to the config file to use +- `-d` specifies the working directory (paths in the config file are relative to it) +- `-w` specifies the temporary working directory that the script uses for building the disk image, and if not set, one is created in /tmp +- `-o` specifies the output disk image file name + diff --git a/docs/reference/xen-arm-debug.rst b/docs/reference/xen-arm-debug.rst new file mode 100644 index 000000000000..048064f0abe6 --- /dev/null +++ b/docs/reference/xen-arm-debug.rst @@ -0,0 +1,33 @@ +************************ +Xen Arm Debug Hypercalls +************************ + +A small set of debug hypercalls are available to help debugging early boot domU issues. + +If you are trying to debug early boot code in your guest kernel, either Dom0 or DomU, you might find the Xen on Arm debug hypercalls useful. They are only available to DEBUG builds, i.e., builds with **Debugging Options ---> Developer Checks** enabled. + +Simply add one of the following hvc instructions to your code, and Xen will print something on the console for you: + +To print the program counter: + +.. code-block:: + + hvc 0xfffd + +To print first byte of register x0: + +.. code-block:: + + hvc 0xfffe + +To prints all registers and stack of the guest: + +.. code-block:: + + hvc 0xffff + +To print program counter and a register: e0 prints x0, e1 prints x1, etc.: + +.. code-block:: + + hvc 0xffe0-0xffef diff --git a/docs/tutorials/booting-xen-u-boot.rst b/docs/tutorials/booting-xen-u-boot.rst new file mode 100644 index 000000000000..46c28dfdf908 --- /dev/null +++ b/docs/tutorials/booting-xen-u-boot.rst @@ -0,0 +1,242 @@ +*********************** +Booting Xen from U-Boot +*********************** + +Booting Xen from U-Boot requires: + +- Loading all the required binaries, manually specifying the loading address for each of them making sure they do not overlap + - Xen, Dom0 kernel, Dom0 ramdisk, device tree binary, any Dom0-less DomUs kernels, ramdisk and partial dtbs for passthrough +- Adding relevant nodes to device tree + - the Dom0 kernel and ramdisk loading addresses need to be specified in device tree under `/chosen` + +=============================== +Dom0 Kernel and ramdisk Modules +================================ + +Xen is passed the dom0 kernel and initrd via a reference in the `/chosen` node of the device tree. + +Each node contains the following properties: + +- compatible + Must always include at least the generic compatiblity string: + + "multiboot,module" + + Optionally a more specific compatible string may be used in addition to the above. One of: + + - "multiboot,kernel" -- the dom0 kernel + - "multiboot,ramdisk" -- the dom0 ramdisk + - "xen,xsm-policy" -- XSM policy blob + + It is normally recommended to include a more specific compatible string (if one applies) in addition to the generic string (which must always be present). + + Xen will assume that the first module which lacks a more specific compatible string is a "multiboot,kernel". + + Xen will examine each module, starting from the second module that lacks a specific compatible string. Xen will check each such module for the XSM Magic number: + + - For a module which has the XSM Magic number: it will be treated by Xen as if its compatible string was "xen,xsm-policy"; + + - For a module which does not have the XSM Magic: the second module lacking a compatible string will be treated by Xen as if its compatible string was "multiboot,ramdisk"; for the third and subsequent modules which lack a specific compatible string, Xen will not apply any special treatment. + + This means if the ramdisk module is present and does not have the compatible string "multiboot,ramdisk", then it must always be the second module. + + .. note:: This XSM Magic detection behavior was introduced by Xen 4.7. Xen 4.6 (and downwards) still requires the XSM module to have the compatible string "xen,xsm-policy". + + Xen 4.4 supported a different set of legacy compatible strings which remain supported such that systems supporting both 4.4 + and later can use a single DTB. + + - "xen,multiboot-module" equivalent to "multiboot,module" + - "xen,linux-zimage" equivalent to "multiboot,kernel" + - "xen,linux-initrd" equivalent to "multiboot,ramdisk" + + For compatibility with Xen 4.4 the more specific "xen,linux-*" names are non-optional and must be included. + +- reg + Specifies the physical address of the module in RAM and the length of the module. + +- bootargs (optional) + + Command line associated with this module. See below for the priority of this field vs. other mechanisms of specifying the bootargs for the kernel. + +Examples +~~~~~~~~ + +A boot module of unspecified type: + +.. code-block:: + + module@0xc0000000 { + compatible = "multiboot,module"; + reg = <0xc0000000 0x1234>; + bootargs = "..."; + }; + +A boot module containing a ramdisk: + +.. code-block:: + + module@0xd0000000 { + compatible = "multiboot,ramdisk", "multiboot,module"; + reg = <0xd0000000 0x5678>; + }; + +The previous examples are compatible with Xen 4.5+ only. + +To be compatible with Xen 4.4 as well use the legacy names: + +.. code-block:: + + module@0xd0000000 { + compatible = "xen,linux-initrd", "xen,multiboot-module"; + reg = <0xd0000000 0x5678>; + }; + +Command Lines +~~~~~~~~~~~~~ + + +Xen also checks for properties directly under /chosen to find suitable command lines for Xen and Dom0. The logic is the following: + +- If xen,xen-bootargs is present, it will be used for Xen. +- If xen,dom0-bootargs is present, it will be used for Dom0. +- If xen,xen-bootargs is _not_ present, but xen,dom0-bootargs is, bootargs will be used for Xen. +- If a kernel boot module is present and has a bootargs property then the top-level bootargs will used for Xen. +- If no Xen specific properties are present, bootargs is for Dom0. +- If xen,xen-bootargs is present, but xen,dom0-bootargs is missing, bootargs will be used for Dom0. + +Most of these cases is to make booting with Xen-unaware bootloaders easier. For those you would hardcode the Xen commandline in the DTB under `/chosen/xen,xen-bootargs` and would let the bootloader set the Dom0 command line by writing bootargs (as for native Linux). + +A Xen-aware bootloader would set xen,xen-bootargs for Xen, xen,dom0-bootargs for Dom0 and bootargs for native Linux. + +=========================================== +Creating Multiple Domains directly from Xen +=========================================== + +It is possible to have Xen create other domains, in addition to dom0, out of the information provided via device tree. A kernel and initrd (optional) need to be specified for each guest. + +For each domain to be created there needs to be one node under `/chosen` with the following properties: + +- compatible + + For domUs: "xen,domain" + +- memory + + A 64-bit integer specifying the amount of kilobytes of RAM to allocate to the guest. + +- cpus + + An integer specifying the number of vcpus to allocate to the guest. + +- vpl011 + + An empty property to enable/disable a virtual pl011 for the guest to use. The virtual pl011 uses SPI number 0 (see GUEST_VPL011_SPI). Please note that the SPI used for the virtual pl011 could clash with the physical SPI of a physical device assigned to the guest. + +- nr_spis + + Optional. A 32-bit integer specifying the number of SPIs (Shared Peripheral Interrupts) to allocate for the domain. If nr_spis is missing, the max number of SPIs supported by the physical GIC is used, or GUEST_VPL011_SPI+1 if vpl011 is enabled, whichever is greater. + +- #address-cells and #size-cells + + Both #address-cells and #size-cells need to be specified because both sub-nodes (described shortly) have reg properties. + + Under the "xen,domain" compatible node, one or more sub-nodes are present + for the DomU kernel and ramdisk. + +The kernel sub-node has the following properties: + +- compatible + + "multiboot,kernel", "multiboot,module" + +- reg + + Specifies the physical address of the kernel in RAM and its length. + +- bootargs (optional) + + Command line parameters for the guest kernel. + +The ramdisk sub-node has the following properties: + +- compatible + + "multiboot,ramdisk", "multiboot,module" + +- reg + + Specifies the physical address of the ramdisk in RAM and its length. + + +Example +~~~~~~~ + +.. code-block:: + + chosen { + domU1 { + compatible = "xen,domain"; + #address-cells = <0x2>; + #size-cells = <0x1>; + memory = <0 131072>; + cpus = <2>; + vpl011; + + module@0x4a000000 { + compatible = "multiboot,kernel", "multiboot,module"; + reg = <0x0 0x4a000000 0xffffff>; + bootargs = "console=ttyAMA0 init=/bin/sh"; + }; + + module@0x4b000000 { + compatible = "multiboot,ramdisk", "multiboot,module"; + reg = <0x0 0x4b000000 0xffffff>; + }; + }; + + domU2 { + compatible = "xen,domain"; + #address-cells = <0x2>; + #size-cells = <0x1>; + memory = <0 65536>; + cpus = <1>; + + module@0x4c000000 { + compatible = "multiboot,kernel", "multiboot,module"; + reg = <0x0 0x4c000000 0xffffff>; + bootargs = "console=ttyAMA0 init=/bin/sh"; + }; + + module@0x4d000000 { + compatible = "multiboot,ramdisk", "multiboot,module"; + reg = <0x0 0x4d000000 0xffffff>; + }; + }; + }; + + +Device Assignment +~~~~~~~~~~~~~~~~~ + +Device Assignment (Passthrough) is supported by adding another module, alongside the kernel and ramdisk, with the device tree fragment corresponding to the device node to assign to the guest. + +The dtb sub-node should have the following properties: + +- compatible + + "multiboot,device-tree" and "multiboot,module" + +- reg + + Specifies the physical address of the device tree binary fragment RAM and its length. + + As an example: + + .. code-block:: + + module@0xc000000 { + compatible = "multiboot,device-tree", "multiboot,module"; + reg = <0x0 0xc000000 0xffffff>; + }; + +The DTB fragment is loaded at 0xc000000 in the example above. It should follow the convention explained in `docs/misc/arm/passthrough.txt`. The DTB fragment will be added to the guest device tree, so that the guest kernel will be able to discover the device. \ No newline at end of file diff --git a/docs/tutorials/configure-networking.rst b/docs/tutorials/configure-networking.rst new file mode 100644 index 000000000000..bdda39449cff --- /dev/null +++ b/docs/tutorials/configure-networking.rst @@ -0,0 +1,3 @@ +********************** +Configuring Networking +********************** \ No newline at end of file diff --git a/docs/tutorials/reduce-latency.rst b/docs/tutorials/reduce-latency.rst new file mode 100644 index 000000000000..5390e2cb9722 --- /dev/null +++ b/docs/tutorials/reduce-latency.rst @@ -0,0 +1,3 @@ +**************** +Reducing Latency +**************** \ No newline at end of file diff --git a/docs/tutorials/running-xen-on-arm/arndale.rst b/docs/tutorials/running-xen-on-arm/arndale.rst new file mode 100644 index 000000000000..3ca1f314d742 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/arndale.rst @@ -0,0 +1,173 @@ +********************************************** +Xen Arm with Virtualization Extensions/Arndale +********************************************** + +The Arndale board is now supported in Xen upstream. + +=================== +Preparing the Board +=================== + +The bootloader provided with the Arndale does not let Xen boot in hypervisor mode, so we will use the u-boot provided by Linaro. + +=========================== +Building Xen and Linux Dom0 +=========================== + +First, build Linux for dom0 to have a device tree to provide to Xen. You can get a tree from Linaro which contains a working configuration for the Arndale. + +.. code-block:: + + git clone -b linux-linaro git://git.linaro.org/kernel/linux-linaro-tracking.git linux + cd linux + ARCH=arm scripts/kconfig/merge_config.sh linaro/configs/linaro-base.conf linaro/configs/distribution.conf\ + linaro/configs/kvm-host.conf linaro/configs/xen.conf linaro/configs/arndale.conf linaro/configs/lt-arndale.conf + make ARCH=arm zImage + make ARCH=arm dtbs + +The device tree used by the Arndale board is located in `arch/arm/boot/dts/exynos5250-arndale.dtb`. + +Then, build Xen on ARM. + +For the moment, xen doesn't build uImage for U-Boot. You can create the image with: + +.. code-block:: + + mkimage -A arm -T kernel -a 0x80200000 -e 0x80200000 -C none -d "$xen_src/xen/xen" xen-uImage + +where, $xen_src is the root directory of your xen git. Note that before commit 47d1a51 (xen: arm: make zImage the default target which we install) it was necessary to use `$xen_src/xen/xen.bin` instead. + +==================== +Booting Xen and Dom0 +==================== + +To boot Xen and Dom0, you can use: + +- PXE: it's easier for development but you need a computer which act as a server; +- copy binary on the SD card: when you don't have network on your board. + +Once you have chosen the boot method, you can start to follow steps in the next sections. When you have finished to configure your board, you can: + +- Save the U-Boot configuration with saveenv command. The next time you want to reboot your board, you won't have to reconfigure U-Boot. +- Boot with boot command. + +=============== +Booting via PXE +=============== + +The following script allow U-boot to download everything via tftp and boot xen. + +1. Setup the PXE Server +2. Copy xen-uImage, the zImage (rename in linux-zImage), exynos5250-arndale.dtb in /tftpboot/ +3. Copy the script in /tftpboot + + .. code-block:: + + wget http://xenbits.xen.org/people/julieng/load-xen-tftp.scr.txt + mkimage -T script -C none -d load-xen-tftp.scr.txt /tftpboot/load-xen-tftp.img + +4. At U-Boot prompt, on your board, you need to set the following variable: + + .. code-block:: + + setenv ipaddr 10.y.y.y + setenv serverip 10.x.x.x + setenv usbethaddr 00:zz:zz:zz:zz:zz + setenv ethaddr 00:zz:zz:zz:zz:zz + setenv xen_addr_r 0x50000000 + setenv kernel_addr_r 0x60000000 + setenv dtb_addr_r 0x42000000 + setenv script_addr_r 0x40080000 + setenv xen_path /xen-uImage + setenv kernel_path /linux-zImage + setenv dtb_path /exynos5250-arndale.dtb + setenv bootcmd 'tftpboot $script_addr_r /load-xen-tftp.img; source $script_addr_r' + setenv xen_bootargs 'sync_console console=dtuart dtuart=/serial@12C20000' + setenv dom0_bootargs 'console=hvc0 ignore_loglevel psci=enable clk_ignore_unused root=/dev/mmcblk1p3' + + with: + + 10.y.y.y the ip addr of the board + 10.x.x.x the ip of a tftp server (or PXE server). + 00:zz:zz:zz:zz:zz the MAC address of the board. + + You can generate it with the following shell command: + + .. code-block:: + + bash -c 'printf "00:16:3e:%02x:%02x:%02x\n" $(( $RANDOM % 256 )) $(( $RANDOM % 256 )) $(( $RANDOM % 256 ))' + + or, using the `www.hellion.org.uk/cgi-bin/randmac.pl <www.hellion.org.uk/cgi-bin/randmac.pl>`__ website. + +=============================== +Booting Directly on the SD Card +=============================== + +1. Copy xen-uImage and the zImage the root directory of you SD card. +2. At U-Boot prompt, on your board, you need to set the following variable: + + .. code-block:: + + setenv kernel_addr_r 0x60000000 + setenv xen_addr_r 0x50000000 + setenv bootcmd_load_linux_mmc 'ext2load mmc 1:0 $kernel_addr_r /zImage' + setenv boot_xen_mmc 'run bootcmd_load_linux_mmc; ext2load mmc 1:0 $xen_addr_r /xen-uImage; bootm $xen_addr_r -' + setenv bootcmd 'run boot_xen_mmc' + +3. Assuming the SD card has only one partition with ext2 filesystem, execute the boot command with the following command: + + .. code-block:: + + boot + +====================================================================== +Alternate SD approach based on Linaro pre-built image for Arndale 5250 +====================================================================== + +1. On your host development machine, install [Linaro prebuilt] to a uSD card (assuming it mounts at /dev/sdb, see linaro instructions for more detail): + +.. code-block:: + + $ wget https://releases.linaro.org/14.03/ubuntu/arndale/arndale-saucy_server_20140323-616.img.gz + $ gunzip < arndale-saucy_server_20140323-616.img.gz | sudo dd bs=64k of=/dev/sdb + +2. Copy your compiled xen-uImage, linux-zImage, and exynos5250-arndale.dtb all to the boot partition on the uSD card. Copy the contents of xen/dist/install to the rootfs partition. Unmount the uSD from your host development machine, insert it in the Arndale. + +3. Start minicom on host development machine, connected to the Arndale via a serial cable: + + .. code-block:: + + $ sudo minicom + +4. Power on Arndale, press reset, watch the minicom console, and hit “enter” to interrupt U-Boot to get the u-Boot prompt. Enter the following into the minicom console. The saveenv stores the environment variables, so you won't have to repeat those the next time around. + + .. code-block:: + + setenv xen_addr_r 0x50000000 + setenv kernel_addr_r 0x60000000 + setenv dtb_addr_r 0x42000000 + setenv xen_bootargs 'sync_console console=dtuart dtuart=/serial@12C20000 dom0_mem=512M' + setenv dom0_bootargs 'console=hvc0 ignore_loglevel psci=enable clk_ignore_unused root=/dev/mmcblk1p3' + saveenv + fatload mmc 0:2 $kernel_addr_r linux-zImage + fatload mmc 0:2 $xen_addr_r xen-uImage + fatload mmc 0:2 $dtb_addr_r exynos5250-arndale.dtb + fdt addr $dtb_addr_r + fdt resize + fdt set /chosen xen,xen-bootargs \"$xen_bootargs\" + fdt set /chosen xen,dom0-bootargs \"$dom0_bootargs\" + fdt mknode /chosen modules + fdt set /chosen/modules '#address-cells' <1> + fdt set /chosen/modules '#size-cells' <1> + fdt mknode /chosen/modules module@0 + fdt set /chosen/modules/module@0 compatible xen,linux-zimage xen,multiboot-module + fdt set /chosen/modules/module@0 reg <$kernel_addr_r 0x00a00000> + bootm $xen_addr_r - $dtb_addr_r + +========= +Resources +========= + +Information about the Arndale board development: `www.arndaleboard.org/wiki/index.php/WiKi <www.arndaleboard.org/wiki/index.php/WiKi>`__. +Linaro page about the Arndale board: `wiki.linaro.org/Boards/Arndale/Setup/PXEBoot <wiki.linaro.org/Boards/Arndale/Setup/PXEBoot>`__ +Alternate SD approach based on Linaro pre-built image (more detailed): `Booting Linux in Xen's Dom0 on Arndale Exynos 5250 <http://www.episodic.cc/2014/06/booting-linux-in-xens-dom0-on-arndale.html>`__ \ No newline at end of file diff --git a/docs/tutorials/running-xen-on-arm/building-xen-arm.rst b/docs/tutorials/running-xen-on-arm/building-xen-arm.rst new file mode 100644 index 000000000000..eec0a27f68f8 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/building-xen-arm.rst @@ -0,0 +1,72 @@ +******************* +Building Xen on Arm +******************* + +There are two major components which need to be built for a Xen system. The Xen hypervisor binary itself and the Xen toolstack. + +=================== +Cross Compiling Xen +=================== + +Cross compiling the Xen hypervisor is simple. Linaro supply cross compilers for both arm32 (`arm-linux-gnueabihf-`) and arm64 (`aarch64-linux-gnu-`) via `linaro-toolchain-binaries <https://launchpad.net/linaro-toolchain-binaries>`__. Alternatively, for 32-bit at least, you can download the arm-unknown-linux-gnueabi compiler from `kernel.org <http://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/>`__. + +Once you have a suitable cross compiler you can compile Xen with: + +.. code-block:: + + $ make dist-xen XEN_TARGET_ARCH=arm32 CROSS_COMPILE=arm-unknown-linux-gnueabihf- + +or, + +.. code-block:: + + $ make dist-xen XEN_TARGET_ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- + +This assumes that the command prefix for you cross compiler is `arm-unknown-linux-gnueabihf-` or `aarch64-linux-gnu-` and that the appropriate `arm-unknown-linux-gnueabihf-gcc` or `aarch64-linux-gnu-gcc` and friends are in your `$PATH`. + +====================== +Building the Toolstack +====================== + +For a complete cross-compilation of the hypervisor and the toolstack, it is recommended to use Yocto, see Xen_on_ARM_and_Yocto. Alternatively, it is possible to use QEMU user to run an ARM64 chroot on a x86 host, i.e. an ARM64 Debian or Ubuntu container on a regular x86 laptop. + +.. code-block:: + + $ apt-get install qemu-user-static + +It installs `/usr/bin/qemu-aarch64-static`. + +Next setup an ARM64 chroot environment on your x86 machine. Follow your distro recommandations. For instance, the following distros offer pre-packaged tarballs ready to be unpackged: + +- Ubuntu: `http://cdimage.ubuntu.com/ubuntu-base/releases/20.04/release/ubuntu-base-20.04-base-arm64.tar.gz <http://cdimage.ubuntu.com/ubuntu-base/releases/20.04/release/ubuntu-base-20.04-base-arm64.tar.gz>`__ +- Alpine Linux: `http://dl-cdn.alpinelinux.org/alpine/v3.11/releases/aarch64/alpine-minirootfs-3.11.6-aarch64.tar.gz <http://dl-cdn.alpinelinux.org/alpine/v3.11/releases/aarch64/alpine-minirootfs-3.11.6-aarch64.tar.gz>`__ + +Assuming that the ARM64 chroot is under `/chroot/distro_arm64`, then you can: + +.. code-block:: + + $ cp /usr/bin/qemu-aarch64-static /chroot/distro_arm64/usr/bin/qemu-aarch64-static + $ chroot /chroot/distro_arm64 + +Now you have a full Arm64 environment running on a regular x86 machine. You can automate all the last steps with the following Docker command (the example is running Debian): + +.. code-block:: + + $ docker run -it -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static arm64v8/debian /bin/bash + +Inside your ARM64 environment you can follow the regular native compilation steps: + +.. code-block:: + + $ cd xen.git + # install build dependencies with apt-get/apk/yum etc. + $ ./configure + $ make -j4 + +=============== +Native Building +=============== + +In order to build the tools a native build environment is required. For 32-bit the developers mainly use the armhf port of Debian, which is present in Wheezy running on an IMX53 based development board, although any ARMv7 development board would do. Note that the build hardware does not need to support the virtualisation extensions, since you don't have to run Xen on the same system as where you build it. + +See `Xen on Raspberry Pi <https://xenproject.org/2020/09/29/xen-on-raspberry-pi-4-adventures/>`__. \ No newline at end of file diff --git a/docs/tutorials/running-xen-on-arm/fastmodel.rst b/docs/tutorials/running-xen-on-arm/fastmodel.rst new file mode 100644 index 000000000000..0176f6b55f09 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/fastmodel.rst @@ -0,0 +1,152 @@ +************************************************* +Xen ARM with Virtualization Extensions/FastModels +************************************************* + +======================= +Fixed Virtual Platforms +======================= + +The primary models in use today by the Xen developers are the Fixed Virtual Platforms (FVP) modules which are available from Arm™, e.g., RTSM_VE_Cortex-A15x2 and RTSM_VE_AEMv8Ax2. + +In addition for ARMv8, Arm also makes a `Foundation Model <#Foundation_Model>`__ freely available. + +If you do not have access to the FVPs or Foundation model (e.g., you are interested in ARMv7), then you may be able to download an evaluation version of the FastModels and build an equivalent model yourself using sgcanvas, see `Building a Model with sgcanvas <.\tutorials\running-xen-on-arm\sgcanvas.rst>`__. + +============ +Known Issues +============ + +Xen boot can be slow on the models because it scrubs the memory. If it is too slow, you can add `no-bootscrub` on the Xen command line. + +=========== +Device Tree +=========== + +The device tree for the ARMv8 foundation model is upstream in Linux. + +Pawel Moll maintains a set of device tree files which describe the fast model platforms. See `arm-dts.git <http://www.linux-arm.org/git?p=arm-dts.git;a=summary>`__. + +This tree contains no build system, therefore the device tree compiler (dtc) should be invoked by hand: + +.. code-block:: + + $ git clone git://linux-arm.org/arm-dts.git arm-dts.git + $ cd arm-dts.git/fast_models + $ dtc -I dts -O dtb -o rtsm_ve-cortex_a15x2.dtb rtsm_ve-cortex_a15x2.dts + $ dtc -I dts -O dtb -o rtsm_ve-aemv8a.dtb rtsm_ve-aemv8a.dts + +You should use the dts file which describes as many CPUs as the model you intend to use (e.g., the x1, x2 or x4 suffix). In the case of the AEM DTS, you should edit it to contain the appropriate number of CPU nodes. + +======================= +Firmware and Boot-wrapper +======================= + +It is common to run the models without real firmware. In this case a boot-wrapper is required in order to provide a suitable boot time environment for Xen (e.g., booting in NS-HYP mode, providing the boot modules, etc.). Bootwrappers are available for both arm32 and arm64; however, their functionality differs significantly. + +arm32 +~~~~~ + +The arm32 boot-wrapper is the more functional version and can make use of semihosting to load the hypervisor, kernel and DTB from the host filesystem at runtime. A version of the boot-wrapper with support for Xen is available in the xen-arm32 branch of `http://xenbits.xen.org/gitweb/?p=people/ianc/boot-wrapper.git;a=summary <http://xenbits.xen.org/gitweb/?p=people/ianc/boot-wrapper.git;a=summary>`__. + +Build the Boot-wrapper +---------------------- + +.. code-block:: + + $ git clone -b xen-arm32 git://xenbits.xen.org/people/ianc/boot-wrapper.git boot-wrapper.git + $ cd boot-wrapper.git + $ make CROSS_COMPILE=arm-linux-gnueabihf- semi + +This produces a `linux-system-semi.axf` binary. This should be passed to the model as the application to run and a `cluster.cpu0.semihosting-cmd_line` option should be passed (with -C) containing the set of modules and their command lines. For example: + +.. code-block:: + + RTSM_VE_Cortex-A15x2 -C cluster.cpu0.semihosting-cmd_line=" + --kernel xen.git/xen/xen \ + --module linux.git/arch/arm/boot/zImage <DOMAIN 0 COMMAND LINE> + --dtb rtsm_ve-cortex_a15x2.dtb -- <XEN COMMAND LINE>" + <MODEL OPTIONS> boot-wrapper.git/linux-system-semi.axf + +The Command line options are as follows: + +--kernel <path-to-kernel> + Provides the "kernel", Xen in this case. +--module <path-to-module> <optional-command-line> + Supplies a boot module. In this case the first module supplied is treated as the domain 0 kernel (in zImage format). The kernel command line should be specified here too. +--dtb <path-to-dtb> + Supplies the Device Tree Blob. + +The final -- token delimits the end of the options after which the kernel (Xen in this case) command line should be supplied. + +.. note:: The entirety of the `cluster.cpu0.semihosting-cmd_line` options should be quoted from the shell. + +arm64 +~~~~~ + +The arm64 version of boot-wrapper is not as fully featured as the arm32 version and does not support semihosting. The required binaries and command lines are built directly into the boot-wrapper which must be rebuilt whenever any component changes. + +The upstream boot-wrapper-aarch64 has Xen support. It can be built as shown below: + +.. code-block:: + + $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git + $ cd boot-wrapper-aarch64 + $ autoreconf -i + $ ./configure --host=aarch64-linux-gnu \ + $ --with-kernel-dir=$KERNEL \ + $ --with-dtb=$KERNEL/arch/arm64/boot/dts/arm/foundation-v8.dtb \ + $ --with-cmdline="console=hvc0 earlycon=pl011,0x1c090000 root=/dev/vda rw" \ + $ --enable-psci \ + $ --with-xen-cmdline="dtuart=serial0 console=dtuart no-bootscrub dom0_mem=512M" \ + $ --with-xen=$XEN \ + $ --with-cpu-ids=0,1,2,3 + $ make + +Where, `$KERNEL` points to the Linux kernel directory, and `$XEN` points to the Xen binary. You need to have the cross-compile toolchain installed on your `$PATH`. + +The resulting `xen-system.axf` binary should be passed to the model as the application to run. For example: + +.. code-block:: + + $ ./Foundation_Platform --image=/path/to/xen-system.axf --block-device=<rootfs> --cores=4 + +If any of Xen, the FDT or the kernel Image change then only the final make step needs to be repeated. + +================ +Foundation Model +================ + +The ARMv8 Foundation Model is a free as in beer AArch64 emulation platform. The use is very similar to the arm64 instructions for the fastmodel using the relevant bootwrapper however the invocation of the model is slightly different: + +.. code-block:: + + ./Foundation_v8pkg/models/Linux64_GCC-4.1/Foundation_v8 \ + --image boot-wrapper-aarch64/xen-system.axf \ + --block-device rootfs.img + +The block device is exposed by the emulated hardware via virtio, therefore the root device will be `/dev/vda`. Make sure to have at least the following options enabled in your kernel config: + +.. code-block:: + + CONFIG_VIRTIO=y + CONFIG_VIRTIO_MMIO=y + CONFIG_VIRTIO_BLK=y + +=============== +FVP AEMv8 Model +=============== + +The FVP AEMv8 Model is a licensed AArch64 emulation platform. It has additional features compared to ARMv8 foundation model. + +.. code-block:: + + model_shell <FVP_AEMv8_install_directory>/models/Linux64_GCC-4.1/RTSM_VE_AEMv8A.so \ + -C motherboard.mmc.p_mmc_file=<aarch64_rootfs_image> \ + boot-wrapper-aarch64/xen-system.axf + + +.. note:: + + 1. The DTS for FVP AEMv8 model is already available in mainline Linux kernel. + 2. For trying XEN on older FVP AEMv8 model we might need to disable VIRTIO BLOCK device from FVP AEMv8 model DTS. + diff --git a/docs/tutorials/running-xen-on-arm/index.rst b/docs/tutorials/running-xen-on-arm/index.rst new file mode 100644 index 000000000000..47661328b0c1 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/index.rst @@ -0,0 +1,17 @@ +****************** +Running Xen on Arm +****************** + +.. toctree:: + :maxdepth: 4 + :hidden: + + requirements + building-xen-arm + fastmodel + odroid + arndale + qemu + raspberry-pi + sgcanvas + \ No newline at end of file diff --git a/docs/tutorials/running-xen-on-arm/midway.rst b/docs/tutorials/running-xen-on-arm/midway.rst new file mode 100644 index 000000000000..9c295db17cd1 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/midway.rst @@ -0,0 +1,94 @@ +********************************************* +Xen Arm with Virtualization Extensions/Midway +********************************************* + +Midway is the codename for the `Calxeda ECX-2000 <http://www.calxeda.com/wp-content/uploads/2013/10/Calxeda-ECX2000-PB-Oct20132.pdf>`__ based server system, featuring four A-15 cores in a micro-server oriented design. + +================== +Supported Versions +================== + +Though the Xen 4.3 release has some ARM support, it will not run on Midway. Beside the missing platform support there were several deficiencies in the code which prevent the 4.3 base (even with fixes) to run on this machine. + +Xen 4.4 supports Midway. There is Midway platform code in Xen, which will be automatically detected and used. To use the earlyprintk feature for early debug output, build Xen with "CONFIG_EARLY_PRINTK=midway". + +On the Linux kernel side you need a version which has all the fixes necessary to fully support Midway in LPAE mode. Kernel 3.12 works pretty well. However you should use the latest available version if possible to get the full support, including SWIOTLB. Configure a multi-platform kernel with support for the Calxeda SoCs, LPAE and at least the xgmac network driver as well as the Highbank SATA driver. + +=========== +Booting Xen +=========== + +The Calxeda primary firmware switches all cores into HYP mode already and installs a PSCI handler in secure state. Then it launches u-boot, which runs completely in HYP mode. So booting Xen works out of the box. The device tree is included in the firmware flash, this will be adjusted by primary firmware to match the runtime configuration (namely the memory size and parameters). There is no need to explicitly load a device tree, and you should avoid doing that. + +To load the Xen binaries, you should load the Xen hypervisor at the address usually used for the (bare-metal) Linux kernel: + +.. code-block:: + + ==> tftp $kernel_addr_r xen-4.4 + +Now tell u-boot where the device tree has been loaded and add the Xen command line: + +.. code-block:: + + ==> fdt addr $fdt_addr; fdt resize + ==> fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=/soc/serial@fff36000" + +Add any additionally needed Xen command line parameters here (like dom0_mem), if required. + +The Dom0 zImage kernel should be loaded at any address supported by Xen, the recommendation is to use 0x1000000 (16MB): + +.. code-block:: + + ==> setenv dom0_addr 0x1000000 + ==> tftp $dom0_addr vmlinuz-3.13-rc3-xen + +To tell Xen about the location of the Dom0 kernel, we will have to add a subnode to the /chosen node in the in-memory device tree: + +.. code-block:: + + ==> fdt mknod /chosen module + ==> fdt set /chosen/module compatible "xen,linux-zimage" "xen,multiboot-module" + ==> fdt set /chosen/module reg <0x0 $dom0_addr 0x0 0x$filesize> + ==> fdt set /chosen/module bootargs "console=hvc0 root=/dev/sda2" + +.. note:: $filesize reflects the file size of the last loaded file in u-boot, so subsequent ext2load/fatload/tftp commands will overwrite this value. So make sure to execute these lines right after the Dom0 kernel load. + +Now you can launch Xen: + +.. code-block:: + + ==> bootz $kernel_addr_r - $fdt_addr + +======================== +Using Network in a Guest +======================== + +The Calxeda firmware needs to know the mac address of the guest, if it plans to use network. Before creating a guest, the following command should be called: + +.. code-block:: + + bridge fdb add xx:xx:xx:xx:xx:xx dev eth0 + +This can be automated by dropping the following into `/etc/xen/scripts/vif-post.d/cxfabric.hook` (the .hook suffix is important and the file must be executable): + +.. code-block:: + + # (De)register the new device with the CX Fabric. Ignore errors from bridge fdb + # since the MAC might already be present etc. + cxfabric() { + local command=$1 + local mac=$(xenstore_read "$XENBUS_PATH/mac") + case $command in + online|add) + log debug "Adding $mac to CXFabric fdb" + do_without_error bridge fdb add $mac dev eth0 + ;; + offline) + log debug "Removing $mac from CXFabric fdb" + do_without_error bridge fdb del $mac dev eth0 + ;; + esac + } + cxfabric $command + +.. note:: The binary bridge is provided by the iproute2 package. On Debian, the package only exists on Jessie and onwards. \ No newline at end of file diff --git a/docs/tutorials/running-xen-on-arm/odroid.rst b/docs/tutorials/running-xen-on-arm/odroid.rst new file mode 100644 index 000000000000..fd02ad7b4c76 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/odroid.rst @@ -0,0 +1,144 @@ +*********************************************** +Xen Arm with Virtualization Extensions/OdroidXU +*********************************************** + +=================== +Preparing the Board +=================== + +The bootloader provided with the OdroidXU does not let Xen boot in hypervisor mode. In the Odroid forums for the XU, one is likely to see many U-Boot blobs (bl1/bl2/tzsw/u-boot) which could possibly be used to let Xen boot in hypervisor mode. To avoid multiple sources, currently a single source is provided which can be accessed from `https://github.com/suriyanr/linux-xen/tree/odroid-3.13.y/sd_fuse <https://github.com/suriyanr/linux-xen/tree/odroid-3.13.y/sd_fuse>`__. + +The `sd_fusing.sh` script found therein can be used to fuse the SD card or eMMC card with the required BL1/BL2/TZSW and U-Boot. + +.. code-block:: + + sd_fusing.sh /dev/mmcblk0 + +Ensure `/dev/mmcblk0` is the correct device which represents the SD card or eMMC card that you will be using. This will let XEN boot in hypervisor mode as well as set CNTFRQ through the trustzone. This is required for domUs to get the correct timer frequency. (dom0's optionally can pick it up from the device tree) + +============ +Building Xen +============ + +Follow Build Xen on Arm to build Xen. For more verbose debug messages from XEN, it is worthwhile to compile Xen as below: + +.. code-block:: + + make dist-xen XEN_TARGET_ARCH=arm32 debug=y CONFIG_EARLY_PRINTK=exynos5250 + +.. note:: + + Currently, Xen cannot build an uImage for U-Boot. You can create the image with: + + .. code-block:: + + mkimage -A arm -T kernel -a 0x80200000 -e 0x80200000 -C none -d "$xen_src/xen/xen" xen4.5-uImage + + where, `$xen_src` is the root directory of your xen git. + +============================================================== +Building a Device Tree for Xen, Linux Dom0 Kernel, and Modules +============================================================== + +We will build Linux for dom0 to have a device tree to provide to Xen. You can get a tree from [1] which contains a working configuration for the Odroid XU. + +.. code-block:: + + git clone -b odroid-3.13.y https://github.com/suriyanr/linux-xen.git --depth=1 + cd linux-xen + make ARCH=arm CROSS_COMPILE=$CROSS_COMPILE odroidxu_xen_defconfig + make ARCH=arm CROSS_COMPILE=$CROSS_COMPILE zImage + make ARCH=arm CROSS_COMPILE=$CROSS_COMPILE dtbs + # Note that CROSS_COMPILE has to be set appropriately or can be left unset if building natively. + +The device tree used by the OdroidXU is located in arch/arm/boot/dts/exynos5410-odroidxu.dtb + +.. code-block:: + + # Now build the dom0 modules + make ARCH=arm CROSS_COMPILE=$CROSS_COMPILE modules + # And install them + make ARCH=arm CROSS_COMPILE=$CROSS_COMPILE INSTALL_MOD_PATH=/media/suriyan/rootfs modules_install + +Note that /media/suriyan/rootfs is where the rootfs of the Odroid XU is mounted in the build box. Your path might vary or you will have to mount it appropriately. + +========================= +Booting from SD/eMMC Card +========================= + +Start with a distribution that you like, say xubuntu 14.04lts from `http://odroid.in/ubuntu_14.04lts/ubuntu-14.04lts-xubuntu-odroid-xu-20140714.img.xz <http://odroid.in/ubuntu_14.04lts/ubuntu-14.04lts-xubuntu-odroid-xu-20140714.img.xz>`__, or ArchLinuxArm, if you prefer from `http://archlinuxarm.org/platforms/armv7/samsung/odroid-xu <http://archlinuxarm.org/platforms/armv7/samsung/odroid-xu>`__. + +Once you have imaged the SD card or eMMC card with your preferred distribution, overwrite the BL1/BL2/TZSW/U-Boot as mentioned in ref:: Preparing the Board. + +Both the distributions mentioned above have a VFAT partition as the first partition. We shall use this partition to populate the XEN specific images - xen4.5-uImage, zImage, exynos5410-odroidxu.dtb - under a directory called `xen`. This is so we avoid overwriting the files that come with the default distribution in case we want to revert back the XEN changes. + +.. code-block:: + + suriyan@Stealth:/media/suriyan/BOOT$ ls -l xen + ... + -rw-r--r-- 1 suriyan suriyan 40467 Dec 8 14:04 exynos5410-odroidxu.dtb + -rw-r--r-- 1 suriyan suriyan 689052 Dec 8 15:15 xen4.5-uImage + -rw-r--r-- 1 suriyan suriyan 4709392 Dec 8 14:04 zImage + +.. note:: The file sizes that you have might be different. The purpose here is to show the directory and the names of the files therein. + +Rename the original boot.ini file in the VFAT partition to `boot.ini.org`. Copy the `boot.ini` found in `https://github.com/suriyanr/linux-xen/blob/odroid-3.13.y/sd_fuse/boot.ini <https://github.com/suriyanr/linux-xen/blob/odroid-3.13.y/sd_fuse/boot.ini>`__ in its place. + +.. note:: + + This `boot.ini` assumes that the rootfs resides in `/dev/mmcblk0p2`. If you have planned it elsewhere, change this value in the line which looks like the below appropriately. + + .. code-block:: + + setenv dom0_bootargs vmalloc=256M console=hvc0 psci=enable earlyprintk debug clk_ignore_unused root=/dev/mmcblk0p2 rootwait rw drm_kms_helper.edid_firmware=edid/1920x1080.fw video=HDMI-A-1:1920x1080MR-32@60 + +Now we can plug the SD/eMMC card to the OdroidXU and boot it up. + +Console Login Prompt +~~~~~~~~~~~~~~~~~~~~ + +hvc0 is used as the console by Xen. You will notice that `console=hvc0` is passed as the kernel boot parameter (will be the same for domU kernel as well). + +To get a login prompt for dom0/domU one has to spawn a tty on hvc0. The below steps work for 14.04* Ubuntu. + +.. code-block:: + + cp /etc/init/tty1.conf /etc/init/hvc0.conf + Replace tty1 with hvc0 in file /etc/init/hvc0.conf + +============================ +Building a Linux DomU Kernel +============================ + +Mainline Linux will be used for this purpose. The options to enable for building the domU kernel is as below: + +.. code-block:: + + make ARCH=arm CROSS_COMPILE=$CROSS_COMPILE exynos_defconfig + make ARCH=arm menuconfig + +When presented with the menu, make sure the below are enabled: + + 1. Kernel Features -> Xen guest support on ARM + 2. Device Drivers -> Block devices -> Xen virtual block device support. + 3. Device Drivers -> Network device support -> Xen network device frontend + 4. Device Drivers -> Xen driver support -> Select all. + 5. System Type -> ARM system type -> Allow multiple platforms to be selected. + 6. System Type -> Multiple platform selection -> ARMv7 based platforms + 7. System Type -> Dummy Virtual Machine. + 8. Device Drivers -> Input Device support -> Miscellaneous devices -> Xen virtual keyboard and mouse support. + +Build the linux kernel: + +.. code-block:: + + make ARCH=arm CROSS_COMPILE=$CROSS_COMPILE zImage + +This zImage can then be used as a Linux domU kernel. + +========= +Resources +========= + +[2] Information about the `OdroidXU board <http://odroid.com/dokuwiki/doku.php?id=en:odroid-xu>`__ +[3] `Odroid XU forum <http://forum.odroid.com/viewforum.php?f=59>`__ \ No newline at end of file diff --git a/docs/tutorials/running-xen-on-arm/qemu.rst b/docs/tutorials/running-xen-on-arm/qemu.rst new file mode 100644 index 000000000000..f267dff04cd8 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/qemu.rst @@ -0,0 +1,121 @@ +********************************************************** +Xen Arm with Virtualization Extensions/qemu-system-aarch64 +********************************************************** + +===================== +QEMU AArch64 Emulator +===================== + +QEMU is an Open Source GPLv2 software emulator. It can emulate a large range of machines of different architectures, including Cortex-A57s based platforms. + +The following steps help you setup QEMU to emulate an ARM64 machine and run Xen inside it. + +Building QEMU +~~~~~~~~~~~~~ + +QEMU v4.0.0 can run Xen out of the box. Build QEMU like this: + +.. code-block:: + + $ ./configure --target-list=aarch64-softmmu --prefix=/usr/local + $ make + $ sudo make install + +Installing Xen +~~~~~~~~~~~~~~ + +QEMU works well with UEFI firmware for the emulated machine. To get the system working, download an Aarch64 UEFI ready distro image, like `xenial-server-cloudimg-arm64-uefi1.img`. + +Then, clone Linux and build the kernel using the defconfig for arm64 (make defconfig && make Image.gz). For building Linux kernel for Arm64 / Arm user might need cross compilers. Below are details with respect to different distros : + +- `OpenSUSE <https://opensuse.pkgs.org/tumbleweed/opensuse-oss/cross-aarch64-gcc7-7.3.1+r258313-1.2.x86_64.rpm.html>`__ +- `UBUNTU <https://packages.ubuntu.com/en/trusty/gcc-aarch64-linux-gnu>`__ +- `Other distros <https://pkgs.org/>`__ + +Steps can be found at `http://events17.linuxfoundation.org/sites/events/files/slides/Shuah_Khan_cross_compile_linux.pdf <http://events17.linuxfoundation.org/sites/events/files/slides/Shuah_Khan_cross_compile_linux.pdf>`__. + +Run QEMU the first time, booting Linux directly (replace the MAC_ADDRESS and the paths). The following command uses user-networking and forwards port 2222 from the host to port 22 inside the virtual machine: + +.. code-block:: + + $ qemu-system-aarch64 \ + -machine virt,gic_version=3 -machine virtualization=true \ + -cpu cortex-a57 -machine type=virt -nographic \ + -smp 4 -m 4000 \ + -kernel /path/to/linux.git/arch/arm64/boot/Image.gz --append "console=ttyAMA0 root=/dev/vda1 init=/bin/sh" \ + -netdev user,id=hostnet0,hostfwd=tcp::2222-:22 -device virtio-net-device,netdev=hostnet0,mac=MAC_ADDRESS \ + -drive if=none,file=/path/to/xenial-server-cloudimg-arm64-uefi1.img,id=hd0 -device virtio-blk-device,drive=hd0 + +This command will give a shell prompt, where user might have to mount the root file system using below command : + +.. code-block:: + + $ mount -o remount,rw /dev/vda1 / + +Once mounted, user would like to set the 'root' password using 'passwd' command and then relaunch the virtual machine using above qemu command, but now remove 'init=/bin/sh'. + +Once login with 'root' user, you may need to set the ssh authorized_keys of host machine, to access the virtual machine via remote. + +Follow `Building Xen on Arm <tutorials\building-xen-arm.rst>`__ to build Xen on Arm. Then, copy the Xen on ARM efi binary and the Linux kernel to the image: + +.. code-block:: + + $ scp -P2222 /path/to/linux.git/arch/arm64/boot/Image.gz root@127.0.0.1:/boot/efi/kernel + $ scp -P2222 /path/to/xen.git/xen/xen.efi root@127.0.0.1:/boot/efi + +Let's write a config file for it: + +.. code-block: + + $ vi /boot/efi/xen.cfg + +Let's use the following config: + +.. code-block:: + + options=console=dtuart noreboot dom0_mem=512M + kernel=kernel root=/dev/vda1 init=/bin/sh rw console=hvc0 + dtb=virt-gicv3.dtb + +Now we need the device tree binary, "virt-gicv3.dtb". QEMU can generate one for you with the following command: + +.. code-block:: + + $ qemu-system-aarch64 \ + -machine virt,gic_version=3 \ + -machine virtualization=true \ + -cpu cortex-a57 -machine type=virt \ + -smp 4 -m 4096 -display none \ + -machine dumpdtb=virt-gicv3.dtb + $ scp -P2222 virt-gicv3.dtb root@127.0.0.1:/boot/efi + +We have written everything we need to the disk image. Let's poweroff the virtual machine and proceed to download the UEFI firmware binary from Linaro: [1]. + +Running QEMU +~~~~~~~~~~~~ + +The following command create a new emulated AArch64 machine with 4 Cortex A57s, 4G of RAM, booting from EFI. The command below uses user networking in QEMU, other network configuration (bridging) are possible. Please change MAC_ADDRESS for the emulated machine and path to the guest image. + +.. code-block:: + + $ qemu-system-aarch64 \ + -machine virt,gic_version=3 \ + -machine virtualization=true \ + -cpu cortex-a57 -machine type=virt \ + -smp 4 -m 4096 -display none \ + -serial mon:stdio \ + -bios /path/to/QEMU_EFI.bin \ + -netdev user,id=hostnet0,hostfwd=tcp::2222-:22 -device virtio-net-device,netdev=hostnet0,mac=MAC_ADDRESS \ + -drive if=none,file=/path/to/xenial-server-cloudimg-arm64-uefi1.img,id=hd0 -device virtio-blk-device,drive=hd0 -boot order=d + +The boot order option will enable to get to UEFI prompt. Typing "FS0:" and then "xen" will boot hypervisor. + +The Xen tools can be built further using below links : + +- `Appendix: chrooting into target file systems <https://wiki.debian.org/Arm64Qemu https://wiki.debian.org/QemuUserEmulation>`__ + +========= +Resources +========= + +[1] `https://web.eecs.umich.edu/~jcma/blog/cjrl7dk7d000brlqjrv1rt7ru/ <https://web.eecs.umich.edu/~jcma/blog/cjrl7dk7d000brlqjrv1rt7ru/>`__ \ No newline at end of file diff --git a/docs/tutorials/running-xen-on-arm/raspberry-pi.rst b/docs/tutorials/running-xen-on-arm/raspberry-pi.rst new file mode 100644 index 000000000000..91e429951935 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/raspberry-pi.rst @@ -0,0 +1,77 @@ + +**************************** +Hacking Xen on Raspberry Pi4 +**************************** + +If you intend to hack Xen on Arm® and would like to use the Raspberry Pi4 (RPi4) to do it, follow these steps to get Xen up and running using U-Boot and TFTP. I like to use TFTP because it makes it extremely fast to update any binary during development. + +.. note:: This `tutorial <https://help.ubuntu.com/community/TFTP>`__ shows you how to set up and configure a TFTP server. To configure a UART connection to get early output from Xen and Linux, see this tutorial `<https://lancesimms.com/RaspberryPi/HackingRaspberryPi4WithYocto_Part1.html>`__. + +1. Use the `rpi-imager` to format an SD card with the regular default Raspberry Pi OS. Mount the first SD card partition and edit `config.txt`. Make sure to add the following: + + .. code-block:: + + kernel=u-boot.bin + + enable_uart=1 + + arm_64bit=1 + +2. Download a suitable U-Boot binary for RPi4 (`u-boot.bin`) from any distro, for instance OpenSUSE. Download the JeOS image, then open it and save `u-boot.bin`: + + .. code-block:: + + xz -d + openSUSE-Tumbleweed-ARM-JeOS-raspberrypi4.aarch64.raw.xz + + kpartx -a + ./openSUSE-Tumbleweed-ARM-JeOS-raspberrypi4.aarch64.raw + + mount /dev/mapper/loop0p1 /mnt + cp /mnt/u-boot.bin /tmp + +3. Place the `u-boot.bin` in the first SD card partition together with `config.txt`. Next time the system boots, you will get a U-Boot prompt that allows you to load Xen, the Linux kernel for Dom0, the Dom0 rootfs, and the device tree from a TFTP server over the network. I automated the loading steps by placing a U-Boot `boot.scr` script on the SD card: + + .. code-block:: + + setenv serverip 192.168.0.1 + + setenv ipaddr 192.168.0.2 + + tftpb 0xC00000 boot2.scr + + source 0xC00000 + + Where: + + – serverip is the IP of your TFTP server + + – ipaddr is the IP of the RPi4 + +4. Use `mkimage` to generate `boot.scr` and place it next to `config.txt` and `u-boot.bin`: + + .. code-block:: + + mkimage -T script -A arm64 -C none -a 0x2400000 -e 0x2400000 -d boot.source boot.scr + + Where: + + – `boot.source` is the input + + – `boot.scr` is the output + + U-Boot will automatically execute the provided `boot.sc`r`, which sets up the network and fetches a second script (`boot2.scr`) from the TFTP server. `boot2.scr` should come with all the instructions to load Xen and the other required binaries. You can generate `b`oot2.scr` using `ImageBuilder <https://wiki.xenproject.org/wiki/ImageBuilder>`__. + +.. important:: + + Make sure to use Xen 4.14 or later. The Linux kernel should be master (or 5.9 when it is out, 5.4-rc4 works.) The Linux ARM64 default config works fine as kernel config. Any 64-bit rootfs should work for Dom0. Use the device tree that comes with upstream Linux for RPi4 (`arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dtb`). RPi4 has two UARTs; the default is bcm2835-aux-uart at address 0x7e215040. It is specified as “serial1” in the device tree instead of serial0. You can tell Xen to use serial1 by specifying on the Xen command line as shown below: + + .. code-block:: + + console=dtuart dtuart=serial1 sync_console + +The Xen command line is provided by the `boot2.scr` script generated by ImageBuilder as “xen,xen-bootargs“. After editing boot2.source you can regenerate `boot2.scr` with mkimage as shown below: + +.. code-block:: + + mkimage -A arm64 -T script -C none -a 0xC00000 -e 0xC00000 -d boot2.source boot2.scr \ No newline at end of file diff --git a/docs/tutorials/running-xen-on-arm/requirements.rst b/docs/tutorials/running-xen-on-arm/requirements.rst new file mode 100644 index 000000000000..20c97cc39342 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/requirements.rst @@ -0,0 +1,135 @@ +************ +Requirements +************ + +==================== +General Requirements +==================== + +- ARM Hardware or Software Model + See the following sections for information on hardware and models which are supported. +- Firmware + Xen requires certain functionality from the system firmware. The primary requirement is that the hypervisor must be launched in Non-Secure Hypervisor mode only. If the stock firmware on a platform does not obey this (most commonly by launching in Secure Supervisor mode) then a firmware update may be required. This support is present in U-Boot 2014.01. + + .. note:: + + Booting secondary processors on an SMP system requires firmware support for the Power State Coordination Interfaces (PSCI). Initial U-Boot support for this interface is available in `https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/log/?h=wip/psci <https://git.kernel.org/cgit/linux/kernel/git/maz/u-boot.git/log/?h=wip/psci>`__. + +- Device Tree + A device tree in the flat device tree format (.dtb). The host platform must be described in a DTB binary passed to Xen at boot time. This will be used by Xen and Dom0. Normally the regular device tree used when booting natively on the platform should be used. + +- Xen + All current work is now merged into the current development branch git://xenbits.xen.org/xen.git. It is recommended to use the latest Xen master branch. + +- Linux Kernel for dom0 + The patches necessary to boot Linux as dom0 under Xen were merged upstream in v3.7. In order to actually start guests a few additional patches were required however these patches have now been included in the v3.8 Linux release. The latest Linus' tree has everything needed to run on Xen on ARM as dom0 and domU. It is recommended to use the latest Linux release where possible. + +- dom0 userspace + The developers are using the armhf port of Debian Wheezy. + +- domU kernel + The patches necessary to boot Linux as a guest under Xen were merged upstream in v3.7. + +================================= +Requirements for Booting Natively +================================= + +Before starting to load Xen it is highly recommended to get the kernel you intend to use as dom0 booting natively (i.e., without Xen underneath). This will let you iron out any driver issues and figure out the necessary kernel command line etc before adding Xen into the mix. + +============================ +Requirements for Booting Xen +============================ + +- ImageBuilder + Many of the details necessary to boot Xen from U-Boot can be generated automatically by ImageBuilder's `uboot-script-gen`, see `ImageBuilder <..reference\imagebuilder.rst>`__. + +- Boot Protocol + Boot requirements are described in `Booting Xen from U-Boot <../tutorials/booting-xen-u-boot.rst>`__ in the Xen tree, which references the Linux arm and arm64 booting documentation. + +- Device Trees + Xen needs the device trees to be in the flat device tree format (the device tree blob or DTB). + + .. note:: It is no longer necessary to build a specific DTB for use with Xen. The Device Tree files shipped with Linux or from the Split Device Tree Repository can be used. + +- Boot Modules + At boot time Xen must be provided with a dom0 kernel blob and an optional dom0 initramfs blob. The bootloader must load these into memory and describe their location in the Device Tree Blob using the bindings specified in `Booting Xen from U-Boot <../tutorials/booting-xen-u-boot.rst>`__. + + These nodes can either be added by hand (by editing and recompiling the .dts file) or by using the `fdt` command in U-Boot to add them dynamically at boot time: + + .. code-block:: + + fdt addr ${fdt_addr} + fdt resize + + fdt set /chosen \#address-cells <1> + fdt set /chosen \#size-cells <1> + + fdt mknod /chosen module@0 + fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module" + fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} > + fdt set /chosen/module@0 bootargs "<DOMAIN 0 COMMAND LINE>" + +- Command Lines + `Booting Xen from U-Boot <../tutorials/booting-xen-u-boot.rst>`__ describes where Xen looks for both its own command line and the command line to pass to domain 0. + +Getting Xen Output +~~~~~~~~~~~~~~~~~~ + +To get output log on the UART, Xen needs to know which UART to use. This should be passed in the hypervisor command line using the "dtuart" parameter. e.g.: + +.. code-block:: + console=dtuart dtuart=myserial + +where, myserial is either an alias to the UART in the device tree (aliases are found in the aliases device tree node) or a full DTB path to the device. As Xen already uses it the UART will be disabled from the point of view of domain 0. + +For instance, this is a dummy device tree (won't work) to use the uart0 in Xen: + +.. code-block:: + + { + choosen { + bootargs = "console=dtuart dtuart=myserial"; + } + aliases { + myserial = &myserial_0; + } + myserial_0: uart0 { + ... configuration of your UART ... + } + } + +Here dtuart is configured using the myserial alias. Alternatively /uart0 (the full path to the device) could have been used. + +.. note:: If you don't see output from Xen, you can enable early printk. This option will turn on platform specific UART and output information before the console is initialized. + +=========== +Dom0 Kernel +=========== + +In general the same kernel configuration as used to boot natively, plus turning on the Xen specific options should work. A good starting point is often the "multi_v7_defconfig" + Xen options. + +If ARM_APPENDED_DTB is enabled then any appended DTB will be used instead of one supplied by Xen and the kernel will crash unless the memory in the DTB matches that location/size supplied by Xen. It is strongly recommended not to append a DTB to your dom0 kernel (or to disable APPENDED_DTB). + +=================== +DomU kernel and DTS +=================== + +Unprivileged guests can be created using xl. A simple VM config file would look like this: + +.. code-block:: + + kernel = "/root/image" + memory = 128 + name = "guest" + vcpus = 1 + disk = [ 'phy:/dev/loop0,xvda,w' ] + extra = "earlyprintk=xenboot console=hvc0 root=/dev/xvda debug rw init=/bin/sh" + +where, "/root/image" is a Linux zImage. + +==================== +Common DomU Pitfalls +==================== + +- Enabling CONFIG_DEBUG_LL in the guest kernel configuration. + Although this option can work for dom0 if configured appropriately for the host it does not work for domU (which cannot see the host UART). The symptoms of this are that the guest console will be silent because the kernel has taken a fault accessing the early UART. This can be confirmed by using the xenctx tool (found in $PREFIX/lib/xen/bin/). The tool takes a numeric domid (not a name, use xl list or xl domid $name) and dumps the VCPU state. A PC of 0x0000000c will usually indicate that an early trap has occurred. \ No newline at end of file diff --git a/docs/tutorials/running-xen-on-arm/sgcanvas.rst b/docs/tutorials/running-xen-on-arm/sgcanvas.rst new file mode 100644 index 000000000000..719b85d545e9 --- /dev/null +++ b/docs/tutorials/running-xen-on-arm/sgcanvas.rst @@ -0,0 +1,61 @@ +****************************** +Building a Model with sgcanvas +****************************** + +========================================== +Download FastModels and Evaluation License +========================================== + +You can download FastModels and an evaluation license from the ARM Info Center. In order to do so you will need to register. Once you have registered you can navigate via the Support drop-down menu, to Resources, Evaluation Products and finally Fast Models. At this point you will be asked to provide a phone number as well as a host MAC address for licensing purposes, you should enter the MAC address of the machine you intend to run the emulator on. Next you should select the Processor model (select Cortex™-A15) and host platform. + +At this point your download should begin and you should be shown your license file which you should download and save + +============ +Installation +============ + +These instructions are based on the `FE000-KT-00002-r7p0-40rel0.tgz` version of FastModels. (Note, this is an older version of FastModels) + +Unpack the tarball and run the setup.bin which is contained. Follow the wizard to install. + +.. note:: If your system is a 64 bit Debian Squeeze you need to install the package ia32-libs to be able to run setup.bin. + +================ +Building a Model +================ + +We use the example models which ship with FastModels. These are equivalent to the FVP. + +If you have access to an AEM license, then you can use `FastModels/FastModelsPortfolio_7.0/examples/RTSM_VE/Build_AEMv7A/RTSM_VE_AEMv7A.sgproj`. If you have a Cortex-A15 evaluation license, use `FastModels/FastModelsPortfolio_7.0/examples/RTSM_VE/Build_Cortex-A15x1/RTSM_VE_Cortex-A15x1.sgproj`. + +To start run, follow these steps: + +1. Run the following command: + + .. code-block:: + + sgcanvas <SGPROJ> + + Using the relevant `.sgproj` file, sgcanvas will start and load the example model. + +2. Select your target environment from the **Project, Active Configuration** menu. Select the environment which best matches your host. + +3. Click the **Build** button, and then hit **yes** to save your changes. + + sgcanvas will compile your model. The output will be similar to the following output: + + `FastModels/FastModelsPortfolio_7.0/examples/RTSM_VE/Build_Cortex-A15x1/Linux64-Release-GCC-4.1/cadi_system_Linux64-Release-GCC-4.1.so` + + Where, `FastModels/FastModelsPortfolio_7.0/examples/RTSM_VE/Build_Cortex-A15x1` corresponds to the example project which you built and `Linux64-Release-GCC-4.1` corresponds to the Active Configuration which you selected. + +.. note:: If your system is a 64 bit Debian Squeeze you need to install the package xutils-dev to be able to compile your model. + +=============== +Running a Model +=============== + +A model is run using the `model_shell` tool, or optionally modeldebugger. To run the model, pass the path to the `cadi_system_Linux64-Release-GCC-4.1.so` as the first argument and the kernel to run (e.g. the boot-wrapper) as the second: + +.. code-block:: + + model_shell FastModels/FastModelsPortfolio_7.0/examples/RTSM_VE/Build_Cortex-A15x1/Linux64-Release-GCC-4.1/cadi_system_Linux64-Release-GCC-4.1.so boot-wrapper.git/linux-system-semi.axf \ No newline at end of file