[ContributorDoc] Add detailed docs for using a local build of SourceKit

[Steffeeen]

As I have struggled a lot with this, I decided to add dedicated documentation for using a local build of SourceKit. As I haven't been able to get tests which use SwiftPMTestProject to run I'm also looking for input on that. Ideally, I would like to change the documentation to include general instructions which work for all tests or if that's not easily possible, at least layout the steps to get these tests to work.

I have already worked out that the toolchain needs to have clang and clangd. The default swift build does not have them in swift-macosx-arm64/bin. However, creating symlinks to the Xcode toolchain clang and clangd works. Another issue is that the swift build does not have the PackageDescription module which causes the parsing of the Package.swift to fail.

Both of these issues can be fixed by including --swift-testing, --swift-testing-macros, --llbuild, --llbuild, --install-all in the build-script invocation. This creates a toolchain-macosx-arm64 directory which contains an Xcode toolchain with clang, clangd and the PackageDescription. When using this toolchain, the tests work. However, I have not been able to figure how to get ninja to update this toolchain when I recompile SourceKit.

New Environment Variable to force in-process Sourcekit

I added a new environment variable SOURCEKIT_LSP_RUN_SOURCEKIT_IN_PROCESS to explicitly force SourceKit to be run in-process on macOS. For this to work correctly, we also have to make a small change in the cmake setup in the main Swift repo. https://github.com/swiftlang/swift/blob/main/cmake/modules/SwiftComponents.cmake#L87-L91 needs to be changed to be:

if(NOT CMAKE_SYSTEM_NAME MATCHES "Darwin")
  list(REMOVE_ITEM _SWIFT_DEFAULT_COMPONENTS "sourcekit-xpc-service")
endif()

instead of

if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
  list(REMOVE_ITEM _SWIFT_DEFAULT_COMPONENTS "sourcekit-inproc")
else()
  list(REMOVE_ITEM _SWIFT_DEFAULT_COMPONENTS "sourcekit-xpc-service")
endif()

This is because we need sourcekit-inproc, even when using XPC. Without this change we don't get a .framework directory for sourcekitdInProc. I haven't opened a PR for this yet, as I'm unsure if this is the correct place and/or way to fix this.

In my opinion, this is better than forcing this in the build-script invocation by adding --extra-cmake-options='-DSWIFT_SOURCEKIT_USE_INPROC_LIBRARY:BOOL=FALSE' as the environment variable approach avoids a reconfigure. The build-script approach would also need a few changes to different CMakeLists.txt to make it work correctly.

I can also move the commit for the environment variable to a new PR, if that's preferred.

[rintaro]

I understand having sourcekitdInProc is convenient for debugging.

if(NOT CMAKE_SYSTEM_NAME MATCHES "Darwin")
  list(REMOVE_ITEM _SWIFT_DEFAULT_COMPONENTS "sourcekit-xpc-service")
endif()

instead of

if(CMAKE_SYSTEM_NAME MATCHES "Darwin")
  list(REMOVE_ITEM _SWIFT_DEFAULT_COMPONENTS "sourcekit-inproc")
else()
  list(REMOVE_ITEM _SWIFT_DEFAULT_COMPONENTS "sourcekit-xpc-service")
endif()

This is because we need sourcekit-inproc, even when using XPC. Without this change we don't get a .framework directory for sourcekitdInProc. I haven't opened a PR for this yet, as I'm unsure if this is the correct place and/or way to fix this.

Yes this is the correct way to do it. We might want to briefly discuss this internally, but feel free to open a PR for it now.

[Steffeeen]

I have opened swiftlang/swift#88159 for the CMake change.

[rintaro]

Since you mentioned above, is it worth to add SOURCEKIT_LOGGING=1 here as well?

[Steffeeen]

I guess it would make sense to note in the docs that one probably wants to also specify SOURCEKIT_LOGGING when using SOURCEKIT_LSP_RUN_SOURCEKITD_IN_PROCESS even tough it is not strictly required.

The SOURCEKIT_LOGGING environment variables also does not seem to documented anywhere. In SourceKit-LSP I only found it in my docs and in the swift repo the only reference I found was in the actual code where it is read. It should probably be documented somewhere (in the SourceKit readme?), and we can then also link to that documentation in the SourceKit-LSP docs.

[rintaro]

(Haven't tried)
Maybe this can be resolved by build-script -r --install-llvm --install-swift (which installs clangd, clang, swiftc etc into e.g. ${build_dir}/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain.

SOURCEKIT_TOOLCHAIN_PATH="$PWD/../build/Ninja-RelWithDebInfoAssert/toolchain-macosx-arm64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr" \
swift test

[Steffeeen]

Building the toolchain works, I now also finally got ninja to correctly update the toolchain. I currently just set the DESTDIR environment variable to point to the toolchain directory before calling ninja:

DESTDIR=../toolchain-macosx-arm64 ninja sourcekitd sourcekit-inproc install-sourcekit-xpc-service install-sourcekit-inproc

But it should probably also be possible to make the install-sourcekit-xpc-service and install-sourcekit-inproc targets install into the toolchain if a toolchain was created. But I currently don't know how.

[Steffeeen]

When this initializer is used, the environment variable would not work currently, as it is not passed to the Toolchain initializer. However, as far as I can tell, this initializer is currently never used and could be removed entirely.