building on colab/ search API

[cmacdonald]

Hi, I didnt manage to get this to build on Colab:

!apt-get install libtool m4 automake
!rm -f out.log
!pip --log out.log  install git+https://github.com/pisa-engine/pypisa.git

The output was

2021-04-01T15:29:14,505   Scanning dependencies of target pypisa
2021-04-01T15:29:14,610   [100%] Building CXX object CMakeFiles/pypisa.dir/src/binder.cpp.o
2021-04-01T15:30:03,942   [100%] Linking CXX shared module ../lib.linux-x86_64-3.7/pypisa.cpython-37m-x86_64-linux-gnu.so
2021-04-01T15:30:04,267   /usr/bin/ld: external/pisa/libpisa.a(tokenizer.cpp.o): relocation R_X86_64_PC32 against symbol `_ZTVN5boost16exception_detail19error_info_injectorINS_7bad_getEEE' can not be used when making a shared object; recompile with -fPIC
2021-04-01T15:30:04,269   /usr/bin/ld: final link failed: Bad value
2021-04-01T15:30:04,269   collect2: error: ld returned 1 exit status
2021-04-01T15:30:04,270   CMakeFiles/pypisa.dir/build.make:96: recipe for target '../lib.linux-x86_64-3.7/pypisa.cpython-37m-x86_64-linux-gnu.so' failed
2021-04-01T15:30:04,270   make[2]: *** [../lib.linux-x86_64-3.7/pypisa.cpython-37m-x86_64-linux-gnu.so] Error 1
2021-04-01T15:30:04,272   CMakeFiles/Makefile2:83: recipe for target 'CMakeFiles/pypisa.dir/all' failed
2021-04-01T15:30:04,272   make[1]: *** [CMakeFiles/pypisa.dir/all] Error 2
2021-04-01T15:30:04,273   Makefile:151: recipe for target 'all' failed
2021-04-01T15:30:04,273   make: *** [all] Error 2

Also, any plans to finish the search() function?

[amallia]

Let me take a look at this error and get back to you.
I wrote this wrapper few months ago and never managed to finish it. We (@JMMackenzie ) were planning to restart development soon...

[cmacdonald]

Here's a notebook to run: https://colab.research.google.com/drive/14Y77x92Puj9mFCkFMVB4ieic67--lEhk?usp=sharing

[JMMackenzie]

Thanks Craig, I am also having the same issue.

/usr/bin/ld: external/pisa/libpisa.a(tokenizer.cpp.o): relocation R_X86_64_PC32 against symbol `_ZTVN5boost16exception_detail19error_info_injectorINS_7bad_getEEE' can not be used when making a shared object; recompile with -fPIC

I ran through the compilation (manually)):

/usr/bin/c++ -march=native -O3 -DNDEBUG  -shared  -o ../lib.linux-x86_64-3.8/pypisa.cpython-38-x86_64-linux-gnu.so CMakeFiles/pypisa.dir/src/binder.cpp.o -Wl,-rpath,/home/jmmacke/pypisa/build/temp.linux-x86_64-3.8/external/pisa/external/tbb_cmake_build/tbb_cmake_build_subdir_release external/pisa/libpisa.a -pthread external/pisa/external/tbb_cmake_build/tbb_cmake_build_subdir_release/libtbb.so.2 external/pisa/external/FastPFor/libFastPFor.a external/pisa/external/libstreamvbyte.a external/pisa/external/libMaskedVByte.a external/pisa/external/libsimdcomp.a gumbo-parser/lib/libgumbo.a external/pisa/external/boost-cmake/libboost_filesystem.a external/pisa/external/boost-cmake/libboost_system.a external/pisa/external/Porter2/libPorter2.a external/pisa/external/KrovetzStemmer/libKrovetzStemmer.a external/pisa/external/wapopp/libwapopp.a external/pisa/external/fmt/libfmt.a
/usr/bin/ld: external/pisa/libpisa.a(tokenizer.cpp.o): relocation R_X86_64_PC32 against symbol `_ZTVN5boost16exception_detail19error_info_injectorINS_7bad_getEEE' can not be used when making a shared object; recompile with -fPIC

After trial and error, removing all links which caused issues led me to:

/usr/bin/c++ -march=native -O3 -DNDEBUG  -shared  -o ../lib.linux-x86_64-3.8/pypisa.cpython-38-x86_64-linux-gnu.so CMakeFiles/pypisa.dir/src/binder.cpp.o -Wl,-rpath,/home/jmmacke/pypisa/build/temp.linux-x86_64-3.8/external/pisa/external/tbb_cmake_build/tbb_cmake_build_subdir_release  -pthread external/pisa/external/tbb_cmake_build/tbb_cmake_build_subdir_release/libtbb.so.2 external/pisa/external/FastPFor/libFastPFor.a external/pisa/external/libMaskedVByte.a external/pisa/external/Porter2/libPorter2.a external/pisa/external/KrovetzStemmer/libKrovetzStemmer.a external/pisa/external/wapopp/libwapopp.a

So, there seems to be linking problems with:

• libpisa.a
• libgumbo.a
• libboost_filesystem.a
• libstreamvbyte.a
• libfmt.a
• libsimdcomp.a
• libboost_system.a

I suspect we're just not setting up the build correctly, but I haven't managed to track down the specific problem just yet.

[JMMackenzie]

I've had some luck adding set_target_properties(pisa PROPERTIES POSITION_INDEPENDENT_CODE TRUE) to CMakeLists.txt but I suspect we need to do something similar for all of the libraries (perhaps).

[cmacdonald]

It took a /long/ time to build on Colab.

For it to have useful utility, I think you need to provide compiled builds. For pytrec_eval_terrier, Sean and I developed the following Github action, which does manylinux, macOs and Windows binary builds to pypi:

https://github.com/terrierteam/pytrec_eval/blob/master/.github/workflows/publish-to-pypi.yml

[JMMackenzie]

Yeah you're right, we're aware of the long build times. PISA is mostly contained in headers, so it takes a really long time to unroll and inline them all.

Thanks for the tip, we'll have a look at your approach, it looks very useful!

[JMMackenzie]

Okay, adding set(CMAKE_POSITION_INDEPENDENT_CODE ON) to PISA's CMakeLists.txt almost solves the issue, but Gumbo is still broken. I have found a way to manually get it working, but can't quite get CMake to do what I want it to do. I suspect it has something to do with the way in which PISA deals with Gumbo and the Gumbo build. @amallia and @elshize -- could either of you try to reproduce?

A small workaround I found is as follows:

I can update the CMakeLists.txt in external/pisa as described above and then run (from the pypisa directory):

$python3.8 setup.py build

Eventually it will error out:

/usr/bin/ld: gumbo-parser/lib/libgumbo.a(libgumbo_la-parser.o): relocation R_X86_64_PC32 against symbol `kGumboEmptyString' can not be used when making a shared object; recompile with -fPIC

At this stage, I can go to the build directory and replace gumbo with the shared object and linking is successful:

$cd build/temp.linux-x86_64-3.8/

$/usr/bin/c++ -march=native -O3 -DNDEBUG  -shared  -o ../lib.linux-x86_64-3.8/pypisa.cpython-38-x86_64-linux-gnu.so CMakeFiles/pypisa.dir/src/binder.cpp.o -Wl,-rpath,/home/jmmacke/pypisa/build/temp.linux-x86_64-3.8/external/pisa/external/tbb_cmake_build/tbb_cmake_build_subdir_release external/pisa/libpisa.a -pthread external/pisa/external/tbb_cmake_build/tbb_cmake_build_subdir_release/libtbb.so.2 external/pisa/external/FastPFor/libFastPFor.a external/pisa/external/libstreamvbyte.a external/pisa/external/libMaskedVByte.a external/pisa/external/libsimdcomp.a gumbo-parser/lib/libgumbo.so.1.0.0 external/pisa/external/boost-cmake/libboost_filesystem.a external/pisa/external/boost-cmake/libboost_system.a external/pisa/external/Porter2/libPorter2.a external/pisa/external/KrovetzStemmer/libKrovetzStemmer.a external/pisa/external/wapopp/libwapopp.a external/pisa/external/fmt/libfmt.a

Then, I can go back to the library directory:

$cd ../lib.linux-x86_64-3.8/
$ls 
pypisa.cpython-38-x86_64-linux-gnu.so

However, if you try to import pypisa, it can't find gumbo:

$python3.8
Python 3.8.0 (default, Feb 25 2021, 22:10:10) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pypisa
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libgumbo.so.1: cannot open shared object file: No such file or directory

So, linking to gumbo here:

 ln -s ../temp.linux-x86_64-3.8/gumbo-parser/lib/*so* .

FInally, we can import pypisa:

$ python3.8
Python 3.8.0 (default, Feb 25 2021, 22:10:10) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pypisa
>>> pypisa.compress()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: compress(): incompatible function arguments. The following argument types are supported:
    1. (arg0: str, arg1: str, arg2: str, arg3: int, arg4: bool) -> None

[elshize]

It took a /long/ time to build on Colab.

Just a note that I'm assuming that it's by default building on one core, so as a temporary solution, we could think of parameterizing that somehow. Just a thought.

But in general, I agree that having binary wheels would be preferrable.

There's one caveat, though. Many integer encoding methods extensively rely on SIMD. Ideally, I would like to see an option to install it manually. I'm not sure how these things would work as a python wrapper, though.

Anyway, I'm making my way through the comments, and I'll try to compile the project. I'll keep you posted.

[elshize]

Ok, I think I figured it out. First of all, we want to avoid using dynamic linking for PISA's dependencies, or else we go insane. So the problem is that cmake doesn't tell the static gumbo library to use -fPIC. So we have to tell it explicily. So besides set(CMAKE_POSITION_INDEPENDENT_CODE ON), we also need to modify this bit:

CONFIGURE_COMMAND ./autogen.sh && ./configure --prefix=${CMAKE_BINARY_DIR}/gumbo-parser CFLAGS=-fPIC

Notice CFLAGS=-fPIC at the end of the line.

With this, I'm able to import the installed python package. I didn't test any functions, though.

[JMMackenzie]

@elshize Perfect, with both of those modifications, it does indeed build for me. Both of these changes need to be done over on PISA itself to get PyPISA building, so we can continue fixing there.