Opened 10 months ago
Last modified 4 weeks ago
#31396 needs_review enhancement
relocatable wheel version of package sage_conf — at Version 63
Reported by:  mkoeppe  Owned by:  

Priority:  major  Milestone:  sage9.5 
Component:  relocation  Keywords:  
Cc:  culler, ghkliem, dimpase, jhpalmieri, slelievre  Merged in:  
Authors:  Matthias Koeppe  Reviewers:  
Report Upstream:  N/A  Work issues:  
Branch:  u/mkoeppe/relocatable_wheel_version_of_package_sage_conf (Commits, GitHub, GitLab)  Commit:  c5a823cb524914ab985eaea09f52a87aa93c03cb 
Dependencies:  #29039, #31409, #31521, #31552, #31562, #31567, #29013  Stopgaps: 
Description (last modified by )
This version of sage_conf
is for making a wheel that packages the precompiled nonPython bits of the Sage distribution (and the wheelhouse local/var/lib/sage/wheels
), making SAGE_ROOT
(and thus SAGE_LOCAL=$SAGE_ROOT/local
) relocatable using Marc Culler's /var/tmp
surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage...
and a symlink).
The sage
script invokes sageconfig
to determine SAGE_ROOT
and SAGE_LOCAL
. In the version of sageconfig
supplied by this version of sage_conf
, we ensure that the symlink from /var/tmp/sage....
to the actual install location is set.
To test:
./bootstrap && (cd src/pkgs/sage_confrelocatable/ && tox v v v)
Specifically for building a wheel on macOS that uses XCode python 3.8:
$ SKIP_CONFIGURE=1 tox e localmacos10.15nohomebrewpython3_xcode  bash (tox e ...) $ (cd src/pkgs/sage_confrelocatable/ && python3 setup.py bdist_wheel platname macosx_10_15_x86_64)
This builds a 850MB wheel, an order of magnitude above the standard file size limit on PyPI; a file size limit increase (requested in https://github.com/pypa/pypisupport/issues/985) has been granted by the PyPI team.
In the meantime, you can use a prebuilt wheel (so far, macOS python3.8 x86_64 only) made available at https://github.com/mkoeppe/sage/releases; install the wheel using
/usr/bin/python3 m pip install user https://github.com/mkoeppe/sage/releases/download/9.3.rc0%2Btrac3139620210328/sage_conf9.3rc0cp38cp38macosx_10_14_6_x86_64.whl
Then proceed with the same instructions as https://pypi.org/project/sageconf/
Followup steps:
 Do not package up the wheelbuilding venv (247MB in
sage_root/local/lib/python3.8/sitepackages/
) (#29013); do not ship anyany
wheels; do not ship the Cython wheel; disable building all sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible;  #31602: split out
sage_root
fromsage_conf
and use installrequires in thesage_conf
sdist/wheel with @ links to GH releases to thesage_root
wheel and the built nonany
wheels.
Change History (63)
comment:1 Changed 10 months ago by
 Branch set to u/mkoeppe/relocatable_wheel_version_of_package_sage_conf
comment:2 Changed 10 months ago by
 Commit set to 9573b7c12afc5de1de743e53ff182d20490b340c
comment:3 Changed 10 months ago by
 Cc ghkliem added
 Description modified (diff)
comment:4 followup: ↓ 7 Changed 10 months ago by
 Description modified (diff)
On macOS, some shared libraries from homebrew still leak into the build:
find src/pkgs/sage_confrelocatable/.tox/python/lib/python3.9/sitepackages/sage_root name "*.so" o name "*.dylib"  xargs otool L libgiac.dylib: /usr/local/opt/gettext/lib/libintl.8.dylib R/library/cluster/libs/*.dylib: /usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 11.0.0, current version 11.0.0) /usr/local/opt/pcre/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.12.0) /usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)
Definitely should disable the R build (#30383).
Also check whether setting CC=gcc sysroot....
helps getting rid of /usr/local
leaking in.
Also need to add enablefatbinary
.
setup.py build_py
should probably use a symlink from /var/tmp...
into ~/.sage
as well so that it is easy to test relocation after removing the symlink.
Also copytree
from sage_root_source
should be changed so it overwrites (updates) the existing contents of SAGE_ROOT.
the tox test should include the test with otool
for selfcontainedness.
comment:5 Changed 10 months ago by
 Commit changed from 9573b7c12afc5de1de743e53ff182d20490b340c to f785d3cde259485597664306e191a21ad0ab9d58
Branch pushed to git repo; I updated commit sha1. New commits:
f785d3c  src/pkgs/sage_confrelocatable/tox.ini: Add environment macos10.15

comment:6 Changed 10 months ago by
Using isysroot helps but giac configuration needs more work  #31403
comment:7 in reply to: ↑ 4 Changed 10 months ago by
Replying to mkoeppe:
setup.py build_py
should probably use a symlink from/var/tmp...
into~/.sage
as well so that it is easy to test relocation after removing the symlink.
Does not work because sage resolves symlinks when it determines SAGE_ROOT.
comment:8 Changed 10 months ago by
 Dependencies changed from #29039 to #29039, #31409
comment:9 Changed 10 months ago by
 Commit changed from f785d3cde259485597664306e191a21ad0ab9d58 to e56a2a956a96bd70dbb443869697e89b7fb2b58a
Branch pushed to git repo; I updated commit sha1. New commits:
6da7ba2  Add variable TARGETS

b2d4ab8  build/pkgs/{r,rpy2}: Downgrade to optional

8732076  src/sage/repl/ipython_tests.py: Mark R interface tests # optional  rpy2

dbdf084  src/sage/structure/sage_object.pyx: Mark R interface test # optional  rpy2

737b21c  src/sage/stats/r.py: Mark all 2 doctests in this file as # optional  rpy2

4403924  src/sage/interfaces/r.py: Mark all tests # optional  rpy2

e56a2a9  Merge branch 't/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional' into t/31396/relocatable_wheel_version_of_package_sage_conf

comment:10 Changed 10 months ago by
 Commit changed from e56a2a956a96bd70dbb443869697e89b7fb2b58a to a2289744baf90e3dcb288f96ef52d471ab8251bb
Branch pushed to git repo; I updated commit sha1. New commits:
a228974  src/pkgs/sage_confrelocatable/setup.py: Move the SAGE_ROOT directory away after building

comment:11 Changed 10 months ago by
 Commit changed from a2289744baf90e3dcb288f96ef52d471ab8251bb to c260942d3085fa369117746c635b48da5a572489
Branch pushed to git repo; I updated commit sha1. New commits:
0cac965  build/pkgs/ecl: Update to 21.2.1, remove patches included in new version

6a62484  delete all merged upstream patches: MR 210, MR 231

62fac00  Merge branch 't/31336/upgrade_ecl_to_21_2_1' into t/31396/relocatable_wheel_version_of_package_sage_conf

c260942  src/pkgs/sage_confrelocatable/setup.py: Use enabledownloadfromupstreamurl

comment:12 Changed 10 months ago by
 Commit changed from c260942d3085fa369117746c635b48da5a572489 to 4f17a047ec84130462cb7f91f28ed284816cfa74
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
dc5e225  build/pkgs/python3/spkgconfigure.m4: On macOS, if the distutils test fails, try using empty ARCHFLAGS

df17ad4  tox.ini: Add configuration factors for specific homebrew python3.x

914ea58  build/pkgs/python3/spkgconfigure.m4: Get rid of stray AC_SUBST

986739b  m4/sage_check_python_for_venv.m4: Fix reason

0b3e70d  build/pkgs/python3/spkgconfigure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it

372adcb  .github/workflows/tox.yml: Update xcode versions

fc8b676  SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting

5ae8044  Merge branch 'u/mkoeppe/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' of git://trac.sagemath.org/sage into t/31396/relocatable_wheel_version_of_package_sage_conf

c5c73a9  src/pkgs/sage_confrelocatable/tox.ini: Define environments macospython3.8, macospython3.7, macospython3_xcode

4f17a04  src/pkgs/sage_confrelocatable/setup.py: Use enablefatbinary

comment:13 Changed 10 months ago by
 Description modified (diff)
comment:14 Changed 10 months ago by
 Commit changed from 4f17a047ec84130462cb7f91f28ed284816cfa74 to 851435289634de431fc8708e0c4c59a37b578b71
Branch pushed to git repo; I updated commit sha1. New commits:
8514352  src/pkgs/sage_confrelocatable/tox.ini: Run delocatelistdeps

comment:15 Changed 10 months ago by
 Description modified (diff)
comment:16 Changed 9 months ago by
 Commit changed from 851435289634de431fc8708e0c4c59a37b578b71 to 1baf390bc2bc5c29a7bfa6d7c73d31f39a1b5e65
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
c3e4093  Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES

4916415  Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default

a35428e  Merge #30383

e098133  src/pkgs/sage_conf/sage_conf.py.in: Add SAGE_SPKG_WHEELS

5dbd712  src/pkgs/sage_confpypi/setup.py: Use 'configure disablenotebook', run 'make build' so that wheels are built

839c534  src/pkgs/sage_confpypi/setup.py: Fix typo

100ff51  src/pkgs/sage_confpypi/MANIFEST.in: Graft sage_root/build/pkgs/sage_docbuild/src

afa2ffd  Add configure option disablesagelib

a4f75b8  src/pkgs/sage_confpypi: Use configure disablesagelib, update README

1baf390  Merge branch 't/29039/pip_installable_package_sage_bootstrap' into t/31396/relocatable_wheel_version_of_package_sage_conf

comment:17 Changed 9 months ago by
 Commit changed from 1baf390bc2bc5c29a7bfa6d7c73d31f39a1b5e65 to 31bba97d13f86c531eeef0a26383b98fa3671c45
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
f04c134  Merge branch 't/30383/new_package_type__optional_enabled_by_default' into t/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional

27c4bb9  Merge #31409

66c2987  build/pkgs/{sage_sws2rst,sage_docbuild}/installrequires.txt: New

7f8ec6c  build/pkgs/sagelib/src/tox.ini: Add factor sitepackages

0283da5  build/make/Makefile.in: Add wheel, setuptools_wheel to PYTHON_TOOLCHAIN to make sure that PEP 517 packages have a complete build system

f720722  build/pkgs/sagelib/src/tox.ini: Add factor nobuildisolation

c451b31  src/setup.cfg.m4 (install_requires): Add sage_conf

6700223  Merge tag '9.3.rc0' into t/30913/sagelib__add_setup_cfg__install_requires_

ac0f563  Merge branch 't/30913/sagelib__add_setup_cfg__install_requires_' into t/29039/pip_installable_package_sage_bootstrap

31bba97  Merge #29039

comment:18 Changed 9 months ago by
 Cc dimpase added
comment:19 Changed 9 months ago by
 Commit changed from 31bba97d13f86c531eeef0a26383b98fa3671c45 to f5f4a157e3a1630fc789f8f61655e123b6573eb6
Branch pushed to git repo; I updated commit sha1. New commits:
f5f4a15  src/pkgs/sage_confrelocatable/setup.py: Build the wheelhouse, but disablenotebook disabler disablesagelib

comment:20 Changed 9 months ago by
 Description modified (diff)
comment:21 Changed 9 months ago by
 Description modified (diff)
comment:22 Changed 9 months ago by
 Commit changed from f5f4a157e3a1630fc789f8f61655e123b6573eb6 to 87db1d0a371b70040b4d7448f0fadc9b50f18b54
Branch pushed to git repo; I updated commit sha1. New commits:
87db1d0  src/pkgs/sage_confrelocatable/setup.py: Use withmp=gmp instead of broken withoutsystemgmp option

comment:23 Changed 9 months ago by
 Dependencies changed from #29039, #31409 to #29039, #31409, #31521
comment:24 Changed 9 months ago by
 Commit changed from 87db1d0a371b70040b4d7448f0fadc9b50f18b54 to b871e09db5d847470fcf33eab2960554e0787bc1
comment:25 Changed 9 months ago by
 Description modified (diff)
comment:26 Changed 9 months ago by
delocatelistdeps
reports the following library dependencies:
/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib /usr/local/Cellar/gmp/6.2.1/lib/libgmp.10.dylib /usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib /usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib /usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib /usr/local/Cellar/littlecms2/2.12/lib/liblcms2.2.dylib /usr/local/Cellar/mpfr/4.1.0/lib/libmpfr.6.dylib /usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib /usr/local/Cellar/readline/8.1/lib/libreadline.8.1.dylib /usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib /usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib /usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib /usr/local/Cellar/zeromq/4.3.3/lib/libzmq.5.dylib /usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib @executable_path/../../../../Python3 liba.dylib libb.dylib libc.dylib
comment:27 Changed 9 months ago by
 Commit changed from b871e09db5d847470fcf33eab2960554e0787bc1 to 74a0ae02a05213aa5ec201245aaf3f55e8c5b037
comment:28 Changed 9 months ago by
 Commit changed from 74a0ae02a05213aa5ec201245aaf3f55e8c5b037 to 5eaaa8788437022cb2f070ef12e461e8e9619532
Branch pushed to git repo; I updated commit sha1. New commits:
aa8ff50  src/pkgs/sage_confrelocatable/setup.py: Use withoutsystemreadline

5574aa3  build/pkgs/giac/spkginstall.in [macOS]: Pass withincludedgettext so that it does not link to Homebrew libintl

f71ba56  configure.ac: If disablenotebook and disablesagelib, do not build zeromq etc.

5eaaa87  src/pkgs/sage_confrelocatable/setup.py: Configure libgd without freetype

comment:29 Changed 9 months ago by
/usr/local/Cellar/gettext/0.21/lib/libintl.8.dylib /usr/local/Cellar/harfbuzz/2.8.0/lib/libharfbuzz.0.dylib /usr/local/Cellar/libtiff/4.2.0/lib/libtiff.5.dylib /usr/local/Cellar/libxcb/1.14_1/lib/libxcb.1.1.0.dylib /usr/local/Cellar/littlecms2/2.12/lib/liblcms2.2.dylib /usr/local/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib /usr/local/Cellar/webp/1.2.0/lib/libwebp.7.dylib /usr/local/Cellar/webp/1.2.0/lib/libwebpdemux.2.dylib /usr/local/Cellar/webp/1.2.0/lib/libwebpmux.3.dylib /usr/local/Cellar/zlib/1.2.11/lib/libz.1.2.11.dylib @executable_path/../../../../Python3 liba.dylib libb.dylib libc.dylib
comment:30 followup: ↓ 36 Changed 9 months ago by
libintl
is still coming in through giac
comment:31 Changed 9 months ago by
 Commit changed from 5eaaa8788437022cb2f070ef12e461e8e9619532 to b7051a99473e4ffa3f2d17c980b619bb45472980
comment:32 followup: ↓ 39 Changed 9 months ago by
The remaining shared libraries from /usr/local
come in through our freetype
(which uses libharfbuzz from /usr/local
), libgd
(which uses libtiff), and pillow
(libtiff and everything else).
We could try not to ship matplotlib
at all in the wheelhouse  it is responsible for pulling in freetype
and pillow
 but it still shares one shared library with some of our packages  libpng
.
comment:33 Changed 9 months ago by
 Commit changed from b7051a99473e4ffa3f2d17c980b619bb45472980 to 41ac551c9b89fb9615e5b52b6164fa0ea67e3a30
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
760b6f0  configure.ac: If disablenotebook and disablesagelib, do not build zeromq etc.

0df51e5  configure.ac (disablenotebook): Disable argon2_cffi

42db661  configure.ac (disablenotebook disablesagelib): Disable ipython

c63ccb0  src/pkgs/sage_confrelocatable/tox.ini (macos): Also use isysroot with g++

1d12382  src/pkgs/sage_confrelocatable/setup.py: Use LIBGD_CONFIGURE='without...almost everything...'

1ef46d0  build/pkgs/giac: Add another patch in patches/autotools

1b0a9ee  build/pkgs/giac: Update to 1.6.0.47p3

41ac551  Merge #31562

comment:34 Changed 9 months ago by
 Dependencies changed from #29039, #31409, #31521 to #29039, #31409, #31521, #31552, #31562
comment:35 Changed 9 months ago by
 Commit changed from 41ac551c9b89fb9615e5b52b6164fa0ea67e3a30 to 00e39560744ad3832aa0614a34b8059a47db5a14
comment:36 in reply to: ↑ 30 Changed 9 months ago by
comment:37 Changed 9 months ago by
 Commit changed from 00e39560744ad3832aa0614a34b8059a47db5a14 to e41932acfaf19f3d19c640d91868696f9b8c5b94
Branch pushed to git repo; I updated commit sha1. New commits:
e41932a  src/pkgs/sage_confrelocatable/tox.ini (macos): Add more SDK paths

comment:38 Changed 9 months ago by
 Work issues set to Refactor through #31567
comment:39 in reply to: ↑ 32 Changed 9 months ago by
Replying to mkoeppe:
The remaining shared libraries from
/usr/local
come in through ourfreetype
(which uses libharfbuzz from/usr/local
)
freetype checks for harfbuzz using pkgconfig, accepts configure withoutharfbuzz
. https://github.com/freetype/freetype/blob/f9f6adb625c48ef15b5d61a3ac1709a068ea95a3/builds/unix/configure.raw#L504
comment:40 Changed 9 months ago by
 Dependencies changed from #29039, #31409, #31521, #31552, #31562 to #29039, #31409, #31521, #31552, #31562, #31567
comment:41 Changed 9 months ago by
 Description modified (diff)
comment:42 Changed 9 months ago by
 Commit changed from e41932acfaf19f3d19c640d91868696f9b8c5b94 to 4ed1bf421731e1cf7d4ad7d1179fa660e5d4364d
Branch pushed to git repo; I updated commit sha1. New commits:
87e2dba  tox.ini: Add localmacosnohomebrew and configuration facors macos{10.14,10.15,11.1 }

0458f23  tox.ini: Add macos{python3_xcode,nohomebrew}python{3.7,3.8 }

39629dd  Merge #31552

9d9c562  Merge #31562

a5f1fc7  Merge #31567

3dd4885  build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no

68c447f  tox.ini (local  bash): Set PS1

4ed1bf4  src/pkgs/sage_confrelocatable: Remove settings that are now set in SAGE_ROOT/tox.ini

comment:43 Changed 9 months ago by
 Work issues Refactor through #31567 deleted
comment:44 Changed 9 months ago by
 Description modified (diff)
comment:45 Changed 9 months ago by
 Description modified (diff)
comment:46 Changed 9 months ago by
 Description modified (diff)
comment:47 Changed 9 months ago by
 Commit changed from 4ed1bf421731e1cf7d4ad7d1179fa660e5d4364d to e4270dd90169c7ba7eb0d9a2c4356dc9c2514941
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
300d054  build/pkgs/pillow/spkginstall.in: Rename environment variable extra_build_ext to PILLOW_BUILD_EXT

8c8d83d  build/pkgs/freetype/spkginstall.in: Pass environment variable FREETYPE_CONFIGURE to configure

12c84a3  tox.ini (macosnohomebrew): Prevent /usr/local from leaking in through boost, freetype, pillow

7966b66  Merge #31409

fadd2b9  Merge #31567

59f41b9  src/pkgs/sage_confrelocatable/setup.py: Resolve SAGE_ROOT

b5edde2  tox.ini (macosnohomebrew): Fix typo in PILLOW_...

e15b152  Merge #31567

dffcdbc  tox.ini (macosnohomebrew): Fix up pillow build by setting ZLIB_ROOT

e4270dd  Merge #31567

comment:48 Changed 9 months ago by
 Commit changed from e4270dd90169c7ba7eb0d9a2c4356dc9c2514941 to 587142f996c7c1457344e960fea682522f00d8e4
Branch pushed to git repo; I updated commit sha1. New commits:
4069297  src/pkgs/sage_confrelocatable/setup.py: Unlink a previous symlink 'sage_root' in the package source tree before setting the new one

3779a25  tox.ini (macosnohomebrew): Disable more libraries to remove dependencies on homebrew

587142f  Merge #31567

comment:49 Changed 9 months ago by
 Commit changed from 587142f996c7c1457344e960fea682522f00d8e4 to c5a823cb524914ab985eaea09f52a87aa93c03cb
Branch pushed to git repo; I updated commit sha1. New commits:
c5a823c  src/pkgs/sage_confrelocatable/setup.py: Make sure that we do not package a copy of the sagelib source tree

comment:50 Changed 9 months ago by
 Description modified (diff)
comment:51 Changed 8 months ago by
 Description modified (diff)
comment:52 Changed 8 months ago by
 Description modified (diff)
comment:53 Changed 8 months ago by
 Cc jhpalmieri slelievre added
comment:54 Changed 8 months ago by
 Status changed from new to needs_review
comment:55 Changed 8 months ago by
 Description modified (diff)
comment:56 Changed 8 months ago by
 Description modified (diff)
comment:57 Changed 8 months ago by
 Status changed from needs_review to needs_work
comment:58 Changed 8 months ago by
 Description modified (diff)
comment:59 Changed 8 months ago by
 Milestone changed from sage9.3 to sage9.4
comment:60 Changed 8 months ago by
 Description modified (diff)
comment:61 Changed 8 months ago by
 Description modified (diff)
comment:62 Changed 8 months ago by
 Dependencies changed from #29039, #31409, #31521, #31552, #31562, #31567 to #29039, #31409, #31521, #31552, #31562, #31567, #29013
comment:63 Changed 8 months ago by
 Description modified (diff)
Branch pushed to git repo; I updated commit sha1. New commits:
Fix up package_data
Closer to working