EsoErik

Tuesday, April 14, 2015

 

Cython Compiler, Called by Distutils!!! YEAH, YOU, CYTHON! I HAVE AN OPTION FOR YOU!

[12:58 PM][ehvatum@pincuslab-2:~/zplrepo/ris_widget]> python3 setup.py build_ext --inplace
running build_ext
skipping 'cython/_ndimage_statistics.cpp' Cython extension (up-to-date)


Suppose that since Cython last built that .cpp file, I modified Cython iteself, and it would now generate significantly different output.  In that case, I want to regenerate the .cpp.

So, how about we not skip rebuilding the .cpp file?  What about that?  I want to not skip it.

[12:58 PM][ehvatum@pincuslab-2:~/zplrepo/ris_widget]> cython --help
Cython (http://cython.org) is a compiler for code written in the
Cython language.  Cython is based on Pyrex by Greg Ewing.

Usage: cython [options] sourcefile.{pyx,py} ...

Options:
  -V, --version                  Display version number of cython compiler
  -l, --create-listing           Write error messages to a listing file
  -I, --include-dir   Search for include files in named directory
                                 (multiple include directories are allowed).
  -o, --output-file    Specify name of generated C file
  -t, --timestamps               Only compile newer source files
  -f, --force        Compile all source files (overrides implied -t)
  -v, --verbose                  Be verbose, print file names on multiple compilation
  -p, --embed-positions          If specified, the positions in Cython files of each
                                 function definition is embedded in its docstring.
  --cleanup               Release interned objects on python exit, for memory debugging.
                                 Level indicates aggressiveness, default 0 releases nothing.
  -w, --working       Sets the working directory for Cython (the directory modules
                                 are searched from)
  --gdb                          Output debug information for cygdb
  --gdb-outdir        Specify gdb debug information output directory. Implies --gdb.

  -D, --no-docstrings            Strip docstrings from the compiled module.
  -a, --annotate                 Produce a colorized HTML version of the source.
  --line-directives              Produce #line directives pointing to the .pyx source
  --cplus                        Output a C++ rather than C file.
  --embed[=]        Generate a main() function that embeds the Python interpreter.
  -2                             Compile based on Python-2 syntax and code semantics.
  -3                             Compile based on Python-3 syntax and code semantics.
  --lenient                      Change some compile time errors to runtime errors to
                                 improve Python compatibility
  --capi-reexport-cincludes      Add cincluded headers to any auto-generated header files.
  --fast-fail                    Abort the compilation on the first error
  --warning-errors, -Werror      Make all warnings into errors
  --warning-extra, -Wextra       Enable extra warnings
  -X, --directive =[,


That sounds good.  We want to feed this "-f" option to the cython invocation distutils makes on our behalf.  Our setup.py file is the logic place to do this, as it kicks off the whole distutils horror show build process.  Let's look at our setup.py file:


from distutils.core import setup
from distutils.extension import Extension
import numpy
import os
import subprocess
import sys

cpp_source = 'cython/_ndimage_statistics_impl.cpp'
cython_source = 'cython/_ndimage_statistics.pyx'
cythoned_source = 'cython/_ndimage_statistics.cpp'
cython_source_deps = ['cython/_ndimage_statistics_impl.h']

include_dirs = [numpy.get_include()]

extra_compile_args = []
extra_link_args = []
define_macros = []

if sys.platform != 'win32':
    extra_compile_args.extend(('-O3', '-march=native'))

try:
    from Cython.Distutils import build_ext

    ext_modules = [Extension('_ndimage_statistics',
                             sources = [cython_source, cpp_source],
                             include_dirs = include_dirs,
                             define_macros = define_macros,
                             language = 'c++',
                             depends = cython_source_deps,
                             extra_compile_args = extra_compile_args,
                             extra_link_args = extra_link_args
                             )]

    setup(name = 'ris_widget',
          cmdclass = {'build_ext' : build_ext},
          ext_modules = ext_modules)
except ImportError:
    print('Cython does not appear to be installed.  Attempting to use pre-made cpp file...')

    ext_modules = [Extension('_ndimage_statistics',
                             sources = [cythoned_source, cpp_source],
                             include_dirs = include_dirs,
                             define_macros = define_macros,
                             language = 'c++',
                             depends = cython_source_deps,
                             extra_compile_args = extra_compile_args,
                             extra_link_args = extra_link_args
                             )]

    setup(name = 'ris_widget',
          ext_modules = ext_modules)

Alright... I guess we can look at distutils.extension.Extension to see if it will accept some kind of cython compiler parameter option.


class Extension:
    """Just a collection of attributes that describes an extension
    module and everything needed to build it (hopefully in a portable
    way, but there are hooks that let you be as unportable as you need).

    Instance attributes:
      name : string
        the full name of the extension, including any packages -- ie.
        *not* a filename or pathname, but Python dotted name
      sources : [string]
        list of source filenames, relative to the distribution root
        (where the setup script lives), in Unix form (slash-separated)
        for portability.  Source files may be C, C++, SWIG (.i),
        platform-specific resource files, or whatever else is recognized
        by the "build_ext" command as source for a Python extension.
      include_dirs : [string]
        list of directories to search for C/C++ header files (in Unix
        form for portability)
      define_macros : [(name : string, value : string|None)]
        list of macros to define; each macro is defined using a 2-tuple,
        where 'value' is either the string to define it to or None to
        define it without a particular value (equivalent of "#define
        FOO" in source or -DFOO on Unix C compiler command line)
      undef_macros : [string]
        list of macros to undefine explicitly
      library_dirs : [string]
        list of directories to search for C/C++ libraries at link time
      libraries : [string]
        list of library names (not filenames or paths) to link against
      runtime_library_dirs : [string]
        list of directories to search for C/C++ libraries at run time
        (for shared extensions, this is when the extension is loaded)
      extra_objects : [string]
        list of extra files to link with (eg. object files not implied
        by 'sources', static library that must be explicitly specified,
        binary resource files, etc.)
      extra_compile_args : [string]
        any extra platform- and compiler-specific information to use
        when compiling the source files in 'sources'.  For platforms and
        compilers where "command line" makes sense, this is typically a
        list of command-line arguments, but for other platforms it could
        be anything.
      extra_link_args : [string]
        any extra platform- and compiler-specific information to use
        when linking object files together to create the extension (or
        to create a new static Python interpreter).  Similar
        interpretation as for 'extra_compile_args'.
      export_symbols : [string]
        list of symbols to be exported from a shared extension.  Not
        used on all platforms, and not generally necessary for Python
        extensions, which typically export exactly one symbol: "init" +
        extension_name.
      swig_opts : [string]
        any extra options to pass to SWIG if a source file has the .i
        extension.
      depends : [string]
        list of files that the extension depends on
      language : string
        extension language (i.e. "c", "c++", "objc"). Will be detected
        from the source extensions if not provided.
      optional : boolean
        specifies that a build failure in the extension should not abort the
        build process, but simply not install the failing extension.
    """

    # When adding arguments to this constructor, be sure to update
    # setup_keywords in core.py.
    def __init__(self, name, sources,
                  include_dirs=None,
                  define_macros=None,
                  undef_macros=None,
                  library_dirs=None,
                  libraries=None,
                  runtime_library_dirs=None,
                  extra_objects=None,
                  extra_compile_args=None,
                  extra_link_args=None,
                  export_symbols=None,
                  swig_opts = None,
                  depends=None,
                  language=None,
                  optional=None,
                  **kw                      # To catch unknown keywords
                 ):
        if not isinstance(name, str):
            raise AssertionError("'name' must be a string")
        if not (isinstance(sources, list) and
                all(isinstance(v, str) for v in sources)):
            raise AssertionError("'sources' must be a list of strings")

        self.name = name
        self.sources = sources
        self.include_dirs = include_dirs or []
        self.define_macros = define_macros or []
        self.undef_macros = undef_macros or []
        self.library_dirs = library_dirs or []
        self.libraries = libraries or []
        self.runtime_library_dirs = runtime_library_dirs or []
        self.extra_objects = extra_objects or []
        self.extra_compile_args = extra_compile_args or []
        self.extra_link_args = extra_link_args or []
        self.export_symbols = export_symbols or []
        self.swig_opts = swig_opts or []
        self.depends = depends or []
        self.language = language
        self.optional = optional

        # If there are unknown keyword options, warn about them
        if len(kw) > 0:
            options = [repr(option) for option in kw]
            options = ', '.join(sorted(options))
            msg = "Unknown Extension options: %s" % options
            warnings.warn(msg)

Unsurprisingly, distutils.extension.Extension.__init__(...) not only doesn't accept any cython-specific options - it also yells about any you might attempt to supply.  I'm an intelligent person, but the docs just don't seem to be helping.  So, I fire up my debugger and wade into our distutils.core.setup(..) call.

In short order, I find myself stepping into the Cython.Distutils.build_ext.build_ext.cython_sources(..) member function.  Part of that function:


        # Setup create_list and cplus from the extension options if
        # Cython.Distutils.extension.Extension is used, otherwise just
        # use what was parsed from the command-line or the configuration file.
        # cplus will also be set to true is extension.language is equal to
        # 'C++' or 'c++'.
        #try:
        #    create_listing = self.cython_create_listing or \
        #                        extension.cython_create_listing
        #    cplus = self.cython_cplus or \
        #                extension.cython_cplus or \
        #                (extension.language != None and \
        #                    extension.language.lower() == 'c++')
        #except AttributeError:
        #    create_listing = self.cython_create_listing
        #    cplus = self.cython_cplus or \
        #                (extension.language != None and \
        #                    extension.language.lower() == 'c++')

        create_listing = self.cython_create_listing or \
            getattr(extension, 'cython_create_listing', 0)
        line_directives = self.cython_line_directives or \
            getattr(extension, 'cython_line_directives', 0)
        no_c_in_traceback = self.no_c_in_traceback or \
            getattr(extension, 'no_c_in_traceback', 0)
        cplus = self.cython_cplus or getattr(extension, 'cython_cplus', 0) or \
                (extension.language and extension.language.lower() == 'c++')
        cython_gen_pxi = self.cython_gen_pxi or getattr(extension, 'cython_gen_pxi', 0)
        cython_gdb = self.cython_gdb or getattr(extension, 'cython_gdb', False)
        cython_compile_time_env = self.cython_compile_time_env or \
            getattr(extension, 'cython_compile_time_env', None)

        # Set up the include_path for the Cython compiler:
        #    1.    Start with the command line option.
        #    2.    Add in any (unique) paths from the extension
        #        cython_include_dirs (if Cython.Distutils.extension is used).
        #    3.    Add in any (unique) paths from the extension include_dirs
        includes = self.cython_include_dirs
        try:
            for i in extension.cython_include_dirs:
                if not i in includes:
                    includes.append(i)
        except AttributeError:
            pass
        for i in extension.include_dirs:
            if not i in includes:
                includes.append(i)

        # Set up Cython compiler directives:
        #    1. Start with the command line option.
        #    2. Add in any (unique) entries from the extension
        #         cython_directives (if Cython.Distutils.extension is used).
        directives = self.cython_directives
        if hasattr(extension, "cython_directives"):
            directives.update(extension.cython_directives)

Sounds promising.  Let's take a look at Cython.Distutils.extension.Extension:


class Extension(_Extension.Extension):
    # When adding arguments to this constructor, be sure to update
    # user_options.extend in build_ext.py.
    def __init__(self, name, sources,
                 include_dirs=None,
                 define_macros=None,
                 undef_macros=None,
                 library_dirs=None,
                 libraries=None,
                 runtime_library_dirs=None,
                 extra_objects=None,
                 extra_compile_args=None,
                 extra_link_args=None,
                 export_symbols=None,
                 #swig_opts=None,
                 depends=None,
                 language=None,
                 cython_include_dirs=None,
                 cython_directives=None,
                 cython_create_listing=False,
                 cython_line_directives=False,
                 cython_cplus=False,
                 cython_c_in_temp=False,
                 cython_gen_pxi=False,
                 cython_gdb=False,
                 no_c_in_traceback=False,
                 cython_compile_time_env=None,
                 **kw):

        # Translate pyrex_X to cython_X for backwards compatibility.
        had_pyrex_options = False
        for key in list(kw.keys()):
            if key.startswith('pyrex_'):
                had_pyrex_options = True
                kw['cython' + key[5:]] = kw.pop(key)
        if had_pyrex_options:
            Extension.__init__(
                self, name, sources,
                include_dirs=include_dirs,
                define_macros=define_macros,
                undef_macros=undef_macros,
                library_dirs=library_dirs,
                libraries=libraries,
                runtime_library_dirs=runtime_library_dirs,
                extra_objects=extra_objects,
                extra_compile_args=extra_compile_args,
                extra_link_args=extra_link_args,
                export_symbols=export_symbols,
                #swig_opts=swig_opts,
                depends=depends,
                language=language,
                no_c_in_traceback=no_c_in_traceback,
                **kw)
            return

        _Extension.Extension.__init__(
            self, name, sources,
            include_dirs=include_dirs,
            define_macros=define_macros,
            undef_macros=undef_macros,
            library_dirs=library_dirs,
            libraries=libraries,
            runtime_library_dirs=runtime_library_dirs,
            extra_objects=extra_objects,
            extra_compile_args=extra_compile_args,
            extra_link_args=extra_link_args,
            export_symbols=export_symbols,
            #swig_opts=swig_opts,
            depends=depends,
            language=language,
            **kw)

        self.cython_include_dirs = cython_include_dirs or []
        self.cython_directives = cython_directives or {}
        self.cython_create_listing = cython_create_listing
        self.cython_line_directives = cython_line_directives
        self.cython_cplus = cython_cplus
        self.cython_c_in_temp = cython_c_in_temp
        self.cython_gen_pxi = cython_gen_pxi
        self.cython_gdb = cython_gdb
        self.no_c_in_traceback = no_c_in_traceback
        self.cython_compile_time_env = cython_compile_time_env

I don't see anything about our force option, but perhaps it will can be a cython_directives entry or even a kw dict entry.  Alright, let's replace our distutils Extension object with a Cython.Distutils Extension object supplied with a cython_directives dict containing force = True:


from distutils.core import setup
import numpy
import os
import subprocess
import sys

cpp_source = 'cython/_ndimage_statistics_impl.cpp'
cython_source = 'cython/_ndimage_statistics.pyx'
cythoned_source = 'cython/_ndimage_statistics.cpp'
cython_source_deps = ['cython/_ndimage_statistics_impl.h']

include_dirs = [numpy.get_include()]

extra_compile_args = []
extra_link_args = []
define_macros = []

if sys.platform != 'win32':
    extra_compile_args.extend(('-O3', '-march=native'))

try:
    from Cython.Distutils import build_ext
    build_ext.force = True
    from Cython.Distutils.extension import Extension

    ext_modules = [Extension('_ndimage_statistics',
                             sources = [cython_source, cpp_source],
                             include_dirs = include_dirs,
                             define_macros = define_macros,
                             language = 'c++',
                             depends = cython_source_deps,
                             extra_compile_args = extra_compile_args,
                             extra_link_args = extra_link_args,
                             cython_directives={
                                 'language_level' : 3,
                                 'force' : True
                                 }
                             )]

    setup(name = 'ris_widget',
          cmdclass = {'build_ext' : build_ext},
          ext_modules = ext_modules)
except ImportError:
    from distutils.extension import Extension
    print('Cython does not appear to be installed.  Attempting to use pre-made cpp file...', file=sys.stderr)

    ext_modules = [Extension('_ndimage_statistics',
                             sources = [cythoned_source, cpp_source],
                             include_dirs = include_dirs,
                             define_macros = define_macros,
                             language = 'c++',
                             depends = cython_source_deps,
                             extra_compile_args = extra_compile_args,
                             extra_link_args = extra_link_args
                             )]

    setup(name = 'ris_widget',
          ext_modules = ext_modules)

Did this fix it?

[06:09 PM][ehvatum@pincuslab-scope:~/zplrepo/ris_widget]> python setup.py build_ext --inplace
running build_ext
skipping 'cython/_ndimage_statistics.cpp' Cython extension (up-to-date)


Nope.  How about supplying force as a kw entry:


from distutils.core import setup
from distutils.extension import Extension
import numpy
import os
import subprocess
import sys

cpp_source = 'cython/_ndimage_statistics_impl.cpp'
cython_source = 'cython/_ndimage_statistics.pyx'
cythoned_source = 'cython/_ndimage_statistics.cpp'
cython_source_deps = ['cython/_ndimage_statistics_impl.h']

include_dirs = [numpy.get_include()]

extra_compile_args = []
extra_link_args = []
define_macros = []

if sys.platform != 'win32':
    extra_compile_args.extend(('-O3', '-march=native'))

try:
    from Cython.Distutils import build_ext

    ext_modules = [Extension('_ndimage_statistics',
                             sources = [cython_source, cpp_source],
                             include_dirs = include_dirs,
                             define_macros = define_macros,
                             language = 'c++',
                             depends = cython_source_deps,
                             extra_compile_args = extra_compile_args,
                             extra_link_args = extra_link_args,
                             cython_directives={'language_level' : 3},
                             force=True
                             )]

    setup(name = 'ris_widget',
          cmdclass = {'build_ext' : build_ext},
          ext_modules = ext_modules)
except ImportError:
    print('Cython does not appear to be installed.  Attempting to use pre-made cpp file...')

    ext_modules = [Extension('_ndimage_statistics',
                             sources = [cythoned_source, cpp_source],
                             include_dirs = include_dirs,
                             define_macros = define_macros,
                             language = 'c++',
                             depends = cython_source_deps,
                             extra_compile_args = extra_compile_args,
                             extra_link_args = extra_link_args
                             )]

    setup(name = 'ris_widget',
          ext_modules = ext_modules)

[10:41 AM][ehvatum@pincuslab-scope:~/zplrepo/ris_widget]> python setup.py build_ext --inplace
/usr/lib64/python3.4/distutils/extension.py:132: UserWarning: Unknown Extension options: 'cython_directives', 'force'
  warnings.warn(msg)
running build_ext
skipping 'cython/_ndimage_statistics.cpp' Cython extension (up-to-date)


Nope.

In fact, no variation of the above will ever work.  Why not?  Let's look at Cython.Distutils.build_ext.build_ext:


class build_ext(_build_ext.build_ext):

    description = "build C/C++ and Cython extensions (compile/link to build directory)"

    sep_by = _build_ext.build_ext.sep_by
    user_options = _build_ext.build_ext.user_options
    boolean_options = _build_ext.build_ext.boolean_options
    help_options = _build_ext.build_ext.help_options

    # Add the pyrex specific data.
    user_options.extend([
        ('cython-cplus', None,
         "generate C++ source files"),
        ('cython-create-listing', None,
         "write errors to a listing file"),
        ('cython-line-directives', None,
         "emit source line directives"),
        ('cython-include-dirs=', None,
         "path to the Cython include files" + sep_by),
        ('cython-c-in-temp', None,
         "put generated C files in temp directory"),
        ('cython-gen-pxi', None,
            "generate .pxi file for public declarations"),
        ('cython-directives=', None,
            "compiler directive overrides"),
        ('cython-gdb', None,
         "generate debug information for cygdb"),
        ('cython-compile-time-env', None,
            "cython compile time environment"),
            
        # For backwards compatibility.
        ('pyrex-cplus', None,
         "generate C++ source files"),
        ('pyrex-create-listing', None,
         "write errors to a listing file"),
        ('pyrex-line-directives', None,
         "emit source line directives"),
        ('pyrex-include-dirs=', None,
         "path to the Cython include files" + sep_by),
        ('pyrex-c-in-temp', None,
         "put generated C files in temp directory"),
        ('pyrex-gen-pxi', None,
            "generate .pxi file for public declarations"),
        ('pyrex-directives=', None,
            "compiler directive overrides"),
        ('pyrex-gdb', None,
         "generate debug information for cygdb"),
        ])

    boolean_options.extend([
        'cython-cplus', 'cython-create-listing', 'cython-line-directives',
        'cython-c-in-temp', 'cython-gdb',
        
        # For backwards compatibility.
        'pyrex-cplus', 'pyrex-create-listing', 'pyrex-line-directives',
        'pyrex-c-in-temp', 'pyrex-gdb',
    ])

    def initialize_options(self):
        _build_ext.build_ext.initialize_options(self)
        self.cython_cplus = 0
        self.cython_create_listing = 0
        self.cython_line_directives = 0
        self.cython_include_dirs = None
        self.cython_directives = None
        self.cython_c_in_temp = 0
        self.cython_gen_pxi = 0
        self.cython_gdb = False
        self.no_c_in_traceback = 0
        self.cython_compile_time_env = None
    
    def __getattr__(self, name):
        if name[:6] == 'pyrex_':
            return getattr(self, 'cython_' + name[6:])
        else:
            return _build_ext.build_ext.__getattr__(self, name)

    def __setattr__(self, name, value):
        if name[:6] == 'pyrex_':
            return setattr(self, 'cython_' + name[6:], value)
        else:
            # _build_ext.build_ext.__setattr__(self, name, value)
            self.__dict__[name] = value

    def finalize_options (self):
        _build_ext.build_ext.finalize_options(self)
        if self.cython_include_dirs is None:
            self.cython_include_dirs = []
        elif isinstance(self.cython_include_dirs, str):
            self.cython_include_dirs = \
                self.cython_include_dirs.split(os.pathsep)
        if self.cython_directives is None:
            self.cython_directives = {}
    # finalize_options ()

    def run(self):
        # We have one shot at this before build_ext initializes the compiler.
        # If --pyrex-gdb is in effect as a command line option or as option
        # of any Extension module, disable optimization for the C or C++
        # compiler.
        if self.cython_gdb or [1 for ext in self.extensions
                                     if getattr(ext, 'cython_gdb', False)]:
            optimization.disable_optimization()

        _build_ext.build_ext.run(self)

    def build_extensions(self):
        # First, sanity-check the 'extensions' list
        self.check_extensions_list(self.extensions)

        for ext in self.extensions:
            ext.sources = self.cython_sources(ext.sources, ext)
            self.build_extension(ext)

    def cython_sources(self, sources, extension):
        """
        Walk the list of source files in 'sources', looking for Cython
        source files (.pyx and .py).  Run Cython on all that are
        found, and return a modified 'sources' list with Cython source
        files replaced by the generated C (or C++) files.
        """
        try:
            from Cython.Compiler.Main \
                import CompilationOptions, \
                       default_options as cython_default_options, \
                       compile as cython_compile
            from Cython.Compiler.Errors import PyrexError
        except ImportError:
            e = sys.exc_info()[1]
            print(("failed to import Cython: %s" % e))
            raise DistutilsPlatformError("Cython does not appear to be installed")

        new_sources = []
        cython_sources = []
        cython_targets = {}

        # Setup create_list and cplus from the extension options if
        # Cython.Distutils.extension.Extension is used, otherwise just
        # use what was parsed from the command-line or the configuration file.
        # cplus will also be set to true is extension.language is equal to
        # 'C++' or 'c++'.
        #try:
        #    create_listing = self.cython_create_listing or \
        #                        extension.cython_create_listing
        #    cplus = self.cython_cplus or \
        #                extension.cython_cplus or \
        #                (extension.language != None and \
        #                    extension.language.lower() == 'c++')
        #except AttributeError:
        #    create_listing = self.cython_create_listing
        #    cplus = self.cython_cplus or \
        #                (extension.language != None and \
        #                    extension.language.lower() == 'c++')

        create_listing = self.cython_create_listing or \
            getattr(extension, 'cython_create_listing', 0)
        line_directives = self.cython_line_directives or \
            getattr(extension, 'cython_line_directives', 0)
        no_c_in_traceback = self.no_c_in_traceback or \
            getattr(extension, 'no_c_in_traceback', 0)
        cplus = self.cython_cplus or getattr(extension, 'cython_cplus', 0) or \
                (extension.language and extension.language.lower() == 'c++')
        cython_gen_pxi = self.cython_gen_pxi or getattr(extension, 'cython_gen_pxi', 0)
        cython_gdb = self.cython_gdb or getattr(extension, 'cython_gdb', False)
        cython_compile_time_env = self.cython_compile_time_env or \
            getattr(extension, 'cython_compile_time_env', None)

        # Set up the include_path for the Cython compiler:
        #    1.    Start with the command line option.
        #    2.    Add in any (unique) paths from the extension
        #        cython_include_dirs (if Cython.Distutils.extension is used).
        #    3.    Add in any (unique) paths from the extension include_dirs
        includes = self.cython_include_dirs
        try:
            for i in extension.cython_include_dirs:
                if not i in includes:
                    includes.append(i)
        except AttributeError:
            pass
        for i in extension.include_dirs:
            if not i in includes:
                includes.append(i)

        # Set up Cython compiler directives:
        #    1. Start with the command line option.
        #    2. Add in any (unique) entries from the extension
        #         cython_directives (if Cython.Distutils.extension is used).
        directives = self.cython_directives
        if hasattr(extension, "cython_directives"):
            directives.update(extension.cython_directives)

        # Set the target_ext to '.c'.  Cython will change this to '.cpp' if
        # needed.
        if cplus:
            target_ext = '.cpp'
        else:
            target_ext = '.c'

        # Decide whether to drop the generated C files into the temp dir
        # or the source tree.

        if not self.inplace and (self.cython_c_in_temp
                or getattr(extension, 'cython_c_in_temp', 0)):
            target_dir = os.path.join(self.build_temp, "pyrex")
            for package_name in extension.name.split('.')[:-1]:
                target_dir = os.path.join(target_dir, package_name)
        else:
            target_dir = None

        newest_dependency = None
        for source in sources:
            (base, ext) = os.path.splitext(os.path.basename(source))
            if ext == ".py":
                # FIXME: we might want to special case this some more
                ext = '.pyx'
            if ext == ".pyx":              # Cython source file
                output_dir = target_dir or os.path.dirname(source)
                new_sources.append(os.path.join(output_dir, base + target_ext))
                cython_sources.append(source)
                cython_targets[source] = new_sources[-1]
            elif ext == '.pxi' or ext == '.pxd':
                if newest_dependency is None \
                        or newer(source, newest_dependency):
                    newest_dependency = source
            else:
                new_sources.append(source)

        if not cython_sources:
            return new_sources

        module_name = extension.name

        for source in cython_sources:
            target = cython_targets[source]
            depends = [source] + list(extension.depends or ())
            if(source[-4:].lower()==".pyx" and os.path.isfile(source[:-3]+"pxd")):
                depends += [source[:-3]+"pxd"]
            rebuild = self.force or newer_group(depends, target, 'newer')
            if not rebuild and newest_dependency is not None:
                rebuild = newer(newest_dependency, target)
            if rebuild:
                log.info("cythoning %s to %s", source, target)
                self.mkpath(os.path.dirname(target))
                if self.inplace:
                    output_dir = os.curdir
                else:
                    output_dir = self.build_lib
                options = CompilationOptions(cython_default_options,
                    use_listing_file = create_listing,
                    include_path = includes,
                    compiler_directives = directives,
                    output_file = target,
                    cplus = cplus,
                    emit_linenums = line_directives,
                    c_line_in_traceback = not no_c_in_traceback,
                    generate_pxi = cython_gen_pxi,
                    output_dir = output_dir,
                    gdb_debug = cython_gdb,
                    compile_time_env = cython_compile_time_env)
                result = cython_compile(source, options=options,
                                        full_module_name=module_name)
            else:
                log.info("skipping '%s' Cython extension (up-to-date)", target)

        return new_sources

Note the enlarged, bolded line.  I have not been able to find any mechanism by which a parameter to Cython.Distutils.extension.Extension.__init__(..) can cause Cython.Distutils.build_ext.force to be set.  Additionally, distutils instantiates build_ext for us, denying us the opportunity to modify build_ext.force before use but after instantiation.  However, we may derive our own, customized build_ext:


from distutils.core import setup
import numpy
import os
import subprocess
import sys

cpp_source = 'cython/_ndimage_statistics_impl.cpp'
cython_source = 'cython/_ndimage_statistics.pyx'
cythoned_source = 'cython/_ndimage_statistics.cpp'
cython_source_deps = ['cython/_ndimage_statistics_impl.h']

include_dirs = [numpy.get_include()]

extra_compile_args = []
extra_link_args = []
define_macros = []

if sys.platform != 'win32':
    extra_compile_args.extend(('-O3', '-march=native'))

try:
    from Cython.Distutils import build_ext
    from Cython.Distutils.extension import Extension

    class build_ext_forced_rebuild(build_ext):
        def __init__(self, *va, **ka):
            super().__init__(*va, **ka)
            self.force = True

    ext_modules = [Extension('_ndimage_statistics',
                             sources = [cython_source, cpp_source],
                             include_dirs = include_dirs,
                             define_macros = define_macros,
                             language = 'c++',
                             depends = cython_source_deps,
                             extra_compile_args = extra_compile_args,
                             extra_link_args = extra_link_args,
                             cython_directives={'language_level' : 3}
                             )]

    setup(name = 'ris_widget',
          cmdclass = {'build_ext' : build_ext_forced_rebuild},
          ext_modules = ext_modules)
except ImportError:
    print('Cython does not appear to be installed.  Attempting to use pre-made cpp file...')
    from distutils.extension import Extension

    ext_modules = [Extension('_ndimage_statistics',
                             sources = [cythoned_source, cpp_source],
                             include_dirs = include_dirs,
                             define_macros = define_macros,
                             language = 'c++',
                             depends = cython_source_deps,
                             extra_compile_args = extra_compile_args,
                             extra_link_args = extra_link_args
                             )]

    setup(name = 'ris_widget',
          ext_modules = ext_modules)

Note the bolded section.  Does this work?

[12:58 PM][ehvatum@pincuslab-scope:~/zplrepo/ris_widget]> python setup.py build_ext --inplace
running build_ext
cythoning cython/_ndimage_statistics.pyx to cython/_ndimage_statistics.cpp
building '_ndimage_statistics' extension
x86_64-pc-linux-gnu-g++ -pthread -fPIC -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -c cython/_ndimage_statistics.cpp -o build/temp.linux-x86_64-3.4/cython/_ndimage_statistics.o -O3 -march=native
In file included from /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:0,
                 from /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/lib64/python3.4/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from cython/_ndimage_statistics.cpp:239:
/usr/lib64/python3.4/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by " \
  ^
x86_64-pc-linux-gnu-g++ -pthread -fPIC -I/usr/lib64/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4 -c cython/_ndimage_statistics_impl.cpp -o build/temp.linux-x86_64-3.4/cython/_ndimage_statistics_impl.o -O3 -march=native
x86_64-pc-linux-gnu-g++ -pthread -shared build/temp.linux-x86_64-3.4/cython/_ndimage_statistics.o build/temp.linux-x86_64-3.4/cython/_ndimage_statistics_impl.o -L/usr/lib64 -lpython3.4 -o /home/ehvatum/zplrepo/ris_widget/_ndimage_statistics.cpython-34.so


Yes, finally, at long last, it does.  I suppose it would have been easier to have made a Python call to unlink (delete) the generated .cpp if I wanted it rebuilt.  But, I'm using distutils to build, aren't I?  And, a build system should be capable of rebuilding, should it not?  Sadly, all build systems are terrible, and distutils isn't actually so bad in comparison to many that I have used.  The little bit of hackery I have under my belt now will be invaluable if I ever need to deal with distutils or distutils + Cython again.
Comments:
You can actually achieve the same effect without monkey patching build_ext by using the cythonize function

from Cython.Build import cythonize
ext_modules = cythonize([Extension('_ndimage_statistics',
sources = [cython_source, cpp_source],
include_dirs = include_dirs,
define_macros = define_macros,
language = 'c++',
depends = cython_source_deps,
extra_compile_args = extra_compile_args,
extra_link_args = extra_link_args,
cython_directives={'language_level' : 3}
)], force=True)

I found that the cython_directives and compiler_directives sometimes need to be defined in the cythonize function args instead, so keep that in mind if the language_level disappears.
 
dongtam
mu private
tim phong tro
http://nhatroso.com/
nhac san cuc manh
tổng đài tư vấn luật
http://dichvu.tuvanphapluattructuyen.com/
văn phòng luật
tổng đài tư vấn luật
dịch vụ thành lập công ty
http://we-cooking.com/
chém gió
trung tâm ngoại ngữ

- Tiểu tử, ta liều mạng với ngươi.

Minh La lạnh lẽo nhìn mười con Cự Long ở giữa không trung, sau đó thân thể thanh mang tràn ngập.

- Tiểu tử, ta liều mạng với ngươi.

Minh La lạnh lẽo nhìn mười con Cự Long, sau đó thân thể thanh mang tràn ngập.

- Ngao.

Một tiếng rít gào vang lên, Minh La cũng khôi phục thiên phú Viễn Cổ Minh Xà bản thể, thân thể khổng lồ không kém Cự Long.

- Hừ, ở nơi này là Thiên Long Phục Ma Trận, xem ngươi làm được gì.

Nhạc Thành quát lạnh.

- Thập phương thiên long tụ, Hạo Thiên Tháp Tru Ma.

 

Post a Comment

Subscribe to Post Comments [Atom]





<< Home

Archives

July 2009   August 2009   September 2009   October 2009   November 2009   December 2009   January 2010   September 2010   December 2010   January 2011   February 2011   April 2011   June 2011   August 2011   February 2012   June 2012   July 2012   August 2012   October 2012   November 2012   January 2014   April 2014   June 2014   August 2014   September 2014   October 2014   January 2015   March 2015   April 2015   June 2015   November 2015   December 2015   January 2016   June 2016   August 2016   January 2017   March 2017   April 2018   April 2019   June 2019   January 2020  

This page is powered by Blogger. Isn't yours?

Subscribe to Posts [Atom]