Wednesday, August 24, 2016

Making Impala connection from Python on Windows (compiling lib SASL)

Time for a new war story...
  1. Install Microsoft Visual C++ Compiler for Python 2.7 (see https://blogs.msdn.microsoft.com/pythonengineering/2016/04/11/unable-to-find-vcvarsall-bat/ for more background)
  2. Copy stdint.h from http://msinttypes.googlecode.com/ (downloaded an archive) to "%USERPROFILE%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\include" (I'm running on Windows 2008 R2 x64). See http://deeplearning.net/software/theano/install_windows.html#visual-studio-and-cuda for more background
  3. Create "%USERPROFILE%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\include\unistd.h"
    #ifndef _UNISTD_H
    #define _UNISTD_H    1
    
    /* Values for the second argument to access.
       These may be OR'd together.  */
    #define R_OK    4       /* Test for read permission.  */
    #define W_OK    2       /* Test for write permission.  */
    //#define   X_OK    1       /* execute permission - unsupported in windows*/
    #define F_OK    0       /* Test for existence.  */
    
    #define access _access
    #define dup2 _dup2
    #define execve _execve
    #define ftruncate _chsize
    #define unlink _unlink
    #define fileno _fileno
    #define getcwd _getcwd
    #define chdir _chdir
    #define isatty _isatty
    #define lseek _lseek
    
    #define ssize_t unsigned int
    
    #define STDIN_FILENO 0
    #define STDOUT_FILENO 1
    #define STDERR_FILENO 2
    
    char *getpass(const char *prompt)
    {    
        return "Hello World!";
    }
    
    
    #endif /* unistd.h  */
    
  4. git clone https://github.com/cyrusimap/cyrus-sasl
  5. Follow https://jira.mongodb.org/browse/CXX-645, specifically:
    1. Use "VS2013 x64 Native Tools Command Prompt"
    2. In cyrus-sasl\lib run
      nmake /f ntmakefile STATIC=no prefix=C:\sasl64
      . Note this uses /MD.
    3. Run
      nmake /f ntmakefile prefix=C:\sasl64 STATIC=no install
      in cyrus-sasl\lib, cyrus-sasl\include, cyrus-sasl\win32\include 
    4. Rename C:\sasl64\lib\libsasl.lib to C:\sasl64\lib\sasl2.lib 
  6. Create requirements.txt
    impyla
    thrift_sasl
    pure-sasl
    
  7. Create Makefile
    SCRIPTS_HOME=$(PYTHON_HOME)/Scripts
    PIP=$(SCRIPTS_HOME)/pip.exe
    
    pip-requirements:
     $(PIP) install -r requirements.txt --global-option=build_ext \
                    --global-option=-IC:\\sasl64\\include \
                    --global-option=-LC:\\sasl64\\lib
  8. Run 'make pip-requirements PYTHON_HOME=C:\\Anaconda2-4.0.0'