Wednesday, August 24, 2016

Making Impala connection from Python on Windows (compiling lib SASL)

Time for a new war story...
  1. Install Microsoft Visual C++ Compiler for Python 2.7 (see https://blogs.msdn.microsoft.com/pythonengineering/2016/04/11/unable-to-find-vcvarsall-bat/ for more background)
  2. Copy stdint.h from http://msinttypes.googlecode.com/ (downloaded an archive) to "%USERPROFILE%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\include" (I'm running on Windows 2008 R2 x64). See http://deeplearning.net/software/theano/install_windows.html#visual-studio-and-cuda for more background
  3. Create "%USERPROFILE%\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\include\unistd.h"
    #ifndef _UNISTD_H
    #define _UNISTD_H    1
    
    /* Values for the second argument to access.
       These may be OR'd together.  */
    #define R_OK    4       /* Test for read permission.  */
    #define W_OK    2       /* Test for write permission.  */
    //#define   X_OK    1       /* execute permission - unsupported in windows*/
    #define F_OK    0       /* Test for existence.  */
    
    #define access _access
    #define dup2 _dup2
    #define execve _execve
    #define ftruncate _chsize
    #define unlink _unlink
    #define fileno _fileno
    #define getcwd _getcwd
    #define chdir _chdir
    #define isatty _isatty
    #define lseek _lseek
    
    #define ssize_t unsigned int
    
    #define STDIN_FILENO 0
    #define STDOUT_FILENO 1
    #define STDERR_FILENO 2
    
    char *getpass(const char *prompt)
    {    
        return "Hello World!";
    }
    
    
    #endif /* unistd.h  */
    
  4. git clone https://github.com/cyrusimap/cyrus-sasl
  5. Follow https://jira.mongodb.org/browse/CXX-645, specifically:
    1. Use "VS2013 x64 Native Tools Command Prompt"
    2. In cyrus-sasl\lib run
      nmake /f ntmakefile STATIC=no prefix=C:\sasl64
      . Note this uses /MD.
    3. Run
      nmake /f ntmakefile prefix=C:\sasl64 STATIC=no install
      in cyrus-sasl\lib, cyrus-sasl\include, cyrus-sasl\win32\include 
    4. Rename C:\sasl64\lib\libsasl.lib to C:\sasl64\lib\sasl2.lib 
  6. Create requirements.txt
    impyla
    thrift_sasl
    pure-sasl
    
  7. Create Makefile
    SCRIPTS_HOME=$(PYTHON_HOME)/Scripts
    PIP=$(SCRIPTS_HOME)/pip.exe
    
    pip-requirements:
     $(PIP) install -r requirements.txt --global-option=build_ext \
                    --global-option=-IC:\\sasl64\\include \
                    --global-option=-LC:\\sasl64\\lib
  8. Run 'make pip-requirements PYTHON_HOME=C:\\Anaconda2-4.0.0'

4 comments:

Unknown said...

Hi Andrii,

I have followed the above steps. I am getting the error while using nmake /f ntmakefile STATIC=no prefix=D:\sasl64

Error
D:\cyrus-sasl\lib>nmake /f ntmakefile STATIC=no prefix=D:\sasl64
.......
....
....
cremental:no /debug ws2_32.lib kernel32.lib user32.lib gdi32.lib winspool.lib co
mdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib /out:"libsa
sl.dll" /implib:"libsasl.lib" /pdb:"libsasl.pdb" auxprop.obj canonusr.obj checkp
w.obj client.obj common.obj config.obj external.obj md5.obj saslutil.obj server.
obj seterror.obj windlopen.obj getsubopt.obj plugin_common.obj libsasl.res
Creating library libsasl.lib and object libsasl.exp
LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or c
orrupt
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 10.0
\VC\Bin\amd64\link.exe"' : return code '0x463'
Stop.

Could you please help regarding the error

Thanks

Andrii Neverov said...

Hi Manjunath,

Have you tried to installing VS 2010 SP1 like described in http://stackoverflow.com/questions/10888391/error-link-fatal-error-lnk1123-failure-during-conversion-to-coff-file-inval?

Thanks,
Andrii

ha62791 said...

For Python 2, you can use pure-sasl to replace sasl and skip the hell of compilation.
impyla uses pure-sasl when it fails to import sasl
Ref:
https://github.com/cloudera/impyla/commit/5e386d6495b580ae88efd81dd0f8927af5157bb8

Just use below 3 lines and it's done
The '--no-deps' option skips installing the sasl dependency

pip install impyla
pip install thrift_sasl --no-deps
pip install pure-sasl


Test configuration:
Windows 7
Anaconda Python 2.7
impyla 0.13.8 (thrift 0.9.3, bitarray 0.8.1, six 1.10.0)
thrift_sasl 0.2.1
pure-sasl 0.3.0
Impala 2.7 (CDH 5.9, auth_mechanism='PLAIN' with user and password)



For Python 3, however, there are two bugs that makes installing impyla on Windows less straightforward:

Parser doesn't support absolute path on windows
https://github.com/eleme/thriftpy/issues/234

TypeError: can't concat bytes to str
https://github.com/cloudera/impyla/issues/238

The thriftpy problem only occurs when you use pip to install
Use:
conda install impyla
OR:
conda install thrfitpy
pip install impyla

if you want to use a newer version of impyla


For the TypeError problem, here's a fix:
After:
pip install thrift_sasl --no-deps
pip install pure-sasl


Go to %site-packages path%\thrift_sasl
For Anaconda it is:
%UserProfile%\AppData\Local\Continuum\Anaconda3\Lib\site-packages\thrift_sasl

Remove __pycache__ if exists

Append below codes before line 94 "self._trans.write(header + body)" in "__init__.py" :

if (type(body) is str):
body = body.encode()


Test configuration:
Windows 7
Anaconda Python 3.5
impyla 0.13.8 (thrift 0.9.3, bitarray 0.8.1, six 1.10.0)
thrift_sasl 0.2.1
pure-sasl 0.3.0
Impala 2.7 (CDH 5.9, auth_mechanism='PLAIN' with user and password)

Unknown said...
This comment has been removed by the author.