python3: Restructure python3 packaging and replace it with autopackaging

See previous commit (python2 version) for more info, since mostly
everything applies here as well.

Old manifest file had several issues:
 - Its unorganized and hard to read and understand it for an average
   human being.
 - When a new package needs to be added, the user actually has to modify
   the script that creates the manifest, then call the script to create
   a new manifest, and then submit a patch for both the script and the
   manifest, so its a little convoluted.
 - Git complains every single time a patch is submitted to the manifest,
   since it violates some of its guidelines.
 - It changes or may change with every release of python, its impossible
   to know if the required files for a certain package have changed
   (it could have more or less dependencies), the only way of doing so
   would be to install and test them all one by one on separate individual
   images, and even then we wouldnt know if they require less dependencies,
   we would just know if an extra dependency is required since it would
   complain, lets face it, this isnt feasible.
 - The same thing happens for new packages, if someone wants to add a new
   package, its dependencies need to be checked manually one by one.

Features/Fixes:
 - A new manifest format is used (JSON), easy to read and understand.
   This file is parsed by the python recipe and python packages
   read from here are passed directly to bitbake during parsing time.
 - It provides an automatic manifest creation task (explained on previous
   commit), which automagically checks for every package dependencies and
   adds them to the new manifest, hence we will have on each package
   exactly what that package needs to be run, providing finer granularity.
  - Dependencies are also checked automagically for new packages
   (explained on previous commit).

This patch has the same features as the python2 version but it differs
in the following ways:
 - Python3 handles precompiled bytecode files  (*.pyc) differently.
   for this reason and since we are cross compiling, wildcards couldnt be
   avoided on python3 (See PEP #3147 [1]).
   Both the manifest and the manifest creation script handle this
   differently, the manifest for python3 has an extra field for cached
   files, which is how it lets the user install the cached files or not
   via : INCLUDE_PYCS = "1" on their local.conf.
 - Shared libraries nomenclature also changed on python3, so again, we
   use wildcards to deal with this issue ( See PEP #3149 [2]):
 - Fixes python3 manifest, python3-core should be base and everything
   should depend on it, hence several packages were deleted:
   python3-enum, re, gdbm, subprocess, signal, readline.
 - When building python3-native it adds as symlink to it called
   nativepython3, which is then isued by the create_manifest task.
 - Fixes [YOCTO #11513] while were at it.

References:
[1] https://www.python.org/dev/peps/pep-3147/
[2] https://www.python.org/dev/peps/pep-3149/

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>

1 files changed, 146 insertions, 0 deletions

diff --git a/meta/recipes-devtools/python/python3/get_module_deps3.py b/meta/recipes-devtools/python/python3/get_module_deps3.py
new file mode 100644
index 0000000000..02931d4511
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/get_module_deps3.py
@@ -0,0 +1,146 @@
+# This script is launched on separate task for each python module
+# It checks for dependencies for that specific module and prints 
+# them out, the output of this execution will have all dependencies
+# for a specific module, which will be parsed an dealt on create_manifest.py
+#
+# Author: Alejandro Enedino Hernandez Samaniego "aehs29" <alejandro.hernandez@intel.com>
+
+# We can get a log per module, for all the dependencies that were found, but its messy.
+debug=False
+
+import sys
+
+# We can get a list of the modules which are currently required to run python
+# so we run python-core and get its modules, we then import what we need
+# and check what modules are currently running, if we substract them from the
+# modules we had initially, we get the dependencies for the module we imported.
+
+# We use importlib to achieve this, so we also need to know what modules importlib needs
+import importlib
+
+core_deps=set(sys.modules)
+
+def fix_path(dep_path):
+    import os
+    # We DONT want the path on our HOST system
+    pivot='recipe-sysroot-native'
+    dep_path=dep_path[dep_path.find(pivot)+len(pivot):]
+
+    if '/usr/bin' in dep_path:
+        dep_path = dep_path.replace('/usr/bin''${bindir}')
+
+    # Handle multilib, is there a better way?
+    if '/usr/lib32' in dep_path:
+        dep_path = dep_path.replace('/usr/lib32','${libdir}')
+    if '/usr/lib64' in dep_path:
+        dep_path = dep_path.replace('/usr/lib64','${libdir}')
+    if '/usr/lib' in dep_path:
+        dep_path = dep_path.replace('/usr/lib','${libdir}')
+    if '/usr/include' in dep_path:
+        dep_path = dep_path.replace('/usr/include','${includedir}')
+    if '__init__.' in dep_path:
+        dep_path =  os.path.split(dep_path)[0]
+    return dep_path
+
+
+# Module to import was passed as an argument
+current_module =  str(sys.argv[1]).rstrip()
+if(debug==True):
+    log = open('log_%s' % current_module,'w')
+    log.write('Module %s generated the following dependencies:\n' % current_module)
+try: 
+    importlib.import_module('%s' % current_module)
+except ImportError as e:
+    if (debug==True):
+        log.write('Module was not found')
+    pass
+
+
+# Get current module dependencies, dif will contain a list of specific deps for this module
+module_deps=set(sys.modules)
+
+# We handle the core package (1st pass on create_manifest.py) as a special case
+if current_module == 'python-core-package':
+    dif = core_deps
+else:
+    # We know this is not the core package, so there must be a difference.
+    dif = module_deps-core_deps
+
+
+# Check where each dependency came from
+for item in dif:
+    dep_path=''
+    try:
+        if (debug==True):
+            log.write('Calling: sys.modules[' + '%s' % item + '].__file__\n')
+        dep_path = sys.modules['%s' % item].__file__
+    except AttributeError as e:
+        # Deals with thread (builtin module) not having __file__ attribute
+        if debug==True:
+            log.write(item + ' ')
+            log.write(str(e))
+            log.write('\n')
+        pass
+    except NameError as e:
+        # Deals with NameError: name 'dep_path' is not defined
+        # because module is not found (wasn't compiled?), e.g. bddsm
+        if (debug==True):
+            log.write(item+' ') 
+            log.write(str(e))                                              
+        pass
+
+    # Site-customize is a special case since we (OpenEmbedded) put it there manually
+    if 'sitecustomize' in dep_path:
+        dep_path = '${libdir}/python2.7/sitecustomize.py'
+        # Prints out result, which is what will be used by create_manifest
+        print (dep_path)
+        continue
+
+    dep_path = fix_path(dep_path)
+
+    import sysconfig
+    soabi=sysconfig.get_config_var('SOABI')
+    # Check if its a shared library and deconstruct it
+    if soabi in dep_path:
+        if (debug==True):
+            log.write('Shared library found in %s' % dep_path)
+        dep_path = dep_path.replace(soabi,'*')
+        print (dep_path)
+        continue
+
+    if (debug==True):
+        log.write(dep_path+'\n')
+    # Prints out result, which is what will be used by create_manifest
+    print (dep_path)
+
+
+    import imp
+    cpython_tag = imp.get_tag() 
+    cached=''
+    # Theres no naive way to find *.pyc files on python3
+    try:
+        if (debug==True):
+            log.write('Calling: sys.modules[' + '%s' % item + '].__cached__\n')
+        cached = sys.modules['%s' % item].__cached__
+    except AttributeError as e:
+        # Deals with thread (builtin module) not having __cached__ attribute
+        if debug==True:
+            log.write(item + ' ')
+            log.write(str(e))
+            log.write('\n')
+        pass
+    except NameError as e:
+        # Deals with NameError: name 'cached' is not defined
+        if (debug==True):
+            log.write(item+' ') 
+            log.write(str(e))                                              
+        pass
+    if cached is not None:
+        if (debug==True):
+            log.write(cached)
+        cached = fix_path(cached)
+        cached = cached.replace(cpython_tag,'*')
+        print (cached)
+
+if debug==True:
+    log.close()