data_smart: allow python snippets to include a dictionary

[YOCTO #14917]

Attempting to use a dictionary in a python code snippet for variable
assignment results in an error. For example attempting something such
as

  IDX = "green"
  VAL = "${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}"

produces the error

  expansion of VAL threw ExpansionError: Failure expanding variable
  VAL, expression was ${@{ 'green': 1, 'blue': 2 }[d.getVar('IDX')]}
  which triggered exception SyntaxError: '{' was never closed (Var
  <VAL>, line 1)

The existing __expand_python_regexp__, "\${@.+?}", will match the
first close curly bracket encountered, resulting in incomplete and
un-parsable code, and thus produce the error. We can correct this by
allowing a single depth of nested curly brackets in
__expand_python_regexp__ by using "\${@(?:{.*?}|.)+?}", which will
match up to and including the matching close curly bracket to the
open, '${@', curly bracket, even if there are one or more singly
nested curly brackets present. This change allows the usecase
described above to function.

This change can't be made on its own though. The old regex would, in
an obscure way, handle the case where a python snippet contained an
unexpandable variable. Since the unexpandable variable is in curly
brackets it would cause incomplete/un-parsable python code and thus
remain unparsed. So something like

  VAL = "${@d.getVar('foo') + ${unsetvar}}"

would remain unparsed as the close curly bracket in "${unsetvar}"
would match and terminate the snippet prematurely. This quirk resulted
in the proper handling of python snippets with unexpanded
variables. With the change to __expand_python_regexp__ the full
snippet will match and be parsed, but to match the old/correct
behavior we would not want to parse it until ${unsetvar} can be
expanded. To ensure the old/correct behavior for python snippets with
unexpanded variables remains in place we add a check for unexpanded
variables in the python snippets before running them.

This handling of unparsed variables brings two benefits. The first we
now have an explicit check visible to all for unexpanded variables
instead of a somewhat hidden behavior. The second is that if there are
multiple python snippets the old behavior would run the code for each
but a single snippet with unexpanded variables would mean all snippets
would remain unparsed, meaning more and repeated processing at a later
time.

For example:
  "${@2*2},${@d.getVar('foo') ${unsetvar}}"
old behavior would give:
  "${@2*2},${@d.getVar('foo') ${unsetvar}}"
new behavior will give:
  "4,${@d.getVar('foo') ${unsetvar}}"

The old behavior would calculate '2*2' but toss the result when the
second snippet would fail to parse resulting in future recalculations
(or fetching from cache), while the new behavior avoids this.

Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

2 files changed, 15 insertions, 1 deletions

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index dd20ca557..62d0c01c4 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -29,7 +29,7 @@ logger = logging.getLogger("BitBake.Data")
 __setvar_keyword__ = [":append", ":prepend", ":remove"]
 __setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>:append|:prepend|:remove)(:(?P<add>[^A-Z]*))?$')
 __expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~:]+?}")
-__expand_python_regexp__ = re.compile(r"\${@.+?}")
+__expand_python_regexp__ = re.compile(r"\${@(?:{.*?}|.)+?}")
 __whitespace_split__ = re.compile(r'(\s)')
 __override_regexp__ = re.compile(r'[a-z0-9]+')
 
@@ -119,6 +119,11 @@ class VariableParse:
             else:
                 code = match.group()[3:-1]
 
+            # Do not run code that contains one or more unexpanded variables
+            # instead return the code with the characters we removed put back
+            if __expand_var_regexp__.findall(code):
+                return "${@" + code + "}"
+
             if self.varname:
                 varname = 'Var <%s>' % self.varname
             else:
diff --git a/lib/bb/tests/data.py b/lib/bb/tests/data.py
index e667c7c7d..8c043b709 100644
--- a/lib/bb/tests/data.py
+++ b/lib/bb/tests/data.py
@@ -60,6 +60,15 @@ class DataExpansions(unittest.TestCase):
         val = self.d.expand("${@5*12}")
         self.assertEqual(str(val), "60")
 
+    def test_python_snippet_w_dict(self):
+        val = self.d.expand("${@{ 'green': 1, 'blue': 2 }['green']}")
+        self.assertEqual(str(val), "1")
+
+    def test_python_unexpanded_multi(self):
+        self.d.setVar("bar", "${unsetvar}")
+        val = self.d.expand("${@2*2},${foo},${@d.getVar('foo') + ' ${bar}'},${foo}")
+        self.assertEqual(str(val), "4,value_of_foo,${@d.getVar('foo') + ' ${unsetvar}'},value_of_foo")
+
     def test_expand_in_python_snippet(self):
         val = self.d.expand("${@'boo ' + '${foo}'}")
         self.assertEqual(str(val), "boo value_of_foo")